NAEP SAIL2: Virtual World for English Language Arts Assessment Development 2014-15

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850–0803. The time required to complete this information collection is estimated to average 90 minutes, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission of this form, please write to: National Assessment of Educational Progress, National Center for Education Statistics, 1990 K Street, NW, Washington, DC 20006.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your participation is voluntary and without the permission of your parent or guardian your answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (20 U.S.C. §9573)]. With your parent or guardian’s permission some of your responses or clips from videos may be selected for use in research reports or presentations.¹

OMB No. 1850-0803 Approval Expires 09/30/2016

Part B. Play Testing Protocol

Script for Introduction to Study, Goals, and Approach

Text written in italics is to be read aloud by the staff facilitator.

NOTE: The facilitator should not read the script word for word, but should be familiar enough with its contents to conduct the play testing sessions in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of the students and adapt the wording or repeat with different wording if they detect that students are having difficulty comprehending the meaning of instructions or questions. Text placed in brackets is generic text that will be tailored to suit this specific research project (Virtual World for ELA).

Hi, our names are __ and we are from the Educational Testing Service. We are conducting research on how students gather, process, and evaluate information from multiple sources, within a virtual, computer-based environment. We are doing this research for a program funded by the federal government called the National Assessment of Educational Progress (NAEP). NAEP is a test taken by students in grades 4, 8, and 12 in the United States. The system you’ll be seeing today isn’t part of the test right now, but we are doing this research to see how it works. Your input will help us make the system work as well as it can for all students. We are going to ask for your opinions based on some activities that we will ask you to do using the system. You are here to give us your thoughts about your experiences of interacting with the virtual world for information gathering. We’d like to hear about what you think is good, what you think is not good, and if you think anything is difficult or confusing.

My colleagues and I will take notes on what you are saying—please be as honest as possible, talk freely with each other, and don’t worry if you find something confusing. You are not being graded on anything—you are helping us by giving us your opinions and thoughts and by interacting with the system so we can see how it works when it is being used. There are no right or wrong thoughts or opinions or actions.

This process is being recorded so that we can review it later. Without the permission of your parent or guardian, your information and responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. § 9573). We will look at what you say later so that we can understand how our system is working and how we can make it better. With parental/guardian permission, some responses or clips from videos may be selected for use in research reports or presentations.

Overall, this session should take about <say length> minutes. If at any time you decide you don’t want to go on, that is your choice, and you may stop. [If the student is no longer interested in participating, thank the student for his/her time and end the interview].

As you work through these activities, [FOR STUDENTS WORKING IN GROUPS] we want you to talk with each other as you go along [or FOR INDIVIDUAL SESSIONS] we want you to think aloud as you go along, so I’ll be asking you questions like “what are you thinking?” so you can tell me what’s in your head as you are thinking about what you are doing. We might also ask you a few questions now and then.

In Play Testing sessions, students will work either individually or in pairs/small groups with a facilitator (plus an additional observer in the case of individual sessions). The think aloud approach will be less formalized than in the cognitive interviews, and there will be more ad hoc questioning by the facilitator as needed (see below), so it doesn’t require a formalized practice session.

For the Virtual World for ELA study, students will be asked to complete tasks created around the manipulation of tools in the Virtual World. A facilitator will work with individual students or small groups and describe what students should do. Tasks will be based on principles of conducting research and inquiry from multiple information sources. Examples of prompts are:

You will help us to test how easy it is to use the interactive tools in the virtual lab to accomplish some basic information-gathering and evaluation tasks, and how quickly you can perform tasks. We would like you to think aloud during your process of completing the tasks, which means we’d like you to say out loud all the thoughts that come into in your head as you work.

The first task you need to complete is to locate sources of information that you think will provide useful information about <topic of task, e.g., whether it is scientifically possible to clone a dinosaur today>. Can you figure out where to find and select information sources that you think are relevant to this task; read and summarize what you learned; use available tools to analyze; evaluate, and compare sources;, and then use those sources to answer a research question or solve a problem?

Once the facilitators have explained the activity and asked the students to begin, they will generally observe them with little interruption or minimal reactive influences on their thinking and will listen to individuals’ verbalized thoughts or small groups’ conversation for insights into what the students are thinking about the items and tasks. Intermittently during the activity, or following a given activity as appropriate, the facilitators can ask targeted questions such as those in this list:

Is this difficult? Why? / Why not?

Are there any things that seem confusing here?

Do you understand what you need to do?

Do you think you can do what is needed here?

Have you done anything like this before?

How could this activity / the system be improved?

Were there aspects of what you just did that you didn’t like? Why?

This activity asked you to solve a problem about <insert focus of activity>. Have you learned about this before, either in school or anywhere outside of school?

Facilitators will observe students, take notes, and will pay attention to verbal answers to the questions above and write them down, but also will look for evidence of engagement, boredom, frustration, misunderstanding, or confusion in what students say to each other and in their facial expressions or body language. If appropriate, staff can use these moments to probe for more information.

Occasionally staff observing play testing sessions will introduce a question to individual students or groups of students talking about the activity to get more information from them, particularly in cases when students talk about issues related to their interest (or non-interest) in the activities, or confusing aspects of an activity or of the system. In such cases, observers might ask something like, “Can you tell me more about what you said?” If students do not provide sufficient comments on targeted parts, a staff member may ask individual students or a group of students if they had any thoughts about those aspects, using questions such as those described above, but focused on specific aspects or issues.

At the end, students will be thanked for their participation and for helping us with our research. Students will be given gift cards for their participation at the conclusion of each play testing session.

Part C. Cognitive Interviews

I. Introduction to Study Script

NOTE: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of the students and adapt the wording or repeat with different wording if they detect that students are having difficulty comprehending meaning. Text placed in brackets is generic text that will be tailored to this specific research project (Virtual World for ELA). In general, protocols will be adjusted and customized in light of how individual students respond.

Introduction:

Hello, my name is ______ and I work for Educational Testing Service. It’s nice to meet you and thank you very much for helping us out today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?

When the student responds, follow up with 2 or 3 questions to get the student used to talking, such as:

That’s interesting—why do you enjoy <subject> so much?
What are you studying in <subject> at the moment?
And what’s the best thing you have studied in <subject> so far?

Good. Well, I think you’ll enjoy what we are going to be doing today. First, let me begin by explaining why I am here and what you are going to be doing. You are taking part in a special study looking at a new type of activity to find out how students interact with a virtual world for information gathering. It is related to the National Assessment of Educational Progress, or NAEP for short, a test that is funded by the U.S. Federal Government and run by a center in the Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will help us test out these new systems and try out new activities that in the future may be used with other students. Overall, this session should take about 90 minutes.

It’s okay if you don’t know how to do any parts of these activities. I will not be grading your work today, and no one will know that it was you who did the task. My goal is to learn how you react to the activities and to the system you will be using, so please just try to do your best.

If at any time you decide you don’t want to go on, that is your choice, and you may stop.

This process is being recorded so that we can review it later. Without the permission of your parent or guardian, your information and responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. § 9573). We will look at what you say later so that we can understand how our test is working and how we can make it better. With parental/guardian permission, some responses or clips from videos may be selected for use in research reports or presentations.

We will work together on a few different activities today.

Do you have any questions?

If the student is no longer interested in participating, thank the student for his/her time and end the interview. After answering any questions and giving further explanation, the facilitator will conduct the think-aloud training.

Okay, let’s move on. I want to make sure you have a good idea of what we’re going to be doing. So I am going to give you some practice activities. The practice activities should help you get used to what we will be doing during the real tasks. They should help you understand how we want you to respond. Do you have any other questions before we start?

II. Concurrent Think-Aloud Method

II a. Concurrent Think-Aloud: Instructions and Modeling Example Script

NOTE: Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of students. Text placed in brackets is generic text that will be tailored to suit this specific research project (Virtual World for ELA).

To help us develop and improve our system, we will ask you to complete some activities. We will be video recording you as you do the activities, and a separate recording will capture all of your responses and actions with the system.

I’m going to ask you to do these activities in a way that may be different from what you are used to. Instead of working quietly, I want you to say aloud whatever is in your head as you work through them. We call this “thinking aloud,” because we are asking you to say everything you are thinking out loud.

In a moment, I will give you an example of what it’s like to think aloud. Then I will give you a chance to practice it. You won’t be graded on anything you say while you are thinking aloud. There are no incorrect thoughts, and everything you think and say is important to us.

Okay, now I’m going to show you how to think out loud—this will help you see how I want you to describe what is in your head as you are working on the activities. When I am finished I’ll ask you to try it, so you can see how it works.

Sample Think-Aloud Demonstration:

The example below shows how the facilitator should model the think aloud method for students. The key is to help them see that what is required is a fluid, naturalistic, continuous verbalization of all thoughts as they occur. Research staff may choose alternative tasks or activities as the basis for the demonstration, as appropriate. For example, since the Virtual World for ELA study uses an interactive tablet-based task, a tablet- or computer-based activity may provide a suitable example. ETS research staff will select an appropriate task or activity to demonstrate think aloud for each study, and facilitators will be trained to model and elicit this kind of naturalistic think-aloud data using the example.

Since we can’t tell what is going in your head, we need you to “Think Aloud.”

Let me give you an example. [Place example in front of student.] Look at this question. It asks me to look at the five animals and choose two that are the most similar. So I am going to do this task and I will think aloud while I work on it.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar:

Beetle
Mouse
Crab
Dolphin
Cow

Okay, so, um…. two animals… most similar... Hmm…. just wondering… what is “similar”? Could mean anything…. not sure how to... Well, ok…. So we have beetle, mouse, crab, dolphin, cow. Well…. Hmm… beetle and crab… kind of similar—well, small, lots of legs… move around fast. But crabs live in water… beetle…? No. Mouse… also fast moving and small, but otherwise, not really—it’s a mammal! Hmm, dolphin… that’s in water… but so not like a crab! So… mouse and cow… both mammals… also dolphins… but really so different from mice and cows… Hmm, seems like a hard question… wish I knew what they want… Okay, decision time…umm…okay, I think… let’s see … mouse and cow. Because they both have fur and four legs and walk around on land. Hmm, that seems weird—they’re not really similar! But, well, all the others are too different… so, okay, that’s it… seems like the best choice to me.

Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do today. The point of the think-aloud is to get at whatever is in your head as you are working. Just say aloud the words and the thoughts that come into in your head, as you go along. Don’t try to have a conversation with me or explain anything to me—just have a conversation with yourself, saying out loud anything that pops into your head as you go along.

II b. Concurrent Think-Aloud: Student Practice Script I

Interviewers should place the practice question in front of the student so that he/she can read it. Some students will be silent after reading the question. Students should be immediately encouraged to say whatever they are thinking.

It may be necessary to remind the student to talk aloud as he/she works through the questions and tasks. If necessary, interviewers should use the “Think-Aloud Hints” shown below to prompt the student, being careful not to lead the student and using the hints as reminders to verbalize but in a non-intrusive way. Since the goal is to gain insights into student cognition as it occurs naturalistically during these activities, it is important to avoid questioning students or otherwise leading or pushing them in any particular direction. It is also important to help students understand that they should not attempt to explain their thinking to the interviewer, in fact they should not even think of the process as describing their thinking for someone else, but rather they should simply say aloud any words that come into their heads, even if they result in broken sentences and unfinished thoughts, so that the process of thinking aloud does not alter or interrupt or the natural flow of the internal thinking process (Ericsson & Simon, 1989; see also Ericsson, 2006; Ericsson & Simon, 1984; Newell & Simon, 1972).

Now I want you to try a think-aloud. You can use this example. Like last time, you have to pick two of these things that you think are the most similar. As you are reading the question and the choices, and as you are figuring out your decision, I want you to say out loud all of the thoughts that come into your head at each moment.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I’ll be reminding you to think aloud if you get quiet because we need to hear all of your thoughts.

Okay, now you try. Go ahead and start working on this question and remember to think aloud as you are doing it.

Which two living things below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the tasks? What questions would you like to ask me? [If the student says he or she feels okay and doesn’t have any questions: Good, then let’s begin our study.] [If the student expresses concerns, says he/she has questions, or appears to be hesitant or reluctant, ask him/her to say more about the concerns or questions, and try to address his/her concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him/her to stop.]

II c. Concurrent Think-Aloud: Student Practice II (Optional)

(Use only if you feel the student would benefit from another think-aloud practice before moving on to the actual questions. It may be beneficial to instead model thinking aloud again for the student rather than doing another practice [see think-aloud demonstration in IIa])

If the student struggles to think aloud, or if the verbalizations are not of the right kind, the interviewer should give the student another opportunity to practice and help to provide guidance in terms of how to make their verbalizations more appropriate. The interviewer should praise the student for the first attempt regardless of how good it was, for example: “Very good—let’s do another one before we start the real tasks. Are you ready? Here is the next practice question. Remember to think out loud as you begin to think about this question and all the way through—say whatever pops into your head as you work through it.” During the practice item, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, below).

Which two of the following objects have the most similar properties? Circle the two that are most similar.

Silver coin
Chocolate coin
Gold coin
Blue plastic coin
Brown plastic coin

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the tasks? What questions would you like to ask me? [If the student says he or she feels ok and doesn’t have any questions: Good, then let’s begin our study.] [If the student expresses concerns, says he/she has questions, or appears to be hesitant or reluctant, ask him/her to say more about the concerns or questions, and try to address his/her concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him/her to stop.]

II d. Concurrent Think-Aloud: Starting the Tasks

Now we will move on to the actual task. Remember, as you are working I’d like you to say aloud everything that you’re thinking, and I may remind you to do that if you are quiet. This part should take about [indicate correct length] minutes. Remember, you will not be graded on what you do and there is no right or wrong way to think aloud, as long as you keep saying any thoughts that you are having. Your thoughts will help us make the activities and the system better. I will also have a few questions after you have finished working.

Do you have any questions before we go on? (Answer any questions the student may ask.)

Because the information you provide is so important to us, I am going to be taking notes while you think aloud and answer the questions.

Ready? OK, let’s begin.

The examples below illustrate the kinds of tasks and activities that students will be asked to work on during the cognitive interviews as they think aloud. During a cognitive interview session, a number of these kinds of activities will be given to each student, as time allows. If students become stuck or are visibly having difficulty, the facilitator will help them out with hints, or if a student is having a lot of difficulty the facilitator can tell them that they have done what is needed and ask them to move on to the next activity or task.

Sample Virtual World for ELA Task Script:

To help us evaluate the Virtual World, we created a simple task that we would like you to carry out in the virtual world. Your task is to locate two useful sources of information about the topic of <insert task-specific topic>, and to use information from those sources to answer questions. To complete this task in the Virtual World, you need to first select and find resources that may provide relevant information. Then you will read or interact with those sources and summarize the information that is important to know. You will also use tools in the environment to evaluate the quality of those sources, and to use information from the sources to support your answer to a research question. Remember to think aloud while doing this and pretend as if I’m not even here. If you fall silent for a while, I will remind you to keep thinking aloud. Are you ready? Then you can go ahead and start.

II e. Concurrent Think-Aloud: Hints for the Interviewer

The goal of the think-aloud method is to capture all the student’s mental processes while working through activities. Interviewers must strive to have the student speak aloud all of his or her thoughts during tasks. If a student is not talking, interviewers should use “continuers” to encourage them to talk. The goal is to get students to verbalize thoughts without putting words in students’ mouths. Care should be taken not to ask questions that lead students’ responses in particular directions or make them rush or change their approach. Interviewers should be as objective and unbiased as possible.

In general, if the student is silent for approximately 5 to 10 seconds, interviewers should use the following as a guide for encouraging the student to describe his or her thoughts, or to help the student elaborate his/her responses.

If the student is not verbalizing enough, interviewers should offer a verbal “nudge” to remind the students to keep talking, such as:

So what’s in your head?
What are you thinking?
What’s in your mind?
What are you thinking right now?
Go on….
Okay….keep going, keep talking

It is important to be responsive and sensitive to each student’s behavior. Students should be encouraged to verbalize as continuously as they can, but their reactions should be closely observed and pushing students should be avoided, especially if a student seems frustrated or uncomfortable. While it is desirable that students articulate as continuously as they can, sometimes students will simply be unable to say what is in their mind. Interviewers must be sensitive to nonverbal signals, if students cannot say any more than they have done.

In addition, during observations if it seems a student is hung up on something, interviewers should note when this occurred.

II f. Notes on Student Actions and Behaviors

As students are proceeding through each activity they will be video recorded. In addition, a screen-capture system and/or video camera focusing on the screen, as well as digital log file capture will record evidence of students’ interactions with the system or task. During the session the interviewer should take notes about any of the student’s expressions or behaviors that may reflect the status of his/her understanding, engagement, or use of the task or system. The following are examples of such behaviors for interviewers to note:

Does the student express signs of confusion, boredom, or excitement?
How does the student use the tools provided?
Does the student rapidly move through the activity or take his/her time?
Does the student spend a lot of time on a particular aspect?

Interviewers should also make a note of any places in the task that appear valuable for follow-up with some additional questions after the task is completed (see section IV).

III. Post-Think-Aloud Follow-Up Questions (Verbal Probing)

After completing the think-aloud process for a task, interviewers will follow up with a brief period of focused retrospective questioning. The post-task questions will comprise:

One standardized post-task question that all students will be asked following all tasks, which is designed to discover whether the student has prior knowledge of the content.

Up to three additional targeted questions. These will be generated by research staff prior to testing.

Standardized Question for All Tasks: Task-Specific Prior Knowledge

Have you studied anything related to this task in school, or have you learned about or come across these things in your own life? [If yes:] Tell me about what you have learned or studied or experienced that is related to what you did today.

Additional Questions: Task-Specific Issues

The purpose of the additional post-task questions is to capture more information on issues such as student actions during the activities, particular aspects of knowledge or skills targeted in the activities, and general reflections about the activities.

The following list shows some sample question frames that research staff will use to build questions. The specifics of each question will be generated via an informal task analysis in which key points for understanding student thinking will be identified.

Let’s think about the part where you [describe key action or aspect]. Can you tell me more about what you were thinking when you were doing that?

This [button or other tool] was on the screen to allow you to [describe action]. Tell me about how and when you used that, or if you didn’t really use it, why was that?

When you saw [information/text/graph/image] at this point, what were you thinking? What was in your mind?

If students’ answers are unclear or not very explicit, the interviewer should use prompts to encourage them to say more, for example:

Can you say a bit more about that?

What else were you thinking?

Anything else?

That’s interesting. Tell me more about that.

Optional: Interviewer-Generated Questions

In addition, if the interviewer noted some especially interesting behavior during the task, he or she can ask additional ad hoc questions about these specific instances, referring to notes made during the task. Interviewers should use their judgment about the need for and value of additional questions, based on the student’s behavior during the task, and based on time constraints.

Some examples of potential interviewer-generated ad hoc questions are:

I noticed on this part of the task that you paused for a while. Can you tell me more about why you paused and what you were thinking at this point?

I noticed on this part of the task that you did not use [describe tool or action]. Can you tell me more about why you didn’t use that?

I noticed that on this part of the task you spent some time looking at/going back-and-forth between [describe images, text, or actions]. Can you tell me more about what was going on at that point?

IV. Debriefing and Thank You for Cognitive Interview

Thank student for his/her time. Provide a gift card.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on these tasks?

Is there anything you would like to ask me about what we did today? [Answer student questions]

Thank you for helping us to study these processes and improve our system.

Part D. Small-Scale TryoutS

I. Introduction to Study Script

Text written in italics is to be read aloud by the facilitator. The facilitator will describe this information to a group of participants, in a school setting. One facilitator can give instructions while another ensures that students are prepared to view the tasks (e.g., setting up computers or tablets appropriately).

NOTE: The facilitator should not read the script word for word, but should be familiar enough with its contents to introduce the tasks in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of students. Text in brackets is generic text that will be tailored based on task specifics.

Introduction Script:

Hello, my name is ______ and I work for ETS. It’s nice to meet you and thank you very much for helping us out today.

Let me begin by explaining why you’re here and what you’re going to be doing today. You are participating in a special study to try out a new virtual world for information gathering. This is part of something called the National Assessment of Educational Progress, or NAEP for short, and is sponsored by the U.S. Department of Education. Today we’ll be asking you to try out a small group of new tasks that are being developed and studied for 8^th/ 9^th graders.

If at any time you decide you do not want to go on, that is your choice and you may stop.

This tryout session is being recorded so researchers can review it later. Your information and responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573).

Today we will aim to work through <say number of tasks>.

Do you have any questions?

Interviewer: Ask the students if they have any questions. If a student is no longer interested in participating, thank the student for his/her time and the student can end the tryout session.

Ok, now, I’m going to give you [number] questions or tasks to complete. I want you to work through them one at a time. Try to finish each task. You aren’t being graded on your work today, so it’s fine if you don’t know the answer or don’t know how to go about any of the activities we are asking you to do. But try to do your best, and try to complete all of the activities, or as much as you can. I won’t be interrupting, I’ll just be here observing, so you can work through the tasks at your own pace.

You will have [indicate correct number of minutes] minutes to do this. You will not be graded on what you do during the task, but please try to work through this as if it was a real test.

Okay, you can go ahead and begin with the first task.

II. examples of tasks or activities in the tryout phase

By the tryout phase, tasks or activities will be self-contained and students will be able to work through them at their normal pace, without interruption and without needing support from the facilitator. As far as possible, tryouts should represent naturalistic test-taking conditions.

The kinds of activities that students will be asked to complete will be similar to those described for earlier phases. However, in the tryout phase, the facilitator will not need to deliver the activities or tasks using verbal instructions, since they will be presented to the students via the system (the Virtual World for ELA). The exact look and feel of the presentation of these will be determined during the later development phases of each project. However, all of the tasks or activities will be presented in a way that is sufficiently self-contained that students can work through them at their own pace, without additional input from the facilitator. Examples of tasks or activities that students will complete in the tryout phase are shown below.

Virtual World for ELA Sample Tryout Task:

Students will use the virtual world tools and resources in order to:

Locate useful and reliable sources which can be used to answer a question about <topic, e.g., whether it is scientifically feasible to clone a dinosaur>.
Synthesize multiple sources on a topic to support and defend an argument directed toward a specific audience.

III. EXAMPLES OF QUESTIONNAIRE ITEMS FOR TRYOUT STUDY

In order to support analysis of data obtained from the tryout study, participants in the tryouts will be asked to complete a brief questionnaire following completion of the assessment tasks designed in the virtual world environment. These questionnaire items will provide information about students’ familiarity and experience with various information literacy and research-related activities, whether in the classroom, or at home. Example questionnaire items to illustrate the format of this instrument appear below. The specific questions to be asked will be determined and finalized prior to the tryout study. These questions are adapted from previous work on information literacy (Katz & Smith-Macklin, 2007).

For each of the following 30 activities, please answer the following two questions:

(1) About how often you have done the activity over the past two years?

(2) How confident are you in your ability to do this activity?

	(1) About how often you have done the activity over the past two years?					(2) How confident are you in your ability to do this activity?
	Never	Rarely	Some-times	Often	Very Often	Not confident in my ability	Somewhat confident in my ability	Confident in my ability
		(a few times a year)	(a few times a month)	(a few times a week)	(several times a week)	Not confident in my ability	Somewhat confident in my ability	Confident in my ability
a. Made changes to your search keywords to find better results from an Internet search	○	○	○	○	○	○	○	○
b. Navigated through a website with multiple links in order to reach a particular destination	○	○	○	○	○	○	○	○
c. Filled in a table to compare information or ideas from various sources	○	○	○	○	○	○	○	○
d. Judged whether a website contains information from an authoritative source	○	○	○	○	○	○	○	○
e. Created an electronic presentation in which you had to integrate text and images or graphics	○	○	○	○	○	○	○	○
f. Efficiently located a specific piece of information within a lengthy website	○	○	○	○	○	○	○	○
g. Evaluated the relevance of sources to your topic	○	○	○	○	○	○	○	○
h. Used a computer to produce visual displays of information to answer a research question	○	○	○	○	○	○	○	○

IV. Debriefing and Thank You

As the tryouts will take place in classroom settings, the following debriefing questions will be distributed in the form of a brief written questionnaire.

Thank you for taking the time to complete this activity. We would like to ask you a few questions about your experience. Please read each question carefully and mark the answer that you think is best.

How much do you agree with each statement?	Strongly Agree	Agree	Disagree	Strongly Disagree
I enjoyed exploring the virtual world environment				
The scenario got my attention and made me interested in completing the task				
The activities were easy to complete				
The directions on the screen were easy to understand				
It was easy for me to know what to do next to move forward in the task				
The vocabulary was easy to understand				
I would enjoy doing more activities like this in the future				
The activity seemed more like a game rather than schoolwork				
The dialogues with game characters are interesting				
I like the onscreen graphics				

Please write your responses to the following questions:

What did you like most about this activity? Why?

What did you like the least about this activity? Why?

Was there anything confusing about the activity? Please explain.

Was there anything especially challenging or difficult about the activity? Please explain.

Finally, what suggestions would you make for improving the virtual world and related activities? Please explain.

Thank student for his/her time. [Provide a gift card, if applicable].

References

Ericsson, K. A. (2006). Protocol analysis and expert thought: Concurrent verbalizations of thinking during experts’ performance on representative tasks. The Cambridge handbook of expertise and expert performance, 223-242.

Ericsson, K. A., & Simon, H. A. (1984). Verbal reports as data. MIT Press.

Ericsson, K. A., & Simon, H. A. (1998). How to study thinking in everyday life: Contrasting think-aloud protocols with descriptions and explanations of thinking. Mind, Culture, and Activity, 5(3), 178-186.

Katz, I. R. and Smith-Macklin, A. (2007). Information and communication technology (ICT) literacy: Integration and assessment in higher education. Journal of Systemics, Cybernetics, and Informatics, 5(4), 50-55. Available at http://www.iiisci.org/Journal/SCI/Abstract.asp?var=&id=P890541

Newell, A., & Simon, H. A. (1972). Human problem solving (Vol. 104, No. 9). Englewood Cliffs, NJ: Prentice-Hall.

1 The clause regarding the potential use of some responses or clips from videos in research reports or presentations will be included only for students participating in play testing and cognitive interviews conducted at ETS. Any data collection conducted at schools will not be videotaped, and recordings of tryouts conducted in any location will not be used in research reports or presentations.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	0000-00-00