National Assessment of Educational Progress (NAEP) Mathematics and Reading Pretesting 2015

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average xx¹ minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 20202-4537. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: NAEP/NCES, U.S. Department of Education, 400 Maryland Avenue, S.W., Washington, DC 20202.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

OMB No. 1850-0803 Approval Expires 09/30/2016

Part B. Cognitive Interviews (Reading)

I. Introduction to Study Script

NOTE: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular task being studied. In general, protocols will be tweaked and customized in light of specific task content and greater knowledge of delivery devices.

Introduction:

Hello, my name is ______ and I work for Educational Testing Service. It’s nice to meet you and thank you very much for helping us out today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?

When the student responds, follow up with 2 or 3 questions to get the student used to talking, such as:

That’s interesting—why do you enjoy <subject> so much?
What are you studying in <subject> at the moment?
And what’s the best thing you have studied in <subject> so far?

Good. Well, I think you’ll enjoy what we are going to be doing today. First, let me begin by explaining why I am here and what you are going to be doing. You are taking part in a special study looking at a new type of activity to find out what students know in reading. It is part of the National Assessment of Educational Progress, or NAEP for short, a test that is funded by the U.S. Federal Government and run by a center in the Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will help us develop better tests for other students. Overall, this session should take about 90 minutes.

It’s okay if you don’t know how to do any parts of the task. I will not be grading your work today, and no one will know that it was you who did the task. My goal is to learn how you react to the task, so please try to do your best.

If at any time you decide you don’t want to go on, that is your choice, and you may stop.

This new test is on the tablet. The instructions about how to do the test are also on the tablet.

This interview is being recorded so we can review it later. What you say will not be told to anyone, or used in any way that could identify you, for any other purpose except as required by law (see 20 U.S.C. § 9573). We will look at what you say later, but only so that we can understand how our test is working and how we can make it better.

We will work together on one task.

Do you have any questions?

After answering any questions and giving further explanation, continue with the think-aloud training. If the student is no longer interested in participating, thank the student for his/her time and end the interview.

Okay, let’s move on. Before we look at the real task, I want to make sure you have a good idea of what we’re going to be doing. So I am going to give you some practice questions. The practice questions should help you get used to what we will be doing during the real task. They should help you understand how we want you to respond. Do you have any other questions before we start?

NOTE: Check the study schedule to establish whether the student should undergo the assigned tasks with a concurrent or retrospective verbal protocol (think-aloud) procedure. If concurrent, the student will think aloud while he or she is doing the task; in those cases, follow the steps described in section II. If retrospective, the student will describe his or her thinking after he or she has completed the task; in those cases, follow the steps described in section III. (NOTE: Given that it is challenging for students to both read passages and think aloud, it is likely that think-alouds will be retrospective for reading tasks.)

II. Concurrent Think-Aloud

II a. Concurrent Think-Aloud: Instructions and Modeling Script

NOTE: If the schedule indicates that the retrospective method is to be used, please skip this section and move to section III.

NOTE: Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular task being studied.

To help us make our test better, we will ask you to complete a task. While you are doing the task, we will be recording what you say out loud.

I’m going to ask you to do the questions in these tasks in a way that may be different from what you are used to. Instead of working quietly, I want you to tell me what you are thinking as you work through the task. We call this “thinking aloud,” because we are asking you to say everything you are thinking out loud.

In a moment, I will give you an example of the think-aloud process. Then I will give you a chance to practice it. You won’t be graded on anything you say while you are thinking aloud. There are no incorrect thoughts, and everything you think and say is important to us.

Okay, now I’m going to show you how to think out loud—this will help you see how I want you to describe what you are thinking as you are working on the task. When I am finished I’ll ask you to try it, so you can see how it works.

Think-Aloud Demonstration:

Since we can’t tell what is going in your head, we need you to “Think Aloud.”

Let me give you an example. [Place example question in front of student.] Look at this question. It asks me to look at the five animals and choose two that are the most similar. So I am going to do this task and I will think aloud while I work on it.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar:

Beetle
Mouse
Crab
Dolphin
Cow

Okay, so I am reading the question…. It says I have to decide on the two animals that are the most similar. Hmm, I am just wondering what do they mean by “similar”? That could mean anything. I’m thinking I’m not sure how I’m going to make that decision. Well, let’s look at the choices. So the choices are: beetle, mouse, crab, dolphin, and cow. Well, as I am reading the choices I am already thinking that there are some things about them that are similar, but there are also differences. I’m thinking the beetle and crab are kind of similar, in a way—well, they both have lots of legs and they kind of move around fast. But one lives in the water and the other doesn’t, so that makes them different. I guess the mouse also moves around fast, but it’s not similar to either the beetle or the crab, except they are all small. Then there is the dolphin that lives in the water, too… but that’s about all it has in common with the crab, so I wouldn’t say they are very similar even though they’re both in the sea. Hmm, so the mouse and the cow are both mammals, they both have fur and they have live babies instead of laying eggs or whatever… oh right, but dolphins are mammals too, aren’t they? Yes, I think they are… but really they seem so different from mice and cows to me. I don’t know… this seems like a hard question… I’m not sure what to choose. Okay, I’m going to try to make a decision…umm…okay, I think I am going to go for… mouse and cow. Because they both have fur and have four legs and they both walk around on the land. That seems weird; they are not really similar! But when I think about all of the others they seem too different from each other in really basic ways, like where they live and their kind of body and that kind of thing… so, yes, even though it was really hard to decide, that seems like the best choice to me, because they are both mammals and they both live on the land. So those things seem like they are important, and that is how I made my decision.

Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do as you are thinking about the task today. The point of the think-aloud is to get at whatever is in your head as you are doing the task. Just say aloud the words and the thoughts that are in your head, as you are thinking about and working on the task.

II b. Concurrent Think-Aloud: Student Practice Script I

Interviewers should place the practice question in front of the student so he/she can read it. Some students will be silent after reading the question. Students should be immediately encouraged to say whatever they are thinking.

It may be necessary to remind the student to talk aloud as he/she works through the question. If necessary, interviewers should use the “Think-Aloud Hints” shown below to prompt the student, being careful not to lead the student. The interviewer needs to be familiar enough with the information to introduce the think-aloud in a natural, conversational manner.

Now you will try a think-aloud. You can use this example. Like last time, you have to pick two of these things that you think are the most similar. As you are reading the question and the choices, and as you are figuring out your decision, I want you to talk out loud to tell me all of the thoughts that are in your head at each moment.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I’ll be reminding you to think aloud if you get quiet because I need to hear all of your thoughts.

Okay, now you try. Go ahead and start working on this question and remember to think aloud as you are doing it.

Which two living things below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the task? What questions would you like to ask me? [If the student says he or she feels okay and doesn’t have any questions: Good, then let’s begin our study.] [If the student expresses concerns, says he/she has questions, or appears to be hesitant or reluctant, ask him/her to say more about the concerns or questions, and try to address his/her concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him/her to stop.]

II c. Concurrent Think-Aloud: Student Practice II (Optional)

(Use only if you feel the student would benefit from another think-aloud practice before moving on to the actual task. It may be beneficial to instead model thinking aloud again for the student rather than doing another practice [see think-aloud demonstration in IIa])

If the student struggles to think aloud, the interviewer should give the student another opportunity to practice. The interviewer should praise the student for the first attempt regardless of how good it was, for example: “Very good—let’s do another one before we start the real task. Are you ready? Here is the next practice question. Remember to think out loud as you begin to think about this question and all the way through—tell me what you are thinking as you work through it.” During the practice question, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, below).

Which two of the following objects have the most similar properties? Circle the two that are most similar.

Silver coin
Chocolate coin
Gold coin
Blue plastic coin
Brown plastic coin

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the task? What questions would you like to ask me? [If the student says he or she feels ok and doesn’t have any questions: Good, then let’s begin our study.] [If the student expresses concerns, says he/she has questions, or appears to be hesitant or reluctant, ask him/her to say more about the concerns or questions, and try to address his/her concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him/her to stop.]

II d. Concurrent Think-Aloud: Starting the Tasks

Now we will move on to the actual task. Remember, as you do all the activities and answer all the questions in the task I’d like you to say aloud everything that you’re thinking, and I may remind you to do that if you are quiet. If this was a real test, this task would probably take you about 30 minutes. It may take you a little longer today because you are thinking aloud. Do not worry about how long it takes.

Remember, you will not be graded on what you do during the task and there is no right or wrong way to think aloud, as long as you keep telling me your thoughts. Your thoughts will help us make the task better. I will also have a few questions after you have finished working on the task.

Do you have any questions before we go on? (Answer any questions the student may ask.)

Because the information you provide is so important to us, I am going to be taking notes while you think aloud and answer the questions.

Here is the task. When you are ready, go ahead and start working on it.

II e. Concurrent Think-Aloud: Hints for the Interviewer

The goal of think-alouds is to capture all the student’s mental processes while working through tasks. Interviewers must strive to have the student speak aloud all of his or her thoughts while doing a task. If a student is continually providing short utterances or not talking, interviewers should use “continuers” to encourage students to be more descriptive. The goal is to get students to verbalize thoughts without putting words in students’ mouths. Care should be taken not to ask questions that lead students’ responses in particular directions or make them rush or change their approach. Interviewers should be as objective and unbiased as possible.

In general, if the student is silent for approximately 5 to 10 seconds, interviewers should use the following as a guide for encouraging the student to describe his or her thoughts, or to help the student elaborate his/her responses.

If the student is not verbalizing enough, interviewers should offer a verbal “nudge” to remind the students to keep talking, such as:

What’s going on in your head right now?
I see you’re looking at the text. What are you thinking?
You seem to be thinking hard—can you tell me what’s in your mind right now?
What are you thinking now?
Any other thoughts?

If the student says something about his or her thoughts but it is difficult to tell exactly what he or she means, or if the student begins to talk but does not say very much, interviewers can urge the student to elaborate, for example:

Tell me a bit more about that.
That’s interesting—can you say more about that?
Go on….
Okay, and what else are/were you thinking?

If more clarity or a more explicit description of a student’s thoughts is desirable:

Can you explain what you mean by that?
Can you say a bit more to help me understand what you mean/are thinking/were thinking?

It is important to be responsive and sensitive to each student’s behavior. If it becomes clear that students cannot tell any more about their thoughts, interviewers should not keep asking. Students should be encouraged to think aloud as much as they can, but their reactions should be closely observed and pushing avoided if students seem frustrated or uncomfortable. While it is desirable that students articulate as much as they can, sometimes students will simply be unable to tell what is in their mind beyond a basic reply such as “I am reading the question.” Interviewers must be sensitive to nonverbal signals, if students cannot tell any more than they have, or if they need a few extra seconds to gather their thoughts to put them into words.

In addition, during observations if it seems a student is hung up on something, interviewers should note when this occurred.

II f. Notes on Student Actions and Behaviors During Tasks

During the session the interviewer should take notes regarding any of the student’s expressions or behaviors that may reflect the status of his/her understanding, engagement, or use of the task. The following are examples of such behaviors for interviewers to note:

Does the student express signs of confusion, boredom, or excitement?
Does the student play the words spoken by the avatars (assuming they are present)? Does the student seem to focus on the avatars (if present)?
Does the student use tools provided in the task such as tapping on tabs or buttons to reveal additional or earlier information or reminders of what to include or think about? Does the student return to the text before responding to questions?
Does the student rapidly move through the screen or take his/her time? Does the student spend a lot of time on a particular aspect?

These kinds of observations should be considered opportunities to prompt students to think aloud and to encourage them to express what is in their minds. Interviewers should also make a note of any places in the task that appear valuable for follow-up with some additional questions after the task is completed (see section IV).

III. Retrospective Think-Aloud

III a. Retrospective Think-Aloud: Instructions and Modeling Script

In this approach, students are not asked to think out loud during the task. Instead, students will be instructed to complete the task as they would under normal testing conditions (silently, without interruption). Then, after the task is completed, they will be asked to describe what they were thinking while reviewing the task on a computer. This approach will most likely be used for reading tasks given that experience with cognitive interviewing at ETS suggests that students have difficulty reading and thinking aloud or writing and thinking aloud. The replay of the task is intended to help cue the student’s recall of the thought processes that occurred during the task as it progressed.

Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular task being studied.

To help us make our test better, we will ask you to complete a task. While you are doing the task, we will be audiotaping what you say aloud. Overall, this session should take about 90minutes.

First you’ll do the task, working at your own pace through all of the activities and questions you will see on the screen. Then, after you have finished, I’m going to ask you to go back over the task with me. As we go back over it and look at the task again, I will ask you to tell me what you were thinking as you were working through the task. We call this “thinking aloud,” because we are asking you to say everything you were thinking during the task out loud. Does that make sense? Do you understand what we’ll be doing?

[Answer any questions and clarify the sequence of steps, if needed.]

Okay, now I’m going to show you how to think out loud—this will help you see how I want you to describe what you were thinking while you were working on the task. When I am finished, I’ll ask you to try it, so you can see how it works.

Think-Aloud Demonstration:

Since we can’t tell what is going on in your head, we need you to “Think Aloud.”

Let me give you an example. [Place example question in front of student.] Look at this question. It asks me to look at the five animals and choose two that are the most similar. So first of all, I am going to do this task just as I normally would for a test question. After I have finished it, I will try to remember what I was thinking as I was working on it, and I’ll tell you what I was thinking.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar:

Beetle
Mouse
Crab
Dolphin
Cow

[The interviewer ‘works’ silently for a minute or two on the question above. The interviewer should enact some behaviors that suggest that he/she is thinking hard about the question and carefully considering the possible answers; for example, pointing with a pencil tip at each option in turn; moving a pencil or fingertip between answer options; returning to the question; giving nonverbal signals indicating deep consideration, uncertainty, etc. Finally the interviewer circles two answers, B and E, and then begins the retrospective think-aloud.]

Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do as you are thinking about the task today. The point of the think-aloud is to get at whatever is in your head as you are doing the tasks. Just say aloud the words and the thoughts that are in your head, as you are thinking about and working on the task.

III b. Retrospective Think-Aloud: Student Practice Script I

Interviewers should place the practice question in front of the student so he/she can read it. Allow the student as much time as needed to answer the question. When he/she indicates he/she has finished, ask the student to begin thinking aloud. Some students will be silent after reading the question. Such students should be asked to say whatever they are thinking. It may be necessary to remind students to talk aloud as they work through the task. If necessary, interviewers should use the “Think-Aloud Hints” shown below to prompt students, being careful not to lead students. The interviewer needs to be familiar enough with the information to encourage the think-aloud in a natural, conversational manner.

Now you will try a think-aloud. You can use this example. Like last time, you have to pick the two things that you think are the most similar. Go ahead and do this task, and when you are finished, let me know.

Once you are done, I will ask you to talk out loud and tell me all of the thoughts that were in your head at each moment. During that phase, if I don’t hear you speaking I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are thinking. I’ll be reminding you to think aloud if you get quiet because I need to hear all of the thoughts you had during the task.

Okay, now you try, you can go ahead and start working on the question. Let me know when you are finished.

Which two organisms below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

[When students indicate they are finished, make sure they have circled two answers, and then ask them to begin describing their thoughts. As necessary, include the appropriate prompting questions, such as the following:]

Okay, so tell me what you were thinking as you were doing this question. What did you think at the beginning, when you started to read the question? … And then what were you thinking? … And what else did you think about? … Anything else? … Can you tell me more about that? … etc.

III c. Retrospective Think-Aloud: Student Practice II (Optional)

If the student struggles to think aloud, the interviewer should give the student another opportunity to practice. The interviewer should praise the student for the first attempt regardless of how good it was, for example: “Very good—let’s do another one before we start the real task. Are you ready? Here is the next practice question. [Place example question in front of student.] Remember that after you have finished answering this question I will ask you to talk out loud to tell me what you were thinking all the way through—so you should tell me what you were thinking as you worked through it. So, first of all, go ahead and do the question, and let me know when you have finished.”

As before, during the post-task think-aloud phase, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, section IIe).

Which two of the following objects have the most similar properties? Circle the two that are most similar

A. Silver coin

B. Chocolate coin

C. Gold coin

D. Blue plastic coin

E. Brown plastic coin

After the think-aloud phase is finished:

Now that you have practiced, do you feel that you understand how you should talk aloud about the thoughts you had while you were doing the task? Is this something you feel okay about doing? [If students say yes:] Good, then let’s begin our study.[If students say no or appear to be hesitant or reluctant, ask them to say more about any reservations they have, and try to address their concerns or uncertainties in a supportive way. If students indicate they do not wish to continue or do not feel comfortable continuing, allow them to stop.]

III d. Retrospective Think-Aloud: Starting the Task

Now we will move on to the actual task. Remember, after you have finished doing all the activities and answering all the questions on the task, I will ask you to say aloud everything that you remember thinking while you were doing the task, and I may remind you to do that if you go quiet. This task should take about 30 minutes.

Do you have any questions before we go on? [Answer any questions the student may ask.]

Because the information you provide is so important to us, I am going to be taking notes while you are doing the task and while you are thinking aloud afterwards.

Here is the task. When you are ready, go ahead and start working on it.

III e. Retrospective Think-Aloud: Prompts and Questions After Completion of Task

Now we will discuss what you did on the task, and I would like you to tell me what you were thinking as we review the task. We will be recording what you are saying as you think about this task. Again, you should feel free to stop at any time, and this information will not be used to grade you. The information will only be used to help improve the task.

Interviewer walks through the task with the student, and asks the student to describe aloud what he/she was thinking at each point in the task.

If the student is not verbalizing enough, interviewers should offer a verbal “nudge” to remind the student to keep talking, such as:

What was going on in your head at this time?
You seemed to be thinking hard—can you tell me what was in your mind when you were working on this part?

If the student says something about his/her thoughts but it is difficult to tell exactly what the student means, or if the student begins to talk but does not say very much, interviewers can urge the student to elaborate, for example:

Tell me a bit more about that.
That’s interesting—can you say more about that?
Go on….
Uh huh, okay, and what else were you thinking?

If more clarity or a more explicit description of a student’s thoughts is desirable:

Can you explain what you mean by that?
Can you say a bit more, to help me understand what you were thinking?

III f. Notes on Student Actions and Behaviors During Tasks

During the session, the interviewer should take notes of any of the student’s expressions or behaviors that may reflect the status of his/her understanding, engagement, or use of the task. The following are examples of such behaviors for interviewers to note:

Does the student express signs of confusion, boredom, or excitement?
Does the student play the words spoken by the avatars (assuming they are present)? Does the student seem to focus on the avatars (if present)?
Does the student use tools provided in the task such as tapping on tabs or buttons to reveal additional or earlier information or reminders of what to include or think about? Does the student return to the text before responding to questions?
Does the student rapidly move through the screen or take his/her time? Does the student spend a lot of time on a particular aspect?

These kinds of observations should be considered opportunities to prompt students to think aloud and to encourage them to express what is in their minds. Interviewers should also make a note of any places in the task that appears valuable for follow-up with some additional questions after the task is completed (see section IV).

IV. Post-Think-Aloud Follow-Up Questions (Verbal Probing)

After completing the think-aloud process for a task, interviewers will follow up with a brief period of focused retrospective questioning. The post-task questions will comprise:

One standardized post-task question that all students will be asked following all tasks, which is designed to discover whether the student has prior knowledge of the content.

Additional targeted questions that are task specific. These will be selected for each task by ETS staff prior to testing. Interviewers will receive the set of questions for the specific task they are examining in each cognitive interview.

Standardized Question for All Tasks: Task-Specific Prior Knowledge

Have you studied anything related to this task in school, or have you learned about or come across these things in your own life? [If yes:] Tell me about what you have learned or studied or experienced that is related to this task.

Additional Questions: Task-Specific Issues

The purpose of the additional post-task questions is to capture more information on issues such as student actions during the task, reactions to particular aspects of the task (for example, avatars if present) , and general reflections about the task. ETS staff will identify questions for interviewers to ask in relation to each task. Interviewers should have the list of post-task questions ready when the student finishes the think-aloud portion for each task. These questions will typically be asked after the student has finished the task. However, occasionally a question will be identified that needs to be answered in the context of the task, while the student is engaged in solving some part of the problem. In those cases, the question will be recorded in session notes and audiotaped along with the rest of the session.

The following list shows some sample question frames that ETS staff will use to build questions related to each task. The specifics of each question will be generated by ETS staff and related to key issues for each task. What did you think about the avatars in the task? Did the task characters make the task easier or help you in any way?

What did you think when you saw this [aspect of the task]?

Overall, how easy or hard was this task?

How interesting did you find this task?

Think about reading tests at school. What do you like or dislike about this task compared to reading tests you take at school?

If students’ answers are unclear or not very explicit, the interviewer should use the same kinds of prompts as identified in the think-aloud protocol to elicit more information, for example:

Can you say a bit more about that?

What else were you thinking?

Anything else?

That’s interesting. Tell me more about that.

Optional: Interviewer-Generated Questions

In addition, if the interviewer noted some especially interesting behavior during the task, he or she can ask additional ad hoc questions about these specific instances, referring to notes made during the task. Interviewers should use their judgment about the need for and value of additional questions, based on the student’s behavior during the task, and based on time constraints.

Some examples of potential interviewer-generated ad hoc questions are:

I noticed on this part of the task that you paused for a while. Can you tell me more about why you paused and what you were thinking at this point?

I noticed that on this part of the task you spent some time looking at/going back-and-forth between the <image/text/question>). Can you tell me more about what was going on there?

V. Eye Tracking

Eye tracking allows data to be gathered about where students focus while taking an assessment. Students will be asked to do tasks on a tablet while a camera monitors where they look.

Apparatus

Eye-tracking systems are video-based systems utilizing high speed cameras and an LED infrared light that illuminates the face (no more powerful than typical sunlight). Infrared cameras are used to capture eye movements. The remote system requires no physical contact with the student: an infrared video camera acquires an image of the eyes and calculates gaze location in real-time. To allow free (but limited) head and body movements while maintaining accuracy of localization, a reflective sticker is placed on the forehead of the student during the eye-tracking session. ETS eye-tracking experts have used this particular system with adults and children as young as 3 years old. It takes a very short time (5 minutes) to set up and calibrate the equipment for each participant.

Procedure

Students’ eye movements will be unobtrusively recorded while they work through the task and (if thinking aloud) articulate their thoughts. Eye tracking data will be reviewed along with other data to analyze student reactions to tasks.

The following describes the sequence that will be followed for eye tracking . Note that the eye-tracking calibration procedure may differ slightly for various hardware setups.

Setup and Calibration

The participating student is greeted by an ETS eye-tracking lab expert. The expert will describe the study procedure and questions or concerns the student may have. The student is then seated in front of the eye tracker, and a sticker is put on his/her forehead to assist the eye tracker to locate the eyes and to compensate for head movements. The system is then calibrated. The student is asked to watch a dot appearing at a number of different locations on the blank screen. The setup and calibration typically take about 5 minutes.

Eye-tracking Session

After a successful calibration, the student starts the assessment session, in which he/she goes through the assessment task on the tablet. Students are free to move their head and body during the assessment within the field of view of the eye-tracking camera.

After the student completes the assessment task, eye tracking will be stopped.

VI. Debriefing and Thank You for Cognitive Interview

Thank student for his/her time. Provide a gift card.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the task?

Is there anything you would like to ask me about what we did today? [Answer student questions]

Thank you for helping us to improve our test.

Part C. Small-Scale Tryout Protocol

Text written in italics is to be read aloud by the interviewer.

NOTE: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text in brackets is generic text that will be tailored based on task specifics.

Introduction Script:

Hello, my name is ______ and I work for EurekaFacts. It’s nice to meet you and thank you very much for helping us out today.

Let me begin by explaining why you’re here and what you’re going to be doing today. You are participating in a special study to try out new <interviewer insert subject> test questions. This is part of something called the National Assessment of Educational Progress, or NAEP for short, and is sponsored by the U.S. Department of Education. Today we’ll be asking you to try out a small group of new test questions that are being developed for 4th/8th graders all over the United States.

If at any time you decide you do not want to go on, that is your choice and you may stop.

Your information and responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573).

You will do one task and then answer some questions about the task and <interviewer insert subject> activities you have done in and out of school.

Do you have any questions?

Interviewer: Ask the students if they have any questions. After answering questions and giving further explanation, continue with the preliminary information questions. If any student is no longer interested in participating, thank the student for his/her time and end that student’s tryout session.

The following should be filled in for all students in a tryout session.

Ref#	Question/Prompt	Response
	Preliminary Information
1	Student ID	(Prefilled)
2	Student’s gender	 Male  Female
3	Name of school
4	Special circumstances that may have affected the tryout session [Do not ask]

For this activity, you will complete one <indicate correct subject> task on tablet.

You will be asked to type in responses or to tap on responses on the screen.

You will have 30 minutes to do the task. You will not be graded on what you do during the task, but please answer these questions as if it was a real test.

When 30 minutes are up, you will be asked to respond to a set of questions about your reactions to the task and your experiences with [indicate either mathematics or reading]. You will have [indicate correct number of minutes] to complete the questions.

Okay, begin work.

Post-task Questions for Mathematics Tasks - Grade 4

1. How hard was this task compared to most other tests you have taken this year in school?

A Easier than other tests

B About as hard as other tests

C Harder than other tests

D Much harder than other tests

2. How hard did you try on this task compared to how hard you tried on most other tests you have taken this year in school?

A Not as hard as on other tests

B About as hard as on other tests

C Harder than on other tests

D Much harder than on other tests

3. How important was it to you to do well on this task?

A Not very important

B Somewhat important

C Important

D Very important

4. How easy or difficult was this task?

A Extremely difficult

B Quite difficult

C Somewhat difficult

D Somewhat easy

E Quite easy

F Extremely easy

5. How many questions do you think you answered correctly on this task?

A None or almost none

B Some but less than half

C About half

D More than half but not all

E All or almost all

6. How similar were the questions on this task to the questions on your math quizzes and tests at school?

A Not similar at all

B Not similar

C Somewhat similar

D Quite similar

E Extremely similar

7. How much effort did you apply to succeed on this task?

A No effort at all

B Very little effort

C Some effort

D Quite a bit of effort

E A lot of effort

8. How stressful was taking this task?

A Not stressful at all

B A little stressful

C Somewhat stressful

D Quite stressful

E Extremely stressful

9. How challenging was taking this task?

A Not challenging at all

B A little challenging

C Somewhat challenging

D Quite challenging

E Extremely challenging

10. This school year, how often did you use the following digital devices to work on math problems, including homework? Select one answer choice on each row.

	Never	About once or twice a year	About once or twice a month	About once or twice a week	Every day or almost every day
a. Hand-held calculator	A	B	C	D	E
b. Desktop or laptop computer	A	B	C	D	E
c. Tablet (for example, Surface Pro, iPad, Kindle Fire)	A	B	C	D	E
d. Smartphone (for example, iPhone, Samsung Galaxy, HTC One)	A	B	C	D	E
e. Other device (Please specify): _____________________	A	B	C	D	E

11. How much does each of the following statements describe a person like you? Select one answer choice on each row.

	Not at all like me	A little bit like me	Somewhat like me	Quite a bit like me	Exactly Like me
a. I enjoy doing math.	A	B	C	D	E
b. I look forward to my math class.	A	B	C	D	E
c. I am interested in the things I learned in math.	A	B	C	D	E
d. I think making an effort in math is worthwhile.	A	B	C	D	E
e. I think math will help me even when I am not in school.	A	B	C	D	E
f. I think it is important to do well in math.	A	B	C	D	E

12. How much do you enjoy solving the following types of math problems? Select one answer choice on each row.

	Enjoy not at all	Enjoy a little bit	Enjoy somewhat	Enjoy quite a bit	Enjoy a lot
a. Addition, subtraction, multiplication, and division	A	B	C	D	E
b. Working with geometric figures like rectangles and squares	A	B	C	D	E
c. Looking at patterns, graphs, and charts	A	B	C	D	E

13. Thinking about math, do you think that you would be able to do each of the following? Do not actually solve the problems. Select one answer choice on each row.

	I definitely can’t	I probably can’t	Maybe	I probably can	I definitely can
a. Estimate the weight of 5 apples using pounds (lbs)	A	B	C	D	E
b. Divide 42 stickers by 6 students	A	B	C	D	E
c. Calculate how many square feet of tile you need to cover your classroom floor	A	B	C	D	E
d. Estimate the area of your classroom floor using square feet	A	B	C	D	E
e. Find the amount of carpet needed to cover a rectangular floor if you know its length and width	A	B	C	D	E
f. Know when to take a turkey out of the oven if it goes in at 10:00 A.M. and it takes 3 hours and 45 minutes to cook	A	B	C	D	E

Post-task Questions for Mathematics Tasks - Grade 8

1. How hard was this task compared to most other tests you have taken this year in school?

A Easier than other tests

B About as hard as other tests

C Harder than other tests

D Much harder than other tests

2. How hard did you try on this task compared to how hard you tried on most other tests you have taken this year in school?

A Not as hard as on other tests

B About as hard as on other tests

C Harder than on other tests

D Much harder than on other tests

3. How important was it to you to do well on this task?

A Not very important

B Somewhat important

C Important

D Very important

4. How easy or difficult was this task?

A Extremely difficult

B Quite difficult

C Somewhat difficult

D Somewhat easy

E Quite easy

F Extremely easy

5. How many questions do you think you answered correctly on this task?

A None or almost none

B Some but less than half

C About half

D More than half but not all

E All or almost all

6. How much do you agree with each of the following statements? Select one answer choice on each row.

	Strongly disagree	Disagree	Somewhat disagree	Somewhat agree	Agree	Strongly agree
a. I applied a lot of effort to succeed on this task.	A	B	C	D	E	F
b. Taking this task was stressful.	A	B	C	D	E	F
c. Taking this task was challenging.	A	B	C	D	E	F
d. I felt time pressure when taking this task.	A	B	C	D	E	F

7. What math class are you taking or did you take this year? Select one or more answer choices.

A Eighth-grade math

B General eighth-grade math

C Algebra I course

D First year of a two-year algebra course

E Second year of a two-year algebra course

F Algebra I (one-year course)

G Algebra II

H Geometry

I Other

8. This school year, how often did you use the following digital devices to work on math problems, including homework? Select one answer choice on each row.

	Never	About once or twice a year	About once or twice a month	About once or twice a week	Every day or almost every day
Hand-held calculator	A	B	C	D	E
Desktop or laptop computer	A	B	C	D	E
Tablet (for example, Surface Pro, iPad, Kindle Fire)	A	B	C	D	E
Smartphone (for example, iPhone, Samsung Galaxy, HTC One)	A	B	C	D	E
Other device (Please specify): __________________________	A	B	C	D	E

9. In your math class this year, how often have you used the following types of calculators? Select one answer choice on each row.

	Never or hardly ever	Less than half of the time	About half of the time	More than half of the time	All or almost all of the time
Basic calculator	A	B	C	D	E
Graphing calculator	A	B	C	D	E
Calculator app on a smartphone	A	B	C	D	E
Calculator on a tablet	A	B	C	D	E
Calculator on a desktop or laptop computer	A	B	C	D	E

10. How much does each of the following statements describe a person like you? Select one answer choice on each row.

	Not at all like me	A little bit like me	Somewhat like me	Quite a bit like me	Exactly Like me
a. I enjoy doing math.	A	B	C	D	E
b. I look forward to my math class.	A	B	C	D	E
c. I am interested in the things I learned in math.	A	B	C	D	E
d. I think making an effort in math is worthwhile.	A	B	C	D	E
e. I think math will help me even when I am not in school.	A	B	C	D	E
f. I think it is important to do well in math.	A	B	C	D	E

11. How much do you enjoy solving the following types of math problems? Select one answer choice on each row.

	Enjoy not at all	Enjoy a little bit	Enjoy somewhat	Enjoy quite a bit	Enjoy a lot
a. Addition, subtraction, multiplication, and division	A	B	C	D	E
b. Finding areas of shapes and figures	A	B	C	D	E
c. Solving for probabilities and events (for example, card, coin, marble, and spinner problems)	A	B	C	D	E
d. Solving equations or simplifying expressions	A	B	C	D	E
e. Constructing and building different types of graphs (for example, bar graph, line graph, or box and whisker plots)	A	B	C	D	E
f. Working with geometric figures like rectangles and squares	A	B	C	D	E
g. Looking at patterns, graphs, and charts	A	B	C	D	E

12. Thinking about math, do you think that you would be able to do each of the following? Do not actually solve the problems. Select one answer choice on each row.

	I definitely can’t	I probably can’t	Maybe	I probably can	I definitely can
a. Estimate the weight of 5 apples using pounds (lbs)	A	B	C	D	E
b. Divide 42 stickers by 6 students	A	B	C	D	E
c. Calculate how many square feet of tile you need to cover your classroom floor	A	B	C	D	E
d. Determine a 20 percent tip of a 67 dollar restaurant dinner bill	A	B	C	D	E
e. Estimate the area of your classroom floor using square feet	A	B	C	D	E
f. Describe the properties shared by every isosceles right triangle	A	B	C	D	E
g. Find the amount of carpet needed to cover a rectangular floor if you know its length and width	A	B	C	D	E
h. Know when to take a turkey out of the oven if it goes in at 10:00 A.M. and it takes 3 hours and 45 minutes to cook	A	B	C	D	E

Post-task Questions for Reading Tasks - Grade 8

1. How hard was this task compared to most other tests you have taken this year in school?

A Easier than other tests

B About as hard as other tests

C Harder than other tests

D Much harder than other tests

2. How hard did you try on this task compared to how hard you tried on most other tests you have taken this year in school?

A Not as hard as on other tests

B About as hard as on other tests

C Harder than on other tests

D Much harder than on other tests

3. How important was it to you to do well on this task?

A Not very important

B Somewhat important

C Important

D Very important

4. How easy or difficult was this task?

A Extremely difficult

B Quite difficult

C Somewhat difficult

D Somewhat easy

E Quite easy

F Extremely easy

5. How many questions do you think you answered correctly on this task?

A None or almost none

B Some but less than half

C About half

D More than half but not all

E All or almost all

6. How similar were the questions on this task to the questions on your English/language arts quizzes and tests at school?

A Not similar at all

B Not similar

C Somewhat similar

D Quite similar

E Extremely similar

7. How much do you agree with each of the following statements? Select one answer choice on each row.

	Strongly disagree	Disagree	Somewhat disagree	Somewhat agree	Agree	Strongly agree
a. I applied a lot of effort to succeed on this task.	A	B	C	D	E	F
b. Taking this task was stressful.	A	B	C	D	E	F
c. Taking this task was challenging.	A	B	C	D	E	F
d. I felt time pressure when taking this task.	A	B	C	D	E	F

8. For school this year, how often do you use the following devices to read a story, article, or book? Select one answer choice on each row.

	Never	About once or twice a year	About once or twice a month	About once or twice a week	Every day or almost every day	Several times a day
a. Desktop computer	A	B	C	D	E	F
b. Laptop computer	A	B	C	D	E	F
c. Tablet (for example, Surface Pro, iPad, Kindle Fire)	A	B	C	D	E	F
d. Smartphone (for example, iPhone, Samsung Galaxy, HTC One)	A	B	C	D	E	F
e. Other device (Please specify): _______________________	A	B	C	D	E	F

9. In your English/language arts class this year, how often do you use a computer or other digital device to do each of the following? Select one answer choice on each row.

	Never	About once or twice a year	About once or twice a month	About once or twice a week	Every day or almost every day
a. Learn and practice vocabulary	A	B	C	D	E
b. Practice spelling and grammar	A	B	C	D	E
c. Access reading-related websites (for example, websites with book reviews and lists of recommended books)	A	B	C	D	E
d. Conduct research for reading and writing projects	A	B	C	D	E
e. Read books or articles	A	B	C	D	E

10. How much does each of the following statements describe a person like you? Select one answer choice on each row.

	Not at all like me	A little bit like me	Somewhat like me	Quite a bit like me	Exactly Like me
a. I read only if I have to.	A	B	C	D	E
b. Reading is one of my favorite activities.	A	B	C	D	E
c. I like talking about books with other people.	A	B	C	D	E
d. I find it difficult to finish books.	A	B	C	D	E
e. I feel happy if I receive a book as a present.	A	B	C	D	E
f. I think reading is important.	A	B	C	D	E
g. I enjoy going to a bookstore or a library.	A	B	C	D	E
h. I read only to get information that I need.	A	B	C	D	E
i. I like to exchange books with my friends.	A	B	C	D	E

11. Which of the following statements are true about the ways in which you and your teachers use computers or other digital devices in your English/language arts class? Select one answer choice on each row.

	Yes	No
a. There are computers or other digital devices that I can use for my English/ language arts schoolwork.	A	B
b. There is Wi-Fi or some other Internet connection that I can use for my English/ language arts schoolwork.	A	B
c. My English/language arts teacher uses computers or other digital devices when teaching my class.	A	B
d. My English/language arts teacher requires us to use computers or other digital devices to complete assignments.	A	B

12. Do you think you would be able to do each of the following when reading? Select one answer choice on each row.

	I definitely can’t	I probably can’t	Maybe	I probably can	I definitely can
a. Figure out the meaning of a word you don’t know by using other words in the text	A	B	C	D	E
b. Explain the meaning of something you have read	A	B	C	D	E
c. Understand the feelings of characters in a story	A	B	C	D	E
d. Figure out the main idea of a text	A	B	C	D	E
e. Figure out the theme of a text	A	B	C	D	E
f. Judge the reliability of sources (for example, how a website might be biased or inaccurate)	A	B	C	D	E
g. Credit others for their ideas (for example, citing sources, using endnotes and footnotes in reports)	A	B	C	D	E
h. Find text in a reading passage to help you answer a question on a test	A	B	C	D	E
i. Recognize when you don’t understand something you are reading	A	B	C	D	E
j. Recognize the difference between fact and opinion in a text	A	B	C	D	E
k. Critique an author’s craft or technique	A	B	C	D	E
l. Use evidence in a text to support my answer	A	B	C	D	E
m. Identify the author’s perspective in a persuasive text	A	B	C	D	E

13. In your English/language arts class this school year, how often have you done each of the following? Select one answer choice on each row.

	Never or hardly ever	Less than half of the time	About half of the time	More than half of the time	All or almost all of the time
a. I came to my English/ language arts class prepared.	A	B	C	D	E
b. I remembered and followed directions from my English/ language arts teacher.	A	B	C	D	E
c. I started working on reading assignments right away rather than waiting until the last minute.	A	B	C	D	E
d. I paid attention and resisted distractions during English/ language arts class.	A	B	C	D	E
e. I stayed on task without reminders from my English/ language arts teacher.	A	B	C	D	E
f. I paid attention in my English/language arts class even when I was not interested.	A	B	C	D	E

14. About how many books are there in your home?

A Few (0–10)

B Enough to fill one shelf (11–25)

C Enough to fill one bookcase (26–100)

D Enough to fill several bookcases (more than 100)

Thank You

Thank students for their time.

1 60 minutes for tryouts. 90 minutes for grade 8 cognitive item cognitive interviews

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Science Volume II Cog Lab Protocols
Author	BlairJ
File Modified	0000-00-00
File Created	2021-01-27