National Assessment of Educational Progress (NAEP) Accessible Booklet Cognitive Items in Reading 2011-2012 Cognitive Interviews

The Paperwork Reduction Act and OMB confidentiality texts are indicated below. Appropriate sections of this information are included in the consent forms and letters contained in the appendices for Volume I. The entire statements will be included on the front of the test materials used in the interviews.

Student Cognitive Interviews

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to take no more than 60 minutes including the time to review instructions and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this research protocol, please write to: U.S. Department of Education, Washington, D.C. 20202-4537.

Your answers may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].

Cognitive Laboratory Script

Text written in italics is to be read aloud by the interviewer.

NOTE: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. Text placed in brackets is generic text that will be tailored to suit the particular task being studied.

Introduction:

Hello, my name is ______ and I work at the University of Illinois. It’s nice to meet you and thank you very much for helping us out today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?
- If student responds with “language arts/reading/writing,” follow up with: Good, then I think you’ll enjoy what we are going to be doing today.
- If student responds with another subject, respond accordingly (e.g., if the student responds with “science,” ask them what they are studying in their science class).

Let me begin by explaining why I am here and what you are going to be doing. You are participating in a special study about <grade 4, grade 8> reading tasks, overseen by the National Center on Education Statistics, of the U.S. Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will be doing two reading tasks.

It’s okay if you do not know how to do the reading tasks. I will not be grading your work today, and no one will know that it was you who did these tasks. My goal is to learn how you react to the tasks, so please try to do your best.

If at any time you decide you do not want to go on, that is your choice and you may stop.

This interview is being recorded so we can review the tapes later. What you say may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573).

We’ll be going through the instructions first. After we do this, you’re going to be given 2 tasks to work on. You will not be graded on your work.

Instead of just reading each task and doing it, I’d like you to “think aloud” as you do the task. We will practice this before you start.

I will also have a few questions after you have finished working on each task. Your thoughts will help us make the tasks better.

So, do you have any questions before we start?

Ask the participant if he/she has any questions. After answering questions and giving further explanation, continue with the think aloud. If the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

OK, let’s begin.

2a. Think Aloud Modeling Script

You are going to be doing a reading task in a way that may be different from what you are used to. Instead of working quietly, I want you to read the questions out loud and then think out loud by telling me what you are thinking as you work on the questions. That includes any planning you may do to answer the questions and anything you are thinking as you come up with ideas for your answer.

I will give you an example of the think-aloud process. Then I will give you a chance to practice it.

You will not be graded on anything you say while you are thinking aloud. There are no wrong thoughts, and everything you think and say is important to us.

Now I’m going to show you how to think out loud. When I am finished you will get a chance to practice thinking aloud.

Think Aloud Explanation:

Remember: the italicized text should not be read word for word. You should be familiar enough with its contents to demonstrate a think aloud in a natural and conversational manner.

Since we can’t tell what is going in your brain, we need you to “think aloud.”

Let me give you an example about making a piece of toast this morning. If I were thinking aloud while I got the toast ready, it would sound something like this.

“What do I want...ok, I want toast. What kind? Maybe this kind with raisins…I love raisins. OK, first plug in the toaster oven, turn it on, close it. OK, done. Hmm (reading the bread package). How come they always have to advertise using this thing about low fat? Wait…serving size 1 piece of bread, Calories: 130… does not seem so bad…Vitamin A 0%, Vitamin C 0%, Iron 8%. So those are good. (Look up at the corner of the ceiling) Wait! Forgot to put the bread in the toaster! OK, OK, put bread in toaster…ow, hot, hot! Do I want butter or…(Ding) Open the toaster, don’t burn myself…I think I want butter, yeah. I’m really hungry…OK, toast is ready, have butter…this is good.”

The point of the think-aloud is not to describe what you are doing or to tell me what you plan to do. Just say out loud the words that are in your head.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I need to hear times when you are thinking in your head that you are confused and times when you are not.

2b. Student Practice

You may need to remind the student to talk aloud as he/she works through the question. If necessary, use the “Think-Aloud Hints” shown below to prompt the student, but be careful not to lead the student.

Now you will do a think-aloud practice. I’m going to ask you a question and you think aloud as you figure out your answer. Remember to say out loud whatever you are thinking. The question is:

“How many windows are in the house or apartment where you live?”

Some participants will be silent after hearing the question. Immediately encourage the participant to say whatever he/she is thinking. You may need to remind the participant that the answer to the question has to be a number.

2c. Additional Interviewer Model

Use only if you feel the student needs to observe another think aloud before moving on to the

actual questions.

Let me give you another example of thinking out loud. Look at this item. [Hand the student a copy of the item.] I am going to think out loud while I answer it.

Read both the question and the answer choices aloud.
Make sense of the question:

The question is asking me which of the following could be the length of the pencil I use in school.

Analyze each option:

All the options have the number 6 in them, but the units are all different.

Option A says 6 feet, which doesn’t make sense because that would be too big.
Option B says 6 pounds, [refer back to question] but the question asks about the length of the pencil and pounds are about weight.

Option C says 6 ounces, which is also weight.
Option D says 6 inches, [Look at your pencil] which would make sense for the length of a pencil.
My answer is D. [Fill in oval for option D on question.]

After the think aloud demonstration and practice, say:

Do you have any questions before we continue? [Answer any questions the student may ask.]

Okay, let’s move on to the actual task.

Because the information you provide is so important to us, I am going to be taking notes while you think aloud and do the task. Remember, you are not getting a grade on the task you do today.

2d. Think-Aloud Hints for the Interviewer

We’re interested in capturing everything that the subject is thinking while doing the task. Your goal is to have the participant speak aloud all his or her thoughts while doing the task. Grade 4 students may pose special challenges, as it may be hard for them to both work on a task and say their thoughts aloud at the same time. Do your best to encourage them, but be aware that you may learn more during the retrospective interview.

If the student is silent for 5 or more seconds or is continually providing short responses or not answering, use “continuers” to encourage the participant to be more descriptive. The goal is to get participants to verbalize their thoughts without “putting words in their mouths.” Don’t ask questions that lead the participant’s response or otherwise make them rush or change their approach to writing—such as planning before they write. You have to be as objective and unbiased as possible, but you may offer a verbal “nudge,” such as:

What are you thinking now?
Any other thoughts?
I see you are looking at the pictures/text. What are you thinking?
I see you have stopped answering. What are you thinking?
I see you are reading the item again. What are you thinking?

NOTE: If the student simply cannot think aloud and answer at the same time (judged by their ongoing difficulty doing so, and discomfort with doing do so), let them proceed with answering the questions, and then rely on the retrospective interview for information about the student’s reaction to the task.

2e. Behavioral Coding Hints for Interviewer

It is important to note when certain behaviors occur that could be evidence of problems with a task. When the following behaviors occur, note the occurrence and when it occurred:

Misreading or rereading a word or words in the task
Skipping a word or words in the task

Generic Protocol

Accessible Reading Cognitive Lab

2011-2012 Grades 4 and 8

Question Identifier: Task Number

Note: The preliminary information should be recorded for a task.

Ref #	Question/Prompt	Response
	Preliminary Information
1	Name of Interviewer
2	Date
3	Student ID	Record student’s unique ID number here.
4	Student’s Gender	 Male  Female
5	Special circumstances that may have affected the interview
6	Do you like studying language arts/reading/writing?	 Yes  No

Start with the introduction and think aloud practice in the cognitive lab script. Then say:

Now, you’re going to do the first reading task.

First you will read a passage to yourself. Then you will answer some questions about the passage. When you answer the questions, I would like you to read each question out loud and to “think aloud” as you answer. Just say what you’re thinking as you answer the questions. Please tell me whatever is going through your mind

You will have 25 minutes to complete the task. You will not be graded on what you answer.

I will also have a few questions after you have finished the task. Your thoughts about these tasks will help NAEP make the tasks better.

So, do you have any questions before we start?

Give the student paper and pencil and the first reading block.

3a. Think Aloud Portion

Ref #	Question/Prompt	Response
	PART 1: Student reads passage silently
1	Please read the passage to yourself.	Record your observations when the student reads the passage.
	PART 2: Student thinks aloud while doing task
2	Now please go ahead and answer the questions, telling me everything you are thinking. There is no right or wrong way to think aloud. I want to know whatever you are thinking as you do the task. I will be taking notes while you are working and talking.
	If necessary, remind the student to say aloud what his/her thought processes are while working. Use the “Think-Aloud Hints” as needed.	Record student’s think-aloud comments here.
3	Does the student reread the passage?	 Yes  No
4	How would you rate the student’s level of motivation while working on the task?	 High  Moderate  Low
5	Based on your observations, did the student demonstrate understanding of the task?	 Yes  No  Student demonstrated some understanding.
6	Did the student return to the passage to find answers?	 Yes  No
	PART 3: Interview with student
7	If a student in your class didn’t understand the passage, how would you explain it? How would you explain this passage to another student?
8	If the passage includes a picture or figure with information the student can be expected to use in his or her response: What did you learn from this picture/figure?
9	If the passage includes a picture or figure with information the student can be expected to use in his or her response: What did you think you were supposed to do with this picture/figure? Did you use it to answer the questions? If yes: Tell me what you thought about when you used it.
10	Before you started reading, what did you think about?
11	Did you go back to the text to find answers? Tell me how you did that.
12	What did you think you were supposed to do with these pictures/figures? Did you use them to answer the questions? If yes: Tell me what you thought about when you used them.
13	Did you change your mind about any of your answers? If yes: Tell me what you thought about when you did that.	 Yes  No
14	Is there anything that you think could make this task easier to understand? If Yes, What is it? Were there any words you did not understand in the task? If Yes, What were they?	 Yes  No Record student’s suggestions.  Yes  No

After completing steps 1 to 14 with first item block, repeat with second item block and add steps 15 and 16.

Ref #	Question/Prompt	Response
15	Have you done tasks like this in school or on other tests? If Yes, Tell me about them.	 Yes  No
16	Do you like to read in school? What kinds of feedback does your teacher give you about your reading? If student does not know word “feedback,” say “comments” from the teacher.	 Yes  No
	Thank you! Now I am going to ask you to do some more work with one of the passages.

3b. Read aloud portion.

After completing the Think Aloud protocol, select one of the item blocks, and one of the questions within the item block, according to the sampling plan provided, and complete the Read Aloud activity below.

Ref #	Question/Prompt	Response
	PART 1: Student reads each part of the task aloud
17	Please read this passage aloud. Now read this question.	Record your observations when the student reads the question.
	PART 2: Review of student answers
18	[for a selected response item] Record the option(s) selected by the student. [for an constructed response item] Record the student’s response.	 Student did not select an option.  Student did not respond.
19	Did the student appear to be certain of his/her answer?	 Yes  No  Student did not respond.
20	How would you rate the student’s level of motivation while working on this question?	 High  Moderate  Low
21	Based on your observations, did the student demonstrate understanding of this question?	 Yes  No  Student demonstrated some understanding.
	PART 3: Interview with student, Follow up with these questions only if the student has not already addressed them during the think-aloud.
22	Are there words or phrases in the question that you found hard to understand? If Yes, Which ones?	 Yes  No List unfamiliar words or phrases:
	[If yes, for each unfamiliar word/phrase, ask:] What does this word/phrase mean in your own words?
23	Did you have any other problems understanding the passage or question?
24	How would you explain what this question/word/sentence means to another student in your class?
25	Add item specific prompts, if any.
26	Is there anything that you think could make this question clearer? If Yes, what is it?	 Yes  No Record student’s suggestions.
27	Have you answered questions or done activities like this one in any of your classes in school or outside of school? if so: Where have you done activities like this one?	 Yes  No Record any courses student mentions.
	Thank you.

3c. Debriefing and Thank You

If it will be necessary to continue the cognitive lab at a later time, say the following:

I see it is time for you to get back to class and we still have some work left to do. If your teachers approve, would you like to do another interview just like this one.

If the student assents, say:

Great, I will let you know when we can meet again.

If the student does not wish to participate in another session, say:

That is fine. Thank you for working with me today.

If the session is complete and you do not need to see the participant again, say:

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the task?

Thank participant for his/her time.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	NATIONAL ASSESSMENT OF
Author	joconnell
File Modified	0000-00-00
File Created	2021-01-31