The National Assessment of Educational Progress (NAEP) 2021 Items Pretesting

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average 60-90 minutes, including the time to review instructions and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission, please write to: National Assessment of Educational Progress (NAEP), National Center for Education Statistics (NCES), Potomac Center Plaza, 550 12th St., SW, 4th floor, Washington, DC 20202.

Authorization and Confidentiality Assurance

National Center for Education Statistics (NCES) is authorized to conduct NAEP by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622).

All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C.§151).

OMB No. 1850-0803 APPROVAL EXPIRES 7/31/2020

Section I: Reading, Mathematics, and Writing Pretesting

A. Cognitive Interview Protocols

Cognitive interviews will be conducted for reading, mathematics, and writing for both grades 4 and 8.

1. Introduction to Study Script

NOTE: Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word-for-word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular items/tasks being studied. In general, protocols will be tweaked and customized in light of specific item and stimulus content.

Introduction:

Hello, my name is ______, and I work for [indicate organization]. I am here with my colleagues [introduce colleagues]. It’s nice to meet you, and thank you very much for helping us out today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?

When the student responds, follow up with two or three questions to get the student used to talking, such as:

That’s interesting—why do you enjoy <subject> so much?
What are you studying in <subject> at the moment?
And what’s the best thing you have studied in <subject> so far?

Good. Well, I think you’ll enjoy what we are going to be doing today. First, let me begin by explaining why I am here and what you are going to be doing. You are taking part in a special study looking at new test questions to find out what students know in <reading OR mathematics OR writing>. It is part of the National Assessment of Educational Progress, or NAEP for short, a test that is funded by the U.S. federal government and run by the National Center for Education Statistics in the Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will help us develop better tests for other students. Overall, this session should take about <length> minutes.

It’s okay if you don’t know how to do any of the questions. I will not be grading your work today. Your answers will help us improve the questions and will not be shared with your parents or school. My goal is to learn how you react to the questions, so please try to do your best.

If at any time you decide you don’t want to go on, that is your choice, and you may stop.

This new test is on the tablet. All instructions about how to do the test are also on the tablet.

This interview is being recorded so we can review and learn from it later. We will also learn from what you tap and other actions you take on the screen.

We will work together on <number of items/tasks>.

Do you have any questions?

After answering any questions and giving further explanation, continue with instructions for additional skill measures and/or think-aloud training. If the student is no longer interested in participating, thank the student for his/her time and end the interview.

Follow the instructions for Additional Skill Measures if applicable here before moving to the next section.

Okay, let’s move on. Before we look at the real questions, I want to make sure you have a good idea of what we’re going to be doing. So, I am going to give you some practice questions. The practice questions should help you get used to what we will be doing during the real questions. They should help you understand how we want you to respond. Do you have any other questions before we start?

After answering any questions and giving further explanation, continue with the think-aloud training. If the student is no longer interested in participating, thank the student for his/her time and end the interview.

2. Instructions for Additional Skill Measures

2a. Instructions for the Read-Aloud Word Recognition Test

A short read-aloud word recognition test will be considered for administration as part of the reading cognitive interview study.

FOR READING ONLY^¹: After answering any questions and giving further explanation, continue with the read-aloud word recognition test.

INSTRUCTIONS FOR THE READ-ALOUD WORD RECOGNITION TEST

I have some lists of words that I want you to read one at a time.
I will listen and make notes while you read. Then I will probably give you another list.
Some of the words will be easy for you and some will probably be very hard.
If you don't know a word right away, try your best to figure it out.
I cannot help you in any way and I cannot tell you if you are right or wrong. Do not worry about how well you do.
Do you have any questions before we go on? [Answer any questions the student may ask.]
Here is the first list. When you are ready, begin reading the words on the list.

3. Concurrent Think-Aloud

The concurrent approach is primarily for mathematics, but not for reading and writing because the cognitive demands of talking while reading or writing would interfere with the verbal processes needed for the task.

3a. Concurrent Think-Aloud: Instructions and Modeling Script

NOTE: Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word-for-word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students, especially 4^th graders, when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular items being studied.

To help us make our test better, we will ask you to complete some questions.

[If using Morae® or other screen capture software]: While you are working on the questions, we will be recording everything that happens on the screen. To help us know how well everything is working, the screen recording will capture all of your responses and movements on the tablet, such as what you tap on and anything you put into text boxes.

I’m going to ask you to do the questions in a way that may be different from what you are used to. Instead of working quietly, I want you to tell me what you are thinking as you work through the questions. We call this a “think-aloud” because we are asking you to say everything you are thinking out loud.

In a moment, I will give you an example of the think-aloud process. Then I will give you a chance to practice it. You won’t be graded on anything you say while you are thinking aloud. There are no incorrect thoughts, and everything you think and say is important to us.

Okay, now I’m going to show you how to think out loud—this will help you see how I want you to describe what you are thinking as you are working on the questions. When I am finished I’ll ask you to try it, so you can see how it works.

Think-Aloud Demonstration:

Since we can’t tell how you are thinking about a question while you are working on it, we need you to think aloud.

Let me give you an example. (Place example question in front of student.) Look at this question. It asks me to look at the five animals and choose two that are the most similar. So, I am going to do this task, and I will think aloud while I work on it.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar.

A. Beetle

B. Mouse

C. Crab

D. Dolphin

E. Cow

Okay, so I am reading the question…It says I have to decide on the two animals that are the most similar. Hmm, what do they mean by “similar?” That could mean anything. Well, I see the choices are beetle, mouse, crab, dolphin, and cow. There are some things about them that are similar, but there are also differences. I’m thinking the beetle and crab are kind of similar, in a way—well, they both have lots of legs and they kind of move around fast. But one lives in the water and the other doesn’t, so that makes them different. I guess the mouse also moves around fast, but it’s not similar to either the beetle or the crab, except they are all small. Then there is the dolphin that lives in the water, too…but that’s about all it has in common with the crab, so I wouldn’t say they are very similar even though they’re both in the sea. Hmm, so the mouse and the cow are both mammals, they both have fur and they have live babies instead of laying eggs or whatever…oh right, but dolphins are mammals too, aren’t they? Yes, I think they are…but really they seem so different from mice and cows to me. I don’t know…this seems like a hard question…I’m not sure what to choose. Okay, I think I am going to go for…mouse and cow, because they are both mammals and they both live on the land.

Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do as you are thinking about the questions today. The point of the think-aloud is to get at whatever is in your head as you are doing the questions. Just say aloud the words and the thoughts that are in your head, as you are thinking about and working on the questions.

3b. Concurrent Think-Aloud: Student Practice Scripts

Practice Script #1

Interviewers should place the practice question in front of the student so he or she can read it. Some students will be silent after reading the question. Students should be immediately encouraged to say whatever they are thinking.

It may be necessary to remind the student to talk aloud as he/she works through the questions. If necessary, interviewers should use the “think-aloud hints” given below to prompt the student, being careful not to lead the student. The interviewer needs to be familiar enough with the information to introduce the think-aloud in a natural, conversational manner.

Now you will try a think-aloud. You can use this example. Like last time, you have to pick two of these things that you think are the most similar. As you are reading the question and the choices, and as you are figuring out your decision, I want you to talk out loud to tell me all of the thoughts that are in your head at each moment.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I’ll be reminding you to think aloud if you get quiet because I need to hear all of your thoughts.

Okay, now you try. Go ahead and start working on this question and remember to think aloud as you are doing it.

Question:

Which two living things below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the questions? What questions would you like to ask me?

If the student says he or she feels okay and doesn’t have any questions, say: Good, then let’s begin our study. If the student expresses concerns, says he or she has questions, or appears to be hesitant or reluctant, ask him or her to say more about the concerns or questions and try to address the concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him or her to stop.

Practice Script #2 (Optional)

Use only if you feel the student would benefit from another think-aloud practice before moving on to the actual questions. It may be beneficial to instead model thinking aloud again for the student rather than doing another practice [see Think-Aloud Demonstration in 3a].

Very good—let’s do another one before we start the real tasks. Are you ready? Here is the next practice question. Remember to think out loud as you begin to think about this question and all the way through. Tell me what you are thinking as you work through it.

During the practice item, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts below).

Question:

Which two of the following objects have the most similar characteristics? Circle the two that are most similar.

A. Silver coin

B. Chocolate coin

C. Gold coin

D. Blue plastic coin

E. Brown plastic coin

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are working on it? What questions would you like to ask me?

If the student says he or she feels okay and doesn’t have any questions: Good, then let’s begin our study. If the student expresses concerns, says he or she has questions, or appears to be hesitant or reluctant, ask him or her to say more about the concerns or questions, and try to address his or her concerns or uncertainties in a supportive way. If the student indicates he or she does not wish to continue or does not feel comfortable continuing, allow the student to stop.

3c. Concurrent Think-Aloud: Starting the Items

Now we will move on to the actual questions.

Remember, as you do all the activities and answer all the questions, I’d like you to say aloud everything that you’re thinking, and I may remind you to do that if you are quiet.

The questions should take about <length> minutes. Remember, you will not be graded on your responses to the questions and there is no right or wrong way to think aloud, as long as you keep telling me your thoughts. Your thoughts will help us make the questions better. I will also have a few questions after you have finished working.

Do you have any questions before we go on? [Answer any questions the student may ask.]

Because the information you provide is so important to us, I am going to be taking notes while you think aloud and answer the questions. Here are the questions. When you are ready, go ahead and start working.

3d. Concurrent Think-Aloud: Hints for the Interviewer

The goal of think-aloud is to capture all the student’s mental processes while working through items/tasks. Interviewers must strive to have the student speak aloud all of his or her thoughts while answering the items. If a student is continually providing short utterances or not talking, interviewers should use “continuers” to encourage students to be more descriptive. The goal is to get students to verbalize thoughts without putting words in students’ mouths. Care should be taken not to ask questions that lead students’ responses, in particular, directions, or make them rush or change their approach. Interviewers should be as objective and unbiased as possible.

In general, if the student is silent for approximately five to 10 seconds, interviewers should use the following as a guide for encouraging the student to describe his or her thoughts or to help the student elaborate on his or her responses.

If the student is not verbalizing enough, interviewers should offer a verbal nudge to remind the students to keep talking, such as:

What’s going on in your head right now?
Remember to keep talking.
You seem to be thinking hard. Can you tell me what’s in your mind right now?
What are you thinking now?

If the student says something about his or her thoughts, but it is difficult to tell exactly what he or she means, or if the student begins to talk but does not say very much, interviewers can urge the student to elaborate. For example:

Tell me a bit more about that.
That’s interesting. Can you say more about that?
Go on…
Okay, and what else are/were you thinking?

If more clarity or a more explicit description of a student’s thoughts is desirable:

Can you explain what you mean by that?
Can you say a bit more to help me understand what you mean/are thinking/were thinking?

It is important to be responsive and sensitive to each student’s behavior. If it becomes clear that students cannot tell any more about their thoughts, interviewers should not keep asking. Students should be encouraged to think aloud as much as they can, but their reactions should be closely observed, and pushing them should be avoided if students seem frustrated or uncomfortable. While it is desirable that students articulate as much as they can, sometimes students will simply be unable to state what is in their mind beyond a basic reply such as, “I am reading the question.” Interviewers must be sensitive to nonverbal signals that may indicate students cannot say any more than they have or that they need a few extra seconds to gather their thoughts to put them into words.

In addition, during observations if it seems a student is hung up on something, interviewers should note when this occurs.

4. Retrospective Think-Aloud

The retrospective approach is used for reading and writing, because the cognitive demands of talking while reading or writing would interfere with the verbal processes needed for the task. Note that the retrospective approach can also be used for mathematics if students have particular difficulty with the concurrent approach. The flexibility to use either think-aloud method has proven very effective in the past.

4a. Retrospective Think-Aloud: Instructions and Modeling Script

Using the retrospective (versus concurrent) think-aloud approach, students are instructed to complete a vocabulary half-block or writing item as they would under normal testing conditions (silently, without interruption). Then, after the work is completed, they will be asked to describe what they were thinking while reviewing their work on the computer. A replay of the block or item (e.g., using software like Camtasia® or Morae®)^² will help cue the student’s recall of the thought processes that occurred during the vocabulary block or writing item as the student progressed through it.

NOTE: Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, interviewers should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that can be tailored to suit the particular writing task being studied.

To help us make our test better, we will ask you to complete <reading questions OR mathematics questions OR writing tasks>. While you are working, we will be recording everything that happens on the screen. The screen recording will capture all of your answers, writing, and movements.

First, you’ll work on the <reading questions OR mathematics questions OR writing tasks> you will see on the screen, working at your own pace. After you have finished, I’m going to ask you to go back over the test with me. As we go back over your work, I will ask you to tell me what you were thinking as you were working on <reading questions OR mathematics questions OR writing tasks>. We call this “thinking aloud,” because we are asking you to say out loud everything you were thinking while you worked.

To help you remember what you were thinking as you worked, we will look at the recording. This way, you will be able to see your screen actions and what you chose for answers or wrote. As we watch the recording, I will be asking you to tell me what you were thinking at different points. Does that make sense? Do you understand what we’ll be doing?

Answer any questions and clarify the sequence of steps, if needed.

In a moment, I will give you an example of the think-aloud process. Then I will give you a chance to practice it. You won’t be graded on anything. There are no incorrect thoughts, and everything you think and say is important to us.

Okay, now I’m going to show you how to think out loud. When I’m finished, I’ll ask you to try it.

Think-Aloud Demonstration:

Since we can’t tell what is going on in your head, we need you to “think aloud.”

Let me give you an example. (Place example question in front of student.) Look at this question. It asks me to choose which two animals are the most similar. I’m going to do this question just as I would for a test. After I have finished, I will try to remember what I was thinking as I was working on the question so I can tell you.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar.

Beetle
Mouse
Crab
Dolphin
Cow

The interviewer “works” silently for about a minute on the question. The interviewer should enact some behaviors that suggest that he or she is thinking hard about the question and carefully considering the possible answers; for example, pointing with a pencil tip at each option in turn; moving a pencil or fingertip between answer options; returning to the question; giving nonverbal signals indicating consideration, uncertainty, etc. Finally, the interviewer circles two answers, B and E, and then begins the retrospective think-aloud.

Okay, so here is what I was thinking as I did this task. First, I read the question…It said I had to decide on the two animals that are the most similar. So, I read the choices—beetle, mouse, crab, dolphin, and cow. I thought the beetle and crab are kind of similar because they both have lots of legs and they move around fast. But one lives in the water and the other doesn’t, so that makes them quite different. I was thinking the mouse also moves around fast but it’s not similar to either the beetle or the crab, except they are all small. Then I was thinking the dolphin lives in the water, but that’s about all it has in common with the crab. So, then I thought, the mouse and the cow are both mammals, they both have fur and they have live babies instead of laying eggs. Even though they are not very similar, the others seemed too different from each other in really basic ways, like where they live and their kind of body. So, mouse and cow seemed like the best choice to me, because they are both mammals and they both live on the land.

Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do as you are thinking about your <reading questions OR mathematics questions OR writing item> today. The point of the think-aloud is to get at whatever was in your head as you were working. Just say aloud the words and the thoughts that are in your head, as you are thinking.

Ok, let’s have you give it a try.

4b. Retrospective Think-Aloud: Student Practice Script I

Interviewers should place the practice question in front of the student so he or she can read it. Allow the student as much time as needed to answer the question. When he or she indicates he or she has finished, ask the student to begin thinking aloud. Some students will be silent after reading the question. Such students should be asked to say whatever they were thinking. If necessary, interviewers should use the “Think-Aloud Hints” shown below to prompt students, being careful not to lead students. The interviewer needs to be familiar enough with the script to encourage the think-aloud in a natural, conversational manner.

Now you will try a think-aloud. I will give you a question like the one I just did. You will read and answer the question and then let me know when you’re done.

Once you are done answering, I will ask you to look back at the question and the answer you chose and try to say out loud the thoughts that were in your head at each moment. If I don’t hear you speaking I’ll ask you to keep talking. I’m telling you that so you won’t think I am criticizing the way you are thinking. I’ll be reminding you to think aloud if you get quiet because I need to hear all of the thoughts you had while reading and answering the question.

Hand students the question at this point.

Okay, now go ahead and start working on the question. Let me know when you are finished.

Question:

Which two organisms below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

When students indicate they are finished, make sure they have circled two answers, and then ask them to begin saying their thoughts out loud. As necessary, include the appropriate prompting questions, such as the following:

Okay, so tell me what you were thinking as you were reading and answering this question. What did you think at the beginning, when you started to read the question?…And then what were you thinking?…And what else did you think about?…Anything else?…Can you tell me more about that?…etc.

4c. Retrospective Think-Aloud: Student Practice II (Optional)

Use only if you feel the student would benefit from another retrospective think-aloud practice before moving on to the actual task. It may be beneficial to instead model thinking aloud again for the student rather than doing another practice (see think-aloud demonstration in section 4a).

If the student struggles to think aloud, the interviewer should give the student another opportunity to practice. The interviewer should praise the student for the first attempt regardless of how good it was, for example, “Very good—let’s do another one before we start the real <reading questions OR mathematics questions OR writing item>. Are you ready? Here is the next practice question. (Place example question in front of student.) Remember that after you have finished answering this question I will ask you to talk out loud to tell me what you were thinking while you were working. Go ahead and do the question, and let me know when you have finished.”

As before, during the post-task think-aloud phase, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts in section 3d).

Question:

Which two of the following objects have the most similar characteristics? Circle the two that are most similar.

A. Silver coin

B. Chocolate coin

C. Gold coin

D. Blue plastic coin

E. Brown plastic coin

After the think-aloud phase is finished:

Now that you have practiced, do you feel that you understand how you should talk aloud about what you were thinking while you were doing the <reading questions OR mathematics questions OR writing task>? Is this something you feel okay about doing?

If students say yes: Good, then let’s begin our study. If students say no or appear to be hesitant or reluctant, ask them to say more about any reservations they have and try to address their concerns or uncertainties in a supportive way. If students indicate they do not wish to continue or do not feel comfortable continuing, allow them to stop.

4d. Retrospective Think-Aloud: Starting the Tasks

Now we will move on to the actual <reading items OR mathematics item OR writing task>. Remember, after you have finished, I will ask you to say aloud everything that you remember thinking while you were working, and I may remind you to do that if you are quiet. This task should take you about <length> minutes, but let me know when you feel you are done working. Remember, you will not be graded on what you do. There is no right or wrong way to think aloud, as long as you keep telling me your thoughts. Your thoughts will help us make the test better. I will also have a few questions after we have finished.

Do you have any questions before we go on? [Answer any questions the student may ask.]

Because the information you provide is so important to us, I am going to be taking notes while you are working and while you are thinking aloud afterwards.

The <reading item OR mathematics questions ORwriting item> is here on the screen. Here is a piece of paper and a pencil you may use if you wish to plan your writing. And, here is a brochure to help you plan and revise your writing if you wish.

When you are ready, go ahead and start working.

5. Post Think-Aloud Follow-Up Questions (Verbal Probing)

After completing the think-aloud process for items, interviewers will follow up with a brief period of focused questioning. The purpose of the additional post-task questions are to capture more information on issues such as student actions during the course of responding to items, particular aspects of <reading OR mathematics OR writing> targeted in the items, and general reflections about the items, stimulus, tools, and/or reading passage. The post-item questions will include targeted questions that are item-specific. These will be selected prior to testing. Interviewers will receive the set of several questions for the items they are examining in each cognitive interview.

5a. Verbal Probing: Examples for Discrete Items

Interviewers will also be trained to generate post-item questions on the fly, for cases where they observe something that they judge needs follow-up probing. The following are some examples of questions that might be asked at this stage. Additional probes of similar nature and targeted to specific items will be further developed and used in cognitive interviews.

Table 1. Example Probes for Discrete Items

Example	Probe	Instructions for Interviewer
1	Were there any words or parts of this <question OR passage> that were confusing?  Yes  No	Ask this probe for all discrete questions.

2	What did you find confusing? What could we do to make the <question OR passage> less confusing?	Ask both probes only if the respondent answered YES to the previous question.

3	Did the <images OR passage OR video OR audio OR stimulus OR tools OR etc.> for the question help you answer the questions?  Yes  No	Ask this probe for all discrete questions.

4	If yes, how did this part help you when answering the question? If no, why not?	Ask one or other probe, according to the answer to the previous question.
5	What did you think this writing task was asking you to do? Tell me in your own words.	Ask this probe for all writing tasks.

6	What did you find easiest in the <question OR passage>?	Ask this probe for all discrete questions.

7	What did you find hardest in the <question OR passage>?	Ask this probe for all discrete questions.

8	Was there anything about the <tool> that you found confusing?  Yes, a little  Yes, somewhat  Yes, a lot  No	Ask this probe for all discrete questions.

5b. Verbal Probing: Hints for the Interviewer

In general, if the student is silent for approximately five to 10 seconds in response to a verbal probe in 5a, interviewers should use the following as a guide for encouraging the student to describe his or her thoughts, or to help the student elaborate on his or her responses.

If the student is not responding to a verbal probe or is not verbalizing enough, interviewers should offer a verbal nudge to remind the students to keep talking, such as:

What’s going on in your head right now?
Remember to keep talking.
You seem to be thinking hard. Can you tell me what’s in your mind right now?
What are you thinking now?

If the student says something about his or her thoughts but it is difficult to tell exactly what he or she means, or if the student begins to talk but does not say very much, interviewers can urge the student to elaborate. For example:

Tell me a bit more about that.
That’s interesting. Can you say more about that?
Go on…
Okay, and what else are/were you thinking?

If more clarity or a more explicit description of a student’s thoughts is desirable:

Can you explain what you mean by that?
Can you say a bit more to help me understand what you mean/are thinking/were thinking?

It is important to be responsive and sensitive to each student’s behavior. If it becomes clear that students cannot tell any more about their thoughts, interviewers should not keep asking.

6. Observing Student Actions and Behaviors

As a student is working through items/tasks, the interviewer and the second observer should take notes regarding any of the student’s expressions or behaviors that may reflect the status of the student’s understanding, engagement, or reactions to items and/or stimulus (e.g., a reading passage). The following are some examples of such behaviors for interviewers to note.

Table 2. Example Notes for Student Actions and Behaviors

Example	Note	Instructions for Interviewer
1	Does the student express signs of confusion, boredom, or interest?  Yes  No	Take notes for all discrete questions.

2	If yes, where?	Take notes only if the respondent answered YES to the previous question.

3	Does the student use tools provided by tapping on tabs or buttons?  Yes  No	Take notes for all discrete questions.

4	Does the student (a) rapidly move through the screen or (b) take his or her time?  (a)  (b)	Take notes for all discrete questions.

5	Does the student spend a lot of time on a particular part of the item or tool or reading passage?  Yes  No	Take notes for all discrete questions.

6	If yes, which part of the item or tool or reading passage?	Take notes only if the respondent answered YES to the previous question.

7	Does the student seem discouraged?  Yes  No	Take notes for all discrete questions.

8	If yes, why?	Take notes only if the respondent answered YES to the previous question.

These kinds of observations should be considered opportunities to prompt students to think aloud and to encourage them to express what they are thinking. Interviewers should also make a note of any reactions to or interactions with items, stimuli, or a reading passage that appear valuable for follow-up with some additional questions after work is completed (see section II.A.3).

7. Debriefing and Thank You for Cognitive Interview

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the questions?

Is there anything you would like to ask me about what we did today? [Answer student questions.]

Thank you for helping us to improve our test.

Thank the student for his or her time. Provide a gift card (both for student and for the parent/legal guardian if the parent/legal guardian brought the child to and from the session).

B. Tryout Protocols

Tryouts will be conducted for reading and mathematics discrete (DI) blocks at both grade 4 and 8, and reading vocabulary half-blocks at grades 4 and 8.

1. Tryout

NOTE: Text written in italics is to be read aloud by the interviewer. The interviewer should not read the script word for word but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text in brackets is generic text that will be tailored based on task specifics.

1a. Tryout: Session Information

The interviewer should complete the information from rows one to three prior to the beginning of the tryout session. Rows four to five are to be completed at the end of the tryout session.

Ref#	Preliminary Information	Data
1	Name of interviewer
2	Date
3	Start time of interview
4	End time of interview
5	Special circumstances that may have affected the tryouts session (Do not ask test taker, just record observations.)

1b. Tryout: Introduction Script

Hello, my name is _______ and I work for [indicate organization]. I am here with my colleagues [introduce colleagues]. It’s nice to meet you, and thank you very much for helping us out today.

Let me begin by explaining why you’re here and what you’re going to be doing today. You are participating in a special study to try out new <reading OR mathematics> test questions. This is part of something called the National Assessment of Educational Progress, or NAEP for short, which is conducted by the National Center for Education Statistics within the U.S. Department of Education. Today we’ll be asking you to try out some questions that are being developed for fourth, eighth, and twelfth graders all over the United States.

If at any time you decide you do not want to go on, that is your choice and you may stop.

What you say will be used only to help us improve these test questions. What we learn from you and other participating students will be summarized in a report where no student names will be listed. After the session, we will use the session’s recording to review what we learned to help us understand how our test is working and how we can make it better. Overall, this session should take about <length> minutes.

Do you have any questions?

Interviewer: If a student is no longer interested in participating, thank the student for his or her time and end the tryout session for that one student only; continue with all others.

Instructions

Ok, now, I’m going to give you a set of questions to complete.

This is a timed activity, which will last for <length of activity>, and everyone will begin at the same time.

All of the instructions will be displayed on your tablet screens.

The tasks and questions you are about to do is something that students, like you, will do on the tablet. However, we aren’t finished making them yet, so there may be some parts that don’t work. Try to work through and complete the questions as best as you can.

You will not be graded on your answers to the questions, but please answer the questions as if it was a real test.

We will all work at the same time. First, I would like you to read the instructions. Once everyone has gone through the instructions, we will begin the questions together. I will start the timer, and I will stop you once the time is up.

Okay, I will come around and open the questions for each one of you, but please do not begin until I say that you may start. [OR: Ok, when I say begin, please press “Start” to begin working.]

If you have any questions about technical problems, please raise your hand and one of us will come to help you.

Okay, now that everyone is ready, you may begin.

1c. Instructions for Reading Skill Measure

Note: We will incorporate a reading skill measure (for example, the Gates MacGinitie Reading Test) as part of reading tryouts.

1d. Tryout: Starting the Items

START THE TIMER NOW. STUDENTS HAVE <60 OR 90> MINUTES TO COMPLETE THE QUESTIONS. STOP STUDENTS AT <60 OR 90> MINUTES EVEN IF THEY HAVE NOT FINISHED.

Wait either for all students to complete the task and/or set of items OR for the full amount of time to be up.

Once students have completed the task and/or set of items, make sure that all students exit the task and submit their answers.

2. Post-Tryout Follow-Up Questions

After completing the tryout tasks and/or set of items, the interface may provide some follow-up questions. Post- tryout questions may include some or all of the following:

A standardized post-task/item question that all students will be asked following all tasks/set of items, which is designed to discover whether the student has prior knowledge of the content.
Additional targeted questions that are task/item specific. These will be selected for each task/item prior to testing.
Standardized <reading OR mathematics> multiple-choice questions that students will be asked following the task/item.
Additional NAEP questionnaire items will be considered to place try-out results into context, such as students’ experience with technology and/or questions about student engagement.

Standardized Question for All Tasks: Task-Specific Prior Knowledge

Have you studied anything related to this task in school, or have you learned about or come across these things in your own life? [If yes: What have you learned or studied or experienced that is related to this task?]

Additional Questions: Item-Specific Issues

The purpose of the additional post-item questions is to capture more information on issues such as student actions during the task, particular aspects of <reading OR mathematics> targeted in the task, and general reflections about the task. The following are some examples of questions that might be asked at this stage.

Did any words in the task confuse you?
Did you understand the directions?
What did you find easiest in the task? What did you find hardest?
What part of the task took you the most time?
Did the <images OR videos OR audios OR etc.> for the task help you answer the questions asked?
Was there anything about the <tool> or the instructions how to use the tool that you found confusing?
What did you find confusing about the <tool>?
What could we do to make the <tool> less confusing?

3. Post-Tryout Discussion Session

At the conclusion of some tryouts, the facilitator may ask the group of students up to three task-specific questions. The following are some examples of questions that might be asked at this stage.

What ways could we change the item(s) to make it better?
Did the <images OR videos OR audios OR etc.> for the task help you answer the questions asked?

4. Debriefing and Thank You for Tryout

Before we finish, I’d like to hear what you all thought about what you’ve been doing.

Is there anything else any of you would like to tell me about working on the test question(s)?

Did you have any problems understanding <any of the questions OR anything>?

Did you have any problems understanding what you were supposed to do to answer any of the questions?

Is there anything that you think could make <this OR these> test question(s) clearer?

Is there anything you would like to ask me about what we did today? [Answer students’ questions.]

Thank participant(s) for their time and escort them to the front desk. Provide gift card(s).

Section II: Questionnaire

The purpose of the cognitive interviews is to inform the development of the student, teacher, and school administrator questionnaires prior to pilot testing. Below are the probes that students, teachers, and school administrators will be asked during the cognitive interview. Please note that some probes may not be addressed if time does not permit.

A. Interviewer Welcome Script and Assent/Consent

The following script does not have to be read verbatim. The interviewer should be familiar enough with the script to introduce the participant to the cognitive interview process in a conversational manner. The text in italics is suggested content for the interviewer to become thoroughly familiar with in advance. The interviewer should project a warm and reassuring tone toward the participant in order to develop a friendly rapport. The interviewer should use conversational language throughout the interview.

After answering questions and giving further explanation, begin the interview with the first item.

Note: Students, teachers, and school administrators will be administered all items and probes. The duration of each cognitive interview will be 90 minutes.³

1. Student Welcome Script

Hello, my name is _______.

Thank you for agreeing to participate in our study. Your help is very important to us. Let me tell you a bit more about what we are going to do today.

I have some questions that students in many different schools across the country will be asked to answer about themselves and their families. The questions will be part of the National Assessment of Educational Progress, or NAEP for short.

Our goal is to make these questions better so that other students can easily understand them. This is why we need your feedback.

We want you to help us find out which questions are easy to understand and which are difficult to understand. There are no right or wrong answers. Your feedback will help the National Center for Education Statistics, or NCES, which is part of the United States Department of Education, make these questions clearer for students like you. In the next hour we will try to get through as many questions as possible.

This interview is being audio recorded so that researchers can review the recordings later. This recording is only meant to help us summarize what we learn about the questions we are trying to improve. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Do you have any questions?

If, for any reason, the participant is no longer interested in participating, thank the participant for his or her time and end the interview.

2. Teacher Welcome Script

Hello, my name is ______.

Thank you for coming here today to help us.

I have some questions that teachers in many different schools will be asked to answer about themselves, their instructional practices, and their schools. The questions will be part of the National Assessment of Educational Progress, or NAEP for short. There are no wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics, or NCES, part of the United States Department of Education, make these questions clearer for teachers like you. In the next hour we will try to get through as many questions as possible.

This interview is being audio recorded so that researchers can review the recordings later. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C.§151).

Do you have any questions?

If, for any reason, the participant is no longer interested in participating, thank the participant for his or her time and end the interview.

3. School Administrator Welcome Script

Hello, my name is ______.

Thank you for coming here today to help us.

I have some questions that school administrators in many different schools will be asked to answer about their school policies and characteristics. The questions will be part of the National Assessment of Educational Progress, or NAEP for short. There are no wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics, or NCES, part of the United States Department of Education, make these questions clearer for school administrators like you. In the next hour we will try to get through as many questions as possible.

Do you have any questions?

If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

B. Instructions and Generic Probes

The interviewer should ask the participant to read and answer the questions, one at a time. Each question will be either one discrete question (multiple choice or free response) or one matrix question (i.e., an item stem with two or more multiple choice sub-items). The respondent is not supposed to read the question aloud, but should work on the question in the same fashion he or she would during a NAEP test or survey administration. If a respondent indicates they do not know a word or phrase, the interviewer can read words aloud and/or explain the meaning of words to the respondent.

For most items, generic probes for discrete and matrix items will include the following:

Generic Probes for Discrete Items (Multiple Choice and Free Response)—Full Set

No.	Probe	Instructions for Interviewer
1	Can you explain to me what the question is asking? Please try to explain without repeating the wording of the item.	Ask this probe for all discrete questions AFTER the respondent has answered the question.
2	Were there any words or parts of this question that were confusing?  Yes  No	Ask this probe for all discrete questions.
3	What did you find confusing? What could we do to make the question less confusing?	Ask both probes only if the respondent answered YES to the previous question.
4	Would you say it was very easy, easy, difficult, or very difficult to answer this question?  Very Easy  Easy  Difficult  Very Difficult	Ask this probe for all discrete questions.
5	How could we make it easier to answer this question?	Ask this probe only if the respondent answered DIFFICULT or VERY DIFFICULT to the previous question.
6	How sure are you about your answer to this item?  Very Unsure  Unsure  Sure  Very Sure	Ask this probe for all discrete questions.

Generic Probes for Matrix Items—Full Set

No.	Probe	Instructions for Interviewer
1	Look back at the first part of the item where you were asked a question. Can you explain to me what this question is asking? Please try to explain without repeating the wording of the item.	Ask this probe AFTER the respondent has answered the entire matrix question. This probe is specifically for the “stem” of the item, not the entire matrix item or a specific part/sub-item.
2	Can you explain to me what [part a., b., c., etc.] means? Please try to explain without repeating the wording of the item.	Ask this probe for all parts/sub-items. Please note probes 2-6 should be asked together for one part/sub-item before moving on to the next part/sub-item.
3	Did you find any words or part of [part a., b., c., etc.] confusing?  Yes  No	Ask this probe for all parts/sub-items.
4	What did you find confusing? What could we do to make [part a., b., c., etc.] less confusing?	Ask both probes only if the respondent answered YES to the previous question.
5	Would you say answering [part a., b., c., etc.] was very easy, easy, difficult, or very difficult?  Very Easy  Easy  Difficult  Very Difficult	Ask this probe for all parts/sub-items.
6	How could we make it easier to answer [part a., b., c., etc.]?	Ask this probe only if the respondent answered DIFFICULT or VERY DIFFICULT to the previous question.
7	How sure are you are you about your answers to this item?  Very Unsure  Unsure  Sure  Very Sure	Ask this probe for all matrix items. Ask this probe once per matrix, after completing all other probes pertaining to parts/sub-items.
8	Were you comfortable answering this item?  Yes  No	Only ask this probe for matrix questions inquiring about potentially sensitive topics. Ask this probe after completing all other probes pertaining to parts/sub-items.
9	How could we make this item more comfortable to answer?	Ask this probe only if the respondent answered NO to the previous probe. This probe should be skipped if the previous probe was not administered.

In some cases, items require less in-depth examination. Existing items subjected to minor revisions, such as updated/clarified response options, require administration in cognitive interviews largely to verify that the minor change has not made the existing item unclear or confusing. In these cases, using a brief set of generic probes can reduce participant burden. For these items, generic probes will be used (listed on the next page).

Generic Probes for Discrete Items (Multiple Choice and Free Response)—Brief Set

No.

Probe

Instructions for Interviewer

Did you find any words or parts of this question that were confusing?

 Yes  No

Ask this probe for all discrete questions AFTER the respondent has answered the question.

What did you find confusing? What could we do to make the question less confusing?

Ask both probes only if the respondent answered YES to the previous question.

Generic Probes for Matrix Items—Brief Set

No.	Probe	Instructions for Interviewer
1	Look back at the first part of the item where you were asked a question. Did you find any words or parts of this question that were confusing?  Yes  No	Ask this probe AFTER the respondent has answered the entire matrix question. This probe is specifically for the “stem” of the item, not the entire matrix item or a specific part/sub-item.
2	What did you find confusing? What could we do to make the question less confusing?	Ask both probes only if the respondent answered YES to the previous question.
3	Did you find any words or part of [part a., b., c., etc.] confusing?  Yes  No	Ask this probe for all parts/sub-items.
4	What did you find confusing? What could we do to make [part a., b., c., etc.] less confusing?	Ask both probes only if the respondent answered YES to the previous question.
5	Were you comfortable answering this item?  Yes  No	Only ask this probe for matrix items inquiring about potentially sensitive topics. Ask this probe after completing all other probes pertaining to parts/sub-items.
6	How could we make this item more comfortable to answer?	Ask this probe only if the respondent answered NO to the previous probe. This probe should be skipped if the previous probe was not administered.

Other existing items may only need to be administered in cog labs to ensure specific terminology or phrases are still relevant (e.g., “word processing”), or because the response options have been changed. In these cases, generic probes will not be administered. Probes specific to the item will be administered in relation to the terminology, phrase, or changed response options to ensure participants are able to provide the feedback.

C. Core Cognitive Interview Items and Probes

1. Student Items—Grades 4, 8, 12 (Cross Grades)

[Grades 4, 8, and 12 Core: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	a) Do you ever use a tablet for reading?	Specific	Required
3	a) [If “yes” to probe #3] What sort of materials do you read on the tablet?	Specific	Conditional
4	b) What is the difference between what part a and part b are asking?	Specific	Required
5	c) Can you explain to me what “e-book reader” means to you?	Specific	Required

[Grades 4, 8, and 12 Core: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	Generic - Brief	Required
2	What types of “art courses” did you think about when answering this item?	Specific	Required
3	If the only art courses you took this school year happened after school or during the weekends, how would you answer this question?	Specific	Required

[Grades 4, 8, and 12 Core: Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	Generic - Brief	Required
2	What types of “music courses” did you think about when answering this item?	Specific	Required
3	If the only music courses you took this school year happened after school or during the weekends, how would you answer this question?	Specific	Required

[Grades 4, 8, and 12 Core: Student #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic – Brief (Including sensitivity probes)	Required
2	a) Can you provide examples of what would make you feel unsafe on your way to or from school?	Specific	Required
3	c) Can you explain what part c of this item is asking about? Please try to explain without repeating the wording of the item.	Specific	Required
4	What does it mean to be “treated fairly” by your teachers?	Specific	Required

2. Student Items—Grades 8 and 12 (Cross Grades)

[Grades 8 and 12 Core: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	General - Brief	Required
2	Can you explain to me what “a language course other than English” means to you?	Specific	Required
3	When responding to this item, would you count language courses taken after school or on the weekends?	Specific	Required

3. Student Items—Grade 12 Specific

[Grade 12 Core: Student #1]

Alternative response option version:

Alternative item version:

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic – Full	Required
2	[If responds “has not prepared me at all” or “has prepared me a little” to item] Why not?	Specific	Conditional
3	Can you explain to me what “the future you want to have after high school” means to you?	Specific	Required
4	Can you explain what “having a job” means to you?	Specific	Required
5	Can you explain what “being in college” means to you?	Specific	Required
6	[Looking at all versions of the item] Please take a look at these alternative versions of the item you just answered. Which version of the item is easiest to answer? Why?	Specific	Required

4. Teacher Items—Grades 4 and 8 (Cross Grades)

[Grades 4 and 8 Core: Teacher #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic – Brief	Required
2	What are some examples of a “professional degree”?	Specific	Required
3	Are there any academic degrees that are missing from this item?	Specific	Required

[Grades 4 and 8 Core: Teacher #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 8.	Generic - Brief	Required
2	Can you explain to me what “the last two years” means to you?	Specific	Required
3	a) Can you explain to me what “basic computer training” means to you?	Specific	Required
4	b) What types of “software applications” did you think about when answering this item?	Specific	Required
5	e) Do you think subitem e is needed, or do the remaining subitems already address this type of training?	Specific	Required

[Grades 4 and 8 Core: Teacher #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic – Full	Required
2	Can you explain what students being “prepared” means to you?	Specific	Required

5. School Items – Grades 4, 8, and 12 (Cross Grades)

[Grades 4, 8, and 12 Core: School #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic - Full	Required
2	h) What are some other examples of “qualitative information”?	Specific	Required

6. Charter School Items—Grades 4, 8, and 12 (Cross Grades)

[Grades 4, 8, and 12 Core: Charter School #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	b) Can you explain to me what “school separation policy” means to you?	Specific	Required
3	c) Do you think that any of these policies overlap or refer to the same policy?	Specific	Required

[Grades 4, 8, and 12 Core: Charter School #2]

The following script should be read to the participant before they provide a response:

This item will only be administered to charter school administrators who answer “yes” to an existing item [INTERVIEWER: show the participant the item in the image below]. If you would have answered “no” to the existing item, choose the first response option for the item you are answering.

Existing item:

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic - Full	Required

[Grades 4, 8, and 12 Core: Charter School #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic - Full	Required

[Grades 4, 8, and 12 Core: Charter School #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic - Full	Required
2	Please explain how you came up with your answer.	Specific	Required
3	Do all students attend school for this many days of the school year, or are some days only for students with specific needs?	Specific	Required

[Grades 4, 8, and 12 Core: Charter School #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	General - Full	Required

D. Reading Cognitive Interview Items and Probes

1. Student Items—Grades 4 and 8 (Cross Grades)

[Grades 4 and 8 Reading: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For full generic probes for matrix items, see page 8.	Generic – Full	Required
2	Can you describe, in your own words, what makes someone a "tutor"?	Specific	Required

[Grades 4 and 8 Reading: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For full generic probes for matrix items, see page 8.	Generic – Full (Stem only)	Full probes only required for item stem; use brief set for sub-items because these sub-items have already been probed for the previous item.
2	Can you describe, in your own words, what the option "I don't receive that type of help" means?	Specific	Required
3	Can you describe, in your own words, what the difference between "understand your English/language arts schoolwork" and "complete your English/language arts schoolwork" is?	Specific	Required

[Grades 4 and 8 Reading: Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For full generic probes for discrete items, see page 14.	Generic - Brief	Required
2	Can you describe, in your own words, what the difference between "borrowing" and "downloading” reading materials is?	Specific	Required

[Grades 4 and 8 Reading: Student #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For full generic probes for matrix items, see page 8.	Generic - Full	Required

[Grades 4 and 8 Reading: Student #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	a) Would you include stories or novels you read online or on an e-book reader (for example, Kindle, Nook, Kobo, Aura) in your answer?	Specific	Required
3	b) Would you include texts you read online or on an e-book reader (for example, Kindle, Nook, Kobo, Aura) in your answer?	Specific	Required
4	c) Would you include texts you read online or on an e-book reader (for example, Kindle, Nook, Kobo, Aura) in your answer?	Specific	Required
5	Read options a through f. Do you think any of the options mean the same thing? For example, does “Read stories or novels” mean the same thing as “Read comic books or graphic novels”?

2. Student Items—Grade 4 Specific

[Grade 4 Reading: Student #1]

Please note, this is a cross-grade item but it will only be taken to cognitive interviews at grade 4 because this item was previously administered at grade 8.

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For full generic probes for matrix items, see page 10.	Generic - Brief	Required
2	a) Can you describe, in your own words, what the word "critique" means?	Specific	Required
3	a) Can you think of examples of crafts or techniques an author can use?	Specific	Required
4	b) Can you describe, in your own words, what the word "evidence" means?	Specific	Required

3. Student Items—Grade 8 Specific

[Grade 8 Reading: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	a) What other examples for features in a text can you think of?	Specific	Required
3	b) What other examples for figurative language in a text can you think of?	Specific	Required
4	c) What other examples for tools the author is using can you think of?	Specific	Required

4. Teacher Items—Grades 4 and 8 (Cross Grades)

[Grades 4 and 8 Reading: Teacher #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Do you think the term "activities" describes the options a through j well? If not, what other terms could you think of to describe the options a through j?	Specific	Required
3	d and e) How would you describe the difference between question d and question e?	Specific	Required
	j) Do you think the chosen examples describe well what "mixed media" is? If not, why? How could we make it clearer?
4	We would like to reduce the number of options a through j in these questions, if possible. Which of the questions, a through j, do you consider the three most important?	Specific	Required
5	Which of the questions in a through j do you consider the three least important?	Specific	Required

[Grades 4 and 8 Reading: Teacher #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Do you think the term "tasks" describes the options a through c well? If not, what other terms could you think of to describe the options a, b, and c?	Specific	Required

[Grades 4 and 8 Reading: Teacher #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Do you think the term "pedagogical techniques or tools" describes the options a through k well? If not, what other terms could you think of to describe the options a through k?	Specific	Required
3	We would like to reduce the number of options a through k in these questions, if possible. Which of the questions a through k do you consider the three most important?	Specific	Required
4	Which of the questions a through k do you consider the three least important?	Specific	Required
5	i) Can you think of examples of self-paced reading programs or apps?	Specific	Required
6	k) Can you think of examples how students might collaborate with peers and experts online?	Specific	Required

[Grades 4 and 8 Reading: Teacher #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Are there any types of text missing on this list that you think should be included?	Specific	Required
3	g) Can you think of other terms to describe texts that include images, infographics, or animations?	Specific	Required

[Grades 4 and 8 Reading: Teacher #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Are there any student groups missing on this list that you think should be included?	Specific	Required

[Grades 4 and 8 Reading: Teacher #6]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Are there any important resources missing on this list that you think should be included?	Specific	Required
3	Can you think of examples of "materials from different curricular areas?	Specific	Required
4	Please read options f, g, h, and i. Are these options overlapping? If so, which options overlap?	Specific	Required

5. Teacher Items—Grade 4 Specific

[Grade 4 Reading: Teacher #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Do you think the term "activities" describes the options a through i well? If not, what other terms could you think of to describe the options a through i?	Specific	Required
3	b) What, if any, do you think the difference between "build reading fluency" and "develop reading fluency" is?	Specific	Required
3	We would like to reduce the number of options a through i in these questions, if possible. Which of the questions a through i do you consider the three most important?	Specific	Required
4	Which of the questions a through i do you consider the three least important?	Specific	Required
5	Do you think option e overlaps with any other options?
6	Please take a moment to review all of the options a through i. Do you think we have left anything off the list that should be included?

6. Teacher Items—Grade 8 Specific

[Grade 8 Reading: Teacher #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Do you think the term "activities" describes the options a through h well? If not, what other terms could you think of to describe the options a through h?	Specific	Required
3	b) What, if any, do you think the difference between "build reading fluency" and "develop reading fluency" is?	Specific	Required
3	We would like to reduce the number of options a through h in these questions, if possible. Which of the questions a through h do you consider the three most important?	Specific	Required
4	Which of the questions a through h do you consider the three least important?	Specific	Required
5	Do you think option d overlaps with any other options?
6	Please take a moment to review all of the options a through h. Do you think anything is missing from the list that should be included?

7. School Items – Grades 4 and 8 (Cross Grades)

[Grades 4 and 8 Reading: School #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Are there any important professional development opportunities missing on this list that you think should be included?	Specific	Required

[Grades 4 and 8 Reading: School #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Are there any important resources missing on this list that you think should be included?	Specific	Required

[Grades 4 and 8 Reading: School #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	Generic - Brief	Required
2	Can you describe, in your own words, what it means to have 'up-to-date' technology and resources?

8. School Items – Grade 4 Specific

[Grade 4 Reading: School #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	Generic - Brief	Required
2	Do you think the list of examples provided is comprehensive? What additional examples can you think of?	Specific	Required

[Grade 4 Reading: School #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Are there any important responsibilities missing on this list that you think should be included?	Specific	Required
3	b and d) When responding to options b and d, did you include students with special needs, gifted and talented students, and/or English language learners in "groups of students"?

[Grade 4 Reading: School #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	Generic - Brief	Required
2	Do you think the list of examples provided is comprehensive? What additional examples can you think of?	Specific	Required

[Grade 4 Reading: School #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Are there any important responsibilities missing on this list that you think should be included?	Specific	Required
3	c) When responding to option c, what kinds of "data collection" were you thinking about?

9. School Items – Grade 8 Specific

[Grade 8 Reading: School #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	Generic - Brief	Required
2	Do you think the list of examples provided is comprehensive? What additional examples can you think of?	Specific	Required

[Grade 8 Reading: School #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Are there any important responsibilities missing on this list that you think should be included?	Specific	Required
3	b and d) When responding to options b and d, did you include students with special needs, gifted and talented students, and/or English language learners in "groups of students"?

[Grade 8 Reading: School #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	Generic - Brief	Required
2	Do you think the list of examples provided is comprehensive? What additional examples can you think of?	Specific	Required

[Grade 8 Reading: School #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Are there any important responsibilities missing on this list that you think should be included?	Specific	Required

E. Mathematics Cognitive Interview Items and Probes

1. Student Items—Grades 4 and 8 (Cross Grades)

[Grades 4 and 8 Mathematics: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	a) Can you give examples of websites or apps a teacher might use in class to teach math?	Specific	Required
3	b) Can you give examples of websites or apps you might use for math assignments?	Specific	Required

[Grades 4 and 8 Mathematics: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	General - Brief	Required

2. Student Items—Grade 4 Specific

[Grade 4 Mathematics: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For full generic probes for matrix items, see page 7.	Generic - Full	Required

[Grade 4 Mathematics: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For full generic probes for matrix items, see page 7.	Generic - Full	Required
2	a) Can you describe to me what “factors” mean to you?	Specific	Required

3. Student Items—Grade 8 Specific

[Grades 8 Mathematics: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	Can you describe to me the difference between Algebra I and Algebra II?	Specific	Required

Please note that the probe(s) only focus on specific words and/or phrases in this item.

[Grades 8 Mathematics: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	How confident are you of your response to this question?	Specific	Required

Please note that the probe(s) only focus on specific words and/or phrases in this item.

[Grades 8 Mathematics: Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	a) Can you explain to me what a “four-function” calculator means to you?	Specific	Required
2	b) Can you explain to me what a “scientific” calculator means to you?	Specific	Required

Please note that the probe(s) only focus on specific words and/or phrases in this item.

[Grades 8 Mathematics: Student #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	Generic - Brief	Required
2	Can you give examples of websites you might use for help with your math assignments?	Specific	Required

[Grades 8 Mathematics: Student #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 7.	Generic - Full	Required
2	d) Can you explain to me what “ratio” means to you?	Specific	Required
3	d) Can you explain to me what “rate” means to you?	Specific	Required
4	f) Can you explain to me what “transforming” a figure in geometry means to you?	Specific	Required

[Grades 8 Mathematics: Student #6]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required

[Grades 8 Mathematics: Student #7]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For full generic probes for matrix items, see page 7.	Generic - Full	Required
2	a) Can you explain to me what “irrational number” means to you?	Specific	Required
3	b) Can you explain to me what “pentagon” means to you?	Specific	Required
4	c) Can you explain to me what a “variable” means to you?	Specific	Required

4. Teacher Items—Grades 4 and 8 (Cross Grades)

[Grades 4 and 8 Mathematics: Teacher #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	b) Can you give specific examples of “other materials provided by your district or school”?	Specific	Required
3	e) Can you give examples of software and apps you use in your math class?	Specific	Required

5. Teacher Items—Grade 4 Specific

[Grade 4 Mathematics: Teacher #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	a) Approximately what percentage of your class does “some students” mean for you?	Specific	Required
3	f) Can you explain to me in your own words what item f means?	Specific	Required

[Grade 4 Mathematics: Teacher #2]

No.	Probe	Type of Probe	Required/Conditional
1	d) Can you explain in your own words what “make assumptions” means when teaching math?	Specific	Required
2	d) Can you provide an example of what this item is describing?	Specific	Required

Please note that the probe(s) only focus on specific words and/or phrases in this item.

6. Teacher Items—Grade 8 Specific

[Grade 8 Mathematics: Teacher #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	a) Approximately what percentage of your class does “some students” mean for you?	Specific	Required
3	f) Can you explain to me in your own words what item f means?	Specific	Required

[Grade 8 Mathematics: Teacher #2]

No.	Probe	Type of Probe	Required/Conditional
1	a) Can you explain in your own words what “make assumptions” means when teaching math?	Specific	Required
2	a) Can you provide an example of what this item is describing?	Specific	Required

Please note that the probe(s) only focus on specific words and/or phrases in this item.

7. School Items – Grade 4 Specific

[Grade 4 Mathematics: School #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Can you explain to me in your own words what the item stem means?	Specific	Required
3	Are there other key algebraic concepts for grades 3 to 6 that should be added to or used in place of “variables, repeating patterns, and proportional reasoning”?	Specific	Required
4	Can you describe to me what the difference between the response options “Not taught” and “No emphasis” means to you?	Specific	Required

[Grade 4 Mathematics: School #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	Generic - Brief	Required
2	Are there other math staff titles that should be included?	Specific	Required
3	Can you describe ways in which additional math staff support students?	Specific	Required

[Grade 4 Mathematics: School #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	Generic - Brief	Required
2	Are there other math staff titles that should be included?	Specific	Required
3	Can you describe ways in which additional math staff support teachers?	Specific	Required

8. School Items – Grade 8 Specific

[Grade 8 Mathematics: School #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	Generic - Brief	Required
2	Do you think the response option categories are adequate for this item?	Specific	Required

[Grade 8 Mathematics: School #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	Generic - Brief	Required
2	Are there other math staff titles that should be included?	Specific	Required
3	Can you describe ways in which additional math staff support students?	Specific	Required

[Grade 8 Mathematics: School #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	Generic - Brief	Required
2	Are there other math staff titles that should be included?	Specific	Required
3	Can you describe ways in which additional math staff support teachers?	Specific	Required

9. Additional Mathematics Terminology Probes

Student Grade 8

No.	Probe	Type of Probe	Required/Conditional
1	Can you explain to me what “write a computer program” means to you?	General	Required
2	Can you explain the difference between “writing a computer program” and “coding”?	General	Required

School Grades 4 and 8

No.	Probe	Type of Probe	Required/Conditional
1	Can you describe to me what “coding classes” means to you? What other terms might you use to describe a class like this?	General	Required

F. Writing Cognitive Interview Items and Probes

1. Student Items—Grades 4, 8, and 12 (Cross Grades)

[Grades 4, 8, and 12 Writing: Student #1]

Alternative version

In a typical week, how many hours do you spend writing on your own and not for school—for example, writing stories or keeping a journal at home?

__________________

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	Can you explain, in your own words, what "writing" means to you? For example, is texting writing?	Specific	Required
2	Besides "writing stories or keeping a journal", what other types of writing do you do on your own and not for school?	Specific	Required
3	[Looking at all versions of the item] Please take a look at the alternative version of the item you just answered. Which version of the item is easiest to answer? Why?	Specific	Required
4	[Looking at the alternate version of the item] Would you say remembering the number of hours a week was very easy, easy, difficult, or very difficult?	Specific	Required

Please note that the probe(s) only focus on specific words and/or phrases in this item.

[Grades 4, 8, and 12 Writing: Student #2]

Alternative version

About how many hours a day do you spend on your English/language arts homework?

Less than 30 minutes a day
30 minutes up to 1 hour a day
1 up to 2 hours a day
2 up to 3 hours a day
3 up to 4 hours a day
More than 4 hours a day

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for discrete items, see page 10.	Generic - Brief	Required
2	[Looking at all versions of the item] Please take a look at the alternative version of the item you just answered. Which version of the item is easiest to answer? Why?	Specific	Required
3	[Looking at the alternate version of the item] Would you say remembering the number of hours a day was very easy, easy, difficult, or very difficult?	Specific	Required

2. Student Items—Grades 8 and 12 (Cross Grades)

[Grades 8 and 12 Writing: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	d) What does “online tool that allows others to edit your work” mean to you?	Specific	Required
3	d) What other types of online tools that allow others to edit your work, if any, can you think of?	Specific	Required

3. Student Items—Grade 4 Specific

[Grade 4 Writing: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	What does the phrase “tools for writing” mean to you in this sentence?	Specific	Required

[Grade 4 Writing: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	What does the phrase “outside of school” mean to you?	Specific	Required
2	What does the phrase "in school" mean to you?	Specific	Required

Please note that the probe(s) only focus on specific words and/or phrases in this item.

[Grade 4 Writing: Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Can you describe, in your own words, what a report is?	Specific	Required

[Grade 4 Writing: Student #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	What does the phrase “in a day” mean to you?	Specific	Required

Please note that the probe(s) only focus on specific words and/or phrases in this item.

4. Student Items—Grade 12 Specific

[Grade 12 Writing: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	Are there other activities you practice in your English/language arts class that are not included in this item?	Specific	Required

[Grade 12 Writing: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic - Full	Required

[Grade 12 Writing: Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required

[Grade 12 Writing: Student #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic - Full	Required
2	b) What does the phrase “sentence structure” mean to you?	Specific	Required

[Grade 12 Writing: Student #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic - Full	Required

[Grade 12 Writing: Student #6]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	b) What does the phrase “cover letter” mean to you?	Specific	Required
3	d) What additional job search websites, if any, can you think of?	Specific	Required
4	e) What does the phrase “professional networking website” mean to you?	Specific	Required
5	e) What additional professional networking websites, if any, can you think of?	Specific	Required

[Grade 12 Writing: Student #7]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required

5. School Items – Grades 4, 8, and 12 (Cross grades)

[Grades 4, 8, and 12 Writing: School #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For brief generic probes for matrix items, see page 10.	Generic - Brief	Required
2	What other types of commonly offered summer programs should be included, if any, to this item?	Specific	Required

6. Additional Writing Terminology Probes

Student Grades 4, 8, and 12

No.	Probe	Type of Probe	Required/Conditional
1	Which term is more familiar to you, "spell-check" or "spell-checker"?	General	Required
2	If your teacher asks you to “write a paper with no spelling, grammar, punctuation, and capitalization mistakes” what does that mean to you?	General	Required
3	If your teacher asks you to “write a plan” what does that mean to you?	General	Required
4	If your teacher asks you to “write an outline” what does that mean to you?	General	Required
5	If your teacher asks you to “complete your writing on a computer” what does that mean to you?	General	Required

Teacher Grades 4 and 8

No.	Probe	Type of Probe	Required/Conditional
1	Which term is more familiar to you, "spell-check" or "spell-checker"?	General	Required
2	What does the phrase “word processing skills” mean to you?	General	Required
3	In your opinion, do “word processing skills” include “keyboarding”?	General	Required
4	[If “no” to probe #4] Why not?	General	Conditional

School Grades 4, 8, and 12

No.	Probe	Type of Probe	Required/Conditional
1	What does the phrase “word processing skills” mean to you?	General	Required
2	In your opinion, do “word processing skills” include “keyboarding”?	General	Required
3	[If “no” to probe #4] Why not?	General	Conditional

G. Additional Cross-subject Terminology Probes

Student Grades 4, 8, and 12

No.	Probe	Type of Probe	Required/Conditional
1	What does “I would be able to do this.” mean to you?	General	Required
2	What does “I can do this.” mean to you?	General	Required
3	What, if any, do you think is the difference between the two sentences?	General	Required

H. Additional Cross-subject Response Options Probes

Grades 4, 8, and 12 – Activity #1

Please look at the six cards in front of you. After you read the cards, try to order them from lowest to highest.

If you think that two or more cards mean the same thing, you can put the cards on top of each other.

Test administrator will put cards (printed out individually) on a pile in front of the student in random order.

No.	Probe	Type of Probe	Required/Conditional
1	Do you think all cards mean something different, or are there cards that mean the same to you?	Specific	Required
2	[Probe about cards that are placed on top of each other/mean the same] Why did you put these cards on top of each other? How could we make these cards more different from each other?	Specific	Conditional
3	[Probe about cards that are placed closer to each other than others] How would you describe the difference between [card] and [card]?	Specific	Required

Grades 4, 8, and 12 – Activity #2

Test administrator will put printed response options in front of the student.

Please read the question on the sheet in front of you and look at the options A through E --- the part of the question with the red circle around it.

Shape1

No.	Probe	Type of Probe	Required/Conditional
1	In your own words, what does "Not at all like me" mean to you?	Specific	Required
2	In your own words, what does "A little bit like me" mean to you?	Specific	Required
3	In your own words, what does "Somewhat like me" mean to you?	Specific	Required
4	In your own words, what does "Quite a bit like me" mean to you?	Specific	Required
5	In your own words, what does "Exactly like me" mean to you?	Specific	Required
6	Do you think all options A-E mean something different, or are there options that mean the same to you?	Specific	Required
7	[For options that mean the same]: How can we make these options more different?	Specific	Conditional

I. Teacher and School Debriefing Probes

Please ask the following probes at the end of each teacher and school administrator cognitive interview.

No.	Probe	Type of Probe	Required/Conditional
1	How helpful, in terms of informing your day-to-day <teaching/school policy> would you find the following information? Student Access to People Resources; □ Not helpful □ Somewhat helpful □ Very helpful Student Access to Technology Resources; □ Not helpful □ Somewhat helpful □ Very helpful Knowing what <reading/math/writing> topics and content teachers from other schools teach their students; □ Not helpful □ Somewhat helpful □ Very helpful Knowing which instructional strategies teachers use when teaching students <reading/math/writing> content; □ Not helpful □ Somewhat helpful □ Very helpful Knowing how <reading/math/writing> teachers assess what students know and can do; □ Not helpful □ Somewhat helpful □ Very helpful Knowing what technological resources other <reading/math/writing> teachers have available to them; □ Not helpful □ Somewhat helpful □ Very helpful Teachers’ educational and training background; □ Not helpful □ Somewhat helpful □ Very helpful Types of professional development offered to teachers; □ Not helpful □ Somewhat helpful □ Very helpful Teachers’ attitudes about their teaching, instruction, and pedagogy; □ Not helpful □ Somewhat helpful □ Very helpful Knowing what outside-of-school activities schools offer to students; □ Not helpful □ Somewhat helpful □ Very helpful	General	Required
2	Of the list above what would be the 3 most helpful pieces of information to know?	General	Required
3	Of the list above what would be the 3 least helpful pieces of information to know?	General	Required
4	Is there any information, not on this list that you would find helpful in terms of informing your day-to-day <teaching/school policy>?	General	Required

J. General Debriefing and Thank You (For all student, teacher, and school administrator participants)

Before we finish, I’d like to hear any other thoughts you have about the survey questions we worked on today.

Is there anything else you would like to tell me about working on the survey questions?

Is there anything that you think could make these survey questions clearer?

Thank participant for his/her time and provide gift card, as appropriate.

1 Reading vocabulary half-blocks will be preceded by a brief read-aloud task consisting of graded word lists to help gauge students’ reading levels.

2 Morae^® software (by TechSmith) can capture a student’s interactive behaviors as they happen, while one or more observers can simultaneously record comments that are time‐locked to the student actions and to the video recording. Adding Morae^®observer software allows observers to be located in a remote location. This is both a convenience for observers and a potential means of reducing student stress or distraction.

3 Please note that the 90 minutes includes time for introductions (maximum 15 minutes), conducting the interview (60 minutes), and debriefing and/or time for additional questions/feedback from the participants (maximum 15 minutes).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Qureshi, Farah
File Modified	0000-00-00
File Created	2021-01-21

The National Assessment of Educational Progress (NAEP) 2021 Items Pretesting

NCES Cognitive, Pilot, and Field Test Studies System

Volume 2 - NAEP 2021 Items Pretesting