NAEP Mathematics, Reading, and Science Hands-on Tasks and National Indian Education Study (NIES) Survey Pretesting

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average xx¹ minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 20202-4537. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: NAEP/NCES, U.S. Department of Education, 400 Maryland Avenue, S.W., Washington, DC 20202.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

OMB No. 1850-0803 Approval Expires 09/30/2016

Part B. Play Testing Protocol (Mathematics, Reading/ELA, and Science Hybrid HOTs)

Introduction to Study and Setup of Item and Task Review Script

Text written in italics is to be read aloud by the staff facilitator.

NOTE: The facilitator should not read the script word for word, but should be familiar enough with its contents to conduct the play testing sessions in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular task being studied.

My name is __ and I am from the Educational Testing Service. I am here with my colleagues [introduce colleagues]. We are developing a new set of tests in [mathematics OR reading/ELA OR science] for a program funded by the federal government called the National Assessment of Educational Progress (NAEP). NAEP is a test taken by students in grades 4, 8, and 12 in the United States. You will help us develop better tests for other students. We are going to ask for your opinions about some interactive items and tasks that we are creating for a NAEP [mathematics OR reading/ELA OR science] test. Right now we have some early drafts of what these items and tasks might look like. You are here to give us your thoughts about these items and tasks. We’d like to hear about what you think is interesting, what you think is not interesting, and if you think any part of an item or task is confusing.

My colleagues and I will take notes on what you are saying about the items and tasks—please be as honest as possible, talk freely with each other, and don’t worry if you find something confusing. You are not being graded on anything—you are helping us by giving us your opinions and thoughts about these items and tasks. There are no right or wrong thoughts.

What you say will not be used in any way that could identify you, except as required by law (see 20 U.S.C. § 9573). We will look at what you say later, but only so that we can understand how our test is working and how we can make it better. Overall, this session should take about [indicate correct length] minutes.

As you work through these items/tasks, we want you to talk with each other as you go along. We might also ask you a few questions now and then.

The facilitators will generally observe the students with little interruption and will listen to the conversation that occurs between group members for insights into what the students are thinking about the items and tasks. Intermittently, as appropriate, the facilitators can ask targeted questions such as those in this list:

Is this item/task interesting? Why?/Why not?

Are there any questions or words that seem confusing here? Did you understand that part?

How would you answer this question? [Ask different group members if their approaches would differ].

How could this item/task be improved? Could it be clearer, or more interesting, for example?

Were there questions in the task you did not like? Why?

This item/task asks you to solve a problem about <insert focus of item or task>. Have you learned about this before, either in school or in your own life?

Did you like working with both the kit and the tablet? Why?

Did you find the video directions for how to do your investigation helpful or not? Why?

Facilitators will observe students, take notes, and will pay attention to verbal answers to the questions above and write them down, but also will look for evidence of engagement, boredom, or confusion in what students say to each other and in their facial expressions or body language. If appropriate, staff can use these moments to probe for more information.

Occasionally staff observing play testing sessions will introduce a question to groups of students talking about the items/tasks to get more information from students, particularly in cases when students talk about issues related to their interest (or non-interest) in the item/task, confusing language in the item/task, or how they might respond to a question on the task. In such cases observers might ask something like, “Can you tell me more about what you said?” Prior to each play testing session, ETS staff may informally identify some key focus areas for each item/task. If students do not provide sufficient comments on targeted parts, a staff member may ask a group of students if they had any thoughts about the particular sections, using questions such as those described above, but focused on specific places or issues in the item/task.

At the end, students will be thanked for their participation and for helping us to improve our tests. Students will be given gift cards for their participation at the conclusion of each play testing session.

Part C. Cognitive Interviews (Mathematics and Reading/ELA)

I. Introduction to Study Script

NOTE: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular task being studied. In general, protocols will be tweaked and customized in light of specific task content and greater knowledge of delivery devices.

Introduction:

Hello, my name is ______ and I work for Educational Testing Service. It’s nice to meet you and thank you very much for helping us out today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?

When the student responds, follow up with 2 or 3 questions to get the student used to talking, such as:

That’s interesting—why do you enjoy <subject> so much?
What are you studying in <subject> at the moment?
And what’s the best thing you have studied in <subject> so far?

Good. Well, I think you’ll enjoy what we are going to be doing today. First, let me begin by explaining why I am here and what you are going to be doing. You are taking part in a special study looking at a new type of activity to find out what students know in [mathematics OR reading/English language arts]. It is part of the National Assessment of Educational Progress, or NAEP for short, a test that is funded by the U.S. Federal Government and run by a center in the Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will help us develop better tests for other students. Overall, this session should take about [indicate correct length] minutes.

It’s okay if you don’t know how to do any parts of the task. I will not be grading your work today, and no one will know that it was you who did the task. My goal is to learn how you react to the task, so please try to do your best.

If at any time you decide you don’t want to go on, that is your choice, and you may stop.

This new test is on the tablet. The instructions about how to do the test are also on the tablet.

This interview is being recorded so we can review it later. We will also be able to capture what you tap on and where you move your fingers. What you say will not be told to anyone, or used in any way that could identify you, for any other purpose except as required by law (see 20 U.S.C. § 9573). We will look at what you say later, but only so that we can understand how our test is working and how we can make it better.

We will work together on [specify number of items/tasks].

Do you have any questions?

After answering any questions and giving further explanation, continue with the think-aloud training. If the student is no longer interested in participating, thank the student for his/her time and end the interview.

Okay, let’s move on. Before we look at the real tasks, I want to make sure you have a good idea of what we’re going to be doing. So I am going to give you some practice questions. The practice questions should help you get used to what we will be doing during the real tasks. They should help you understand how we want you to respond. Do you have any other questions before we start?

NOTE: Check the study schedule to establish whether the student should undergo the assigned tasks with a concurrent or retrospective verbal protocol (think-aloud) procedure. If concurrent, the student will think aloud while he or she is doing the task; in those cases, follow the steps described in section II. If retrospective, the student will describe his or her thinking after he or she has completed the task, while watching a recording of the task that was made as they proceeded through it; in those cases, follow the steps described in section III. (NOTE: Given that it is challenging for students to both read passages and think aloud, it is likely that think-alouds will be retrospective for reading/ELA tasks.)

II. Concurrent Think-Aloud

II a. Concurrent Think-Aloud: Instructions and Modeling Script

NOTE: If the schedule indicates that the retrospective method is to be used, please skip this section and move to section III.

NOTE: Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular task being studied.

To help us make our test better, we will ask you to complete some tasks. While you are doing the tasks, we will be recording everything that happens on the screen. The screen recording will capture all of your responses and movements on the tablet such as what you tap on and anything you put into text boxes.[If the student is to be video recorded, add:] We will also be videotaping you as you complete the task.

I’m going to ask you to do the questions in these tasks in a way that may be different from what you are used to. Instead of working quietly, I want you to tell me what you are thinking as you work through the task. We call this “thinking aloud,” because we are asking you to say everything you are thinking out loud.

In a moment, I will give you an example of the think-aloud process. Then I will give you a chance to practice it. You won’t be graded on anything you say while you are thinking aloud. There are no incorrect thoughts, and everything you think and say is important to us.

Okay, now I’m going to show you how to think out loud—this will help you see how I want you to describe what you are thinking as you are working on the task. When I am finished I’ll ask you to try it, so you can see how it works.

Think-Aloud Demonstration:

Since we can’t tell what is going in your head, we need you to “Think Aloud.”

Let me give you an example. [Place example question in front of student.] Look at this question. It asks me to look at the five animals and choose two that are the most similar. So I am going to do this task and I will think aloud while I work on it.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar:

Beetle
Mouse
Crab
Dolphin
Cow

Okay, so I am reading the question…. It says I have to decide on the two animals that are the most similar. Hmm, I am just wondering what do they mean by “similar”? That could mean anything. I’m thinking I’m not sure how I’m going to make that decision. Well, let’s look at the choices. So the choices are: beetle, mouse, crab, dolphin, and cow. Well, as I am reading the choices I am already thinking that there are some things about them that are similar, but there are also differences. I’m thinking the beetle and crab are kind of similar, in a way—well, they both have lots of legs and they kind of move around fast. But one lives in the water and the other doesn’t, so that makes them different. I guess the mouse also moves around fast, but it’s not similar to either the beetle or the crab, except they are all small. Then there is the dolphin that lives in the water, too… but that’s about all it has in common with the crab, so I wouldn’t say they are very similar even though they’re both in the sea. Hmm, so the mouse and the cow are both mammals, they both have fur and they have live babies instead of laying eggs or whatever… oh right, but dolphins are mammals too, aren’t they? Yes, I think they are… but really they seem so different from mice and cows to me. I don’t know… this seems like a hard question… I’m not sure what to choose. Okay, I’m going to try to make a decision…umm…okay, I think I am going to go for… mouse and cow. Because they both have fur and have four legs and they both walk around on the land. That seems weird; they are not really similar! But when I think about all of the others they seem too different from each other in really basic ways, like where they live and their kind of body and that kind of thing… so, yes, even though it was really hard to decide, that seems like the best choice to me, because they are both mammals and they both live on the land. So those things seem like they are important, and that is how I made my decision.

Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do as you are thinking about the tasks today. The point of the think-aloud is to get at whatever is in your head as you are doing the tasks. Just say aloud the words and the thoughts that are in your head, as you are thinking about and working on the tasks.

II b. Concurrent Think-Aloud: Student Practice Script I

Interviewers should place the practice question in front of the student so he/she can read it. Some students will be silent after reading the question. Students should be immediately encouraged to say whatever they are thinking.

It may be necessary to remind the student to talk aloud as he/she works through the questions and tasks. If necessary, interviewers should use the “Think-Aloud Hints” shown below to prompt the student, being careful not to lead the student. The interviewer needs to be familiar enough with the information to introduce the think-aloud in a natural, conversational manner.

Now you will try a think-aloud. You can use this example. Like last time, you have to pick two of these things that you think are the most similar. As you are reading the question and the choices, and as you are figuring out your decision, I want you to talk out loud to tell me all of the thoughts that are in your head at each moment.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I’ll be reminding you to think aloud if you get quiet because I need to hear all of your thoughts.

Okay, now you try. Go ahead and start working on this question and remember to think aloud as you are doing it.

Which two living things below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the tasks? What questions would you like to ask me? [If the student says he or she feels okay and doesn’t have any questions: Good, then let’s begin our study.] [If the student expresses concerns, says he/she has questions, or appears to be hesitant or reluctant, ask him/her to say more about the concerns or questions, and try to address his/her concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him/her to stop.]

II c. Concurrent Think-Aloud: Student Practice II (Optional)

(Use only if you feel the student would benefit from another think-aloud practice before moving on to the actual questions. It may be beneficial to instead model thinking aloud again for the student rather than doing another practice [see think-aloud demonstration in IIa])

If the student struggles to think aloud, the interviewer should give the student another opportunity to practice. The interviewer should praise the student for the first attempt regardless of how good it was, for example: “Very good—let’s do another one before we start the real tasks. Are you ready? Here is the next practice question. Remember to think out loud as you begin to think about this question and all the way through—tell me what you are thinking as you work through it.” During the practice item, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, below).

Which two of the following objects have the most similar properties? Circle the two that are most similar.

Silver coin
Chocolate coin
Gold coin
Blue plastic coin
Brown plastic coin

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the tasks? What questions would you like to ask me? [If the student says he or she feels ok and doesn’t have any questions: Good, then let’s begin our study.] [If the student expresses concerns, says he/she has questions, or appears to be hesitant or reluctant, ask him/her to say more about the concerns or questions, and try to address his/her concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him/her to stop.]

II d. Concurrent Think-Aloud: Starting the Tasks

Now we will move on to the actual task. Remember, as you do all the activities and answer all the questions in the task I’d like you to say aloud everything that you’re thinking, and I may remind you to do that if you are quiet. This task should take about [indicate correct length] minutes. Remember, you will not be graded on what you do during the task and there is no right or wrong way to think aloud, as long as you keep telling me your thoughts. Your thoughts will help us make the tasks better. I will also have a few questions after you have finished working on the task.

Do you have any questions before we go on? (Answer any questions the student may ask.)

Because the information you provide is so important to us, I am going to be taking notes while you think aloud and answer the questions.

Here is the task. When you are ready, go ahead and start working on it.

II e. Concurrent Think-Aloud: Hints for the Interviewer

The goal of think-alouds is to capture all the student’s mental processes while working through tasks. Interviewers must strive to have the student speak aloud all of his or her thoughts while doing the tasks. If a student is continually providing short utterances or not talking, interviewers should use “continuers” to encourage students to be more descriptive. The goal is to get students to verbalize thoughts without putting words in students’ mouths. Care should be taken not to ask questions that lead students’ responses in particular directions or make them rush or change their approach. Interviewers should be as objective and unbiased as possible.

In general, if the student is silent for approximately 5 to 10 seconds, interviewers should use the following as a guide for encouraging the student to describe his or her thoughts, or to help the student elaborate his/her responses.

If the student is not verbalizing enough, interviewers should offer a verbal “nudge” to remind the students to keep talking, such as:

What’s going on in your head right now?
I see you’re looking at the task [or screen/figure/chart/text]. What are you thinking?
You seem to be thinking hard—can you tell me what’s in your mind right now?
What are you thinking now?
Any other thoughts?

If the student says something about his or her thoughts but it is difficult to tell exactly what he or she means, or if the student begins to talk but does not say very much, interviewers can urge the student to elaborate, for example:

Tell me a bit more about that.
That’s interesting—can you say more about that?
Go on….
Okay, and what else are/were you thinking?

If more clarity or a more explicit description of a student’s thoughts is desirable:

Can you explain what you mean by that?
Can you say a bit more to help me understand what you mean/are thinking/were thinking?

It is important to be responsive and sensitive to each student’s behavior. If it becomes clear that students cannot tell any more about their thoughts, interviewers should not keep asking. Students should be encouraged to think aloud as much as they can, but their reactions should be closely observed and pushing avoided if students seem frustrated or uncomfortable. While it is desirable that students articulate as much as they can, sometimes students will simply be unable to tell what is in their mind beyond a basic reply such as “I am reading the question.” Interviewers must be sensitive to nonverbal signals, if students cannot tell any more than they have, or if they need a few extra seconds to gather their thoughts to put them into words.

In addition, during observations if it seems a student is hung up on something, interviewers should note when this occurred.

II f. Notes on Student Actions and Behaviors During Tasks

As a student is proceeding through a task, a screen-capture system will record evidence of his or her actions on the task (such as where a student taps the screen, how he or she moves through the content and the questions, and any text or numbers entered into fields or boxes). During the session the interviewer should take notes regarding any of the student’s expressions or behaviors that may reflect the status of his/her understanding, engagement, or use of the task. The following are examples of such behaviors for interviewers to note:

Does the student express signs of confusion, boredom, or excitement?
Does the student use tools provided in the task such as tapping on tabs or buttons to reveal additional or earlier information, digital notepads, digital calculators, or reminders of what to include or think about?
Does the student rapidly move through the screen or take his/her time? Does the student spend a lot of time on a particular aspect?

These kinds of observations should be considered opportunities to prompt students to think aloud and to encourage them to express what is in their minds. Interviewers should also make a note of any places in the task that appear valuable for follow-up with some additional questions after the task is completed (see section IV).

III. Retrospective Think-Aloud

III a. Retrospective Think-Aloud: Instructions and Modeling Script

In this approach, students are not asked to think out loud during the task. Instead, students will be instructed to complete the task as they would under normal testing conditions (silently, without interruption). Then, after the task is completed, they will be asked to describe what they were thinking while reviewing the task on a computer. This approach will most likely be used for reading/ELA tasks given that experience with cognitive interviewing at ETS suggests that students have difficulty reading and thinking aloud or writing and thinking aloud. The replay of the task is intended to help cue the student’s recall of the thought processes that occurred during the task as it progressed.

Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular task being studied.

To help us make our test better, we will ask you to complete some tasks. While you are doing the tasks, we will be recording everything that happens on the screen. The screen recording will capture all of your responses and movements on the tablet, such as what you tap on. [If the student is to be video recorded, add:] We will also be videotaping you, as you complete the task. Overall, this session should take about [indicate correct length] minutes.

First you’ll do the task, working at your own pace through all of the activities and questions you will see on the screen. Then, after you have finished, I’m going to ask you to go back over the task with me. As we go back over it and look at the task again, I will ask you to tell me what you were thinking as you were working through each of the sections. We call this “thinking aloud,” because we are asking you to say everything you were thinking during the task out loud. To help you remember what you were thinking as you worked on it the first time, we will look at the recording we just made of you doing the task. This way, you will be able to see all of the screen taps and all of the answers that you gave as you were doing the task. We will watch the recording, and as we are going through it, I will be asking you to tell me what you were thinking at each point. Does that make sense? Do you understand what we’ll be doing?

[Answer any questions and clarify the sequence of steps, if needed.]

Okay, now I’m going to show you how to think out loud—this will help you see how I want you to describe what you were thinking while you were working on the task. When I am finished, I’ll ask you to try it, so you can see how it works.

Think-Aloud Demonstration:

Since we can’t tell what is going on in your head, we need you to “Think Aloud.”

Let me give you an example. [Place example question in front of student.] Look at this question. It asks me to look at the five animals and choose two that are the most similar. So first of all, I am going to do this task just as I normally would for a test question. After I have finished it, I will try to remember what I was thinking as I was working on it, and I’ll tell you what I was thinking.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar:

Beetle
Mouse
Crab
Dolphin
Cow

[The interviewer ‘works’ silently for a minute or two on the question above. The interviewer should enact some behaviors that suggest that he/she is thinking hard about the question and carefully considering the possible answers; for example, pointing with a pencil tip at each option in turn; moving a pencil or fingertip between answer options; returning to the question; giving nonverbal signals indicating deep consideration, uncertainty, etc. Finally the interviewer circles two answers, B and E, and then begins the retrospective think-aloud.]

III b. Retrospective Think-Aloud: Student Practice Script I

Interviewers should place the practice question in front of the student so he/she can read it. Allow the student as much time as needed to answer the question. When he/she indicates he/she has finished, ask the student to begin thinking aloud. Some students will be silent after reading the question. Such students should be asked to say whatever they are thinking. It may be necessary to remind students to talk aloud as they work through the questions and tasks. If necessary, interviewers should use the “Think-Aloud Hints” shown below to prompt students, being careful not to lead students. The interviewer needs to be familiar enough with the information to encourage the think-aloud in a natural, conversational manner.

Now you will try a think-aloud. You can use this example. Like last time, you have to pick the two things that you think are the most similar. Go ahead and do this task, and when you are finished, let me know.

Once you are done, I will ask you to talk out loud and tell me all of the thoughts that were in your head at each moment. During that phase, if I don’t hear you speaking I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are thinking. I’ll be reminding you to think aloud if you get quiet because I need to hear all of the thoughts you had during the task.

Okay, now you try, you can go ahead and start working on the question. Let me know when you are finished.

Which two organisms below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

[When students indicate they are finished, make sure they have circled two answers, and then ask them to begin describing their thoughts. As necessary, include the appropriate prompting questions, such as the following:]

Okay, so tell me what you were thinking as you were doing this task. What did you think at the beginning, when you started to read the question? … And then what were you thinking? … And what else did you think about? … Anything else? … Can you tell me more about that? … etc.

III c. Retrospective Think-Aloud: Student Practice II (Optional)

If the student struggles to think aloud, the interviewer should give the student another opportunity to practice. The interviewer should praise the student for the first attempt regardless of how good it was, for example: “Very good—let’s do another one before we start the real tasks. Are you ready? Here is the next practice question. [Place example question in front of student.] Remember that after you have finished answering this question I will ask you to talk out loud to tell me what you were thinking all the way through—so you should tell me what you were thinking as you worked through it. So, first of all, go ahead and do the question, and let me know when you have finished.”

As before, during the post-task think-aloud phase, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, section IIe).

Which two of the following objects have the most similar properties? Circle the two that are most similar

A. Silver coin

B. Chocolate coin

C. Gold coin

D. Blue plastic coin

E. Brown plastic coin

After the think-aloud phase is finished:

Now that you have practiced, do you feel that you understand how you should talk aloud about the thoughts you had while you were doing the tasks? Is this something you feel okay about doing? [If students say yes:] Good, then let’s begin our study.[If students say no or appear to be hesitant or reluctant, ask them to say more about any reservations they have, and try to address their concerns or uncertainties in a supportive way. If students indicate they do not wish to continue or do not feel comfortable continuing, allow them to stop.]

III d. Retrospective Think-Aloud: Starting the Tasks

Now we will move on to the actual tasks. Remember, after you have finished doing all the activities and answering all the questions on a task, I will ask you to say aloud everything that you remember thinking while you were doing the task, and I may remind you to do that if you go quiet. This task should take about [indicate correct time] minutes. Remember, you will not be graded on what you do during the task and there is no right or wrong way to think aloud, as long as you keep telling me your thoughts. Your thoughts will help us make the tasks better. I will also have a few questions after we have finished.

Do you have any questions before we go on? [Answer any questions the student may ask.]

Because the information you provide is so important to us, I am going to be taking notes while you are doing the task and while you are thinking aloud afterwards.

Here is the first task. When you are ready, go ahead and start working on it.

III e. Retrospective Think-Aloud: Prompts and Questions After Completion of Task

Now we will look at what you did on the task, and I would like you to tell me what you were thinking as we review what you did. We will be recording what you are saying as you think about this task. Again, you should feel free to stop at any time, and this information will not be used to grade you. The information will only be used to help improve the task. I will play the task back from the beginning now, and I want you to talk about the thoughts you had as you were working through it. From time to time, I might pause the video of the task playback, to give you enough time to explain what you were thinking at that point or if I need to ask you any extra questions. I will let you know when I am pausing the playback.

Interviewer plays and watches the recording of the task in real time with the student, and asks the student to describe aloud what he/she was thinking at each point in the task. The interviewer should pause the recording whenever more time is needed, making sure to say “I’ll just pause this for a moment” or “I am pausing the task here,” so that the student understands that the playback has been paused.

If the student is not verbalizing enough, interviewers should offer a verbal “nudge” to remind the student to keep talking, such as:

What was going on in your head at this time?
I see you were looking at the task [or screen/passage/chart/image]. What were you thinking?
You seemed to be thinking hard—can you tell me what was in your mind right then?

If the student says something about his/her thoughts but it is difficult to tell exactly what the student means, or if the student begins to talk but does not say very much, interviewers can urge the student to elaborate, for example:

Tell me a bit more about that.
That’s interesting—can you say more about that?
Go on….
Uh huh, okay, and what else were you thinking?

If more clarity or a more explicit description of a student’s thoughts is desirable:

Can you explain what you mean by that?
Can you say a bit more, to help me understand what you were thinking?

III f. Notes on Student Actions and Behaviors During Tasks

As a student is proceeding through the task, a screen-capture system will record evidence of his or her actions on the task (such as where a student taps, how he or she moves through the task, and any text entered into fields or boxes). During the session, the interviewer should take notes of any of the student’s expressions or behaviors that may reflect the status of his/her understanding, engagement, or use of the task. The following are examples of such behaviors for interviewers to note:

Does the student express signs of confusion, boredom, or excitement?
Does the student use tools provided in the task such as tapping on tabs or buttons to reveal additional or earlier information, digital notepads, digital calculators, or reminders of what to include or think about?
Does the student rapidly move through the screen or take his/her time? Does the student spend a lot of time on a particular aspect?

These kinds of observations should be considered opportunities to prompt students to think aloud and to encourage them to express what is in their minds. Interviewers should also make a note of any places in the task that appears valuable for follow-up with some additional questions after the task is completed (see section IV).

IV. Post-Think-Aloud Follow-Up Questions (Verbal Probing)

After completing the think-aloud process for a task, interviewers will follow up with a brief period of focused retrospective questioning. The post-task questions will comprise:

One standardized post-task question that all students will be asked following all tasks, which is designed to discover whether the student has prior knowledge of the content.

Up to three additional targeted questions that are task specific. These will be selected for each task by ETS staff prior to testing. Interviewers will receive the set of up to three questions for the specific task they are examining in each cognitive interview.

Standardized Question for All Tasks: Task-Specific Prior Knowledge

Have you studied anything related to this task in school, or have you learned about or come across these things in your own life? [If yes:] Tell me about what you have learned or studied or experienced that is related to this task.

Additional Questions: Task-Specific Issues

The purpose of the additional post-task questions is to capture more information on issues such as student actions during the task, particular aspects of [reading comprehension, computation, OR other appropriate focus] targeted in the task, and general reflections about the task. ETS staff will identify up to three questions for interviewers to ask in relation to each task. Interviewers should have the list of post-task questions ready when the student finishes the think-aloud portion for each task. These questions will typically be asked after the student has finished the task. However, occasionally a question will be identified that needs to be answered in the context of the task, while the student is engaged in solving some part of the problem. In those cases, the question will be added to the script and interview protocol for that task, and it will be placed beneath the screenshot showing precisely at which point during the task it should be asked. The general format and goals of these in-task questions will be the same as the post-task questions; only their timing will differ.

The following list shows some sample question frames that ETS staff will use to build questions related to each task. The specifics of each question will be generated by ETS staff via an informal task analysis in which key points in the task for understanding student thinking, or key issues for each task, are identified. Up to three additional questions will be generated by ETS and given to the interviewer for each task.

Let’s return to this section where you were asked to highlight a part of the passage supporting your answer to this question (interviewer points to particular screen with student looking on). I noticed you highlighted this part here—can you tell me more about what you were thinking when you chose that part to support your answer?

Let’s go back to the part where you were asked to pull together information and perspectives from several different sources to develop and write about your own point of view. I noticed you did write about your own point of view—can you tell me more about what you were thinking when you wrote your piece?

Let’s return to this section where you were asked to make a prediction (interviewer points to particular tablet screen with student looking on). I noticed you did make a prediction—can you tell me more about what you were thinking when you made that prediction?

At this point you were asked to design an investigation. I noticed you selected one set of variables to vary, but others were not varied—can you tell me more about what you were thinking when you made those choices?

Let’s go back to the part where you were asked to use information from the task to explain how you got to your answer or how you made your decision. I noticed you did draw a conclusion—can you tell me more about what you were thinking when you wrote that conclusion?

This button was on the screen to allow you to look at <information> at any time. Tell me about how and when you used this button, or if you didn’t really use it, why was that?

This tab allowed you to go back to <earlier screen> at any time. Tell me about how you used this tab, or if you didn’t really use it, why was that?

When you were shown <information/graph/image> at this point, what were you thinking? What was in your mind?

When you had to put the earlier information together to <draw a conclusion/make a decision>, how did you go about doing that?

At this point, you needed to <summarize the passage/interpret what you had found>. Did you learn how to do anything like this in school? Have you been taught how to do this, or learned a particular way to do this?

If students’ answers are unclear or not very explicit, the interviewer should use the same kinds of prompts as identified in the think-aloud protocol to elicit more information, for example:

Can you say a bit more about that?

What else were you thinking?

Anything else?

That’s interesting. Tell me more about that.

Optional: Interviewer-Generated Questions

In addition, if the interviewer noted some especially interesting behavior during the task, he or she can ask additional ad hoc questions about these specific instances, referring to notes made during the task. Interviewers should use their judgment about the need for and value of additional questions, based on the student’s behavior during the task, and based on time constraints.

Some examples of potential interviewer-generated ad hoc questions are:

I noticed on this part of the task that you paused for a while. Can you tell me more about why you paused and what you were thinking at this point?

I noticed on this part of the task that you did not use the prompt next to the response box. Can you tell me more about why you did not use that prompt?

I noticed that on this part of the task you spent some time looking at/going back-and-forth between the <image/table/question>). Can you tell me more about what was going on there?

V. Eye Tracking

Eye tracking allows data to be gathered about where students focus while taking an assessment. Students will be asked to do tasks on a tablet while a camera monitors where they look.

Apparatus

Eye-tracking systems are video-based systems utilizing high speed cameras and an LED infrared light that illuminates the face (no more powerful than typical sunlight). Infrared cameras are used to capture eye movements. The remote system requires no physical contact with the student: an infrared video camera acquires an image of the eyes and calculates gaze location in real-time. To allow free (but limited) head and body movements while maintaining accuracy of localization, a reflective sticker is placed on the forehead of the student during the eye-tracking session. ETS eye-tracking experts have used this particular system with adults and children as young as 3 years old. It takes a very short time (5 minutes) to set up and calibrate the equipment for each participant.

Procedure

First, students will be asked to work through a task in a normal way, without interruption. During this phase their eye movements are unobtrusively recorded and all events on the screen are captured in real time.

Next, using a retrospective verbal protocol, students are asked to go back over the task a second time and attempt to reflect on, and describe out loud, what they were thinking as they went through the task, and these verbalizations are recorded.

To help the student recall their thinking as it occurred, a “movie” (screen capture) of the complete task, including the student’s own movements and responses, will be played back for the student. A moving cursor indicating their gaze patterns will be overlaid onto these screen images, so that the student can see exactly where and how he/she was looking while he/she was doing the task the first time. Seeing their own eye movements acts as a prompt, helping students reconstruct their thinking at each point in the task.

These “movies” can be speeded up or slowed down by the facilitator as necessary to allow focus on specific parts of tasks, or to allow sufficient time for students to verbalize their thoughts in full before moving on to later task sections.

The following describes the sequence that will be followed for eye tracking and the subsequent retrospective think-aloud. Note that the eye-tracking calibration procedure may differ slightly for various hardware setups.

Setup and Calibration

The participating student is greeted by an ETS eye-tracking lab expert. The expert will describe the study procedure and questions or concerns the student may have. The student is then seated in front of the eye tracker, and a sticker is put on his/her forehead to assist the eye tracker to locate the eyes and to compensate for head movements. The system is then calibrated. The student is asked to watch a dot appearing at a number of different locations on the blank screen. The setup and calibration typically take about 5 minutes.

Shape2

Eye-tracking Session

After a successful calibration, the student starts the assessment session, in which he/she goes through the assessment task on the tablet. Students are free to move their head and body during the assessment within the field of view of the eye-tracking camera.

Along with the eye-tracking data, the system also records the following information: key presses, finger movements on the screen, screen changes (as a video file), and audio (optional). The experimenter takes notes during the session about noticeable behaviors or events.

Cued Retrospective Verbal Report

After the student completes the assessment task, eye tracking will be stopped. Depending on the design, a retrospective think-aloud may be used. To help remedy memory decay and to prevent fabrication, the screencast video is replayed with the student’s eye. As the student watches the playback, he/she is asked to report on his/her thoughts.

I am going to show you how you did the task by showing you where you were looking as you went through the task. This is done by showing you a video playback of the task. This video will show your eye movements, finger movements, and keyboard movements. The red cross (+) that you will see on the screen will show where you were looking as you worked through the task. The movements and keystrokes are those you made as you worked. We are interested in what was on your mind at that moment in the task: why were your eyes at a particular spot in the task?

We want you to think about what was happening at that time shown in the video, for example, “Here, I was reading the question and then it looks as though I moved to the part where the answers were and looked over the options.”

Try not to just reread the directions or your answers. Instead, talk about your goals and plans at the time when you were working on that part of the task. Tell us what was in your mind as you read, answered questions, and thought about the task.

To help students with the verbal protocol task, they are informed that they will be asked to speak continuously and that if they stop talking, they will be prompted with terms such as “please keep talking” and “please tell me more about that.” Students’ verbal reports are recorded for later coding and/or transcription.

VI. Debriefing and Thank You for Cognitive Interview

Thank student for his/her time. Provide a gift card.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the task?

Is there anything you would like to ask me about what we did today? [Answer student questions]

Thank you for helping us to improve our test.

Part D. Small-Scale Tryout Protocol

Text written in italics is to be read aloud by the interviewer.

NOTE: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text in brackets is generic text that will be tailored based on task specifics.

Introduction Script:

Hello, my name is ______ and I work for EurekaFacts. It’s nice to meet you and thank you very much for helping us out today.

Let me begin by explaining why you’re here and what you’re going to be doing today. You are participating in a special study to try out new <interviewer insert subject> test questions. This is part of something called the National Assessment of Educational Progress, or NAEP for short, and is sponsored by the U.S. Department of Education. Today we’ll be asking you to try out a small group of new test questions that are being developed for 4th/8th graders all over the United States.

If at any time you decide you do not want to go on, that is your choice and you may stop.

This tryout session is being recorded so researchers can review it later. Your information and responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573).

We will work together to do <say number of tasks>.

Do you have any questions?

Interviewer: Ask the student if he/she has any questions. After answering questions and giving further explanation, continue with the preliminary information questions. If the student is no longer interested in participating, thank the student for his/her time and end the tryout session.

First I want to ask you few questions…

Ref#	Question/Prompt	Response
	Preliminary Information
1	Name of interviewer	(Prefilled)
2	Date	(Prefilled)
3	Student ID	(Prefilled)
4	Student’s gender	 Male  Female
5	Name of school
6	School district
7	Special circumstances that may have affected the tryout session [Do not ask]
8	Start time of tryout

Ok, now, I’m going to give you, <number of questions> questions or tasks to complete.

When we are finished making the test, students will take it on tablets. However, we aren’t finished yet, so there may be some parts of the tasks that don’t work yet. You will be asked to type in responses or to tap on responses on the screen, but the computer will not save your answers. I’ll be sitting next to you and writing your answers down on a sheet of paper.

You will have [indicate correct number of minutes] minutes to do this task. You will not be graded on what you do during the task, but please answer these questions as if it was a real test.

Okay, let’s begin.

Debriefing and Thank You

Before we finish, I’d like to hear [any/other] thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the test questions?

Did you have any problems understanding [other/any of] the questions?

Did you have any problems understanding what you were supposed to do to answer [other/any of the] questions?

Is there anything that you think could make [this/these] test question(s) clearer?

Thank student for his/her time.

Part E. NIES Survey Questionnaire Cognitive Interviews

I. Interviewer Welcome Script and Assent/Consent

The following scripts² should not be read verbatim. You, as the interviewer, need to be familiar enough with the scripts below to introduce the think-aloud process in a conversational manner. Text written in italics is suggested content for you to be thoroughly familiar with in advance. You should project a warm and reassuring manner toward the participant to develop a friendly rapport and as such should use conversational language throughout.

Student Participant Script

Hello, my name is <name >.

Thank you for coming here today to help us.

I have some questions that students like you in many different schools will be asked to answer about themselves, their education, and their families. The questions you will be asked will be part of a survey for the National Assessment of Educational Progress (NAEP) research study. Some of the questions may not be easy to understand or answer. There are no wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), part of the U.S. Department of Education, make these questions clearer for students like you. This interview is being recorded so researchers can review the recordings later. No one else will see or hear the recording. Participant responses are voluntary and will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by the Education Sciences Reform Act of 2002 ((ESRA 2002) 20 U.S.C., § 9573). Overall, this session should take about [indicate correct length] minutes.

Do you have any questions?

After answering questions and giving further explanation, continue with an explanation and demonstration of the think-aloud process. If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

Teacher Participant Script

Hello, my name is <name>.

Thank you for coming here today to help us.

I have some questions that teachers like you in many different schools will be asked to answer about themselves, their instructional practices, and their schools. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. There are no wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), part of the U.S. Department of Education, make these questions clearer for teachers like you. This interview is being recorded so researchers can review the recordings later. Participant responses are voluntary and will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by the Education Sciences Reform Act of 2002 ((ESRA 2002) 20 U.S.C., § 9573). Overall, this session should take about [indicate correct length] minutes.

Do you have any questions?

School Administrator Participant Script

Hello, my name is <name >.

Thank you for coming here today to help us.

I have some questions that school administrators like you in many different schools will be asked to answer about their school policies and characteristics. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. There are no wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), part of the U.S. Department of Education, make these questions clearer for school administrators like you. This interview is being recorded so researchers can review the recordings later. Participant responses are voluntary and will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by the Education Sciences Reform Act of 2002 ((ESRA 2002) 20 U.S.C., § 9573). Overall, this session should take about [indicate correct length] minutes.

Do you have any questions?

II. Think-Aloud Instructions and Practice

The following scripts should not be read verbatim. You need to be familiar enough with the information to introduce the think-aloud in a conversational manner.

We want to learn what people think about when answering questions. As you answer each question, I’d like you to think aloud. All that means is to just read the question aloud and then say what you’re thinking as you answer the question. Please tell me whatever is going through your mind as you answer the question.

Let me give you an example about making a piece of toast this morning. If I were thinking aloud while I got the toast ready, it would sound something like this.

“Bread drawer. Muffin or white bread? White bread. Where is the knife? Oh, the sink. Cut. Plug in the toaster oven, turn it on, close it. OK, Medium. Hmm (reading the white bread package). Serving size 1 piece of bread, Calories: 130; Calories from fat 10 – I guess that’s not so bad. Vitamin A 0%, Vitamin C 0%, Calcium 8%, Iron 8% So aren’t there B vitamins in bread? Hmm. (Look up at the corner of the ceiling) Wait! I forgot to put the bread in the toaster! OK, OK, put bread in toaster. Do I want butter or…(Ding) Open the toaster, don’t burn yourself! Use the knife to pull out the bread. Table, sit down, eat.”

The point of the think-aloud is to simply say aloud the words that are in your head.

If I don’t hear you speaking, I’ll ask you to start talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I need to hear times when you are confused (and what is confusing you) and times when you understand what you are expected to do.

Sometimes it’s helpful to do a short “think-aloud” practice. I’m going to ask you a simple question and ask you to think aloud as you decide on your answer. The question is:

“How many different kinds of fruit did you eat yesterday?”

Interviewer Note: Some participants will be silent after hearing the question. Immediately encourage the participant to say whatever he/she is thinking. You may need to remind the participant that the answer to the question has to be a number by asking:

“Can you count the different kinds of fruit you ate yesterday?”

After finishing your think-aloud for each question, or after you’ve gone through all the questions in the survey, I’d like to hear any other comments — good or bad— you have about the question and any changes that would make the question easier to understand or answer.

So, do you have any questions before we start?

Interviewer Note: After answering questions and giving further explanation, begin the interview with the first item.

III. NIES: Items and Probes

After the participant thinks aloud while answering each question, the item-specific follow-up questions or probes will be asked, which are categorized as “required” or as “conditional”. Required probes are questions that will be asked to all participants. Probes that are categorized as “conditional” are questions that will only be asked if a specific response was received.

Note: In order to keep respondent burden low and ensure valid results, each student will only answer a subset of all items and probes presented in the following. The student samples (40 students at each grade level) are sufficiently large to allow for such a design. The maximum duration of each cognitive interview will be 60 minutes.

Student Questionnaire – Grade 4

[Grade 4 NIES–Student #1]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “American Indian or Alaska Native history” mean to you? Can you give me examples of “American Indian or Alaska Native history” that you have learned?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In answer choice B, what does the phrase “Family members” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	[Conditional probe: If the participant selects answer choice B, then ask the follow-up questions]: Do you have any family members who are also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
7	[Conditional probe: If the participant selects answer choice F, then ask the follow-up question]: Who is the person?	Collect further information about who else may teach the content.	Specific	Conditional
8	What does the phrase “most of what you know” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
9	Which answer choice would you select if multiple people had taught you about American Indian or Alaska Native history? How would you decide which answer choice to select?	Collect further validity information	Specific	Required

[Grade 4 NIES–Student #2]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	In answer choice A, what does the phrase “American Indian or Alaska Native traditions” mean to you? Can you give me examples of “American Indian or Alaska Native traditions” that you have learned?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In answer choice B, what does the phrase “Family members” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	[Conditional probe: If the participant selects answer choice B, then ask the follow-up questions]: Do you have any family members who are also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
7	[Conditional probe: If the participant selects answer choice F, then ask the follow-up question]: Who is the person?	Collect further information about who else may teach the content.	Specific	Conditional

[Grade 4 NIES–Student #3]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “American Indian or Alaska Native arts and crafts” mean to you? Can you give me examples of “American Indian or Alaska Native arts and crafts” that you have learned?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In answer choice B, what does the phrase “Family members” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	[Conditional probe: If the participant selects answer choice B, then ask the follow-up questions]: Do you have any family members who are also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
7	[Conditional probe: If the participant selects answer choice F, then ask the follow-up question]: Who is the person?	Collect further information about who else may teach the content.	Specific	Conditional

[Grade 4 NIES–Student #4]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “American Indian or Alaska Native language” mean to you? Can you give me an example of “American Indian or Alaska Native language” that you can speak or know about?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	How many American Indian or Alaska Native languages do you speak?	Record the response.	Specific	Required
5	How would you answer this question if you speak more than one American Indian or Alaska Native language?	Probe comprehension of the item.	Specific	Required
6	What is the difference between answer choices A and B?	Probe whether the participant differentiates answer choices A and B.	Specific	Required
7	If there were no courses offered in your school that are taught in an American Indian or Alaska Native language, which answer choice would you select?	Further probe whether the participant differentiates answer choices A and B.	Specific	Required

[Grade 4 NIES–Student #5]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	How many American Indian or Alaska Native languages do you speak?	Record the response.	Specific	Required
4	How would you answer this question if you speak more than one American Indian or Alaska Native language?	Probe comprehension of the item.	Specific	Required
5	In answer choice B, what does the phrase “I can speak a few words or phrases” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
6	In answer choice C, what does the phrase “I can speak well” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
7	Do you think that additional answer choices should be added between B and C? If yes, can you give me examples of what those answer choices would be?	Probe how the participant interprets the answer choices.	Specific	Required

[Grade 4 NIES–Student #6]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	How many American Indian or Alaska Native languages do you speak?	Record the response.	Specific	Required
4	How would you answer this question if you speak more than one American Indian or Alaska Native language?	Probe comprehension of the item.	Specific	Required
5	[Conditional probe: If the participant selects answer choice B or C, then ask the follow-up question]: Can you give me an example of “American Indian or Alaska Native language” that you can read?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
6	In answer choice B, what does the phrase “I can read a few words or phrases” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
7	In answer choice C, what does the phrase “I can read well” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
8	Do you think that additional answer choices should be added between B and C? If yes, can you give me examples of what those answer choices would be?	Probe how the participant interprets the answer choices.	Specific	Required

[Grade 4 NIES–Student #7]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	How many American Indian or Alaska Native languages do you speak?	Record the response.	Specific	Required
4	How would you answer this question if you speak more than one American Indian or Alaska Native language?	Probe comprehension of the item.	Specific	Required
5	What does the phrase “about an American Indian or Alaska Native language” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	Can you give me examples of what you consider to be part of “American Indian or Alaska Native language”?	Probe the participant’s comprehension of the phrase.	Specific	Required
7	In answer choice B, what does the phrase “Family members” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
8	In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?	Probe the participant’s comprehension of the phrase.	Specific	Required
9	[Conditional probe: If the participant selects answer choice B, then ask the follow-up questions]: Do you have any family members who are also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
10	[Conditional probe: If the participant selects answer choice F, then ask the follow-up question]: Who is the person?	Collect further information about who else may teach the content.	Specific	Conditional

[Grade 4 NIES–Student #8]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “American Indian or Alaska Native systems of counting” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up question]: Can you give me examples of “American Indian or Alaska Native systems of counting” that you know about?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
5	Do you think that answer choices A through D are clear? If not, can you give me examples of other answer choices that would be more clear?	Probe how the participant interprets the answer choice.	Specific	Required

[Grade 4 NIES–Student #9]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	In answer choice B, what does the phrase “Family members” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	[Conditional probe: If the participant selects answer choice B, then ask the follow-up questions]: Do you have any family members who are also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
6	[Conditional probe: If the participant selects answer choice F, then ask the follow-up question]: Who is the person?	Collect further information about who else may teach the content.	Specific	Conditional

[Grade 4 NIES–Student #10]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “American Indian or Alaska Native symbols and designs” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up question]: Can you give me examples of “American Indian or Alaska Native symbols and designs” that you know about?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
5	Do you think that answer choices A through D are clear? If not, can you give me examples of other answer choices that would be more clear?	Probe how the participant interprets the answer choice.	Specific	Required

[Grade 4 NIES–Student #11]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	In answer choice B, what does the phrase “Family members” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	[Conditional probe: If the participant selects answer choice B, then ask the follow-up questions]: Do you have any family members who are also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
6	[Conditional probe: If the participant selects answer choice F, then ask the follow-up question]: Who is the person?	Collect further information about who else may teach the content.	Specific	Conditional

[Grade 4 NIES–Student #12]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What time frame were you thinking about when you answered this question?	Probe the participant’s comprehension of the phrase “during the 4th grade.”	Specific	Required
4	What does the phrase “assemblies or presentations” mean to you? Can you give me examples of “assemblies or presentations”?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	What does the phrase “elders or leaders from an American Indian or Alaska Native tribe or group” mean to you? Can you give me examples of “elders or leaders”?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	[Conditional probe: If the participant selects answer choice A or B, then ask the follow-up question]: How many times have you attended these assemblies or presentations in your school?	Collect further information about attendance to assemblies or presentations.	Specific	Conditional
7	[Conditional probe: If the participant selects answer choice C, then ask the follow-up questions]: Is the reason you have not attended assemblies or presentations during 4^th grade because they are not available? If not, please explain why you have not attended assemblies or presentations during 4^th grade.	Collect further information about attendance to assemblies or presentations.	Specific	Conditional
8	Do you think that answer choices A and B are clear? If not, can you give me examples of other answer choices that would be more clear?	Probe how the participant interprets the answer choice.	Specific	Required

[Grade 4 NIES–Student #13]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	[Conditional probe: If the participant selects answer choice A, then ask the follow-up question]: What types of places have you attended to learn about American Indian or Alaska Native people?	Collect further information about places attended to learn.	Specific	Conditional
4	[Conditional probe: If the participant selects answer choice B or C, then ask the follow-up question]: How do you know that other 4^th-grade students attended, or that activities were not offered to 4^th-grade students?	Probe if the participant knows what other 4^th-grade students did or the activities were not offered to 4^th-grade students.	Specific	Conditional
5	[Conditional probe: If the participant selects answer choice A, then ask the follow-up question]: Do you know whether other 4th-grade students had the activities or the activities were not offered to 4th-grade students?	Probe if the participant knows what other 4th-grade students did or the activities were not offered to 4th-grade students.	Specific	Conditional

[Grade 4 NIES–Student #14]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	[Conditional probe: If the participant selects answer choice A or B, then ask the follow-up question]: How many times have you used these materials in school?	Test whether other answer choices should be provided.	Specific	Conditional
4	What does the phrase “online resources” mean to you? Can you give me examples of “online resources”?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	What does the phrase “in school” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	Do you think that answer choices A and B are clear? If not, can you give me examples of other answer choices that would be more clear?	Probe how the participant interprets the answer choice.	Specific	Required

[Grade 4 NIES–Student #15]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	[Conditional probe: If the participant selects answer choice A or B, then ask the follow-up question]: How many times have you used these materials outside of school?	Test whether other answer choices should be provided and Record the answer choice selected.	Specific	Conditional
4	What does the phrase “outside of school” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	Do you think that answer choices A and B are clear? If not, can you give me examples of other answer choices that would be more clear?	Probe how the participant interprets the answer choice.	Specific	Required

[Grade 4 NIES–Student #16]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “school library, media center, or resource center” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	What does the phrase “have access” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required

[Grade 4 NIES–Student #17]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “when my teacher talks about” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	Which answer choice would you select if your teacher had not talked about American Indian or Alaska Native history or culture? Why would you select this answer choice?	Precondition of the following probes.	Specific	Required

[Grade 4 NIES–Student #18]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	[Conditional probe: If the participant selects answer choice B or C, then ask the follow-up question]: Can you give me examples of how you put a lot of effort into your schoolwork?	Probe comprehension of the item.	Specific	Conditional

[Grade 4 NIES–Student #19]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “one of the best students” mean to you?	Probe how the participant defines “best student.”	Specific	Required

[Grade 4 NIES–Student #20]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “being challenged” mean to you? Can you give me examples of you being challenged in your classes?	Probe how the participant interprets the phrase.	Specific	Required

[Grade 4 NIES–Student #21]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “belong at school” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice B or C, then ask the follow-up question]: Can you give me examples of when you feel you “belong at school”?	Probe how the participant interprets the phrase.	Specific	Conditional

[Grade 4 NIES–Student #22]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	Specific	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “someone else who lives in your community or is a friend of your family” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up question]: Can you give me examples of how these people have helped you with your schoolwork? Who is this person?	Probe participant to provide additional examples.	Specific	Conditional

[Grade 4 NIES–Student #23]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	[Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up questions]: Can you give me examples of how you have helped other students with their schoolwork? Are the other students that you have helped with their schoolwork in 4^th grade? If not, what grade are the students that you have helped with their schoolwork in?	Probe participant to provide additional examples.	Specific	Conditional

[Grade 4 NIES–Student #24]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “put in enough effort” mean to you? Can you give me examples of how you could put in enough effort?	Probe how the participant interprets the phrase.	Specific	Required
4	What does the phrase “succeed in school” mean to you? Can you give me examples of what “succeed in school” would look like?	Probe how the participant interprets the phrase.	Specific	Required

[Grade 4 NIES–Student #25]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “trying hard in school” mean to you? Can you give me examples of what “trying hard in school” would be?	Probe how the participant interprets the phrase.	Specific	Required
4	What does the phrase “a good life” mean to you? Can you give me examples of what “a good life” would look like?	Probe how the participant interprets the phrase.	Specific	Required
5	What does the phrase “when I grow up” mean to you? Can you give me examples of what being “grown up” means to you?	Probe how the participant interprets the phrase.	Specific	Required

Student Questionnaire – Grade 8

[Grade 8 NIES–Student #1]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “American Indian or Alaska Native history” mean to you? Can you give me examples of “American Indian or Alaska Native history” that you have learned?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In answer choice B, what does the phrase “Family members” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	[Conditional probe: If the participant selects answer choice B, then ask the follow-up questions]: Do you have any family members who are also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
7	[Conditional probe: If the participant selects answer choice F, then ask the follow-up question]: Who is the person?	Collect further information about who else may teach the content.	Specific	Conditional
8	What does the phrase “most of what you know” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
9	Which answer choice would you select if multiple people had taught you about American Indian or Alaska Native history? How would you decide which answer choice to select?	Collect further validity information	Specific	Required

[Grade 8 NIES–Student #2]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	In answer choice A, what does the phrase “American Indian or Alaska Native traditions” mean to you? Can you give me examples of “American Indian or Alaska Native traditions” that you have learned?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In answer choice B, what does the phrase “Family members” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	[Conditional probe: If the participant selects answer choice B, then ask the follow-up questions]: Do you have any family members who are also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
7	[Conditional probe: If the participant selects answer choice F, then ask the follow-up question]: Who is the person?	Collect further information about who else may teach the content.	Specific	Conditional

[Grade 8 NIES–Student #3]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	In answer choice B, what does the phrase “Family members” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	[Conditional probe: If the participant selects answer choice B, then ask the follow-up questions]: Do you have any family members who are also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
6	[Conditional probe: If the participant selects answer choice F, then ask the follow-up question]: Who is the person?	Collect further information about who else may teach the content.	Specific	Conditional

[Grade 8 NIES–Student #4]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “American Indian or Alaska Native language” mean to you? Can you give me an example of “American Indian or Alaska Native language” that you can speak or know about?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	How many American Indian or Alaska Native languages do you speak?	Record the response.	Specific	Required
5	How would you answer this question if you speak more than one American Indian or Alaska Native language?	Probe comprehension of the item.	Specific	Required
6	What is the difference between answer choices A and B?	Probe whether the participant differentiates answer choices A and B.	Specific	Required
7	If there were no courses offered in your school that are taught in an American Indian or Alaska Native language, which answer choice would you select?	Further probe whether the participant differentiates answer choices A and B.	Specific	Required

[Grade 8 NIES–Student #5]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	How many American Indian or Alaska Native languages do you speak?	Record the response.	Specific	Required
4	How would you answer this question if you speak more than one American Indian or Alaska Native language?	Probe comprehension of the item.	Specific	Required
5	In answer choice B, what does the phrase “I can speak a few words or phrases” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
6	In answer choice C, what does the phrase “I can speak well” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
7	Do you think that additional answer choices should be added between B and C? If yes, can you give me examples of what those answer choices would be?	Probe the participant’s comprehension of the answer choice.	Specific	Required

[Grade 8 NIES–Student #6]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	How many American Indian or Alaska Native languages do you speak?	Record the response.	Specific	Required
4	How would you answer this question if you speak more than one American Indian or Alaska Native language?	Probe comprehension of the item.	Specific	Required
5	[Conditional probe: If the participant selects answer choice B or C, then ask the follow-up question]: Can you give me an example of “American Indian or Alaska Native language” that you can read?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
6	In answer choice B, what does the phrase “I can read a few words or phrases” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
7	In answer choice C, what does the phrase “I can read well” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
8	Do you think that additional answer choices should be added between B and C? If yes, can you give me examples of what those answer choices would be?	Probe the participant’s comprehension of the answer choice.	Specific	Required

[Grade 8 NIES–Student #7]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	How many American Indian or Alaska Native languages do you speak?	Record the response.	Specific	Required
4	How would you answer this question if you speak more than one American Indian or Alaska Native language?	Probe comprehension of the item.	Specific	Required
5	What does the phrase “about an American Indian or Alaska Native language” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	Can you give me examples of what you consider to be part of “American Indian or Alaska Native language”?	Probe the participant’s comprehension of the phrase.	Specific	Required
7	In answer choice B, what does the phrase “Family members” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
8	In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?	Probe the participant’s comprehension of the phrase.	Specific	Required
9	[Conditional probe: If the participant selects answer choice B, then ask the follow-up questions]: Do you have any family members who are also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
10	[Conditional probe: If the participant selects answer choice F, then ask the follow-up question]: Who is the person?	Collect further information about who else may teach the content.	Specific	Conditional

[Grade 8 NIES–Student #8]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	In answer choice A, what does the phrase “American Indian or Alaska Native systems of counting” mean to you? Can you give me examples of “American Indian or Alaska Native systems of counting” that you know about?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In answer choice B, what does the phrase “Family members” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	[Conditional probe: If the participant selects answer choice B, then ask the follow-up questions]: Do you have any family members who are also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
7	[Conditional probe: If the participant selects answer choice F, then ask the follow-up question]: Who is the person?	Collect further information about who else may teach the content.	Specific	Conditional

[Grade 8 NIES–Student #9]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	In answer choice A, what does the phrase “American Indian or Alaska Native symbols and designs” mean to you? Can you give me examples of “American Indian or Alaska Native symbols and designs” that you know about?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In answer choice B, what does the phrase “Family members” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	[Conditional probe: If the participant selects answer choice B, then ask the follow-up questions]: Do you have any family members who are also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?	Probe the participant’s comprehension of the phrase.	Specific	Conditional
7	[Conditional probe: If the participant selects answer choice F, then ask the follow-up question]: Who is the person?	Collect further information about who else may teach the content.	Specific	Conditional

[Grade 8 NIES–Student #10]

Shape3

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. through d.?	Record the answer choice selected.	General	Required
3	[Conditional probe: If the participant selects answer choice B or C, then ask the follow-up question]: How do you know that other 8^th-grade students attended, or that activities were not offered to 8^th-grade students?	Probe if the participant knows what other 8^th-grade students did or the activities were not offered to 8^th-grade students.	Specific	Conditional
4	[Conditional probe: If the participant selects answer choice A, then ask the follow-up questions]: Do you know whether other 8^th-grade students had the activities?	Probe if the participant knows what other 8^th-grade students did or the activities were not offered to 8^th-grade students.	Specific	Conditional
5	For questions a. through d., how would you answer this question if you did not attend these activities and did not know whether other 8^th graders attended these activities?	Probe how the participant interprets the item.	Specific	Required

[Grade 8 NIES–Student #11]

Revised/VH039972 During 8th grade, how often have you used any of the following materials in school? Fill in one oval on each line.
	Never or hardly ever	Once or twice a month	Once or twice a week	Every day or almost every day
Books, videos, or other materials (including online resources) about American Indian or Alaska Native history, traditions, or cultures (way of life, customs)	A	B	C	D	Revised/VH039974
Books, videos, or other materials (including online resources) not related to American Indian or Alaska Native people, traditions, or cultures	A	B	C	D	Revised/VH040079
Books, videos, or other materials (including online resources) about current issues that are important to American Indian or Alaska Native people	A	B	C	D	Revised/VH040083

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. through c.?	Record the answer choice selected.	General	Required
3	In question b., what does the phrase “books, videos, or other materials not related to American Indian or Alaska Native people, traditions, or cultures” mean to you? Can you give me examples of these books, videos, or other materials?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In question a., what does the phrase “online resources” mean to you? Can you give me examples of “online resources”?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	What does the phrase “in school” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	Have you used any of the books, videos, or materials listed in questions a. through c. outside of school, but not for “in school” class work or assignments? If yes, do you consider those activities to be “in school” class work or assignments?	Probe how the participant interprets the phrase.	Specific	Required

[Grade 8 NIES–Student #12]

Revised/VH040104 During 8th grade, how often have you used any of the following materials outside of school? Fill in one oval on each line.
	Never or hardly ever	Once or twice a month	Once or twice a week	Every day or almost every day
Books, videos, or other materials (including online resources) about American Indian or Alaska Native history, traditions, or cultures (way of life, customs)	A	B	C	D	Revised/VH040106
Books, videos, or other materials (including online resources) not related to American Indian or Alaska Native history, traditions, or cultures	A	B	C	D	Revised/VH040109
Books, videos, or other materials (including online resources) about current issues that are important to American Indian or Alaska Native people	A	B	C	D	Revised/VH040111

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. through c.?	Record the answer choice selected.	General	Required
3	What does the phrase “outside of school” mean to you? Do you consider “outside of school” to include all activities that are for schoolwork and not for schoolwork?”	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In question b., what does the phrase “books, videos, or other materials not related to American Indian or Alaska Native people, traditions, or cultures” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required

[Grade 8 NIES–Student #13]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “school library, media center, or resource center” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	What does the phrase “have access” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required

[Grade 8 NIES–Student #14]

Revised/VC996624 Here are some sentences about reading. Fill in one oval on each line to show whether the sentence describes a person like you.
	This is not like me.	This is a little like me.	This is a lot like me.
When my teacher talks about American Indian or Alaska Native history or culture, I try to read more about it.	A	B	C	Revised/ VH043704
I enjoy reading about American Indian or Alaska Native people.	A	B	C	Trend/ VC996627
I enjoy reading about people who have different traditions and cultures (ways of life, customs) than I have.	A	B	C	Trend/ VC996628

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what question a. is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for question a.?	Record the answer choice selected.	General	Required
3	In question a. what does the phrase “when my teacher talks about” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	Which answer choice would you select if your teacher had not talked about American Indian or Alaska Native history or culture? Why would you select this answer choice?	Precondition of the following probes.	Specific	Required

[Grade 8 NIES–Student #15]

New/VH040130 Here are some sentences about your school. Fill in one oval on each line to show whether the sentence describes a person like you.
	This is not like me.	This is a little like me.	This is a lot like me.
I put a lot of effort into my schoolwork.	A	B	C	New/ VH040131
I want to be one of the best students in my class.	A	B	C	New/ VH040134
I enjoy being challenged in my classes.	A	B	C	New/ VH040137
I feel that I belong at school.	A	B	C	New/ VH040138

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question a. is asking?	Probe comprehension of the item.	General	Required
2	[Conditional probe: For question a., if the participant selects answer choice B or C, then ask the follow-up question]: Can you give me examples of how you put a lot of effort into your schoolwork?	Probe comprehension of the item.	Specific	Conditional
3	Can you tell me, in your own words, what question b. is asking?	Probe comprehension of the item.	General	Required
4	Can you tell me, in your own words, what question c. is asking?	Probe comprehension of the item.	General	Required
5	Can you tell me, in your own words, what question d. is asking?	Probe comprehension of the item.	General	Required
6	Which answer choice would you select for questions a. through d.?	Record the answer choice selected.	General	Required
7	In question b., what does the phrase “one of the best students” mean to you?	Probe how the participant defines “best student.”	Specific	Required
8	In question c., what does the phrase “being challenged” mean to you? Can you give me examples of you “being challenged in your classes”?	Probe how the participant interprets the phrase.	Specific	Required
9	In question d., what does the phrase “belong at school” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
10	[Conditional probe: For question d. if the participant selects answer choice B or C, then ask the follow-up question]: Can you give me examples of when you feel you “belong at school”?	Probe comprehension of the phrase.	Specific	Conditional

[Grade 8 NIES–Student #16]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	[Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up question]: Can you give me examples of how you have helped other students with their schoolwork? Are the other students that you have helped with their schoolwork in 8^th grade? If not, what grade are the students that you have helped with their schoolwork in?	Probe participant to provide additional examples.	Specific	Conditional

[Grade 8 NIES–Student #17]

Trend/VC996635 How often do any of the following people help you with your schoolwork? For example, they might help you to study for a test, help you with a school project, or go over your homework with you. Fill in one oval on each line.
	Never or hardly ever	Once or twice a month	Once or twice a week	Every day or almost every day
A parent or someone else from my family	A	B	C	D	Trend/ VC996636
A teacher or another adult from my school	A	B	C	D	Trend/ VC996637
Another student	A	B	C	D	Trend/ VC996639
A tribal representative or elder	A	B	C	D	New/VH018268
Someone else who lives in my community or is a friend of my family	A	B	C	D	Trend/ VC996640

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	Specific	Required
2	Can you tell me, in your own words, what question d. is asking?	Probe comprehension of the item.	Specific	Required
3	Can you tell me, in your own words, what question e. is asking?	Probe comprehension of the item.	Specific	Required
4	Which answer choice would you select for questions d. and e.?	Record the answer choice selected.	General	Required
5	In question e., what does the phrase “someone else who lives in your community or is a friend of your family” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
6	[Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up questions]: Can you give me examples of these people who have helped you with your schoolwork? Can you give me examples of how these people have helped you?	Probe how the participant interprets the item.	Specific	Conditional

[Grade 8 NIES–Student #18]

Revised/ VH040140 During 8th grade, have you participated in any of the following activities organized by your school? Fill in one oval on each line.
	Yes	No
Sports team	A	B	Revised/VH040143
Drama club	A	B	Revised/VH040146
Music club, orchestra, or band	A	B	Revised/VH040147
Cultural dances or other American Indian or Alaska Native activities (for example, drum groups)	A	B	Revised/VH040150
Academic club (for example, math club)	A	B	Revised/VH040152

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. through e.?	Record the answer choice selected.	General	Required
3	In question a., what does the phrase “sports team” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	In question b., what does the phrase “drama club” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
5	In question c., what does the phrase “music club, orchestra, or band” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
6	In question d., what does the phrase “cultural dances or other American Indian or Alaska Native activities” mean to you? Do you think the example “drum groups” is an example of cultural dances? If not, can you give me examples of what “cultural dances” would be?	Probe how the participant interprets the phrase.	Specific	Required
7	In question e., what does the phrase “academic club” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
8	In addition to the activities listed in questions a. through e., do you think any additional activities should be included? If yes, can you give me examples of additional activities?	Probe the participant to provide specific examples.	Specific	Conditional

[Grade 8 NIES–Student #19]

Trend/ VC996644 During 8th grade, how many times have you talked to each of the following people about the classes you should take in high school or about what you want to do after high school? Fill in one oval on each line.
	Never	One time	Two or three times	Four or more times
A family member	A	B	C	D	Trend/VC996645
A teacher	A	B	C	D	Trend/VC996646
A school counselor	A	B	C	D	Trend/VC996647
Another student	A	B	C	D	Trend/VC996648
A tribal representative or elder	A	B	C	D	New/VH026209
Someone else who lives in my community or is a friend of my family	A	B	C	D	Trend/VH026211

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Can you tell me, in your own words, what question e. is asking?	Probe comprehension of the item.	Specific	Required
3	Which answer choice would you select for question e.?	Record the answer choice selected.	General	Required
4	[Conditional probe: If the participant selects answer choice A, then ask the follow-up questions]: Do you have any family members who are also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?	Probe the participant’s comprehension of the phrase.	Specific	Conditional

[Grade 8 NIES–Student #20]

New/VH040157 How much do you agree with each of the following statements? Fill in one oval on each line.
	Strongly disagree	Disagree	Agree	Strongly agree
If I put in enough effort I will succeed in school.	A	B	C	D	New/VH040158
If I put in enough effort in school I will get into college.	A	B	C	D	New/VH040160
If I put in enough effort in school I will get a good job.	A	B	C	D	New/VH040161

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. through c.?	Record the answer choice selected.	General	Required
3	What does the phrase “put in enough effort” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	In question a., what does the phrase “succeed in school” mean to you? Can you give me examples of what “succeed in school” would look like?	Probe how the participant interprets the phrase.	Specific	Required
5	In question b., what does the phrase “get into college” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
6	In question c., what does the phrase “get a good job” mean to you? Can you give me examples of what “a good job” would be?	Probe how the participant interprets the phrase.	Specific	Required

[Grade 8 NIES–Student #21]

New/VH040216

To what extent is each of the following a problem in your school? Fill in one oval on each line.

	Not at all	Small extent	Moderate extent	Large extent
Student tardiness	A	B	C	D	New/VH040217
Student health problems	A	B	C	D	New/VH040219
Student misbehavior in class	A	B	C	D	New/VH040221
Physical conflicts among students	A	B	C	D	New/VH040223
Bullying	A	B	C	D	New/VH040226
Low student motivation	A	B	C	D	New/VH040227
Low teacher expectations	A	B	C	D	New/VH040231

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. though g.?	Record the answer choice selected.	General	Required
3	What does the phrase “student tardiness” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	In question a., what does the phrase “student health problems” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
5	In question c., what does the phrase “student misbehavior in class” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
6	In question d., what does the phrase “physical conflicts among students” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
7	In question e., what does the phrase “bullying” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
8	In question f., what does the phrase “low student motivation” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
9	In question g., what does the phrase “low teacher expectations” mean to you?	Probe how the participant interprets the phrase.	Specific	Required

Teacher Questionnaire – Grade 4

[Grade 4 NIES–Teacher #1]

New/VH040247

During the last two years, to what extent have you implemented culturally specific instructional practices for American Indian or Alaska Native students in your classroom?

Not at all
Small extent
Moderate extent
Large extent

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	Do you consider “the last two years” to be a reasonable time frame? If not, can you give me examples of what a reasonable time frame would be?	Probe how the participant interprets the phrase.	Specific	Required
4	Do you consider “the last two years” to be this school year and last school year, or to be the last 24 months? If not, can you give me examples of what “the last two years” would be?	Probe how the participant interprets the phrase.	Specific	Required
5	What does the phrase “culturally specific instructional practices for American Indian or Alaska Native students” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	[Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up question]: Can you give me an example for culturally specific instructional practices for American or Alaska Native students that you have implemented in your classroom?	Probe the participant to provide specific examples.	Specific	Conditional

[Grade 4 NIES–Teacher #2]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	Do you consider “the last two years” to be a reasonable time frame? If not, can you give me examples of what a reasonable time frame would be?	Probe how the participant interprets the phrase.	Specific	Required
4	Do you consider “the last two years” to be this school year and last school year, or to be the last 24 months? If not, can you give me examples of what “the last two years” would be?	Probe how the participant interprets the phrase.	Specific	Required
5	What does the phrase “professional or community-based development programs” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
6	What does the phrase “in-service classes” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
7	When you answered this question, did you consider activities that were not in-service classes and workshops or online classes? If yes, can you give me examples of what these other activities are?	Probe the participant to provide specific examples.	Specific	Required
8	[Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up questions]: What development programs have you attended? Were any of the programs you attended taught as online classes? Have you attended development programs that are not through in-service classes?	Probe how the participant interprets the phrase.	Specific	Conditional
G	[Instructions for interviewer: After completing the required probes above, skip to question #5 only if the participant selects answer choice A for this question.]	N/A	N/A	N/A

[Grade 4 NIES–Teacher #3]

New/VH040281

To what extent have you implemented lessons learned from these professional or community-based development programs in your classroom?

Not at all
Small extent
Moderate extent
Large extent

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “lessons learned” from these programs mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up question]: Can you give me examples of how you have implemented lessons learned from these professional or community-based development programs in your classroom?	Probe participant to provide additional examples.	Specific	Conditional

[Grade 4 NIES–Teacher #4]

Revised/VH040284

Who sponsored the professional or community-based development programs you attended in the last two years? Fill in all ovals that apply.

State
District
Tribal education department
Indian education professional associations
College or university
Other (please specify): ________________

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choices would you select?	Record the answer choice selected.	General	Required
3	[Conditional probe: If the participant selects answer choice F, then ask the follow-up question]: What is this funding source?	Collect further information on who has sponsored programs.	Specific	Conditional

[Grade 4 NIES–Teacher #5]

Trend/VE012668 To what extent do you use the following to assess student progress? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
State assessments	A	B	C	D	Trend/VE012670
District assessments	A	B	C	D	Trend/VE012672
Assessments developed by American Indian or Alaska Native organizations	A	B	C	D	Trend/VE012673
Tests supplied by textbook publishers (for example, end of unit or chapter tests)	A	B	C	D	Trend/VE012674
Teacher-made tests or quizzes	A	B	C	D	Trend/VE012675
Performance-based assessments	A	B	C	D	Trend/VE012676
Group projects	A	B	C	D	Trend/VE012678
Oral responses of students during class discussions	A	B	C	D	Trend/VE012681
Assessments to evaluate English language proficiency	A	B	C	D	New/VH040300

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for question i.?	Record the answer choice selected.	Specific	Required
3	In question i., what does the phrase “assessments to evaluate English language proficiency” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice B, C, or D for question i., then ask the follow-up question]: Can you give me examples of how you have used “assessments to evaluate English language proficiency” to assess student progress?	Probe participant to provide additional examples.	Specific	Conditional
5	In question c., what specific organizations did you consider when answering this question?	Probe how the participant interprets the item.	Specific	Required
6	When answering this question did you consider NIES information about student progress? If yes, in which question specifically did you consider NIES information about student progress?	Probe how the participant interprets the item.	Specific	Required

[Grade 4 NIES–Teacher #6]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
G	[Instructions for interviewer: If the participant selects answer choice A, then proceed to question #7. If the participant selects answer choice B, then skip to question #8.]	N/A	N/A	N/A

[Grade 4 NIES–Teacher #7]

Revised/VH040381 How often do you integrate materials about the following topics into your reading/language arts lessons? Fill in one oval on each line.
	Never	At least once a year	At least once a month	At least once a week	Every day or almost every day
American Indian or Alaska Native culture or history	A	B	C	D	E	Revised/VH040385
Current issues affecting American Indian or Alaska Native people or communities	A	B	C	D	E	Revised/VH040386

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. and b.?	Record the answer choice selected.	General	Required
3	In question a., what does the phrase “American Indian or Alaska Native culture or history” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In question b., what does the phrase “current issues affecting American Indian or Alaska Native people or communities” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	[Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b., or to both, then ask the follow-up question]: Can you give me examples of how you have integrated materials about the topics into your reading/language arts lessons?	Probe participant to provide additional examples.	Specific	Conditional

[Grade 4 NIES–Teacher #8]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
G	[Instructions for interviewer: If the participant selects answer choice A, then proceed to question #9. If the participant selects answer choice B, then skip to question #10.]	N/A	N/A	N/A

[Grade 4 NIES–Teacher #9]

Revised/VH040387 How often do you integrate materials about the following topics into your mathematics lessons? Fill in one oval on each line.
	Never	At least once a year	At least once a month	At least once a week	Every day or almost every day
American Indian or Alaska Native culture or history	A	B	C	D	E	Revised/VH040390
Current issues affecting American Indian or Alaska Native people or communities	A	B	C	D	E	Revised/VH040392

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. and b.?	Record the answer choice selected.	General	Required
3	[Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b., or to both, then ask the follow-up question]: Can you give me examples of how you have integrated materials about the topics into your mathematics lessons?	Probe participant to provide additional examples.	Specific	Conditional

[Grade 4 NIES–Teacher #10]

New/VH043953

How much do you agree with each of the following statements about the materials available in your school library, media center, or resource center? Fill in one oval on each line.

	Strongly disagree	Disagree	Agree	Strongly agree
The number of books and materials available for 4^th grade students is sufficient.	A	B	C	D	New/VH043957
The quality of the books and materials available for 4^th grade students is satisfactory.	A	B	C	D	New/VH043959
The number of books and materials specific to American Indian and Alaska Native culture available for 4^th grade students is sufficient.	A	B	C	D	New/VH043964
The quality of the books and materials specific to American Indian and Alaska Native culture available for 4^th grade students is satisfactory.	A	B	C	D	New/VH043966

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. through d.?	Record the answer choice selected.	General	Required
3	In questions a. and c., what does the phrase “is sufficient” mean to you?	Probe comprehension in this context.	Specific	Required
4	In questions b. and d., what does the phrase “is satisfactory” mean to you?	Probe comprehension in this context.	Specific	Required
5	In questions b. and d., what does the phrase “the quality of the books and materials” mean to you?	Probe comprehension in this context.	Specific	Required
6	In questions c. and d., what does the phrase “specific to American Indian and Alaska Native cultures” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required

[Grade 4 NIES–Teacher #11]

New/VH040411

To what extent is each of the following a problem in your school? Fill in one oval on each line.

	Not at all	Small extent	Moderate extent	Large extent
Student absenteeism	A	B	C	D	New/VH040413
Student tardiness	A	B	C	D	New/VH040414
Student health problems	A	B	C	D	New/VH040416
Student misbehavior in class	A	B	C	D	New/VH040422
Physical conflicts among students	A	B	C	D	New/VH040424
Bullying	A	B	C	D	New/VH040425
Low student aspirations	A	B	C	D	New/VH040427
Low teacher expectations	A	B	C	D	New/VH040428
Low family involvement	A	B	C	D	New/VH040430

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. through i.?	Record the answer choice selected.	General	Required
3	In question a., what does the phrase “student absenteeism” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In question b., what does the phrase “student tardiness” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	In question c., what does the phrase “student health problems” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	In question d., what does the phrase “student misbehavior in class” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
7	In question e., what does the phrase “physical conflicts among students” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
8	In question f., what does the phrase “bullying” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
9	In question g., what does the phrase “low student aspirations” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
10	In question h., what does the phrase “low teacher expectations” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
11	In question i., what does the phrase “low family involvement” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required

Teacher Questionnaire – Grade 8

[Grade 8 NIES–Teacher #1]

New/VH040247

During the last two years, to what extent have you implemented culturally specific instructional practices for American Indian or Alaska Native students in your classroom?

Not at all
Small extent
Moderate extent
Large extent

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	Do you consider “the last two years” to be a reasonable time frame? If not, can you give me examples of what a reasonable time frame would be?	Probe how the participant interprets the phrase.	Specific	Required
4	Do you consider “the last two years” to be this school year and last school year, or to be the last 24 months? If not, can you give me examples of what “the last two years” would be?	Probe how the participant interprets the phrase.	Specific	Required
5	What does the phrase “culturally specific instructional practices for American Indian or Alaska Native students” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	[Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up question]: Can you give me an example for culturally specific instructional practices for American or Alaska Native students that you have implemented in your classroom?	Probe the participant to provide specific examples.	Specific	Conditional

[Grade 8 NIES–Teacher #2]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	Do you consider “the last two years” to be a reasonable time frame? If not, can you give me examples of what a reasonable time frame would be?	Probe how the participant interprets the phrase.	Specific	Required
4	Do you consider “the last two years” to be this school year and last school year, or to be the last 24 months? If not, can you give me examples of what “the last two years” would be?	Probe how the participant interprets the phrase.	Specific	Required
5	What does the phrase “professional or community-based development programs” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
6	What does the phrase “in-service classes” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
7	When you answered this question, did you consider activities that were not in-service classes and workshops or online classes? If yes, can you give me examples of what these other activities are?	Probe the participant to provide specific examples.	Specific	Required
8	[Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up questions]: What development programs have you attended? Were any of the programs you attended taught as online classes? Have you attended development programs that are not through in-service classes?	Probe how the participant interprets the phrase.	Specific	Conditional
G	[Instructions for interviewer: After completing the required probes above, skip to question #5 only if the participant selects answer choice A for this question.]	N/A	N/A	N/A

[Grade 8 NIES–Teacher #3]

New/VH040281

To what extent have you implemented lessons learned from these professional or community-based development programs in your classroom?

Not at all
Small extent
Moderate extent
Large extent

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “lessons learned” from these programs mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up question]: Can you give me examples of how you have implemented lessons learned from these professional or community-based development programs in your classroom?	Probe participant to provide additional examples.	Specific	Conditional

[Grade 8 NIES–Teacher #4]

Revised/VH040284

Who sponsored the professional or community-based development programs you attended in the last two years? Fill in all ovals that apply.

State
District
Tribal education department
Indian education professional associations
College or university
Other (please specify): ________________

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	[Conditional probe: If the participant selects answer choice F, then ask the follow-up question]: What is this funding source?	Collect further information on who has sponsored programs.	Specific	Conditional

[Grade 8 NIES–Teacher #5]

Trend/VE012668 To what extent do you use the following to assess student progress? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
State assessments	A	B	C	D	Trend/VE012670
District assessments	A	B	C	D	Trend/VE012672
Assessments developed by American Indian or Alaska Native organizations	A	B	C	D	Trend/VE012673
Tests supplied by textbook publishers (for example, end of unit or chapter tests)	A	B	C	D	Trend/VE012674
Teacher-made tests or quizzes	A	B	C	D	Trend/VE012675
Performance-based assessments	A	B	C	D	Trend/VE012676
Group projects	A	B	C	D	Trend/VE012678
Oral responses of students during class discussions	A	B	C	D	Trend/VE012681
Assessments to evaluate English language proficiency	A	B	C	D	New/VH040300

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for question i.?	Record the answer choice selected.	General	Required
3	In question i., what does the phrase “assessments to evaluate English language proficiency” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice B, C, or D for question i., then ask the follow-up question]: Can you give me examples of how you have used “assessments to evaluate English language proficiency” to assess student progress?	Probe participant to provide additional examples.	Specific	Conditional
5	In question C, what specific organizations did you consider when answering this question?	Probe how the participant interprets the question.	Specific	Required
6	When answering this question did you consider NIES information about student progress? If yes, in which question specifically did you consider NIES information about student progress?	Probe how the participant interprets the question.	Specific	Required

[Grade 8 NIES–Teacher #6]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
G	[Instructions for interviewer: If the participant selects answer choice A, then proceed to question #7. If the participant selects answer choice B, then skip to question #8.]	N/A	N/A	N/A

[Grade 8 NIES–Teacher #7]

Revised/VH040381 How often do you integrate materials about the following topics into your reading/language arts lessons? Fill in one oval on each line.
	Never	At least once a year	At least once a month	At least once a week	Every day or almost every day
American Indian or Alaska Native culture or history	A	B	C	D	E	Revised/VH040385
Current issues affecting American Indian or Alaska Native people or communities	A	B	C	D	E	Revised/VH040386

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. and b.?	Record the answer choice selected.	General	Required
3	In question a., what does the phrase “American Indian or Alaska Native culture or history” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In question b., what does the phrase “current issues affecting American Indian or Alaska Native people or communities” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	[Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b., or to both, then ask the follow-up question]: Can you give me examples of how you have integrated materials about the topics into your reading/language arts lessons?	Probe participant to provide additional examples.	Specific	Conditional

[Grade 8 NIES–Teacher #8]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
G	[Instructions for interviewer: If the participant selects answer choice A, then proceed to question #9. If the participant selects answer choice B, then skip to question #10.]	N/A	N/A	N/A

[Grade 8 NIES–Teacher #9]

Revised/VH040387 How often do you integrate materials about the following topics into your mathematics lessons? Fill in one oval on each line.
	Never	At least once a year	At least once a month	At least once a week	Every day or almost every day
American Indian or Alaska Native culture or history	A	B	C	D	E	Revised/VH040390
Current issues affecting American Indian or Alaska Native people or communities	A	B	C	D	E	Revised/VH040392

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. and b.?	Record the answer choice selected.	General	Required
3	[Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b., or to both, then ask the follow-up question]: Can you give me examples of how you have integrated materials about the topics into your mathematics lessons?	Probe participant to provide additional examples.	Specific	Conditional

[Grade 8 NIES–Teacher #10]

New/VH040395

How much do you agree with each of the following statements about the materials available in your school library, media center, or resource center? Fill in one oval on each line.

	Strongly disagree	Disagree	Agree	Strongly agree
The number of books and materials available for 8^th grade students is sufficient.	A	B	C	D	New/VH040399
The quality of the books and materials available for 8^th grade students is satisfactory.	A	B	C	D	New/VH040401
The number of books and materials specific to American Indian and Alaska Native culture available for 8^th grade students is sufficient.	A	B	C	D	New/VH040402
The quality of the books and materials specific to American Indian and Alaska Native culture available for 8^th grade students is satisfactory.	A	B	C	D	New/VH040404

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. through d.?	Record the answer choice selected.	General	Required
3	In questions a. and c., what does the phrase “is sufficient” mean to you?	Probe comprehension in this context.	Specific	Required
4	In questions b. and d., what does the phrase “is satisfactory” mean to you?	Probe comprehension in this context.	Specific	Required
5	In questions b. and d., what does the phrase “the quality of the books and materials” mean to you?	Probe comprehension in this context.	Specific	Required
6	In questions c. and d., what does the phrase “specific to American Indian and Alaska Native cultures” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required

[Grade 8 NIES–Teacher #11]

New/VH040406

What proportion of your American Indian or Alaska Native students will complete the 8^th grade?

None
A few
Some
Most
All
I don’t know.

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What proportion of your 8^th grade students are American Indian or Alaska Native? To what extent does having a high or low proportion of American Indian or Alaska Native students influence the answer choice you selected?	Probe how the participant calculated the answer.	Specific	Required
4	[Conditional probe: If the participant selects answer choice A, B, C, D, or E, then ask the follow-up question]: Please explain why you selected the answer choice. In other words, how did you figure out the answer?	Probe how the participant calculated the answer.	Specific	Conditional
5	Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?	Probe whether the participant is able to answer the question.	Specific	Required
6	Do you think that answer choices A through F are clear? If not, can you give me examples of other answer choices that would be more clear?	Probe how the participant interprets the answer choice.	Specific	Required

[Grade 8 NIES–Teacher #12]

New/VH040408

What proportion of your American Indian or Alaska Native students will be prepared for high school?

None
A few
Some
Most
All
I don’t know.

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What proportion of your 8^th grade students are American Indian or Alaska Native? To what extent does having a high or low proportion of American Indian or Alaska Native students influence the answer choice you selected?	Probe how the participant calculated the answer.	Specific	Required
4	[Conditional probe: If the participant selects answer choice A, B, C, D, or E, then ask the follow-up question]: Please explain why you selected the answer choice. In other words, how did you figure out the answer?	Probe how the participant calculated the answer.	Specific	Conditional
5	Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?	Probe whether the participant is able to answer the question.	Specific	Required
6	Do you think that answer choices A through F are clear? If not, can you give me examples of other answer choices that would be more clear?	Probe how the participant interprets the answer choice.	Specific	Required

[Grade 8 NIES–Teacher #13]

New/VH040411

To what extent is each of the following a problem in your school? Fill in one oval on each line.

	Not at all	Small extent	Moderate extent	Large extent
Student absenteeism	A	B	C	D	New/VH040413
Student tardiness	A	B	C	D	New/VH040414
Student health problems	A	B	C	D	New/VH040416
Teen pregnancies	A	B	C	D	New/VH040417
Drug or alcohol use by students	A	B	C	D	New/VH040421
Student misbehavior in class	A	B	C	D	New/VH040422
Physical conflicts among students	A	B	C	D	New/VH040424
Bullying	A	B	C	D	New/VH040425
Low student aspirations	A	B	C	D	New/VH040427
Low teacher expectations	A	B	C	D	New/VH040428
Low family involvement	A	B	C	D	New/VH040430

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. through k.?	Record the answer choice selected.	General	Required
3	In question a., what does the phrase “student absenteeism” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In question b., what does the phrase “student tardiness” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	In question c., what does the phrase “student health problems” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	[Conditional probe: If the participant selects answer choice B, C, or D to question d., then ask the follow-up question]: Can you give me examples of how teen pregnancies are a problem in your school?	Record the answer choice selected.	Specific	Conditional
7	In question e., what does the phrase “drug or alcohol use by students” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
8	In question f., what does the phrase “student misbehavior in class” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
9	In question g., what does the phrase “physical conflicts among students” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
10	In question h., what does the phrase “bullying” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
11	In question i., what does the phrase “low student aspirations” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
12	In question j., what does the phrase “low teacher expectations” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
13	In question k., what does the phrase “low family involvement” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required

School Questionnaire – Grade 4

[Grade 4 NIES–School #1]

Revised/VH040432

Which of the following describes your school best? Please fill in one oval only.

Regular public school
Charter public school
Bureau of Indian Education contracted or grant school
Bureau of Indian Education operated school
Other nonpublic school

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	Do you think that you need to select more than one answer choice? If yes, please explain why.	Probe whether the answer choices are mutually exclusive since the question has been revised to be multiple-choice single-selection.	Specific	Required
4	In answer choice E, what does the phrase “other nonpublic school” mean to you? Can you give me examples of “other nonpublic schools?”	Probe how the participant interprets the phrase.	Specific	Required

[Grade 4 NIES–School #2]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “boarding school” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice A, then ask the follow-up questions]: Are there any students at your school that do not board? If yes, what proportion of your students do not board? To what extent does having a high or low proportion of students who do not board influence the answer choice you selected?	Probe how the participant calculated the answer.	Specific	Conditional

[Grade 4 NIES–School #3]

New/VH040437

Is your school located on a reservation/on tribal land?

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “located on a reservation/on tribal land” mean to you?	Probe how the participant interprets the phrase.	Specific	Required

[Grade 4 NIES–School #4]

New/VH040477

Is your school an American Indian or Alaska Native language immersion school?

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “an American Indian or Alaska Native language immersion school” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice A, then ask the follow-up question]: What proportion of students in your school participates in your schools’ language immersion program?	Collect further information about the proportion of students participating in the language immersion program.	Specific	Conditional

[Grade 4 NIES–School #5]

Revised/VH040574 Are families of your American Indian or Alaska Native students involved with your school in the following ways? Fill in one oval on each line.
	Yes	No
Academic club (for example, math club)	A	B	Revised/VH040576
Arts club	A	B	Revised/VH040577
Cultural dances or activities (for example, drum groups)	A	B	Revised/VH040578
Drama club	A	B	Revised/VH040580
Making school curriculum decisions	A	B	Revised/VH040581
Open houses or back-to-school nights	A	B	Revised/VH040584
Parent-teacher conferences	A	B	Revised/VH040588
Parent-teacher organizations	A	B	Revised/VH040589
School sports	A	B	Revised/VH040592
Volunteer programs	A	B	Revised/VH040596
Other (please specify): __________________________________	A	B	Revised/VH040597

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. through k.?	Record the answer choice selected.	General	Required
3	Are your answer choices for questions a. through k. specific to the families of your grade 4 American Indian or Alaska Native students, or do they apply to the families of all American Indian or Alaska Native students in your school?	Probe how the participant interprets the phrase.	Specific	Required
4	In question a., what does the phrase “academic club” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
5	In question b., what does the phrase “arts club” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
6	In question c., what does the phrase “cultural dances or activities” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
7	[Conditional probe: If the participant selects answer choice A for question c., then ask the follow-up question]: Can you give me examples of cultural dances or activities? Do you consider “drum groups” a good example of “cultural dances or activities”?	Probe how the participant interprets the phrase.	Specific	Conditional
8	In question d., what does the phrase “drama club” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
9	In question e., what does the phrase “making school curriculum decisions” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
10	In question f., what does the phrase “open houses or back-to-school nights” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
11	In question g., what does the phrase “Parent-teacher conferences” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
12	In question h., what does the phrase “Parent-teacher organizations” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
13	In question i., what does the phrase “school sports” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
14	In question j., what does the phrase “Volunteer programs” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
15	What proportion of your students are American Indian or Alaska Native? To what extent does having a high or low proportion of American Indian or Alaska Native students in your school influence the answer choice you selected?	Probe how the participant calculated the answer.	Specific	Conditional
16	[Conditional probe: If the participant selects answer choice A for question k, then ask the follow-up question]: What are the other ways in which families of your American Indian or Alaska Native students are involved with your school?	Probe participant to provide additional examples.	Specific	Conditional

[Grade 4 NIES–School #6]

Trend/VC962952 How many of the following courses and programs about American Indian or Alaska Native traditions and culture are offered at your school each year? Fill in one oval on each line.
	None	1	2	3	4 or more
Year-long course	A	B	C	D	E	Trend/VC962954
Semester-long course	A	B	C	D	E	Trend/VC962956
Workshop or unit	A	B	C	D	E	Trend/VC962957
Clubs	A	B	C	D	E	Trend/VC962958
Other programs (for example, study groups before or after regular class periods)	A	B	C	D	E	Revised/VH018332

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for question e.?	Record the answer choice selected.	General	Required
3	What does the phrase “study groups before or after regular class periods” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice B, C, D, or E to question e., then ask the follow-up question]: Can you give me examples of other programs offered at your school?	Probe how the participant interprets the phrase.	Specific	Conditional
5	Do you think that answer choices A through E are clear? If not, can you give me examples of other answer choices that would be more clear?	Probe how the participant interprets the answer choice.	Specific	Required

[Grade 4 NIES–School #7]

Trend/VC963029 Considering all of the students in your school, to what extent is each of the following a problem in your school? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
Student absenteeism	A	B	C	D	Trend/VC963030
Student tardiness	A	B	C	D	Trend/VC963031
Student health problems	A	B	C	D	Trend/VC963032
Student misbehavior in class	A	B	C	D	Trend/VC963034
Physical conflicts among students	A	B	C	D	Trend/VC963035
Bullying	A	B	C	D	Trend/VC963037
Low student aspirations	A	B	C	D	Trend/VC963038
Low teacher expectations	A	B	C	D	Trend/VC963039
Low family involvement	A	B	C	D	Trend/VC963040
Inadequate transportation for students	A	B	C	D	New/VH040475

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for question j.?	Record the answer choice selected.	General	Required
3	In question a., what does the phrase “student absenteeism” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In question b., what does the phrase “student tardiness” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	In question c., what does the phrase “student health problems” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	In question d., what does the phrase “student misbehavior in class” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
7	In question e., what does the phrase “physical conflicts among students” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
8	In question f., what does the phrase “bullying” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
9	In question g., what does the phrase “low student aspirations” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
10	In question h., what does the phrase “low teacher expectations” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
11	In question i., what does the phrase “low family involvement” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
12	In question j., what does the phrase “inadequate transportation” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required

[Grade 4 NIES–School #8]

Trend/VC963041 Which of the following types of student and family services are offered at your school on a regular basis? Fill in one oval on each line.
	Offered	Not offered
Health services for students such as a school nurse	A	B	Trend/VC963042
Health services for families such as a community clinic	A	B	Trend/VC963043
Social or counseling services for students	A	B	Trend/VC963045
Social or counseling services for families	A	B	Trend/VC963047
Adult education programs for all ages	A	B	Trend/VC963048
Tutoring services for students	A	B	New/VH017944
Career counseling services for students	A	B	New/VH017945

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions f. and g.?	Record the answer choice selected.	General	Required
3	What does the phrase “a regular basis” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In question f., what does the phrase “tutoring service for students” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	[Conditional probe: If the participant selects answer choice A to question f., then ask the follow-up question]: Can you give me examples of the tutoring services offered at your school?	Probe participant to provide additional examples.	Specific	Conditional
6	In question g., what does the phrase “career counseling services for students” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
7	[Conditional probe: If the participant selects answer choice A to question g., then ask the follow-up question]: Can you give me examples of the career counseling services offered at your school?	Probe participant to provide additional examples.	Specific	Conditional

[Grade 4 NIES–School #9]

New/VH040496

To what extent do your school’s 4^th grade curricula emphasize preparing American Indian or Alaska Native students for life in their community, tribe, group, pueblo, or village?

Not at all
Small extent
Moderate extent
Large extent

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “preparing American Indian or Alaska Native students for life in their community, tribe, group, pueblo, or village” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up question]: Can you give me examples of how your school’s 4^th grade curricula emphasize preparing American Indian or Alaska Native students for life in their community, tribe, group, pueblo or village?	Probe participant to provide additional examples.	Specific	Conditional
5	What does the word “pueblo” mean to you?	Probe the participant’s comprehension of the word.	Specific	Required
6	Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?	Probe whether the participant is able to answer the question.	Specific	Required

School Questionnaire – Grade 8

[Grade 8 NIES–School #1]

Revised/VH040432

Which of the following describes your school best? Please fill in one oval only.

Regular public school
Charter public school
Bureau of Indian Education contracted or grant school
Bureau of Indian Education operated school
Other nonpublic school

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	Do you think that you need to select more than one answer choice? If yes, please explain why.	Probe whether the answer choices are mutually exclusive since the question has been revised to be multiple-choice single-selection.	Specific	Required
4	In question e., what does the phrase “other nonpublic school” mean to you? Can you give me examples of “other nonpublic schools?”	Probe how the participant interprets the phrase.	Specific	Required

[Grade 8 NIES–School #2]

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “boarding school” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice A, then ask the follow-up questions]: Are there any students at your school that do not board? If yes, what proportion of your students do not board? To what extent does having a high or low proportion of students who do not board influence the answer choice you selected?	Probe how the participant calculated the answer.	Specific	Conditional

[Grade 8 NIES–School #3]

New/VH040437

Is your school located on a reservation/on tribal land?

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “located on a reservation/on tribal land” mean to you?	Probe how the participant interprets the phrase.	Specific	Required

[Grade 8 NIES–School #4]

New/VH040477

Is your school an American Indian or Alaska Native language immersion school?

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	What does the phrase “an American Indian or Alaska Native language immersion school” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice A, then ask the follow-up question]: What proportion of students in your school participates in your schools’ language immersion program?	Collect further information about the proportion of students participating in the language immersion program.	Specific	Conditional

[Grade 8 NIES–School #5]

Revised/VH040574 Are families of your American Indian or Alaska Native students involved with your school in the following ways? Fill in one oval on each line.
	Yes	No
Academic club (for example, math club)	A	B	Revised/VH040576
Arts club	A	B	Revised/VH040577
Cultural dances or activities (for example, drum groups)	A	B	Revised/VH040578
Drama club	A	B	Revised/VH040580
Making school curriculum decisions	A	B	Revised/VH040581
Open houses or back-to-school nights	A	B	Revised/VH040584
Parent-teacher conferences	A	B	Revised/VH040588
Parent-teacher organizations	A	B	Revised/VH040589
School sports	A	B	Revised/VH040592
Volunteer programs	A	B	Revised/VH040596
Other (please specify): __________________________________	A	B	Revised/VH040597

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. through k.?	Record the answer choice selected.	General	Required
3	Are your answer choices for questions a. through k. specific to the families of your grade 8 American Indian or Alaska Native students, or do they apply to the families of all American Indian or Alaska Native students in your school?	Probe how the participant interprets the phrase.	Specific	Required
4	In question a., what does the phrase “academic club” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
5	In question b., what does the phrase “arts club” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
6	In question c., what does the phrase “cultural dances or activities” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
7	[Conditional probe: If the participant selects answer choice A for question c., then ask the follow-up question]: Can you give me examples of cultural dances or activities? Do you consider “drum groups” a good example of “cultural dances or activities”?	Probe how the participant interprets the phrase.	Specific	Conditional
8	In question d., what does the phrase “drama club” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
9	In question e., what does the phrase “making school curriculum decisions” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
10	In question f., what does the phrase “open houses or back-to-school nights” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
11	In question g., what does the phrase “Parent-teacher conferences” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
12	In question h., what does the phrase “Parent-teacher organizations” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
13	In question i., what does the phrase “school sports” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
14	In question j., what does the phrase “Volunteer programs” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
15	What proportion of your students are American Indian or Alaska Native? To what extent does having a high or low proportion of American Indian or Alaska Native students in your school influence the answer choice you selected?	Probe how the participant calculated the answer.	Specific	Conditional
16	[Conditional probe: If the participant selects answer choice A for question k, then ask the follow-up question]: What are the other ways in which families of your American Indian or Alaska Native students are involved with your school?	Probe participant to provide additional examples.	Specific	Conditional

[Grade 8 NIES–School #6]

Trend/VC962952 How many of the following courses and programs about American Indian or Alaska Native traditions and culture are offered at your school each year? Fill in one oval on each line.
	None	1	2	3	4 or more
Year-long course	A	B	C	D	E	Trend/VC962954
Semester-long course	A	B	C	D	E	Trend/VC962956
Workshop or unit	A	B	C	D	E	Trend/VC962957
Clubs	A	B	C	D	E	Trend/VC962958
Other programs (for example, study groups before or after regular class periods)	A	B	C	D	E	Revised/VH018332

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for question e.?	Record the answer choice selected.	General	Required
3	What does the phrase “study groups before or after regular class periods” mean to you?	Probe how the participant interprets the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice B, C, D, or E for question e., then ask the follow-up question]: Can you give me examples of other programs offered at your school?	Probe participant to provide additional examples.	Specific	Conditional
5	Do you think that answer choices A through E are clear? If not, can you give me examples of other answer choices that would be more clear?	Probe how the participant interprets the answer choice.	Specific	Required

[Grade 8 NIES–School #7]

Trend/VC963029 Considering all of the students in your school, to what extent is each of the following a problem in your school? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
Student absenteeism	A	B	C	D	Trend/VC963030
Student tardiness	A	B	C	D	Trend/VC963031
Student health problems	A	B	C	D	Trend/VC963032
Teen pregnancies	A	B	C	D	New/VH040470
Drug or alcohol use by students	A	B	C	D	Trend/VC963033
Student misbehavior in class	A	B	C	D	Trend/VC963034
Physical conflicts among students	A	B	C	D	Trend/VC963035
Bullying	A	B	C	D	Trend/VC963037
Low student aspirations	A	B	C	D	Trend/VC963038
Low teacher expectations	A	B	C	D	Trend/VC963039
Low family involvement	A	B	C	D	Trend/VC963040
Inadequate transportation for students	A	B	C	D	New/VH040475

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions d. and l.?	Record the answer choice selected.	General	Required
3	In question a., what does the phrase “student absenteeism” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	In question b., what does the phrase “student tardiness” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
5	In question c., what does the phrase “student health problems” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	In question e., what does the phrase “drug or alcohol use by students” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
7	In question f., what does the phrase “student misbehavior in class” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
8	In question g., what does the phrase “physical conflicts among students” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
9	In question h., what does the phrase “bullying” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
10	In question i., what does the phrase “low student aspirations” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
11	In question j., what does the phrase “low teacher expectations” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
12	In question k., what does the phrase “low family involvement” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
13	In question l., what does the phrase “inadequate transportation” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
14	[Conditional probe: If the participant selects answer choice B, C, or D to question d., then ask the follow-up question]: Can you give me examples of how teen pregnancies are a problem in your school?	Probe participant to provide additional examples.	Specific	Conditional

[Grade 8 NIES–School #8]

Trend/VC963041 Which of the following types of student and family services are offered at your school on a regular basis? Fill in one oval on each line.
	Offered	Not offered
Health services for students such as a school nurse	A	B	Trend/VC963042
Health services for families such as a community clinic	A	B	Trend/VC963043
Social or counseling services for students	A	B	Trend/VC963045
Social or counseling services for families	A	B	Trend/VC963047
Adult education programs for all ages	A	B	Trend/VC963048
Tutoring services for students	A	B	New/VH017944
Career counseling services for students	A	B	New/VH017945

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions f. and g.?	Record the answer choice selected.	General	Required
3	What does the phrase “a regular basis” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
3	In question f., what does the phrase “tutoring service for students” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice A for question f., then ask the follow-up question]: Can you give me examples of the tutoring services offered at your school?	Probe participant to provide additional examples.	Specific	Conditional
5	In question g., what does the phrase “career counseling services for students” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
6	[Conditional probe: If the participant selects answer choice A for question g., then ask the follow-up question]: Can you give me examples of the career counseling services offered at your school?	Probe participant to provide additional examples.	Specific	Conditional

[Grade 8 NIES–School #9]

Revised/VH040486

What percentage of your 8th grade students dropped out of school during the last school year?

0 – 2 %
3 – 5 %
6 – 10 %
11 – 20 %
21 – 30 %
31 – 40 %
41 – 50 %
More than 50 %

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select?	Record the answer choice selected.	General	Required
3	Please explain why you selected the answer choice. In other words, how did you calculate the answer?	Probe how the participant calculated the answer.	Specific	Required
4	Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?	Probe whether the participant is able to answer the question.	Specific	Required

[Grade 8 NIES–School #10]

New/VH040507 To what extent do your school’s 8th grade curricula emphasize preparing American Indian or Alaska Native students for the following? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
Prepare students for life in their community, tribe, group, pueblo, or village	A	B	C	D	New/VH040511
Prepare students for the workforce	A	B	C	D	New/VH040518
Prepare students for high school and college	A	B	C	D	New/VH040522

Item-Specific Probes:

No.	Probe	Purpose	Type of Probe	Required/ Conditional
1	Can you tell me, in your own words, what the question is asking?	Probe comprehension of the item.	General	Required
2	Which answer choice would you select for questions a. through c.?	Record the answer choice selected.	General	Required
3	In question a., what does the phrase “prepare students for life in their community, tribe, group, pueblo, or village” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
4	[Conditional probe: If the participant selects answer choice B, C, or D for question a., then ask the follow-up question]: Can you give me examples of how your school’s 8^th grade curricula emphasize preparing American Indian or Alaska Native students for life in their community, tribe, group, pueblo or village?	Probe participant to provide additional examples.	Specific	Conditional
5	What does the word “pueblo” mean to you?	Probe the participant’s comprehension of the word.	Specific	Required
6	In question b., what does the phrase “prepare students for the workforce” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
7	[Conditional probe: If the participant selects answer choice B, C, or D for question b., then ask the follow-up question]: Can you give me examples of how your school’s 8^th grade curricula emphasize preparing American Indian or Alaska Native students for workforce?	Probe participant to provide additional examples.	Specific	Conditional
8	In question c., what does the phrase “prepare students for high school and college” mean to you?	Probe the participant’s comprehension of the phrase.	Specific	Required
9	[Conditional probe: If the participant selects answer choice B, C, or D for question c., then ask the follow-up question]: Can you give me examples of how your school’s 8^th grade curricula emphasizes preparing American Indian or Alaska Native students for high school and college?	Probe participant to provide additional examples.	Specific	Conditional
10	Which answer choice would you select for questions a. through c. if the curricula is focused on high school only?	Record the answer choice selected.	Specific	Required
11	Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?	Probe whether the participant is able to answer the question.	Specific	Required

Debriefing and Thank You

Before we finish, I’d like to hear [any/other] thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the survey questions?

Did you have any problems understanding [other/any of] the questions?

Did you have any problems understanding what you were supposed to do to answer [other/any of the] questions?

Is there anything that you think could make [this/these] survey question(s) clearer?

Thank participant for his/her time and provide gift card, as appropriate.

1 60 minutes for play testing, tryouts, grade 4 cognitive item cognitive interviews, and NIES survey question cognitive interviews

90 minutes for grade 8 cognitive item cognitive interviews

2 All student interviews will be conducted in-person. Teacher and school administrator interviews will primarily be conducted in-person; however some interviews may be conducted over the phone. The same scripts will be utilized for teacher and school administrator in-person and telephone interviews.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Science Volume II Cog Lab Protocols
Author	BlairJ
File Modified	0000-00-00
File Created	2021-01-27