National Center for Education Statistics
National Assessment of Educational Progress
Volume II
Items and Protocols
National Assessment of Educational Progress (NAEP) Pretesting of Scenario-based Tasks (SBTs) for Pilot in 2019 Grades 4 and 8
OMB# 1850-0803 v.193
April 2017
Part A. Paperwork Burden Statement 1
Part B. Mathematics and Reading Task Pretesting 2
I. Introduction to Study Script 4
II a. Think-Aloud: Instructions and Modeling Script 5
II b. Think-Aloud: Student Practice Script I 6
II c. Think-Aloud: Student Practice II (Optional) 7
II d. Think-Aloud: Starting the Items 8
II e. Think-Aloud: Hints for the Interviewer 8
II f. Notes on Student Actions and Behaviors during Cognitive Lab 9
III. Post-Think-Aloud Follow-Up Questions (Verbal Probing) 10
IV. Debriefing and Thank You for Cognitive Interview 10
Small-Scale Tryout Protocol 10
Post-Tryout Follow-Up Questions 12
Post-Tryout Discussion Session 13
The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.
Paperwork Burden Statement, OMB Information
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to be average xx1 minutes, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission of form, please write to: National Assessment of Educational Progress (NAEP), National Center for Education Statistics (NCES), PCP, 550 12th St., SW, 4th floor, Washington, DC 20202.
This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.
The information collected will be used for statistical purposes only. In accordance with the Confidential Information Protection provisions of Title V, Subtitle A, Public Law 107-347 and other applicable Federal laws, responses will be kept confidential and will not be disclosed in identifiable form to anyone other than employees or agents. By law, every NCES employee as well as every agent, such as contractors and NAEP coordinators, has taken an oath and is subject to a jail term of up to 5 years, a fine of $250,000, or both if he or she willfully discloses ANY identifiable information about participants. Electronic transmission of your information will be monitored for viruses, malware, and other threats by Federal employees and contractors in accordance with the Cybersecurity Act of 2015.
OMB No. 1850-0803 Approval Expires 07/31/2019
Introduction to Study and Setup of Item Review Script
Text written in italics is to be read aloud by the staff facilitator.
NOTE: The facilitator should not read the script word-for-word, but should be familiar enough with its contents to conduct the playtesting sessions in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular items being studied.
We are developing a new set of tests in <mathematics OR reading> for a program funded by the federal government called the National Assessment of Educational Progress (NAEP). NAEP is a test taken by students in grades 4, 8, and 12 in the United States. You will help us develop better tests for other students. We are going to ask for your opinions about some interactive questions that we are creating for a NAEP <mathematics OR reading> test. Right now we have some early drafts of what these questions might look like. You are here to give us your thoughts about these questions. We’d like to hear about what you think is interesting, what you think is not interesting, and if you think any part of a question is confusing.
[For students working in pairs or groups]:
My colleagues and I will take notes on what you are saying about the questions. Please be as honest as possible, talk freely with each other, and don’t worry if you find something confusing. You are not being graded on anything; you are helping us by giving us your opinions and thoughts about these questions. There are no right or wrong thoughts.
[For students working on their own]:
My colleagues and I will take notes on what you are doing as you work and saying about the questions. Please be as honest as possible, talk freely with me, and don’t worry if you find something confusing. You are not being graded on anything; you are helping us by giving us your opinions and thoughts about these questions. There are no right or wrong reactions.
What you say will be used only to help us improve tasks and questions for a future study. After this session, we will use the session’s recording to review any thoughts you share with us to help us understand how our test is working and how we can make it better. Overall, this session should take about [indicate correct length] minutes.
As you work through these questions, we want you to talk with each other or [for students working on their own] with me as you go along. We might also ask you a few questions now and then.
Do you have any questions?
After answering any questions and giving further explanation, continue with the activity. If a student is no longer interested in participating, thank the student for his/her time and end the activity for that student. Provide a gift card (both for student and for the parent if the parent brought the child to and from the session).
The facilitators will generally observe the students with little interruption and will listen to the conversation that occurs between group members or observations students make to the facilitator for insights into what the students are thinking about the questions and any associated stimulus. Intermittently, as appropriate, the facilitators can ask targeted questions such as those in this list:
|
Facilitators will observe students, take notes, and pay attention to verbal answers to any questions like those suggested above (and write them down). They will look for evidence of engagement, boredom, or confusion in what students say to each other (or to the facilitator) and in their facial expressions or body language. If appropriate, staff can use these moments to probe for more information.
Occasionally staff observing playtesting sessions may introduce a question to students talking about the items to get more information from students, particularly in cases when students talk about issues related to their interest (or non-interest) in the item or task, confusing language in the item/task, or how they might respond to a question on the task. In such cases observers might ask something like, “Can you tell me more about what you said?” Prior to each playtesting session, some key focus areas for each item/task may be identified. If students do not provide sufficient comments on targeted parts, a staff member may ask students if they had any thoughts about the particular sections, using questions such as those described above but focused on specific places or issues in the item or task.
At the end, time allowing, facilitators may ask students additional questions about the items and students’ reactions. Once that is complete, students will be thanked for their participation and for helping us to improve the tests. Gift cards will be provided at the conclusion of each playtesting session (both for students and parents if the parent brought their child to and from the session.)
NOTE: The interviewer should not read the script word-for-word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular items being studied. In general, protocols will be tweaked and customized in light of specific item and stimulus content.
Introduction:
Hello, my name is ______ and I work for [indicate organization]. I am here with my colleagues [introduce colleagues]. It’s nice to meet you, and thank you very much for helping us out today.
Create small talk to build rapport with the student by asking a question, such as:
When the student responds, follow up with 2 or 3 questions to get the student used to talking, such as:
Good. Well, I think you’ll enjoy what we are going to be doing today. First, let me begin by explaining why I am here and what you are going to be doing. You are taking part in a special study looking at new test questions to find out what students know in <mathematics OR reading>. It is part of the National Assessment of Educational Progress, or NAEP for short, a test that is funded by the U.S. Federal Government and run by the National Center for Education Statistics in the Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will help us develop better tests for other students. Overall, this session should take about <indicate correct length> minutes.
It’s okay if you don’t know how to do any of the questions. I will not be grading your work today, and no one will know that it was you who answered the questions. My goal is to learn how you react to the questions, so please try to do your best.
If at any time you decide you don’t want to go on, that is your choice, and you may stop.
This new test is on the tablet. All instructions about how to do the test are also on the tablet.
This interview is being recorded so we can review and learn from it later. We will also learn from what you tap on the screen and where you move your fingers.
We will work together on <specify number of items/tasks>.
Do you have any questions?
After answering any questions and giving further explanation, continue with the think-aloud training. If the student is no longer interested in participating, thank the student for his/her time and end the interview.
Okay, let’s move on. Before we look at the real questions, I want to make sure you have a good idea of what we’re going to be doing. So, I am going to give you some practice questions. The practice questions should help you get used to what we will be doing during the real questions. They should help you understand how we want you to respond. Do you have any other questions before we start? |
NOTE: Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word-for-word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular items being studied.
If students are doing reading questions for which they must read a passage, students should only be prompted to think aloud based on this training for the questions, NOT while they are reading the passage. The modeling script below is for questions (items), NOT for reading extended passages.
To help us make our test better, we will ask you to complete some questions.
[If using Morae or other screen capture software]: While you are doing the questions, we will be recording everything that happens on the screen. To help us know how well everything is working, the screen recording will capture all of your responses and movements on the tablet such as what you tap on and anything you put into text boxes.
I’m going to ask you to do the questions in a way that may be different from what you are used to. Instead of working quietly, I want you to tell me what you are thinking as you work through the questions. We call this “a think-aloud” because we are asking you to say everything you are thinking out loud.
In a moment, I will give you an example of the think-aloud process. Then I will give you a chance to practice it. You won’t be graded on anything you say while you are thinking aloud. There are no incorrect thoughts, and everything you think and say is important to us.
Okay, now I’m going to show you how to think out loud—this will help you see how I want you to describe what you are thinking as you are working on the questions. When I am finished I’ll ask you to try it, so you can see how it works.
Think-Aloud Demonstration:
Since we can’t tell how you are thinking about a question and answering it, we need you to think aloud.
Let me give you an example. (Place example question in front of student.) Look at this question. It asks me to look at the five animals and choose two that are the most similar. So, I am going to do this task, and I will think aloud while I work on it.
Question:
Which two animals below do you think are similar? Circle the two that you think are most similar.
Okay, so I am reading the question…It says I have to decide on the two animals that are the most similar. Hmm, I am just wondering, what do they mean by “similar?” That could mean anything. I’m thinking I’m not sure how I’m going to make that decision. Well, let’s look at the choices. So, the choices are beetle, mouse, crab, dolphin, and cow. Well, as I am reading the choices I am already thinking that there are some things about them that are similar, but there are also differences. I’m thinking the beetle and crab are kind of similar, in a way—well, they both have lots of legs and they kind of move around fast. But one lives in the water and the other doesn’t, so that makes them different. I guess the mouse also moves around fast, but it’s not similar to either the beetle or the crab, except they are all small. Then there is the dolphin that lives in the water, too…but that’s about all it has in common with the crab, so I wouldn’t say they are very similar even though they’re both in the sea. Hmm, so the mouse and the cow are both mammals, they both have fur and they have live babies instead of laying eggs or whatever…oh right, but dolphins are mammals too, aren’t they? Yes, I think they are…but really they seem so different from mice and cows to me. I don’t know…this seems like a hard question…I’m not sure what to choose. Okay, I’m going to try to make a decision…umm…okay, I think I am going to go for…mouse and cow. Because they both have fur and have four legs and they both walk around on the land. That seems weird; they are not really similar! But when I think about all of the others they seem too different from each other in really basic ways, like where they live and their kind of body and that kind of thing…so, yes, even though it was really hard to decide, that seems like the best choice to me, because they are both mammals and they both live on the land. So those things seem like they are important, and that is how I made my decision.
Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do as you are thinking about the questions today. The point of the think-aloud is to get at whatever is in your head as you are doing the questions. Just say aloud the words and the thoughts that are in your head, as you are thinking about and working on the questions. |
Interviewers should place the practice question in front of the student so he or she can read it. Some students will be silent after reading the question. Students should be immediately encouraged to say whatever they are thinking.
It may be necessary to remind the student to talk aloud as he/she works through the questions. If necessary, interviewers should use the “think-aloud hints” given below to prompt the student, being careful not to lead the student. The interviewer needs to be familiar enough with the information to introduce the think-aloud in a natural, conversational manner.
Now you will try a think-aloud. You can use this example. Like last time, you have to pick two of these things that you think are the most similar. As you are reading the question and the choices, and as you are figuring out your decision, I want you to talk out loud to tell me all of the thoughts that are in your head at each moment.
If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I’ll be reminding you to think aloud if you get quiet because I need to hear all of your thoughts.
Okay, now you try. Go ahead and start working on this question and remember to think aloud as you are doing it.
Which two living things below do you think are most similar? Circle the two that you think are most similar.
After the student has finished: Now that you have practiced, how do you feel about thinking aloud while you are doing the questions? What questions would you like to ask me? [If the student says he or she feels okay and doesn’t have any questions, say: Good, then let’s begin our study.] [If the student expresses concerns, says he or she has questions, or appears to be hesitant or reluctant, ask him/her to say more about the concerns or questions, and try to address the concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him or her to stop.] |
Use only if you feel the student would benefit from another think-aloud practice before moving on to the actual questions. It may be beneficial to instead model thinking aloud again for the student rather than doing another practice [see Think-Aloud Demonstration in II a].
If the student struggles to think aloud, the interviewer should give the student another opportunity to practice. The interviewer should praise the student for the first attempt regardless of how good it was, for example:
Very good—let’s do another one before we start the real tasks. Are you ready? Here is the next practice question. Remember to think out loud as you begin to think about this question and all the way through—tell me what you are thinking as you work through it.
During the practice item, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, below).
Now we will move on to the actual questions.
If student is doing reading questions with an associated passage: You will be reading a passage for doing some reading questions. You do not need to think aloud as you read the passage. However, when you are answering the questions, just like in our practice, I’d like you to say aloud everything that you’re thinking. I may remind you to do that if you are quiet.
If student is NOT reading a passage: Remember, as you do all the activities and answer all the questions, I’d like you to say aloud everything that you’re thinking, and I may remind you to do that if you are quiet.
The questions should take about <indicate correct length> minutes. Remember, you will not be graded on your responses to the questions and there is no right or wrong way to think aloud, as long as you keep telling me your thoughts. Your thoughts will help us make the questions better. I will also have a few questions after you have finished working.
Do you have any questions before we go on? <Answer any questions the student may ask.> Because the information you provide is so important to us, I am going to be taking notes while you think aloud and answer the questions. Here are the questions. When you are ready, go ahead and start working. |
The goal of think-alouds is to capture all the student’s mental processes while working through items. Interviewers must strive to have the student speak aloud all of his or her thoughts while doing the items. If a student is continually providing short utterances or not talking, interviewers should use “continuers” to encourage students to be more descriptive. The goal is to get students to verbalize thoughts without putting words in students’ mouths. Care should be taken not to ask questions that lead students’ responses in particular directions or make them rush or change their approach. Interviewers should be as objective and unbiased as possible.
In general, if the student is silent for approximately 5 to 10 seconds, interviewers should use the following as a guide for encouraging the student to describe his or her thoughts, or to help the student elaborate on his or her responses.
If the student is not verbalizing enough, interviewers should offer a verbal nudge to remind the students to keep talking, such as:
If the student says something about his or her thoughts but it is difficult to tell exactly what he or she means, or if the student begins to talk but does not say very much, interviewers can urge the student to elaborate. For example:
If more clarity or a more explicit description of a student’s thoughts is desirable:
|
It is important to be responsive and sensitive to each student’s behavior. If it becomes clear that students cannot tell any more about their thoughts, interviewers should not keep asking. Students should be encouraged to think aloud as much as they can, but their reactions should be closely observed and pushing them should be avoided if students seem frustrated or uncomfortable. While it is desirable that students articulate as much as they can, sometimes students will simply be unable to state what is in their mind beyond a basic reply such as “I am reading the question.” Interviewers must be sensitive to nonverbal signals that may indicate students cannot say any more than they have or that they need a few extra seconds to gather their thoughts to put them into words.
In addition, during observations if it seems a student is hung up on something, interviewers should note when this occurred.
As a student is working through items, the interviewer should take notes regarding any of the student’s expressions or behaviors that may reflect the status of the student’s understanding, engagement, or reactions to items and/or stimulus (e.g., a reading passage). The following are examples of such behaviors for interviewers to note.
Does the student express signs of confusion, boredom, or excitement?
Does the student use tools provided by tapping on tabs or buttons?
Does the student rapidly move through the screen or take his or her time?
Does the student spend a lot of time on a particular item or tool or reading passage?
Does the student seem discouraged?
These kinds of observations should be considered opportunities to prompt students to think aloud and to encourage them to express what is in their minds. Interviewers should also make a note of any reactions to or interactions with items, stimuli, or a reading passage that appear valuable for follow-up with some additional questions after work is completed (see section III).
After completing the think-aloud process for items, interviewers will follow up with a brief period of focused retrospective questioning. The purpose of the additional post-task questions is to capture more information on issues such as student actions during the course of responding to items, particular aspects of [mathematics or reading] targeted in the items/task, and general reflections about the items, stimulus, tools, and or/reading passage. The post-item questions will include targeted questions that are item-specific. These will be selected prior to testing. Interviewers will receive the set of up to five questions for the items they are examining in each cognitive interview.
Interviewers will also be trained to generate post-item questions on the fly, for cases where they observe something that they judge needs follow-up probing. The following are some examples of questions that might be asked at this stage.
Did any words in the item(s)/task/stimulus/passage confuse you?
What ways could we change the item(s)/task/stimulus to make it better?
Did the [images, video, audio, stimulus, tools, etc.] for the item(s)/task help you answer the questions?
What did you find easiest in the task? What did you find hardest?
What part of the task took you the most time?
Thank the student for his or her time. Provide a gift card (both for student and for the parent if the parent brought the child to and from the session). |
Text written in italics are to be read aloud by the interviewer.
NOTE: The interviewer should not read the script word for word but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text in brackets is generic text that will be tailored based on task specifics.
Session Information
The interviewer should complete the information from rows one to three prior to the beginning of the tryout session. Rows four to five are to be completed at the end of the tryout session.
Ref# |
Preliminary Information |
Data |
1 |
Name of interviewer |
|
2 |
Date |
|
3 |
Start time of interview |
|
4 |
End time of interview |
|
5 |
Special circumstances that may have affected the tryouts session (Do not ask test taker, just record observations.) |
|
Introduction Script:
Hello, my name is _______ and I work for EurekaFacts. I am here with my colleagues [introduce colleagues]. It’s nice to meet you, and thank you very much for helping us out today.
Let me begin by explaining why you’re here and what you’re going to be doing today. You are participating in a special study to try out new <mathematics OR reading> test questions. This is part of something called the National Assessment of Educational Progress, or NAEP for short, and is sponsored by the U.S. Department of Education. Today we’ll be asking you to try out some questions that are being developed for fourth, eighth, and twelfth graders all over the United States.
If at any time you decide you do not want to go on, that is your choice and you may stop.
What you say will be used only to help us improve these test questions. What we learn from you and other participating students will be summarized in a report where no student names will be listed. After the session, we will use the session’s recording to review what we learned to help us understand how our test is working and how we can make it better. Overall, this session should take about [indicate correct length] minutes.
Do you have any questions?
Interviewer: If a student is no longer interested in participating, thank the student for his or her time and end the tryout session for that one student only; continue with all others.
Instructions
Ok, now, I’m going to give you a set of questions to complete.
This is a timed activity, which will last for <indicate length of activity>, and everyone will begin at the same time.
All of the instructions will be displayed on your tablet screens.
The tasks and questions you are about to do is something that students, like you, will do on the tablet. However, we aren’t finished making them yet, so there may be some parts that don’t work. Try to work through and complete the questions as best as you can.
You will not be graded on your answers to the questions, but please answer the questions as if it was a real test.
We will all work at the same time, first I would like you to read the instructions, once everyone has gone through the instructions, we will begin the questions together. I will start the timer and I will stop you once time is up.
Okay, I will come around and open the questions for each one of you, but please do not begin until I say that you may start. [OR: Ok, when I say begin, please press “Start” to begin working.]
If you have any questions about technical problems, please raise your hand and one of us will come to help you.
Okay, now that everyone is ready, you may begin.
START THE TIMER NOW. STUDENTS HAVE <60 or 90> MINUTES TO COMPLETE THE QUESTIONS. STOP STUDENTS AT <60 or 90> MINUTES EVEN IF THEY HAVE NOT FINISHED.
Wait either for all students to complete the task and/or set of items OR for the full amount of time to be up.
Once students have completed the task and/or set of items, make sure that all students exit the task and submit their answers.
After completing the tryout tasks and/or set of items, the interface may provide some follow-up questions. Post- tryout questions may include some or all of the following:
A standardized post-task/item question that all students will be asked following all tasks/set of items, which is designed to discover whether the student has prior knowledge of the content.
Up to three additional targeted questions that are task/item specific. These will be selected for each task/item prior to testing.
A set of no more than seven standardized <mathematics or reading> multiple-choice questions that students will be asked following the task/item.
Standardized Question for All Tasks: Task-Specific Prior Knowledge
Have you studied anything related to this task in school, or have you learned about or come across these things in your own life? [If yes: What have you learned or studied or experienced that is related to this task?]
Additional Questions: Task-Specific Issues
The purpose of the additional post-task questions is to capture more information on issues such as student actions during the task, particular aspects of [mathematics or reading] targeted in the task, and general reflections about the task. The following are some examples of questions that might be asked at this stage.
Did any words in the task confuse you?
Did you understand the directions?
What did you find easiest in the task? What did you find hardest?
What part of the task took you the most time?
Did the [images, videos, audios, etc.] for the task help you answer the questions asked?
At the conclusion of some tryouts, the facilitator may ask the student up to three task-specific questions. The following are some examples of questions that might be asked at this stage.
What ways could we change the task to make it better?
Did the [images, videos, audios, etc.] for the task help you answer the questions asked?
Before we finish, I’d like to hear what you all thought about what you’ve been doing.
Is there anything else any of you would like to tell me about working on the test question?
Did you have any problems understanding <any of the questions/anything>?
Did you have any problems understanding what you were supposed to do to answer any of the questions?
Is there anything that you think could make <this/these> test question[s] clearer?
Is there anything you would like to ask me about what we did today? <Answer students’ questions.>
Thank participant(s) for their time and escort them to the front desk. Provide gift card(s).
1 60 minutes for grade 4 and 90 minutes for grade 8.
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
File Title | Science Volume II Cog Lab Protocols |
Author | BlairJ |
File Modified | 0000-00-00 |
File Created | 2021-01-22 |