National Assessment of Educational Progress (NAEP) Science Cognitive Interviews and Play Test Studies 2013

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average xx¹ minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 20202-4537. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: NAEP/NCES, U.S. Department of Education, 400 Maryland Avenue, S.W., Washington, DC 20202.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

OMB No. 1850-0803 Approval Expires 09/30/2013

Part B. Background Questionnaire Cognitive Interviews

I. Interviewer Welcome Script and Assent/Consent

The following scripts should not be read verbatim. You, as the interviewer, need to be familiar enough with the scripts below to introduce the think-aloud process in a conversational manner. Text written in italics is suggested content for you to be thoroughly familiar with in advance. You should project a warm and reassuring manner toward the participant to develop a friendly rapport and as such should use conversational language throughout.

Student (under age 18) Participant Script

Hello, my name is <name >.

Thank you for coming here today to help us.

I have some questions that students like you in many different schools will be asked to answer about themselves, their education, and their families. The questions you will be asked will be part of a survey for the National Assessment of Educational Progress (NAEP) research study. Some of the questions may not be easy to understand or answer. There are no wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), part of the U.S. Department of Education, make these questions clearer for students like you. This interview is being recorded so researchers can review the recordings later. No one else will see or hear the recording. Your feedback may be used only for statistical purposes and may not be used for any other purpose except as required by law.

Do you have any questions?

After answering questions and giving further explanation, continue with an explanation and demonstration of the think-aloud process. If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

Student (age 18 or older) Participant Script

Hello, my name is <name >.

Thank you for coming here today to help us.

I have some questions that students like you in many different schools will be asked to answer about themselves, their education, and their families. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. There are no wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), part of the U.S. Department of Education, make these questions clearer for students like you. This interview is being recorded so researchers can review the recordings later. Your feedback will be used only for statistical purposes and may not be used for any other purpose except as required by law (20 U.S.C., § 9573).

Do you have any questions?

Teacher Participant Script

Hello, my name is <name>.

Thank you for coming here today to help us.

I have some questions that teachers like you in many different schools will be asked to answer about themselves, their instructional practices, and their schools. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. There are no wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), part of the U.S. Department of Education, make these questions clearer for teachers like you. This interview is being recorded so researchers can review the recordings later. Your feedback will be used only for statistical purposes and may not be used for any other purpose except as required by law (20 U.S.C., § 9573).

Do you have any questions?

School Administrator Participant Script

Hello, my name is <name >.

Thank you for coming here today to help us.

I have some questions that school administrators like you in many different schools will be asked to answer about their school policies and characteristics. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. There are no wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), part of the U.S. Department of Education, make these questions clearer for school administrators like you. This interview is being recorded so researchers can review the recordings later. Your feedback will be used only for statistical purposes and may not be used for any other purpose except as required by law (20 U.S.C., § 9573).

Do you have any questions?

II. Think-Aloud Instructions and Practice

The following scripts should not be read verbatim. You need to be familiar enough with the information to introduce the think-aloud in a conversational manner.

We want to learn what people think about when answering questions. As you answer each question, I’d like you to think aloud. All that means is to just read the question aloud and then say what you’re thinking as you answer the question. Please tell me whatever is going through your mind as you answer the question.

Let me give you an example about making a piece of toast this morning. If I were thinking aloud while I got the toast ready, it would sound something like this.

“Bread drawer. Muffin or white bread? White bread. Where is the knife? Oh, the sink. Cut. Plug in the toaster oven, turn it on, close it. OK, Medium. Hmm (reading the white bread package). Serving size 1 piece of bread, Calories: 130; Calories from fat 10 – I guess that’s not so bad. Vitamin A 0%, Vitamin C 0%, Calcium 8%, Iron 8% So aren’t there B vitamins in bread? Hmm. (Look up at the corner of the ceiling) Wait! I forgot to put the bread in the toaster! OK, OK, put bread in toaster. Do I want butter or…(Ding) Open the toaster, don’t burn yourself! Use the knife to pull out the bread. Table, sit down, eat.”

The point of the think-aloud is to simply say aloud the words that are in your head.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I need to hear times when you are confused (and what is confusing you) and times when you understand what you are expected to do.

Sometimes it’s helpful to do a short “think-aloud” practice. I’m going to ask you a simple question and ask you to think aloud as you decide on your answer. The question is:

“How many different kinds of fruit did you eat yesterday?”

Interviewer Note: Some participants will be silent after hearing the question. Immediately encourage the participant to say whatever he/she is thinking. You may need to remind the participant that the answer to the question has to be a number by asking:

“Can you count the different kinds of fruit you ate yesterday?”

After finishing your think-aloud for each question, or after you’ve gone through all the questions in the survey, I’d like to hear any other comments — good or bad— you have about the question and any changes that would make the question easier to understand or answer.

So, do you have any questions before we start?

Interviewer Note: After answering questions and giving further explanation, begin the interview with the first item.

III. Science: Items and Probes

Interviewer Note: Proceed in the following manner for each of the survey items in this section:

Direct the participant to the item and ask him/her to read the question (and any preceding or following instructions) out loud
Instruct the participant to answer the question and to think aloud while doing so
Record the participant’s comments
Ask generic and item-specific probes, as appropriate, before going on to the next item

Format Note:

Some items are grade specific and will be asked only of participants for that grade. The grade, subject, and participant associated with the items are identified in brackets (e.g., [Grade 4 Science–Student #1]) preceding the item.
Each item to be tested is presented on its own page.
For items used in multiple grades, instances of “grey” highlighting indicates the item sequence number used in the other grade(s).

Student Questionnaire–Grade 4

[Grade 4 Science–Student #1]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
How would you answer this question?
What does the phrase “activities or projects” mean to you?
What activities or projects have you done to learn about electricity? What have you learned from these activities or projects about electricity?
What does the phrase “circuits and energy” mean to you? Have you learned about “circuits and energy” from these activities or projects?
Would you have answered this question differently if the example had been “batteries and light” instead of “circuits and energy”?

[If the participant answers “yes,” then ask the follow-up question]:
How would you have answered this question?

[For the following two items, each version should have an equal chance to be presented to participants first. In other words, 5 of the 10 participants should see item #2 first and item #3 second, the other 5 of the 10 participants should see item #3 first and item #2 second. Please alternate which item is shown first to participants.]

[Grade 4 Science–Student #2]

Item-Specific Probes:

[Between items #2 and #3, if this item is shown to the participant first, then ask the probe]: Can you tell me, in your own words, what the question is asking?
Which answer choice would you select? Please explain why you would select that choice.
How would you have answered this question if you had done activities with chemicals only at certain times during the school year? For example, if you had done this every day for three weeks, but hadn’t done it since then, how would you have answered this question?
[If the participant answers B, C, D, or E, then ask the follow-up question]:
When were these activities or projects carried out in this school year? Can you give me examples of something you learned about chemicals?
[Between items #2 and #3, if this item is shown to the participant second, then ask the probe. Please point to the two sets of answer choices for this question to the participant]:
Which of the two versions of the answer choices made more sense to you? Why?

[Between items #2 and #3, if this item is shown to the participant first, then say the sentence to the participant]: The next question will give you different ways to answer this question.

[Grade 4 Science–Student #3]

[Between items #2 and #3, if this item is shown to the participant first, then ask the probe]: Can you tell me, in your own words, what the question is asking?
Which answer choice would you select? Please explain why you would select that choice.
How would you have answered this question if you had done activities with chemicals only at certain times during the school year? For example, if you had done this every day for three weeks, but hadn’t done it since then, how would you have answered this question?
What does the word “rarely” mean to you? What does the word “sometimes” mean to you? What does the word “often” mean to you?
[Between items #2 and #3, if this item is shown to the participant second, then ask the probe. Please point to the two sets of answer choices for this question to the participant]:
Which of the two versions of the answer choices made more sense to you? Why?

[Between items #2 and #3, if this item is shown to the participant first, then say the sentence to the participant]: The next question will give you different ways to answer this question.

[Grade 4 Science–Student #4]

Item-Specific Probes:

[Among items #4, #5, and #6, if this item is shown to the participant first, then ask the probe]: Can you tell me, in your own words, what the question is asking?
[Between items #4, #5, and #6, if this item is shown to the participant second or third, then ask the probe]: This is like the last question, but it has different answer choices. Which answer choice would you select? Please explain why you would select that choice.
[Among items #4, #5, and #6, if this item is shown to the participant first, then ask the probe]: What does the phrase “scientific tools” mean to you? Can you give me examples of some scientific tools that you have used?
How would you have answered this question if you had used scientific tools only at certain times during the school year? For example, if you had used <scientific tool, other than a computer, that student has mentioned in previous response> every day for three weeks, but hadn’t used it since then.
[If student did not provide an example of a tool that can be used in the previous question, follow up with this probe]:
How would you have answered this question if you had used scientific tools only at certain times during the school year? For example, if you had used a microscope every day for three weeks, but haven’t used it since then, how would you have answered this question?
[Among items #4, #5, and #6, if this item is shown to the participant last, then ask the probe (Please point to the three sets of answer choices for this question to the participant)]: Which of the three versions of the answer choices made more sense to you? Why?

[Among items #4, #5, and #6, if this item is shown to the participant first, then say the sentence]: The next two questions will give you different ways to answer this question.

[Among items #4, #5, and #6, if this item is shown to the participant second, then say the sentence]: Now, please look at the next question.

[Grade 4 Science–Student #5]

Item-Specific Probes:

[Among items #4, #5, and #6, if this item is shown to the participant first, then ask the probe]: Can you tell me, in your own words, what the question is asking?
[Between items #4, #5, and #6, if this item is shown to the participant second or third, then ask the probe]: This is like the last question, but it has different answer choices. Which answer choice would you select? Please explain why you would select that choice.
[Among items #4, #5, and #6, if this item is shown to the participant first, then ask the probe]: What does the phrase “scientific tools” mean to you? Can you give me examples of some scientific tools that you have used?
How would you have answered this question if you had used scientific tools only at certain times during the school year? For example, if you had used <scientific tool, other than a computer, that student has mentioned in previous response> every day for three weeks, but haven’t used it since then.
[If student did not provide an example of a tool that can be used in the previous question, follow up with this probe]:
How would you have answered this question if you had used scientific tools only at certain times during the school year? For example, if you had used a microscope every day for three weeks, but haven’t used it since then, how would you have answered this question?
What does the word “rarely” mean to you? What does the word “sometimes” mean to you? What does the word “often” mean to you?
[Among items #4, #5, and #6, if this item is shown to the participant last, then ask the probe (Please point to the three sets of answer choices for this question to the participant)]: Which of the three versions of the answer choices made more sense to you? Why?

[Among items #4, #5, and #6, if this item is shown to the participant first, then say the sentence]: The next two questions will give you different ways to answer this question.

[Among items #4, #5, and #6, if this item is shown to the participant second, then say the sentence]: Now, please look at the next question.

[Grade 4 Science–Student #6]

[Among items #4, #5, and #6, if this item is shown to the participant first, then ask the probe]: Can you tell me, in your own words, what the question is asking?
[Between items #4, #5, and #6, if this item is shown to the participant second or third, then ask the probe]: This is like the last question, but it has different answer choices. Which answer choice would you select? Please explain why you would select that choice.
[Among items #4, #5, and #6, if this item is shown to the participant first, then ask the probe]: What does the phrase “scientific tools” mean to you? Can you give me examples of some scientific tools that you have used?
How would you have answered this question if you had used scientific tools only at certain times during the school year? For example, if you had used <scientific tool, other than a computer, that student has mentioned in previous response> every day for three weeks, but haven’t used it since then.
[If student did not provide an example of a tool that can be used in the previous question, follow up with this probe]:
How would you have answered this question if you had used scientific tools only at certain times during the school year? For example, if you had used a microscope every day for three weeks, but haven’t used it since then, how would you have answered this question?
What does the phrase “a little” mean to you? What does the phrase “a lot” mean to you?
[Among items #4, #5, and #6, if this item is shown to the participant last, then ask the probe (Please point to the three sets of answer choices for this question to the participant)]: Which of the three versions of the answer choices made more sense to you? Why?

[Among items #4, #5, and #6, if this item is shown to the participant first, then say the sentence]: The next two questions will give you different ways to answer this question.

[Among items #4, #5, and #6, if this item is shown to the participant second, then say the sentence]: Now, please look at the next question.

[Grade 4 Science–Student #7]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Which answer choice would you select? In selecting your choice, did you include books or magazines about science that you have read outside of school, such as at home or in a public library?
Do you read books or magazines about science that are not for schoolwork? For example, you might read books or magazines only for your own curiosity or pleasure, but not for school assignment.
When you selected the answer choice, have you included the books or magazines about science that you have read that are not for schoolwork?
[If the participant answers B, C, D, or E, then ask the follow-up question]:
Where do you most often read books or magazines about science—in your classroom, in your school library, at home, in a public library, or in other places?

[Grade 4 Science–Student #8]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Which answer choice would you select? Please explain why you would select that choice.
What does the word “presented” mean to you?
In this school year, have you shown things that you have learned about science to your class? Can you give me an example of how you showed what you have learned about science to your class?
What does it mean to “show what you have learned”? Is that different than to “present what you have learned”?

[Grade 4 Science–Student #9]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Which answer choice would you select? Please explain why you would select that choice.
What does the phrase “talked about” mean to you? What does the phrase “measurements or results from your science activities or project” mean to you?
[If the participant answers B, C, D, or E, then ask the follow-up question]:
Who did you “talk about” measurements or results with? Why did you talk about measurements or results with these people?
[The focus of the cognitive lab is to find out whether the participant can comprehend the item, especially the words “talked about,” without being provided with examples such as peers, classmates, teachers, or lab partners. If the participant does not specify with whom they talked, then the interviewer should ask the follow-up question]:
Have you talked about these measurements or results with your peers, classmates, teachers, or lab partners?

[Grade 4 Science–Student #10]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Which answer choice would you select? Please explain why you would select that choice.
What does the word “report” mean to you?

[Grade 4 Science–Student #11]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Which answer choice would you select? Please explain why you would select that choice.
What would you call the written work that you do for a science activity or project? Would you call the written work a “report,” a “science journal,” a “lab write-up,” or anything else?
What does the phrase “science journal” mean to you?
What does the phrase “lab write-up” mean to you?

[Grade 4 Science–Student #12]

[Grade 8 Science–Student #6]

[Grade 12 Science–Student #6]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
When answering the question, did you think about your school’s library, a public library, or both?
When you read this question, what did you think “library resources” meant? Can you provide examples of “library resources” that you have used for science?
When you read this question, did you think of computers as library resources? Did you think of the Internet as a library resource?
[If the participant did not mention the following items (i.e., books, magazines, encyclopedias, and videos) in responding to probe 3, then ask the follow-up question]:
When you read this question, did you think of books? Did you think of magazines? Did you think of encyclopedias? Did you think of videos?

[Grade 4 Science–Student #13]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “science activities” mean to you?
What does the phrase “not for schoolwork” mean to you?
Can you provide examples of science activities that you or your classmates might have done or might do that are “not for schoolwork”?
[If the participant provides examples, then ask the follow-up questions]:

Why did you say this is “not for schoolwork”?

[Grade 4 Science–Student #14]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “science fair” mean to you?
Can you give me an example of a science fair that you or your classmates have participated in or might participate in?

[Grade 4 Science–Student #15]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “science club” mean to you?
Can you give me an example of a science club that you or your classmates have participated in or might participate in?

[Grade 4 Science–Student #16]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “science competition” mean to you?
Can you give me an example of a science competition that you or your classmates have participated in or might participate in?
Have you ever heard of a robotics competition?
[Ask the following question if participant answers “Yes” to probe 4]:
Have you ever participated in a robotics competition?
[The focus of the following probe is to find out whether the participant can differentiate among the activities (science fair, science club, science competition) in the three items just completed. The interviewer needs to make sure that the participant understands that the “three questions” referred to in the following question are the three items asking about students’ participation in the three science activities]:
The last three questions are about a science fair, a science club, and a science competition. Do you think these are different things? Why?

[Grade 4 Science–Student #17]

[Grade 8 Science–Student #10]

[Grade 12 Science–Student #9]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “science museum” mean to you?
What museums do you think of as science museums? What museums can you think of that are not science museums?
Can you give me an example of a science museum that you or your classmates have visited or might visit?
[If the participant answers “Yes,” then ask the follow-up question]:
Who did you go with to the science museum this school year?

The next two questions will ask about visiting science museums in different ways.

[Grade 4 Science–Student #18]

[Grade 8 Science–Student #11]

[Grade 12 Science–Student #10]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “on a school trip” mean to you? Can you describe a time when you visited a science museum on a school trip?
How often did you visit a science museum this school year on a school trip?

[Grade 4 Science–Student #19]

[Grade 8 Science–Student #12]

[Grade 12 Science–Student #11]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “not on a school trip” mean to you? Can you describe a time when you visited a science museum that was “not on a school trip”?
How often did you visit a science museum this school year that was not on a school trip?

Student Questionnaire–Grade 8

[Grade 8 Science–Student #1]

[Grade 12 Science–Student #1]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
How would you answer each part of this question?
What does the phrase “hands-on activities or projects” mean to you?
In part (b), what activities or projects have you done to learn about electricity? What have you learned from these activities or projects about electricity?
What does the phrase “circuits and energy” mean to you? Have you learned about “circuits and energy” in activities or projects that you have done?
Would you have answered part (b) differently if the example had been “batteries and light” instead of “circuits and energy”?

[If the participant answers “yes,” then ask the follow-up question]:
How would you have answered part (b)?

The next two questions will give you different ways to answer this question.

[Grade 8 Science–Student #2]

[Grade 12 Science–Student #2]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
How would you have answered each part of this question if you had only done activities or projects at certain times during the school year? For example, if you had done projects about living things every day for three weeks, but hadn’t done it since then, how would you have answered this question?

The next question will give you different ways to answer this question.

[Grade 8 Science–Student #3]

[Grade 12 Science–Student #3]

Item-Specific Probes:

What does the word “rarely” mean to you? What does the word “sometimes” mean to you? What does the word “often” mean to you?
How would you have answered each part of this question if you had done activities or projects only at certain times during the school year? For example, if you had done projects about living things every day for three weeks, but hadn’t done it since then, how would you have answered this question?
[Please point to the three sets of answer choices for this question to the participant]
Which of the three versions of the answer choices made more sense to you? Why?

[Grade 8 Science–Student #4]

[Grade 12 Science–Student #4]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Do you read books or magazines about science that are not for schoolwork? For example, you might read books or magazines only for your own curiosity or pleasure, but not for school assignment.
When you selected the answer choice for part (b), have you included the books or magazines about science that you have read that are not for schoolwork?
What does the phrase “science topics” mean to you?
Would you answer parts (b), (c), or (d) differently without the word “topics”? That is, if the parts read:
1. Read a book or magazine about science
2. Use the Internet to learn about science
3. Watch a movie, video, or DVD about science
[Ask the following question for each sub-item b, c, and d for which the participant answers “Yes”]: Why would you answer the question differently without the word “topics”?

[Grade 8 Science–Student #5]

[Grade 12 Science–Student #5]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
In parts (c) and (d), what does the phrase “talk about” mean to you?
[If the participant answers B, C, D, or E to part (c) or (d), then ask the follow-up question]:
Who have you talked to? Why did you talk about these measurements/results with these people?
[The focus of the cognitive lab is to find out whether the participant can comprehend the item, especially the words “talked about,” without being provided with examples such as peers, classmates, teachers, or lab partners. If the participant does not specify with whom they talked, then the interviewer should ask the follow-up questions]:
Have you talked about these measurements or results with your peers, classmates, teachers, or lab partners?
In part (h), what does the phrase “lab write-up” mean to you?
[If the participant answers B, C, D, or E to part (h), then ask the follow-up question]:
Can you give me an example of a lab write-up that you have written?
In part (i), what does the phrase “science journal” mean to you?
[If the participant answers B, C, D, or E to part (i), then ask the follow-up question]:
Can you give me an example of a science journal that you have written?

[Grade 4 Science–Student #12]

[Grade 8 Science–Student #6]

[Grade 12 Science–Student #6]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
When answering the question, did you think about your school’s library, a public library, or both?
When you read this question, what did you think “library resources” meant? Can you provide examples of “library resources” that you have used for science?
When you read this question, did you think of computers as library resources? Did you think of the Internet as a library resource?
[If the participant did not mention the following items (i.e., books, magazines, encyclopedias, and videos) in responding to probe 3, then ask the follow-up question]:
When you read this question, did you think of books? Did you think of magazines? Did you think of encyclopedias? Did you think of videos?

[Grade 8 Science–Student #7]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
In part (a), what does the phrase “science-related activities” mean to you? What does the phrase “not for schoolwork” mean to you?
Can you provide an example of science-related activities that you or your classmates have done or might do that are not for schoolwork?
[If the participant provides an example, ask the follow-up question]:

Why did you say this is “not for schoolwork”?

[Grade 8 Science–Student #8]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
In this question, what does the word “covered” mean to you?
[Ask the following question for each sub-item for which the participant answers “Yes”]:
Please describe how the topic was covered in your science class this year.
Instead of choosing between “Yes” and “No,” do you think more choices, such as “small extent,” “moderate extent,” and “large extent,” would be needed to describe how the topics being covered in your classes?

[Grade 8 Science–Student #9]

[Grade 12 Science–Student #8]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Which answer choices would you select? Please explain why you would select those choices.
What does the phrase “science fair” mean to you?
Can you give me an example of a science fair that you or your classmates have participated in or might participate in?
What does the phrase “science club” mean to you?
Can you give me an example of a science club that you or your classmates have participated in or might participate in?
What does the phrase “science competition” mean to you?
Can you give me an example of a science competition that you or your classmates have participated in or might participate in?

[Grade 4 Science–Student #17]

[Grade 8 Science–Student #10]

[Grade 12 Science–Student #9]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “science museum” mean to you?
What museums do you think of as science museums? What museums can you think of that are not science museums?
Can you give me an example of a science museum that you or your classmates have visited or might visit?
[If the participant answers “Yes,” then ask the follow-up question]:
Who did you go with to the science museum this school year?

The next two questions will ask about visiting science museums in different ways.

[Grade 4 Science–Student #18]

[Grade 8 Science–Student #11]

[Grade 12 Science–Student #10]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “on a school trip” mean to you? Can you describe a time when you visited a science museum on a school trip?
How often did you visit a science museum this school year on a school trip?

[Grade 4 Science–Student #19]

[Grade 8 Science–Student #12]

[Grade 12 Science–Student #11]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “not on a school trip” mean to you? Can you describe a time when you visited a science museum that was “not on a school trip”?
How often did you visit a science museum this school year that was not on a school trip?

Student Questionnaire–Grade 12

[Grade 8 Science–Student #1]

[Grade 12 Science–Student #1]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
How would you answer each part of this question?
What does the phrase “hands-on activities or projects” mean to you?
In part (b), what activities or projects have you done to learn about electricity? What have you learned from these activities or projects about electricity?
What does the phrase “circuits and energy” mean to you? Have you learned about “circuits and energy” in activities or projects that you have done?
Would you have answered part (b) differently if the example had been “batteries and light” instead of “circuits and energy”?

[If the participant answers “yes,” then ask the follow-up question]:
How would you have answered part (b)?

The next two questions will give you different ways to answer this question.

[Grade 8 Science–Student #2]

[Grade 12 Science–Student #2]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
How would you have answered each part of this question if you had only done activities or projects at certain times during the school year? For example, if you had done projects about living things every day for three weeks, but hadn’t done it since then, how would you have answered this question?

The next question will give you different ways to answer this question.

[Grade 8 Science–Student #3]

[Grade 12 Science–Student #3]

Item-Specific Probes:

What does the word “rarely” mean to you? What does the word “sometimes” mean to you? What does the word “often” mean to you?
How would you have answered each part of this question if you had done activities or projects only at certain times during the school year? For example, if you had done projects about living things every day for three weeks, but hadn’t done it since then, how would you have answered this question?
[Please point to the three sets of answer choices for this question to the participant]
Which of the three versions of the answer choices made more sense to you? Why?

[Grade 8 Science–Student #4]

[Grade 12 Science–Student #4]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Do you read books or magazines about science that are not for schoolwork? For example, you might read books or magazines only for your own curiosity or pleasure, but not for school assignment.
When you selected the answer choice for part (b), have you included the books or magazines about science that you have read that are not for schoolwork?
What does the phrase “science topics” mean to you?
Would you answer parts (b), (c), or (d) differently without the word “topics”? That is, if the parts read:
1. Read a book or magazine about science
2. Use the Internet to learn about science
3. Watch a movie, video, or DVD about science
[Ask the following question for each sub-item b, c, and d for which the participant answers “Yes”]: Why would you answer the question differently without the word “topics”?

[Grade 8 Science–Student #5]

[Grade 12 Science–Student #5]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
In parts (c) and (d), what does the phrase “talk about” mean to you?
[If the participant answers B, C, D, or E to part (c) or (d), then ask the follow-up question]:
Who have you talked to? Why did you talk about these measurements/results with these people?
[The focus of the cognitive lab is to find out whether the participant can comprehend the item, especially the words “talked about,” without being provided with examples such as peers, classmates, teachers, or lab partners. If the participant does not specify with whom they talked, then the interviewer should ask the follow-up questions]:
Have you talked about these measurements or results with your peers, classmates, teachers, or lab partners?
In part (h), what does the phrase “lab write-up” mean to you?
[If the participant answers B, C, D, or E to part (h), then ask the follow-up question]:
Can you give me an example of a lab write-up that you have written?
In part (i), what does the phrase “science journal” mean to you?
[If the participant answers B, C, D, or E to part (i), then ask the follow-up question]:
Can you give me an example of a science journal that you have written?

[Grade 4 Science–Student #12]

[Grade 8 Science–Student #6]

[Grade 12 Science–Student #6]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
When answering the question, did you think about your school’s library, a public library, or both?
When you read this question, what did you think “library resources” meant? Can you provide examples of “library resources” that you have used for science?
When you read this question, did you think of computers as library resources? Did you think of the Internet as a library resource?
[If the participant did not mention the following items (i.e., books, magazines, encyclopedias, and videos) in responding to probe 3, then ask the follow-up question]:
When you read this question, did you think of books? Did you think of magazines? Did you think of encyclopedias? Did you think of videos?

[Grade 12 Science–Student #7]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
In part (a), what does the phrase “science-related activities” mean to you? What does the phrase “not for schoolwork” mean to you?
Can you provide an example of science-related activities that you or your classmates have done or might do that are not for schoolwork?
[If the participant provides an example, ask the follow-up question]:

Why did you say this is “not for schoolwork”?

[Grade 8 Science–Student #9]

[Grade 12 Science–Student #8]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Which answer choices would you select? Please explain why you would select those choices.
What does the phrase “science fair” mean to you?
Can you give me an example of a science fair that you or your classmates have participated in or might participate in?
What does the phrase “science club” mean to you?
Can you give me an example of a science club that you or your classmates have participated in or might participate in?
What does the phrase “science competition” mean to you?
Can you give me an example of a science competition that you or your classmates have participated in or might participate in?

[Grade 4 Science–Student #17]

[Grade 8 Science–Student #10]

[Grade 12 Science–Student #9]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “science museum” mean to you?
What museums do you think of as science museums? What museums can you think of that are not science museums?
Can you give me an example of a science museum that you or your classmates have visited or might visit?
[If the participant answers “Yes,” then ask the follow-up question]:
Who did you go with to the science museum this school year?

The next two questions will ask about visiting science museums in different ways.

[Grade 4 Science–Student #18]

[Grade 8 Science–Student #11]

[Grade 12 Science–Student #10]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “on a school trip” mean to you? Can you describe a time when you visited a science museum on a school trip?
How often did you visit a science museum this school year on a school trip?

[Grade 4 Science–Student #19]

[Grade 8 Science–Student #12]

[Grade 12 Science–Student #11]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “not on a school trip” mean to you? Can you describe a time when you visited a science museum that was “not on a school trip”?
How often did you visit a science museum this school year that was not on a school trip?

Teacher Questionnaire–Grade 4

[Grade 4 Science–Teacher #1]

[Grade 8 Science–Teacher #1]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “student groupings” mean to you?
In part (b), what does the phrase “students’ learning preferences” mean to you?
In part (c), what does the phrase “students’ readiness needs” mean to you?
How would you answer these questions if you group students for some lessons, but not for other lessons?
Do you use all the three grouping methods listed? Are there any other grouping methods that you have used or you might use?
Do you think asking a general question with only “yes” and “no” answer choices, that is, “Do you create groups within this class for science instruction on the basis of ability?” would be sufficient? In other words, do you think it is necessary to ask about the three grouping methods?
Are the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent” appropriate for describing the extent of instructional grouping strategies? What would you suggest for improving the choices?

[Grade 4 Science–Teacher #2]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “upper grade levels” in part (d) mean to you?
What does the phrase “studying science” in part (d) mean to you?
[One of the focuses of the cognitive lab is to find out whether the participant includes middle school in his/her consideration of “upper grade levels.” If this information is not clear in the participant’s answer, the interviewer should ask the follow-up question]:
Do you include middle school in your understanding of “upper grade levels”?
What does the phrase “knowledge and skills needed for studying science in upper grade levels” mean to you?
How do you determine what knowledge and skills will be needed beyond 4^th grade? In other words, what sources of information do you use to make the decision?
If part (d) had been phrased as “prepare students for further study in science,” would you answer the question the same way?
What does the phrase “prepare students for further study in science” mean to you?

[Grade 4 Science–Teacher #3]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Which answer choice would you select for each part? Please explain why you would select the choice.

The next question asks about this in a different way.

[Grade 4 Science–Teacher #4]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Does the way the question is asked make a difference in how you would answer any of the parts listed? If so, why?
Are the distinctions “not at all,” “small extent,” “moderate extent,” and “large extent” appropriate for describing what is provided by the school?
For these questions what does the phrase “small extent” mean to you? What does the phrase “moderate extent” mean to you? What does the phrase “large extent” mean to you?
Which version of this question and answer choices (A, B, C, D) do you think is closest to how the material is provided in your school?
Can you suggest other answer choices that would provide more specific information about the resources provided by the school? Please quantify how many of each resource your school provides for each part where you answered B, C, or D.

[Grade 4 Science–Teacher #5]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
In part (c), what does the phrase “vary the pace of instruction for some students” mean to you?
If part (c) had been phrased as “pace my teaching differently for some students,” would you answer the question the same way? Why or why not?
Which way of phrasing part (c) do you think is better for describing how the instructional strategy is used? Why?

[Grade 4 Science–Teacher #6]

[Grade 8 Science–Teacher #6]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Do you make a distinction between “science fair” and “science competition”? If so, how would you explain the difference?
[If the participant answers B, C, or D to any part, then ask the follow-up question]:
Can you give examples of how you provided these learning experiences?
[If the participant answers B, C, or D to part (d), then ask the follow-up question]:
What types of science museums do you take students to?
Are the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent,” sufficient for describing the extent of the opportunities you provide? What would you suggest for improving the choices?
Would using frequencies, such as certain times a week/a semester/a school year, be better answer choices for this question? If so, what time frequencies would you suggest?

Teacher Questionnaire–Grade 8

[Grade 4 Science–Teacher #1]

[Grade 8 Science–Teacher #1]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “student groupings” mean to you?
In part (b), what does the phrase “students’ learning preferences” mean to you?
In part (c), what does the phrase “students’ readiness needs” mean to you?
How would you answer these questions if you group students for some lessons, but not for other lessons?
Do you use all the three grouping methods listed? Are there any other grouping methods that you have used or you might use?
Do you think asking a general question with only “yes” and “no” answer choices, that is, “Do you create groups within this class for science instruction on the basis of ability?” would be sufficient? In other words, do you think it is necessary to ask about the three grouping methods?
Are the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent” appropriate for describing the extent of instructional grouping strategies? What would you suggest for improving the choices?

[Grade 8 Science–Teacher #2]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What does the phrase “upper grade levels” in part (d) mean to you?
What does the phrase “studying science” in part (d) mean to you?
What does the phrase “knowledge and skills needed for studying science in upper grade levels” mean to you?
How do you determine what knowledge and skills will be needed beyond 8^th grade? In other words, what sources of information do you use to make the decision?
If part (d) had been phrased as “prepare students for further study in science,” would you answer the question the same way?
What does the phrase “prepare students for further study in science” mean to you?

[Grade 8 Science–Teacher #3]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Which answer choice would you select for each part? Please explain why you would select the choice.

The next question asks about this in a different way.

[Grade 8 Science–Teacher #4]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Does the way the question is asked make a difference in how you would answer any of the parts listed? If so, why?
Are the distinctions “not at all,” “small extent,” “moderate extent,” and “large extent” appropriate for describing what is provided by the school?
For these questions what does the phrase “small extent” mean to you? What does the phrase “moderate extent” mean to you? What does the phrase “large extent” mean to you?
Which version of this question and answer choices (A, B, C, D) do you think is closest to how the material is provided in your school?
Can you suggest other answer choices that would provide more specific information about the resources provided by the school? Please quantify how many of each resource your school provides for each part where you answered B, C, or D.

[Grade 8 Science–Teacher #5]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
In part (c), what does the phrase “vary the pace of instruction for some students” mean to you?
If part (c) had been phrased as “pace my teaching differently for some students,” would you answer the question the same way? Why or why not?
Which way of phrasing part (c) do you think is better for describing how the instructional strategy is used? Why?

[Grade 4 Science–Teacher #6]

[Grade 8 Science–Teacher #6]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Do you make a distinction between “science fair” and “science competition”? If so, how would you explain the difference?
[If the participant answers B, C, or D to any part, then ask the follow-up question]:
Can you give examples of how you provided these learning experiences?
[If the participant answers B, C, or D to part (d), then ask the follow-up question]:
What types of science museums do you take students to?
Are the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent,” sufficient for describing the extent of the opportunities you provide? What would you suggest for improving the choices?
Would using frequencies, such as certain times a week/a semester/a school year, be better answer choices for this question? If so, what time frequencies would you suggest?

School Questionnaire–Grade 4

[Grade 4 Science–School #1]

[NOTE: Skip pattern will be removed when questions are sent for cog lab.]

Item-Specific Probes:

Describe the position or role of a “science specialist or coach.”
Which answer choice would you select? Please explain why you would select that choice.
[If the participant answers A or B, then ask the follow-up questions]:
What are the responsibilities of the science specialist or coach in your school?
Please elaborate on the availability of the science specialist or coach to fourth-grade teachers at your school.
The responsibilities of a science specialist or coach might include developing curriculum, implementing research findings on science instruction, helping teachers with content and pedagogical knowledge, and helping students with science learning. Is there a different term or title that is used in your school for describing individuals who have these responsibilities?

[Grade 4 Science–School #2]

Item-Specific Probes:

In part (d), what does the phrase “translate science standards and current research into classroom practice” mean to you? Can you think of a way to describe this activity that might be more meaningful to school administrators?
In part (e), what does the phrase “provide science enrichment to some student groups” mean to you?
Do you think that the parts of this question cover the main responsibilities of a science specialist or coach in your school? Are there additional responsibilities that a science specialist or coach might have, and what are they?
Are the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent,” sufficient for describing the extent of science specialist or coach responsibilities in your school? What would you suggest for improving the choices?

[Grade 4 Science–School #3]

[Grade 8 Science–School #3]

[Grade 12 Science–School #3]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Do you make a distinction between “science fair” and “science competition”? If so, how would you explain the difference?
[If the participant answers B, C, or D to any part, then ask the follow-up question]:
Can you provide a few examples of how your school has provided the opportunities for students?
[If the participant answers B, C, or D to part (d), then ask the follow-up question]:
What types of science museums does your school take students to?
Are the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent,” sufficient for describing the extent of the opportunities your school provides? What would you suggest for improving the choices?
Would using frequencies, such as certain times a week/a semester/a school year, be better answer choices for this question? If so, what time frequencies would you suggest?

School Questionnaire–Grade 8

[Grade 8 Science–School #1]

[NOTE: Skip pattern will be removed when questions are sent for cog lab.]

Item-Specific Probes:

Describe the position or role of a “science specialist or coach.”
Which answer choice would you select? Please explain why you would select that choice.
[If the participant answers A or B, then ask the follow-up questions]:
What are the responsibilities of the science specialist or coach in your school?
Please elaborate on the availability of the science specialist or coach to eighth-grade teachers at your school.
The responsibilities of a science specialist or coach might include developing curriculum, implementing research findings on science instruction, helping teachers with content and pedagogical knowledge, and helping students with science learning. Is there a different term or title that is used in your school for describing individuals who have these responsibilities?

[Grade 8 Science–School #2]

Item-Specific Probes:

In part (d), what does the phrase “translate science standards and current research into classroom practice” mean to you? Can you think of a way to describe this activity that might be more meaningful to school administrators?
In part (e), what does the phrase “provide science enrichment to some student groups” mean to you?
Do you think that the parts of this question cover the main responsibilities of a science specialist or coach in your school? Are there additional responsibilities that a science specialist or coach might have, and what are they?
Are the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent,” sufficient for describing the extent of science specialist or coach responsibilities in your school? What would you suggest for improving the choices?

[Grade 4 Science–School #3]

[Grade 8 Science–School #3]

[Grade 12 Science–School #3]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Do you make a distinction between “science fair” and “science competition”? If so, how would you explain the difference?
[If the participant answers B, C, or D to any part, then ask the follow-up question]:
Can you provide a few examples of how your school has provided the opportunities for students?
[If the participant answers B, C, or D to part (d), then ask the follow-up question]:
What types of science museums does your school take students to?
Are the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent,” sufficient for describing the extent of the opportunities your school provides? What would you suggest for improving the choices?
Would using frequencies, such as certain times a week/a semester/a school year, be better answer choices for this question? If so, what time frequencies would you suggest?

School Questionnaire–Grade 12

[Grade 12 Science–School #1]

[NOTE: Skip pattern will be removed when questions are sent for cog lab.]

Item-Specific Probes:

Describe the position or role of a “science specialist or coach.”
Which answer choice would you select? Please explain why you would select that choice.
[If the participant answers A or B, then ask the follow-up questions]:
What are the responsibilities of the science specialist or coach in your school?
Please elaborate on the availability of the science specialist or coach to twelfth-grade teachers at your school.
The responsibilities of a science specialist or coach might include developing curriculum, implementing research findings on science instruction, helping teachers with content and pedagogical knowledge, and helping students with science learning. Is there a different term or title that is used in your school for describing individuals who have these responsibilities?

[Grade 12 Science–School #2]

Item-Specific Probes:

In part (d), what does the phrase “translate science standards and current research into classroom practice” mean to you? Can you think of a way to describe this activity that might be more meaningful to school administrators?
In part (e), what does the phrase “provide science enrichment to some student groups” mean to you?
Do you think that the parts of this question cover the main responsibilities of a science specialist or coach in your school? Are there additional responsibilities that a science specialist or coach might have, and what are they?
Are the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent,” sufficient for describing the extent of science specialist or coach responsibilities in your school? What would you suggest for improving the choices?

[Grade 4 Science–School #3]

[Grade 8 Science–School #3]

[Grade 12 Science–School #3]

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Do you make a distinction between “science fair” and “science competition”? If so, how would you explain the difference?
[If the participant answers B, C, or D to any part, then ask the follow-up question]:
Can you provide a few examples of how your school has provided the opportunities for students?
[If the participant answers B, C, or D to part (d), then ask the follow-up question]:
What types of science museums does your school take students to?
Are the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent,” sufficient for describing the extent of the opportunities your school provides? What would you suggest for improving the choices?
Would using frequencies, such as certain times a week/a semester/a school year, be better answer choices for this question? If so, what time frequencies would you suggest?

IV. Generic Probes and Think-Aloud Hints for the Interviewer

If the participant is silent for 5 or more seconds, use the following as a guideline for encouraging the participant to read the question aloud and say his/her thoughts, or to help him/her elaborate the responses.

Please read the question aloud and tell me what you are thinking.
You mentioned __________, tell me more about that.
You mentioned __________, what was that like for you?

We are interested in capturing all the participant’s mental processes while answering these questions. Your goal is to have the participant speak aloud all his or her thoughts while answering the questions. By asking follow-up probes after each item, you will ensure that the data collected are as complete as possible. Provide positive feedback by telling the participant when he or she has given helpful information. The participant may provide more detailed responses if given positive feedback such as:

This information is very helpful.
The example you just gave is very useful.
Thank you for providing such a thorough response.

If a participant is continually providing short responses or not answering, use “continuers” to encourage the participant to be more descriptive. The goal is to get participants to verbalize their thoughts without “putting words in their mouth.” Do not lead the participant’s response by asking a question that implies or contains its own answer such as:

Leading Question: Do you dislike science class like most students?
Neutral Question: How do you feel about science class?

You have to be as objective and unbiased as possible, but you may offer a verbal “nudge,” such as:

What are you thinking now?
Any other thoughts?
Tell me how you came to pick that answer.

Use your best judgment. If a participant exhibits difficulty reading or understanding an item, probe the participant without biasing the response.

Can you tell me in your own words what that question was asking? [comprehension and interpretation probe]
What does the word [term] mean to you as it’s used in this question? [comprehension and interpretation probe]

Use your best judgment. If a participant appears to have guessed or seems less than certain of the answer he/she has given, probe the participant.

How much would you say you know about [topic]? [confidence probe]
How sure are you of your answer? [confidence probe]
How did you come up with that answer? [recall strategy and bias probe]
Was this easy or hard to answer? [comprehension and recall probe]
How easy or difficult is it to remember [topic]? [recall probe]
Did you find the list of answer choices easy or hard? [response probe]
How easy or hard was it to choose an answer from that list of choices? [response probe]

Use your best judgment. If a participant appears uncomfortable with a word or answering a question, probe the participant.

The question uses the word [term]. Does that sound OK to you, or would you choose something different? [sensitivity probe]
In general, how do you feel about this question? [sensitivity probe]

Use your best judgment. After testing an item, using the above generic probes to understand why the question as written is not clear, you may reword the question in a way that the participant will understand based on shared interviewer feedback or previous interview experiences. If this is done, be sure to document precisely the reworded question that was tested and test the item using relevant generic probes.

In addition to generic probes, interviewer protocols will include item-specific probes. Not all items have an item-specific probe. Item-specific probes are provided only if the researchers deemed it important to learn more about a cognitive process or issue that might not be raised naturally from the think-aloud process or generic probing. If the issue to be inquired through the item-specific probe was addressed, or if time is running short, the item-specific probe does not need to be asked. Use your best judgment.

V. Data Collection and Coding Procedure

Record your notes into the provided data entry spreadsheet or software program after the interview. As described in your training session, you are asked to record information to identify problems with participants’ survey response processes, which includes explicitly coding the occurrence of six different threats to obtaining valid and reliable survey responses.

Did the participant skip over, misread, or reread any word(s)?
Did the participant exhibit difficulty reading an item?
Did the participant exhibit difficulty understanding the question?
Did the participant exhibit difficulty understanding the meaning of particular words or concepts?
Did the participant exhibit difficulty remembering the question or answer choices?
Did the participant exhibit difficulty or discomfort answering the question?

In addition to the explicit coding to identify problems, you should also record in the data entry sheets descriptions of identified problems, summarization of participant verbal reports, participant responses to item-specific probes, and any other information that you identified as potentially useful.

VI. Debriefing and Thank You

The following scripts should not be read verbatim. Only ask questions below that have not already been addressed by the participant.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

[Grade 12 Students, Teachers, and School Administrators Only] If you had not been making comments and answering my questions, about how long do you think it would have taken you to fill out this questionnaire?

How did you feel about the questions you were asked?

Did you feel the questions were easy or difficult to understand?

Did you feel the questions were easy or difficult to answer?

[Teachers and School Administrators Only] Overall, did the language used in the questionnaire items seem appropriate?

Thank participant for his/her time. Provide gift card.

Part C. SICT Play Testing

Introduction to Study and Setup of Task Review Script

Text written in italics is to be read aloud by the staff facilitator.

NOTE: The facilitator should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. Text placed in brackets is generic text that should be tailored to suit the particular task being studied.

My name is __ and I am from the Educational Testing Service in Princeton. I am here with my colleagues [introduce colleagues]. We are developing a new set of tests in science for a program funded by the federal government called the National Assessment of Educational Progress (NAEP). NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will help us develop better tests for other students. We are going to ask for your opinions about some interactive tasks that we are creating for a NAEP science test. Eventually our tests will be on the computer, but right now we have some early drafts of what these tasks might look like. You are here to give us your thoughts about these tasks, tell us what you find interesting—or perhaps what is not interesting—and what seems clear or what is confusing.

My colleagues and I will take notes on what you are saying about the task—please be as honest as possible, talk freely with each other, and don’t be concerned if you find something confusing. You are not being graded on anything—you are helping us by voicing any opinions or thoughts about these tasks. There are no right or wrong thoughts.

What you say will not be used in any way that could identify you, except as required by law (see 20 U.S.C. § 9573). We will look at what you say later, but only so that we can understand how our test is working and how we can make it better.

As you work through these tasks, we want you to talk with each other as you go along. We might also ask you a few questions now and then.

The facilitators will generally observe the students with little interruption and will listen to the conversation that occurs between group members for insights into what the students are thinking about the tasks. Intermittently, as appropriate, the facilitators can ask targeted questions such as those in this list:

Is this task interesting? Why?/Why not?

Are there any questions or words that seem confusing here? Did you understand that part?

How would you answer this question? [Ask different group members if their approaches would differ].

How could this task be improved? Could it be clearer, or more interesting, for example?

This task asks you to solve a problem about <insert natural phenomenon targeted in the task>. Have you learned about this before, either in school or in your own life?

Staff facilitators will observe students, take notes, and will pay attention to verbal answers to the questions above and write them down, but also will look for evidence of engagement, boredom, or confusion in what students say to each other and in their facial expressions or body language. If appropriate, staff can use these moments to probe for more information.

Occasionally staff observing play testing sessions will introduce a question to groups of students talking about the tasks to get more information from them, particularly in cases when students talk about issues related to their interest (or non-interest) in the task, confusing language in the task, or how they might respond to a question on the task. In such cases observers might ask something like, “Can you tell me more about what you said?”. Prior to each play testing session, ETS staff may informally identify some key focus areas for each task. If students do not provide sufficient comments on targeted parts, a staff member may ask a group of students if they had any thoughts about the particular sections, using questions such as those described above, but focused on specific places or issues in the task.

At the end, students will be thanked for their participation and for helping us to improve our tests. Students will be given gift cards for their participation.

Part D. SICT Cognitive Interviews

I. Introduction to Study Script

NOTE: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. Text placed in brackets is generic text that should be tailored to suit the particular task being studied.

Introduction:

Hello, my name is ______ and I work for EurekaFacts. It’s nice to meet you and thank you very much for helping us out today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?

When the student responds, follow up with 2 or 3 questions to get the student used to talking, such as:

That’s interesting—why do you enjoy <subject> so much?
What are you studying in <subject> at the moment?
And what’s the best thing you have studied in <subject> so far?

Good. Well, I think you’ll enjoy what we are going to be doing today. First, let me begin by explaining why I am here and what you are going to be doing. You are taking part in a special study looking at a new type of activity to find out what students know in science. It is part of the National Assessment of Educational Progress, or NAEP for short, a test that is funded by the U.S. Federal Government and run by a center in the Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will help us develop better tests for other students.

It’s okay if you don’t know how to do any parts of the task. I will not be grading your work today, and no one will know that it was you who did the task. My goal is to learn how you react to the task, so please try to do your best.

If at any time you decide you don’t want to go on, that is your choice, and you may stop.

This new test is on the computer. The instructions about how to do the test are also on the computer.

This interview is being recorded so we can review the tapes later. We will also be able to capture what you click on and where you move your mouse. What you say will not be told to anyone, or used in any way that could identify you, for any other purpose except as required by law (see 20 U.S.C. § 9573). We will look at what you say later, but only so that we can understand how our test is working and how we can make it better.

We will work together for no more than an hour and a half.

Do you have any questions?

After answering any questions and giving further explanation, continue with the think-aloud training. If the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

Okay, let’s move on. Before we look at the real tasks, I want to make sure you have a good idea of what we’re going to be doing. So I am going to give you some practice tasks. The practice tasks should help you get used to what we will be doing during the real tasks. They should help you understand how we want you to respond. Do you have any other questions before we start?

NOTE: Check the ETS-provided SICT study schedule to establish whether the student should undergo the assigned tasks with a concurrent or retrospective verbal protocol (think-aloud) procedure. If concurrent, the student will think aloud while he or she is doing the task; in those cases, follow the steps described in section II. If retrospective, the student will describe his or her thinking after he or she has completed the task, while watching a recording of the task that was made as he or she proceeded through it; in those cases, follow the steps described in section III.

II. Science ICT Concurrent Think-Aloud

IIa. Science ICT Concurrent Think-Aloud: Instructions and Modeling Script

NOTE: If the schedule indicates that the retrospective method is to be used, please skip this section and move to section III.

Before launching the tasks, the interviewer should open Morae Recorder and begin screen capture. The settings should be adjusted to allow recording of all on-screen events that occur during the task, including typed responses, mouse clicks, and mouse movements. It should also be set to record the student, via the webcam. Audio settings should be verified to ensure that any verbalizations will be captured. (In other words, does the microphone work, and is the software set up correctly to capture the student’s voice?) If audio quality is poor with the audio device attached to the webcam, a digital audio recording device should be used to supplement the webcam microphone.

Note: Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. Text placed in brackets is generic text that should be tailored to suit the particular task being studied.

To help us make our test better, we will ask you to complete some tasks. While you are doing the tasks, we will be recording everything that happens on the screen. The screen recording will capture all of your responses and movements on the computer such as what you click on and where you drag your mouse. We will also be videotaping you, as you complete the task, through the webcam.

I’m going to ask you to do the questions in these tasks in a way that may be different from what you are used to. Instead of working quietly, I want you to tell me what you are thinking as you work through the task. We call this “thinking-aloud,” because we are asking you to say everything you are thinking out loud.

In a moment, I will give you an example of the think-aloud process. Then I will give you a chance to practice it. You won’t be graded on anything you say while you are thinking aloud. There are no incorrect thoughts, and everything you think and say is important to us.

Okay, now I’m going to show you how to think out loud—this will help you see how I want you to describe what you are thinking as you are working on the task. When I am finished I’ll ask you to try it, so you can see how it works.

Think-Aloud Demonstration:

Since we can’t tell what is going in your head, we need you to “Think Aloud.”

Let me give you an example. Look at this question. It asks me to look at the five types of animals and choose two that are the most similar. So I am going to do this task and I will think aloud while I work on it.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar:

Beetle
Mouse
Crab
Dolphin
Cow

Okay, so I am reading the question…. It says I have to decide on the two animals that are the most similar. Hmm, I am just wondering what do they mean by “similar”? That could mean anything. I’m thinking I’m not sure how I’m going to make that decision. Well, let’s look at the choices. So the choices are: beetle, mouse, crab, dolphin, and cow. Well, as I am reading the choices I am already thinking that there are some things about them that are similar, but there are also differences, obviously. I’m thinking the beetle and crab are kind of similar, in a way—well, they both have lots of legs and they kind of move around fast. But one lives in the water and the other doesn’t, so that makes them quite different. I suppose the mouse also moves around fast, but it’s not similar to either the beetle or the crab, except they are all small. Then there is the dolphin that lives in the water, too… but that’s about all it has in common with the crab, so I wouldn’t say they are very similar even though they’re both in the sea. Hmm, so the mouse and the cow are both mammals, they both have fur and they have live babies instead of laying eggs or whatever… oh right, but dolphins are mammals too, aren’t they? Yes, I think they are…. but really they seem so different from mice and cows to me. I don’t know… this seems like a hard question… I’m not sure what to choose. Okay, I’m going to try to make a decision…umm…okay, I think I am going to go for… mouse and cow. Because they both have fur and have four legs and they both walk around on the land. That seems weird; they are not very similar at all really! But when I think about all of the others they seem too different from each other in really basic ways, like where they live and their kind of body and so on… so, yes, although I found it really hard to decide, that seems like the best choice to me, because they are both mammals and they both live on the land. So those things seem like they are important, and that is how I made my decision.

Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do as you are thinking about the tasks today. The point of the think-aloud is to get at whatever is in your head as you are doing the tasks. Just say aloud the words and the thoughts that are in your head, as you are thinking about and working on the tasks.

IIb. Science ICT Concurrent Think-Aloud: Student Practice Script I

Place the practice question in front of the student so he/she can read it. Some participants will be silent after reading the question. Immediately encourage the participant to say whatever he/she is thinking.

You may need to remind the student to talk aloud as he/she works through the questions and tasks. If necessary, use the “Think-Aloud Hints” shown below to prompt the student, but be careful not to lead the student. You need to be familiar enough with the information to introduce the think-aloud in a natural, conversational manner.

Now you will try a think-aloud. You can use this example. Like last time, you have to pick two of these things that you think are the most similar. As you are reading the question and the choices, and as you are thinking about the possibilities and figuring out your decision, I want you to talk out loud to tell me all of the thoughts that are in your head at each moment.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I’ll be reminding you to think aloud if you get quiet because I need to hear all of your thoughts.

Okay, now you try. Go ahead and start working on this question and remember to think aloud as you are doing it.

Which two living things below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the tasks? What questions would you like to ask me? [If the student says he or she feels ok and doesn’t have any questions: Good, then let’s begin our study.] [If the student expresses concerns, says he/she has questions, or appears to be hesitant or reluctant, ask him/her to say more about the concerns or questions, and try to address his/her concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him/her to stop.]

IIc. Science ICT Concurrent Think-Aloud: Student Practice II (Optional)

(Use only if you feel the student needs to practice another Think-Aloud before moving on to the actual questions.)

If the student struggles to think aloud, the interviewer should give the student another opportunity to practice. The interviewer should praise the student for the first attempt regardless of how good it was, for example: “Very good—let’s do another one before we start the real tasks. Are you ready? Here is the next practice question. Remember to think out loud as you begin to think about this question and all the way through—tell me what you are thinking as you work through it.” During the practice item, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, below).

Which two of the following objects have the most similar properties? Circle the two that are most similar.

Silver coin
Chocolate coin
Gold coin
Blue plastic coin
Brown plastic coin

After the student has finished:

IId. Science ICT Concurrent Think-Aloud: Starting the Science Tasks

Now we will move on to the actual test questions. Remember, when you answer the questions I’d like you to say aloud everything that you’re thinking, and I may remind you to do that if you go quiet. This task should take about [20 or 35] minutes. Remember, you will not be graded on what you do during the task and there is no right or wrong way to think aloud, as long as you keep telling me your thoughts. Your thoughts will help us make the tasks better. I will also have a few questions after you have finished working on the task.

Do you have any questions before we go on? (Answer any questions the student may ask.)

Because the information you provide is so important to us, I am going to be taking notes while you think aloud and answer the questions.

Here is the task. When you are ready, go ahead and start working on it.

IIe. Science ICT Concurrent Think-Aloud: Hints for the Interviewer

We are interested in capturing all the student’s mental processes while working through tasks. Your goal is to have the student speak aloud all of his or her thoughts while doing the tasks. If a student is continually providing short utterances or not talking, you should use “continuers” to encourage him or her to be more descriptive. The goal is to get the student to verbalize thoughts without putting words in his or her mouth. Don’t ask questions that lead the participant’s response in a particular direction or make him or her rush or change his or her approach. You have to be as objective and unbiased as possible.

In general, if the student is silent for approximately 5 to 10 seconds, you should use the following as a guide for encouraging the student to describe his or her thoughts, or to help the student elaborate his or her responses.

If the student is not verbalizing enough, you should offer a verbal “nudge” to remind them to keep talking, such as:

What’s going on in your head right now?
I see you’re looking at the task [or screen/figure/chart/text]. What are you thinking?
You seem to be thinking hard—can you tell me what’s in your mind right now?
What are you thinking now?
Any other thoughts?

If the student says something about his or her thoughts but it is difficult to tell exactly what he or she means, or if he or she begins to talk but doesn’t say very much and you want him or her to elaborate on what he or she is saying, you can ask for more, for example:

Tell me a bit more about that.
That’s interesting—can you say more about that?
Go on….
Uh huh, okay, and what else are/were you thinking?

If you need more clarity or a more explicit description of the student’s thoughts:

Can you explain what you mean by that?
Can you say a bit more, to help me understand what you mean/are thinking/were thinking?

It is important to be responsive and sensitive to the student’s behavior. If it becomes clear that the student cannot tell you any more about his or her thoughts, do not keep asking. Try to encourage the student to think aloud as much as he or she can, but observe his or her responses and don’t keep pushing if he or she seems frustrated or uncomfortable. Although we want to keep students talking as much as they can, sometimes a student will simply be unable to tell you what is in his or her mind beyond a basic reply such as “I am reading the question.” Be sensitive to students’ nonverbal signals, if they say they cannot tell you any more than they have, or if they need a few extra seconds to gather their thoughts in order to put them into words.

In addition, during your observations if it seems a student is hung up on something, it’s important to note when this occurred.

IIf. Science ICT Notes on Student Actions and Behaviors During Tasks

As a student is proceeding through the task, a screen-capture system will record evidence of his or her actions on the task (such as where a student clicks, how he or she moves the mouse, and any text typed into fields or boxes). During the session, we would like you, as the interviewer, to take notes of any of the student’s expressions or behaviors that you think may reflect the status of his or her understanding, engagement, or use of the SICT. The following are examples of such behaviors for interviewers to note:

Does the student express signs of confusion, boredom, or excitement?
Does the student use tools provided in the task such as clicking on tabs or buttons to reveal additional or earlier information, digital notepads, digital calculators, or reminders of what to include or think about?
Does the student rapidly move through the screen or take his or her time? Does the student spend a lot of time on a particular aspect?

As you observe students, you should consider these kinds of observations as an opportunity to prompt them to think aloud, in order to encourage them to express what is in their minds. You should also make a note of any places in the task that you feel would be valuable to follow up with some additional questions after the task is completed (see section IV).

III. Science ICT Retrospective Think-Aloud

IIIa. Science ICT Retrospective Think-Aloud: Instructions and Modeling Script

In this approach, students are not asked to think out loud during the task. Instead, students will be instructed to complete the task as they normally would under naturalistic testing conditions (silently, without interruption). Then, after the task is completed, they will be asked to describe what they were thinking while reviewing the task on a computer. The replay of the task is intended to help cue the student’s recall of the thought processes that occurred during the task as it progressed.

Before launching the tasks, the interviewer should open Morae Recorder and begin screen capture. The settings should be adjusted to allow recording of all on-screen events that occur during the task, including typed responses, mouse clicks, and where the mouse is dragged on the screen. The software should also be set to record the student, via the webcam. Audio settings should be checked to ensure that any verbalizations will be captured (in other words does the recording work to capture the voice). If audio quality is poor with the audio devise attached to the webcam,a digital audio recording device should be used to supplement the webcam microphone.

Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. Text placed in brackets is generic text that should be tailored to suit the particular task being studied.

To help us make our test better, we will ask you to complete some tasks. While you are doing the tasks, we will be recording everything that happens on the screen. The screen recording will capture all of your responses and movements on the computer, such as what you click on and where you drag the mouse. We will also be videotaping you, as you complete the task, through the webcam.

I’ll get you to do the task in the normal way, working at your own pace through all of the questions you will see on the screen. Then, after you have finished, I’m going to ask you to go back over the task with me. As we go back over it and look at the task again, I will ask you to tell me what you were thinking as you were working through each of the sections. We call this “thinking aloud,” because we are asking you to say everything you were thinking during the task out loud. To help you remember what you were thinking as you worked on it the first time, we will look at the recording we just made of you doing the task. So you will be able to see all of the mouse clicks and all of the answers that you gave as you were doing the task. We will watch the recording, and as we are going through it, I will be asking you to tell me what you were thinking at each point. Does that make sense? Do you understand what we’ll be doing?

[Answer any questions and clarify the sequence of steps, if needed.]

Okay, now I’m going to show you how to think out loud—this will help you see how I want you to describe what you were thinking while you were working on the task. When I am finished, I’ll ask you to try it, so you can see how it works.

Think-Aloud Demonstration:

Since we can’t tell what is going on in your head, we need you to “Think Aloud.”

Let me give you an example. Look at this question. It asks me to look at the five types of animals and choose two that are the most similar. So first of all, I am going to do this task just as I normally would for a test question. After I have finished it, I will try to remember what I was thinking as I was working on it, and I’ll tell you what I was thinking.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar:

Beetle
Mouse
Crab
Dolphin
Cow

[The interviewer ‘works’ silently for a minute or two on the question above. The interviewer should enact some behaviors that suggest that he/she is thinking hard about the question and carefully considering the possible answers; for example, pointing with a pencil tip at each option in turn; moving a pencil or fingertip between answer options; returning to the question; giving nonverbal signals indicating deep consideration, uncertainty, etc. Finally the interviewer circles two answers, B and E, and then begins the retrospective Think-Aloud.]

Okay, so here is what I was thinking as I did this task. Well, first of all I read the question…. It said I had to decide on the two animals that are the most similar. So at first I was wondering what do they mean by “similar”? That could mean anything. I was thinking I wasn’t sure how I was going to make that decision. So I thought, well, let’s look at the choices. And the choices were: beetle, mouse, crab, dolphin, and cow. Well, as I was reading the choices I was already thinking that there are some things about them that are similar, but there are also differences, obviously. I was thinking the beetle and crab are kind of similar, in a way—well, they both have lots of legs and they kind of move around fast. But one lives in the water and the other doesn’t, so that makes them quite different. I was thinking I suppose the mouse also moves around fast but it’s not similar to either the beetle or the crab, except they are all small. Then I was thinking there is the dolphin that lives in the water, too… but that’s about all it has in common with the crab, so I didn’t think they were very similar even though they’re both in the sea. Hmm, so then I thought, the mouse and the cow are both mammals, they both have fur and they have live babies instead of laying eggs or whatever… oh right, but then I realized dolphins are mammals too, aren’t they? I was thinking they are…. but really they seem so different to mice and cows to me. I didn’t really know… this seemed like a hard question… I wasn’t sure what to choose. So in the end, I just tried to make a decision…and I decided to go for… mouse and cow. Because they both have fur and have four legs and they both walk around on the land. That seemed a bit weird; they are not very similar at all really! But when I thought about all of the others they seemed too different from each other in really basic ways, like where they live and their kind of body and so on… so, yes, although I found it really hard to decide, that seemed like the best choice to me, because they are both mammals and they both live on the land. So those things seemed like they were important, and that is how I made my decision.

Can you see how I tried to go back over the task and remember all of the steps in my thinking, and then I said out loud all of my thoughts? That is what I will be asking you to try to do after you complete each of the tasks today. The point of the think-aloud is to get at what was in your head as you were doing the tasks. Just watch the replay as it goes along and try to recall and say out loud the words and the thoughts that were in your head as you worked through the task.

IIIb. Science ICT Retrospective Think-Aloud: Student Practice Script I

Place the practice question (shown below) in front of the student so he or she can read it. Allow the student as much time as needed to answer the question. When he or she indicates he or she has finished, ask him or her to begin thinking aloud. You may need to remind the student to keep talking and describing the thoughts he or she had during the task. If necessary, use the “Think-Aloud Hints” (see section IIe) to prompt the student, but you should be careful not to lead him or her. You need to be familiar enough with these hints to facilitate and encourage the think-aloud process in a natural, conversational manner.

Now you will try a think-aloud. You can use this example. Like last time, you have to pick the two of these things that you think are the most similar. Go ahead and do this task, and when you are finished, let me know.

Once you are done, I will ask you to talk out loud and tell me all of the thoughts that were in your head at each moment. During that phase, if I don’t hear you speaking I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are thinking. I’ll be reminding you to think aloud if you get quiet because I need to hear all of the thoughts you had during the task.

Okay, now you try, you can go ahead and start working on the question. Let me know when you are finished.

Which two organisms below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

(When the student indicates he or she has finished, make sure he or she has circled two answers, and then ask him or her to begin describing his or her thoughts. As necessary, include the appropriate prompting questions, such as the following:)

Okay, so tell me what you were thinking as you were doing this task. What did you think at the beginning, when you started to read the question? … And then what were you thinking? … And what else did you think about? … Anything else? … Can you tell me more about that? … etc.

IIIc. Science ICT Retrospective Think-Aloud: Student Practice II (Optional)

(Use only if you feel the student needs to practice another think-aloud before moving on to the actual questions.)

If the student struggles to think aloud, the interviewer should give the student another opportunity to practice. The interviewer should praise the student for the first attempt regardless of how good it was, for example: “Very good—let’s do another one before we start the real tasks. Are you ready? Here is the next practice question. Remember that after you have finished answering this question I will ask you to talk out loud to tell me what you were thinking all the way through—so you should tell me what you were thinking as you worked through it. So, first of all, go ahead and do the question, and let me know when you have finished.”

As before, during the post-task think-aloud phase, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, section IIe).

Which two of the following objects have the most similar properties? Circle the two that are most similar

Silver coin
Chocolate coin
Gold coin
Blue plastic coin
Brown plastic coin

After the think-aloud phase is finished:

Now that you have practiced, do you feel that you understand how you should talk aloud about the thoughts you had while you were doing the tasks? Is this something you feel okay about doing? [If student says yes:] Good, then let’s begin our study.[If student says no or appears to be hesitant or reluctant, ask him or her to say more about any reservations he or she has, and try to address his or her concerns or uncertainties in a supportive way. If the student indicates he or she does not wish to continue or does not feel comfortable continuing, allow him/her to stop.]

IIId. Science ICT Retrospective Think-Aloud: Starting the Science Tasks

Now we will move on to the actual test questions. Remember, after you have finished answering all the questions on a task, I will ask you to say aloud everything that you remember thinking while you were doing the task, and I may remind you to do that if you go quiet. This task should take about [15 or 30] minutes. Remember, you will not be graded on what you do during the task and there is no right or wrong way to think aloud, as long as you keep telling me your thoughts. Your thoughts will help us make the tasks better. I will also have a few questions after we have finished.

Do you have any questions before we go on? [Answer any questions the student may ask.]

Because the information you provide is so important to us, I am going to be taking notes while you are doing the task and while you are thinking aloud afterwards.

Here is the first task. When you are ready, go ahead and start working on it.

IIIe. Science ICT Retrospective Think-Aloud: Prompts and Questions after Completion of Task

Now we will look at what you did on the task, and I would like you to tell me what you were thinking as we review what you did. We will be recording what you are saying as you reflect on this task. Again, you should feel free to stop at any time, and this information will not be used to grade you. The information will only be used to help improve the test. I will play the task back from the beginning now, and I want you to talk about the thoughts you had as you were working through it. From time to time, I might pause the video of the task playback, to give you enough time to explain what you were thinking at that point or if I need to ask you any extra questions. I will let you know when I am pausing the playback.

Interviewer plays and watches the recording of the task in real time with student, and asks student to describe aloud what he or she was thinking at each point in the task. The interviewer should pause the recording whenever more time is needed, making sure to say “I’ll just pause this for a moment” or “I am pausing the task here,” so that the student understands that the playback has been paused.

If the student is not verbalizing enough, you should offer a verbal “nudge” to remind him or her to keep talking, such as:

What was going on in your head at this time?
I see you were looking at the task [or screen/figure/chart/text]. What were you thinking?
You seemed to be thinking hard—can you tell me what was in your mind right then?

If the student says something about his or her thoughts but it is difficult to tell exactly what he or she means or if he or she begins to talk but doesn’t say very much, and you want him or her to elaborate on what he or she is saying, you can ask for more, for example:

Tell me a bit more about that.
That’s interesting—can you say more about that?
Go on….
Uh huh, okay, and what else were you thinking?

If you need more clarity or a more explicit description of student’s thoughts:

Can you explain what you mean by that?
Can you say a bit more, to help me understand what you were thinking?

IIIf. Science ICT Notes on Student Actions and Behaviors During Tasks

Does the student express signs of confusion, boredom, or excitement?
Does the student use tools provided in the task such as clicking on tabs or buttons to reveal additional or earlier information, digital notepads, digital calculators, or reminders of what to include or think about?
Does the student rapidly move through the screen or take his or her time? Does the student spend a lot of time on a particular aspect?

IV. Science ICT: Post-Think-Aloud Follow-Up Questions (Verbal Probing)

After completing the Think-Aloud process for a task, interviewers will follow-up with a brief period of focused retrospective questioning. The post-task questions will comprise:

One standardized post-task question that all students will be asked following all tasks, which is designed to discover whether the student has prior knowledge of the content.

Up to three additional targeted questions that are task-specific. These will be selected for each task by ETS staff prior to testing. Interviewers will receive the set of questions ahead of time for the specific task they are examining in each cognitive interview.

Standardized Question For all Tasks: Task-Specific Prior Knowledge

Have you studied anything related to this task in school, or have you learned about these things in your own life? [If yes:] Tell me about what you have learned or studied that is related to this task.

Additional Questions: Task-Specific Issues

The purpose of the additional post-task questions is to capture more information on issues such as student actions during the task, particular aspects of science inquiry targeted in the task, and general reflections about the task. Interviewers should have the list of post-task questions ready to be asked when the student finishes the think-aloud portion for each task.

The following list shows some sample question frames that ETS staff will use to build questions related to each task. The specifics of each question will be generated by ETS staff by an informal task analysis which identifies key points in the task for understanding student thinking, or key issues for each task. The post-task probes will focus on three broad types of issues: 1) usability issues; 2) design issues (design features other than usability); and 3) validity issues. Validity-related probes will probably make up the majority of post-task questions, since this is the main focus of the cognitive interviews. These questions will attempt to establish whether specific parts of the task appear to be eliciting the kinds of thinking among students that they are designed to capture. Design-related probes and usability-related probes will focus more on surface features of the task and will probably be used less often, but may sometimes be appropriate if there is concern or interest from ETS staff about a design-related issue in a particular task. The questions for each task will be determined ahead of time by ETS cognitive scientists and science subject experts, in conjunction with assessment designers. They will be compiled and sent by the ETS cognitive interviews administrator to EurekaFacts at least one week in advance of each task being tested, to allow time for the interviewers to become familiar with the probes.

Below is a list of sample question frames. Up to three additional questions similar to these will be generated by ETS and given to the interviewer for each task:

Let’s return to this section where you were asked to make a prediction (interviewer points to particular screen with student looking on). I noticed you did make a prediction—can you tell me more about what you were thinking when you made that prediction?

At this point you were asked to design an investigation. I noticed you selected one set of variables to vary, but others were not varied—can you tell me more about what you were thinking when you made those choices?

Let’s go back to the part where you were asked to use information from the task to explain how you got to your answer or how you made your decision. I noticed you did draw a conclusion—can you tell me more about what you were thinking when you wrote that conclusion?

This button was on the screen to allow you to look at <information> at any time. Tell me about how and when you used this button, or if you didn’t really use it, why was that?

This tab allowed you to go back to <earlier screen> at any time. Tell me about how you used this tab, or if you didn’t really use it, why was that?

This task had quite a lot of things that you could control and interact with. Did you understand how to use all of them? Was there anything you had trouble with, or anything you thought wasn’t obvious enough?

When you were shown <information/graph/image> at this point, what were you thinking? What was in your mind?

When you had to put the earlier information together to <draw a conclusion/make a decision> how did you go about doing that?

At this point, you needed to <make sense of the data/interpret what you had found>. Did you learn how to do anything like in school? Have you been taught how to do this, or learned a particular way to do this?

If students’ answers are unclear or not very explicit, the interviewer should use the same kinds of prompts as indentified in the think-aloud protocol to elicit more information, for example:

Can you say a bit more about that?

What else were you thinking?

Anything else?

That’s interesting, tell me more about that.

Optional: Interviewer-Generated Questions

In addition, if the interviewer noted some especially interesting behavior during the task, he or she can ask additional ad hoc questions about these specific instances, referring to notes made during the task. Interviewers should use their judgment about the need for and value of additional questions, according to the student’s behavior during the task, and according to time constraints.

Some examples of potential interviewer-generated ad hoc questions are:

I noticed on this part of the task that you paused for a while. Can you tell me more about why you paused and what you were thinking at this point?

I noticed on this part of the task that you did not use the prompt next to the response box. Can you tell me more about why you did not use that prompt?

I noticed that on this part of the task you spent some time looking at/back-and-forth between the <graph/table/question>). Can you tell me more about what was going on there?

V. Debriefing and Thank You for Cognitive Interview

Thank participant for his/her time. Provide gift card.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the task?

Is there anything you would like to ask me about what we did today? [Answer student questions]

Thank you for helping us to improve our test.

1 60 minutes for background questionnaire cognitive interviews and SICT play testing

90 minutes for SICT cognitive interviews

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Science Volume II Cog Lab Protocols
Author	BlairJ
File Modified	0000-00-00
File Created	2021-01-31