National Assessment of Educational Progress (NAEP) Pretesting of Survey and Cognitive Items for Pilot in 2017 and 2018 Updated

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average xx¹ minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission of this information collection, please write to: National Assessment of Educational Progress, National Center for Education Statistics, 1990 K Street, N.W., Washington, DC 20006.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

OMB No. 1850-0803 Approval Expires 09/30/2016

Part B. Student, Teacher, and School Questionnaire Cognitive Interviews

The purpose of the cognitive interviews is to inform the development of the core student, teacher, and school administrator questionnaires prior to pilot testing. Below are the probes that students, teachers, and school administrators will be asked during the cognitive interview. Please note that some probes may not be addressed if time does not permit.

Interviewer Welcome Script and Assent/Consent

The following script does not have to be read verbatim. You, as the interviewer, should be familiar enough with the script to introduce the participant to the cognitive interview process in a conversational manner. The text in italics is suggested content for you to become thoroughly familiar with in advance. You should project a warm and reassuring manner toward the participant in order to develop a friendly rapport and should use conversational language throughout the interview.

After answering questions and giving further explanation, begin the interview with the first item.

Note: Students, teachers, and school administrators will be administered all items and probes. The duration of each cognitive interview will be 90 minutes.

Student Welcome Script:

Hello, my name is <name>.

Thank you for agreeing to participate in our study. Your help is very important to us. Let me tell you a bit more about what we are going to do today.

I have some questions that students in many different schools across the country will be asked to answer about themselves and their families. The questions will be part of the National Assessment of Educational Progress (NAEP).

Some of the questions may be difficult to understand or answer. Our goal is to make these questions better so that other students can easily understand them. That is why we need your feedback.

We want you to help us find out which questions are easy to understand and which are difficult to understand. There are no right or wrong answers. Your feedback will help the National Center for Education Statistics (NCES), which is part of the United States Department of Education, make these questions clearer for students like you.

This interview is being audio recorded so that researchers can review the recordings later. No one else will hear the recording. Participant responses are voluntary and will be used only for research purposes. They will not be disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C. § 9573].

Do you have any questions?

If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

Teacher Welcome Script

Hello, my name is <name>.

Thank you for coming here today to help us.

I have some questions that teachers in many different schools will be asked to answer about themselves, their instructional practices, and their schools. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. There are no wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), part of the United States Department of Education, make these questions clearer for teachers like you. This interview is being audio recorded so that researchers can review the recordings later. Participant responses are voluntary, will be used only for research purposes, and will not be disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C. § 9573].

Do you have any questions?

If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

School Administrator Welcome Script

Hello, my name is <name>.

Thank you for coming here today to help us.

I have some questions that school administrators in many different schools will be asked to answer about their school policies and characteristics. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. There are no wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), part of the United States Department of Education, make these questions clearer for school administrators like you. This interview is being audio recorded so that researchers can review the recordings later. Participant responses are voluntary, will be used only for research purposes, and will not be disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C. § 9573].

Do you have any questions?

If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

Instructions and Generic Probes

The interviewer should ask the participant to read and answer the question on the tablet screen. There can be one discrete question (multiple choice or free response) or one matrix question on a screen. The participant is not supposed to read the question aloud, but should work on the question in the same fashion he/she would during a NAEP test administration/survey completion. If a participant indicates they do not know a word or phrase the interviewer can provide/read words for the participant.

Generic Probes for Discrete Items (Multiple Choice and Free Response)

No.	Probe	Instructions for Interviewer
1	Can you tell me, in your own words, what the question is asking?	Ask this probe AFTER the participant has answered the question on the screen.
2	Were there any words or parts of this question that were confusing?  Yes  No	Ask this probe for all discrete questions.
3	What did you find confusing? What could we do to make the question less confusing?	Ask this probe only if the participant answered YES to the previous question.
4	Would you say it was very easy, easy, difficult, or very difficult to answer this question?  Very Easy  Easy  Difficult  Very Difficult	Ask this probe for all discrete questions.
5	How could we make it easier to answer this question?	Ask this probe only if the participant answered DIFFICULT or VERY DIFFICULT to the previous question.

Generic Probes for Matrix Items

No.	Probe	Instructions for Interviewer
1	Looking back at the first part of the question, can you tell me in your own words what this question is asking you to do?	Ask this probe AFTER the participant has answered the entire matrix question on the screen. This probe is specifically for the “stem” of the item, not the entire matrix item or a specific option/sub-item.
2	Can you tell me, in your own words, what [option a., b., c., etc.] means to you?	Ask this probe for all options/sub-items. Please note probes 2-6 should be asked together for one option/sub-item before moving on to the next option/sub-item.
3	Did you find any words or part of [option a., b., c., etc.] confusing?  Yes  No	Ask this probe for all options/sub-items.
4	What did you find confusing? What could we do to make [option a., b., c., etc.] less confusing?	Ask both probes only if the participant answered YES to the previous question.
5	Would you say answering [option a., b., c., etc.] was very easy, easy, difficult, or very difficult?  Very Easy  Easy  Difficult  Very Difficult	Ask this probe for all options/sub-items.
6	How could we make it easier to answer [option a., b., c., etc.]?	Ask this probe only if the student answered DIFFICULT or VERY DIFFICULT to the previous question.

Core: Items and Probes

Teacher Items – Grade 8

[Grade 8 Core: Teacher #1]

TBD

Have you been awarded tenure by the school, district, or diocese where you currently teach?

Yes
No
My school, district, or diocese does not award tenure.

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required
2	Who awards tenure to the teachers at your school?	Specific	Required

School Items – Grade 8

[Grade 8 Core: School #1]

TBD

Does your school, district, or diocese offer tenure to teachers?

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required
2	Who awards tenure to the teachers at your school?	Specific	Required
3	Are there other entities that award tenure that should be included in the wording of this item?	Specific	Required

Mathematics: Items and Probes

Student Items – Grades 8

[Grade 8 Mathematics: Student #1]

TBD

How often do you play digital games (for example, apps, video games, or computer games) outside of school that involve math?

Never
About once or twice a year
About once or twice a month
About once or twice a week
Every day or almost every day

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Mathematics: Student #2]

TBD

How often do you use a computer, tablet, smartphone or gaming console outside of school to play educational games that involve math?

Never
About once or twice a year
About once or twice a month
About once or twice a week
Every day or almost every day

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required
2	Which games did you think of when you read the term "educational games”?	Specific	Required
3	Can you think of examples of educational games that use math?	Specific	Required

[Grade 8 Mathematics: Student #3]

TBD

How often do you use a computer, tablet, smartphone or gaming console outside of school to play popular games that involve math?

Never
About once or twice a year
About once or twice a month
About once or twice a week
Every day or almost every day

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required
2	Which games did you think of when you read the term "popular games”?	Specific	Required
3	Can you think of examples of popular games that use math?	Specific	Required

[Grade 8 Mathematics: Student #4]

TBD

Have you ever participated in a math study group outside of school?

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required
2	Can you give an example of a math study group outside of school?	Specific	Required

[Grade 8 Mathematics: Student #5]

TBD

Over the past seven days, how many days have you participated in a math study group outside of school? Enter the number of days.

_________

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Mathematics: Student #6]

TBD

Have you ever participated in a math study group at school?

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required
2	Can you give an example of a math study group in school?	Specific	Required

[Grade 8 Mathematics: Student #7]

TBD

Over the past seven days, how many days have you participated in a math study group at school? Enter the number of days.

_________

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Mathematics: Student #8]

TBD Thinking about math, do you think that you would be able to do each of the following? Do not actually solve the problems. Select one answer choice on each row.
	I definitely can’t	I probably can’t	Maybe	I probably can	I definitely can
Draw a bar graph to represent the number of desks, chairs, and tables in your classroom	A	B	C	D	E	TBD
Use a box plot to represent the height, in inches, of all of the students in your class	A	B	C	D	E	TBD
Determine the probability of picking a blue marble without looking if you know the number of marbles of each color in a bag	A	B	C	D	E	TBD
List all of the different possible outcomes when a coin is flipped three times	A	B	C	D	E	TBD
Create an expression that represents the average number of miles you run in a week if you run 100 miles in w weeks	A	B	C	D	E	TBD
Create an expression to show how old you will be in n years if you are 8 years old today	A	B	C	D	E	TBD
Create an expression that shows number of wheels on n wagons if each wagon has 4 wheels	A	B	C	D	E	TBD
Explain how you know the number of books you can buy if you know the price of each book and you have $35	A	B	C	D	E	TBD

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Mathematics: Student #9]

TBD Thinking about math, do you think that you would be able to do each of the following? Select one answer choice on each row.
	I definitely can’t	I probably can’t	Maybe	I probably can	I definitely can
Decide if your classmate's math work is correct or incorrect	A	B	C	D	E	TBD
Give an example to show that a math statement is false	A	B	C	D	E	TBD

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Mathematics: Student #10]

TBD Thinking about math, do you think that you would be able to do each of the following? Select one answer choice on each row.
	I definitely can’t	I probably can’t	Maybe	I probably can	I definitely can
Use correct words, symbols, and notation to communicate ideas about math	A	B	C	D	E	TBD
Use correct math notation when showing your work	A	B	C	D	E	TBD
Explain to your classmate how you solved a math problem	A	B	C	D	E	TBD
Use definitions of geometric shapes to support an argument	A	B	C	D	E	TBD

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do you understand what the word "notation" means in option b?	Specific	Required

Teacher Items – Grades 8

[Grades 8 Mathematics: Teacher #1]

TBD In your mathematics class this year, do you use any of the following instructional materials? Select one circle in each row.
	Yes	No, I do not prefer to use this resource	No, this resource is not available to me.
Textbooks provided by your district or school	A	B	C	TBD
Other materials provided by your district or school	A	B	C	TBD
Materials found on the Internet	A	B	C	TBD
Digital manipulatives	A	B	C	TBD
Digital games	A	B	C	TBD
Interactive whiteboard	A	B	C	TBD
Physical manipulatives (for example, ruler, protractor, compass)	A	B	C	TBD
Flip chart	A	B	C	TBD
Paper and pencil workbooks	A	B	C	TBD
Other materials (Please specify): _________________	A	B	C	TBD

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	When answering this question, what context were you thinking about?	Specific	Required
3	Are there any materials that are not listed, that should be?	Specific	Required
4	Are there any materials listed that can be removed?	Specific	Required

[Grades 8 Mathematics: Teacher #2]

TBD Thinking about your eighth-grade mathematics classes this year, how much emphasis did you place on teaching your students each of the following? Select one circle in each row.
	No emphasis	Very little emphasis	Some emphasis	Quite a bit of emphasis	A lot of emphasis
Use clear and precise language when students are discussing their problem solving and reasoning	A	B	C	D	E	TBD
Use models to explain calculations	A	B	C	D	E	TBD
Represent a problem situation in multiple ways including numbers, words, pictures, and charts	A	B	C	D	E	TBD
Use models to examine real life and mathematical examples	A	B	C	D	E	TBD

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	For sub-item b, When thinking about the word "language", were you thinking of "oral" or "written" language?	Specific	Required

[Grades 8 Mathematics: Teacher #3]

TBD Thinking about your eighth-grade mathematics classes this year, how often do you incorporate the following practices when teaching your students? Select one circle in each row.
	Never or hardly ever	Once in a while	Sometimes	Often	Always or almost always
Pose questions during a lesson that will help students to reason and make sense of mathematical connections and relationships	A	B	C	D	E	TBD
Use overall learning goals for student learning to guide instructional decisions	A	B	C	D	E	TBD
Provide opportunities for students to productively struggle with mathematical ideas and relationships	A	B	C	D	E	TBD
Build procedural understanding to encourage the use of multiple problem-solving strategies	A	B	C	D	E	TBD

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grades 8 Mathematics: Teacher #4]

TBD Thinking about your eighth-grade mathematics classes this year, how much emphasis did you place on teaching your students each of the following? Select one circle in each row.
	No emphasis	Very little emphasis	Some emphasis	Quite a bit of emphasis	A lot of emphasis
Use definitions and notation precisely	A	B	C	D	E	TBD
Justify and explain their reasoning	A	B	C	D	E	TBD
Identify and correct flawed mathematical reasoning	A	B	C	D	E	TBD
Construct arguments using tools, graphs, or diagrams	A	B	C	D	E	TBD
Make, test, and validate conjectures	A	B	C	D	E	TBD
Use examples or counterexamples to support or refute a mathematical conjecture	A	B	C	D	E	TBD
Engage in deductive reasoning and informal proof	A	B	C	D	E	TBD

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	For option a, would this question be easier to answer if the word "precisely" were changed to "fluently"?	Specific	Required

[Grade 8 Mathematics: Teacher #5]

TBD Thinking about your eighth-grade mathematics classes this year, how much emphasis did you place on teaching your students each of the following? Select one circle in each row.
	No emphasis	Very little emphasis	Some emphasis	Quite a bit of emphasis	A lot of emphasis
Evaluate a problem-solving process	A	B	C	D	E	TBD
Create equations	A	B	C	D	E	TBD
Relate what your students know to the real world and make sense of it mathematically	A	B	C	D	E	TBD
Use appropriate terminology when referring to the number system, functions, geometric figures, and data displays	A	B	C	D	E	TBD
Evaluate the conclusions of other students	A	B	C	D	E	TBD
Examine patterns in tables and graphs to describe relationships	A	B	C	D	E	TBD

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

School – Grade 8

[Grade 8 Mathematics: School #1]

TBD To what extent is your school’s mathematics program structured according to the following resources? Select one circle in each row.
	Not at all	Small Extent	Moderate extent	Large extent
State curriculum standards or frameworks	A	B	C	D	TBD
District curriculum standards or curriculum guides	A	B	C	D	TBD
Results from district assessments	A	B	C	D	TBD
Results from state assessments	A	B	C	D	TBD
Results from school-level assessments	A	B	C	D	TBD
Recommendations from school mathematics department or teachers	A	B	C	D	TBD
Discretion of individual teachers	A	B	C	D	TBD
Commercially designed programs	A	B	C	D	TBD
Internet resources	A	B	C	D	TBD

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Are there any resources that are not listed, that should be?	Specific	Required

Reading Items and Probes

Student Items – Grade 8

[Grade 8 Reading: Student #1]

VH260228

In this school year, how often do you borrow reading materials (such as books or magazines) from your classroom library, school library, or media center?

Never
About once or twice a year
About once or twice a month
About once or twice a week
Every day or almost every day

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required
2	In this question, what does the phrase “classroom library” mean to you?	Specific	Required

[Grade 8 Reading: Student #2]

VH260407

On a typical school day, how much time do you use a computer or other digital device to do your English/language arts schoolwork and homework?

Less than 30 minutes
30 minutes up to 1 hour
1 up to 2 hours
2 up to 3 hours
3 up to 4 hours
More than 4 hours

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required
2	In this question, what do you think is meant by “digital device”?	Specific	Required

Teacher – Grade 8

[Grade 8 Reading: Teacher #1]

VH334360 When you teach English/language arts, how often do you use the following strategies? Select one circle in each row.
	Never or hardly ever	Once in a while	Sometimes	Often	Always or almost always
I teach reading as a whole-class activity.	A	B	C	D	E	VH334361
I create student groups with the same achievement level.	A	B	C	D	E	VH334362
I create student groups with different achievement levels.	A	B	C	D	E	TBD
I create groups by random assignment.	A	B	C	D	E	VH334363
I allow students to choose their own groups.	A	B	C	D	E	VH334368
I use individualized instruction for reading.	A	B	C	D	E	VH334364
I ask students to work independently on an assigned plan or goal.	A	B	C	D	E	TBD
I ask students to work independently on a goal they choose themselves.	A	B	C	D	E	TBD
Other (specify): _____	A	B	C	D	E	VH334367

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	In options g and h, what does the word “goal” mean to you?	Specific	Required
3	Do you think the word “goal” is too vague?	Specific	Required
4	If yes: How could we make this word more specific?	Specific	Conditional
5	If the participant choose B-E for option i: In option i, you included an additional strategy. Could you please explain or define that strategy?	Specific	Conditional
6	Do you think there are any strategies missing from this item that would be important to include?	General	Required

[Grade 8 Reading: Teacher #2]

TBD In your eighth-grade English/language arts class this year, how often do your students do each of the following? Select one circle in each row.
	Never	About once or twice a year	About once or twice a month	About once or twice a week	Every day or almost every day
Build and practice vocabulary	A	B	C	D	E	TBD
Increase reading fluency	A	B	C	D	E	TBD
Increase reading comprehension	A	B	C	D	E	TBD
Access reading-related websites	A	B	C	D	E	TBD
Conduct research for reading projects	A	B	C	D	E	TBD

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do these response options make sense for the question you are being asked?	Specific	Required

School – Grade 8

[Grade 8 Reading: School #1]

TBD In addition to English/language arts teachers, does your school have the following personnel to assist with English/language arts class instruction for eighth-grade English language learners (ELL)? Select one circle in each row.
	Yes	No
Certified ELL/bilingual education teachers	A	B	TBD
Reading specialists or literacy coaches	A	B	TBD
Speech pathologists	A	B	TBD
Paraprofessionals or teacher aides who are trained to work with students who are ELL	A	B	TBD
Paraprofessionals or teacher aides who are not trained to work with students who are ELL	A	B	TBD
Parent volunteers	A	B	TBD

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	In your school, do reading specialists support students only, teachers only, or both students and teachers?	General	Required
3	In your school, do literacy coaches support students only, teachers only, or both students and teachers?	General	Required
4	Are there any additional personnel that should be included in this item?	General	Required

Note: If the participant selects “No” for option/sub-item “B” please skip Grade 8 Reading: School #2.

[Grade 8 Reading: School #2]

TBD To what extent are each of the following a responsibility of the reading specialist(s) available to eighth-grade students at your school? Select one circle in each row.
	Not at all	Small extent	Moderate extent	Large extent
Provide one-on-one help to students on various reading topics	A	B	C	D	TBD
Provide one-on-one help to students at various achievement levels	A	B	C	D	TBD

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	In option a, what does the phrase “various reading topics” mean to you?	Specific	Required
3	Do you think the phrase “various reading topics” is too vague?	Specific	Required
4	If yes: How could we make this phrase more specific?	Specific	Conditional
5	Are there any additional responsibilities that should be included in this item?	General	Required

Social Sciences Items and Probes

Civics Student Items – Grade 8

[Grade 8 Social Sciences: Civics Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	How would you describe “civics or United States government”? What types of information would a student learn in this type of class?	Specific	Required
3	Why might a student answer “I don’t know” to this question? Do you think this option is necessary? Would “I don’t remember” be a more accurate option?	Specific	Required

[Grade 8 Social Sciences: Civics Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	How would you describe “social studies”? What types of information would a student learn in this type of class?	Specific	Required
3	Do these response options make sense for the question that is being asked? How would you answer this question if you studied a certain topic a lot for only a couple of weeks? The question below presents the same question with slightly different response options. Which of these two questions do you find easier to answer? Why?	Specific	Required

[Grade 8 Social Sciences: Civics Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	d. Can you describe, in your own words, what "civics and United State government topics" means to you? Can you provide an example?	Specific	Required

[Grade 8 Social Sciences: Civics Student #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Civics Student #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	a. Can you describe, in your own words, what "position" means to you? Is this word the same or different from the words "opinions" or "thoughts"? Would this question be easier to answer if these words were used instead?	Specific	Required
3	d. Can you describe, in your own words, what an "issue in the community" means to you? What would be an example of an "issue in the community"?	Specific	Required

[Grade 8 Social Sciences: Civics Student #6]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/ Conditional

For generic probes for matrix items, see page 4

General

Required

Do these response options make sense for the question that is being asked? How would you answer this question if you studied a certain topic a lot for only a couple of weeks?

The question below presents the same question with slightly different response options. Which of these two questions do you find easier to answer? Why?

Specific

Required

[Grade 8 Social Sciences: Civics Student #7]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Civics Student #8]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do these response options make sense for the question you are being asked? The question below presents the same question with slightly different response options. Which of these two questions do you find easier to answer? Why?	Specific	Required
3-5	Sub-items a. – c. Can you describe a recent assignment of this kind?	Specific	Required

[Grade 8 Social Sciences: Civics Student #9]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do these response options make sense for the question you are being asked? The question below presents the same question with slightly different response options. Which of these two questions do you find easier to answer? Why?	Specific	Required
3-4	Sub-items a. – b. Can you describe a recent assignment of this kind?	Specific	Required

[Grade 8 Social Sciences: Civics Student #10]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Civics Student #11]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	b. Can you describe, in your own words, what "multimedia" means to you?	Specific	Required

[Grade 8 Social Sciences: Civics Student #12]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Civics Student #13]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Civics Student #14]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Civics Student #15]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	When answering this question, do you consider school clubs to be an "outside of school" activity?	Specific	Required
3	Do these response options make sense for the question you are being asked? The questions below (Alternative Questions A and B) present the same question with slightly different response options. Which of these three questions do you find easiest to answer? Why?	Specific	Required
4	c. Can you describe, in your own words, what “public officials” are?	Specific	Required
5	d. Can you describe, in your own words, what a “community” is?	Specific	Required
6	e. Can you describe, in your own words, what “public affairs” means?	Specific	Required
7	g. Can you describe, in your own words, what a “community organization” is? Can you provide an example?	Specific	Required

Alternative Question A

Alternative Question B

[Grade 8 Social Sciences: Civics Student #16]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Civics Student #17]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Civics Student #18]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	What type of “community” did you think about when you were answering these questions?	Specific	Required

[Grade 8 Social Sciences: Civics Student #19]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	What type of “community” did you think about when you were answering these questions?	Specific	Required

[Grade 8 Social Sciences: Civics Student #20]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Civics Student #21]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Civics Student #22]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	e. Can you describe, in your own words, what “public affairs” means?

[Grade 8 Social Sciences: Civics Student #23]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Civics Student #24]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	a. Can you describe, in your own words, what "social issues" means to you? Can you provide another example of a "social issue"?	Specific	Required
3	e. Can you describe, in your own words, what “facilitate” means to you?	Specific	Required

[Grade 8 Social Sciences: Civics Student #25]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Can you describe, in your own words, what "prepared" means to you? Do you include only schoolwork in this question, or are there other things you need to be prepared for class?	Specific	Required

Geography Student Items – Grade 8

[Grade 8 Social Sciences: Geography Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	How would you describe “geography”? What types of information would a student learn in this type of class?	Specific	Required
3	Why might a student answer “I don’t know” to this question? Do you think this option is necessary? Would “I don’t remember” be a more accurate option?	Specific	Required

[Grade 8 Social Sciences: Geography Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	How would you describe “social studies”? What types of information would a student learn in this type of class?	Specific	Required
3	Do these response options make sense for the question that is being asked? How would you answer this question if you studied a certain topic a lot for only a couple of weeks?	Specific	Required

[Grade 8 Social Sciences: Geography Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	d. Can you describe, in your own words, what “geography topics” means to you? Can you provide an example?	Specific	Required

[Grade 8 Social Sciences: Geography Student #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Geography Student #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Geography Student #6]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/ Conditional

For generic probes for matrix items, see page 4

General

Required

Do these response options make sense for the question that is being asked? How would you answer this question if you studied a certain topic a lot for only a couple of weeks?

The question below presents the same question with slightly different response options. Which of these two questions do you find easier to answer? Why?

Specific

Required

[Grade 8 Social Sciences: Geography Student #7]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	a. When answering this question, did you think of people who move to different parts of the world by choice? Did you think of people who are forced to move?	Specific	Required

[Grade 8 Social Sciences: Geography Student #8]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2-4	Sub-items a. – c. Can you describe a recent assignment of this kind?	Specific	Required

[Grade 8 Social Sciences: Geography Student #9]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do these response options make sense for the question you are being asked? The question below presents the same question with slightly different response options. Which of these two questions do you find easier to answer? Why?	Specific	Required
3-4	Sub-items a. – b. Can you describe a recent assignment of this kind?	Specific	Required

[Grade 8 Social Sciences: Geography Student #10]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Geography Student #11]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	f. Can you describe, in your own words, what "source" means to you? What is an example of a "source"?	Specific	Required

[Grade 8 Social Sciences: Geography Student #12]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Geography Student #13]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Geography Student #14]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Geography Student #15]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	When answering this question, did you consider school clubs to be an "outside of school" activity?
3	a. Can you give examples of other online mapping tools? Do you think there are differences between an "online map" and an "online mapping tool"? If so, how are they different?	Specific	Required
4	b. Can you describe, in your own words, what "physical systems" are? Can you provide other examples of "physical systems"?	Specific	Required
5	c. Can you describe, in your own words, what "geographic issue" means to you? Can you provide an example?	Specific	Required
6	d. Can you describe, in your own words, what “environmental conservation” means to you?	Specific	Required
7	f. Can you give examples of ways of looking up geographic information on the web?	Specific	Required

[Grade 8 Social Sciences: Geography Student #16]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Geography Student #17]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Geography Student #18]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Geography Student #19]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Geography Student #20]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Geography Student #21]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Can you describe, in your own words, what "prepared" means to you? Do you include only schoolwork in this question, or are there other things you need to be prepared for class?	Specific	Required

U.S. History Student Items – Grade 8

[Grade 8 Social Sciences: U.S. History Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	How would you describe “United States history”? What types of information would a student learn in this type of class?	Specific	Required
3	Why might a student answer “I don’t know” to this question? Do you think this option is necessary? Would “I don’t remember” be a more accurate option?	Specific	Required

[Grade 8 Social Sciences: U.S. History Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	How would you describe “social studies”? What types of information would a student learn in this type of class?	Specific	Required
3	Do these response options make sense for the question that is being asked? How would you answer this question if you studied a certain topic a lot for only a couple of weeks?	Specific	Required

[Grade 8 Social Sciences: U.S. History Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	c. Can you describe, in your own words, what “primary sources” are?	Specific	Required
3	e. Can you describe, in your own words, what "United States history topics" means to you? Can you provide an example?	Specific	Required

[Grade 8 Social Sciences: U.S. History Student #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: U.S. History Student #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	b. Can you describe, in your own words, what "historical debate" means to you?	Specific	Required

[Grade 8 Social Sciences: U.S. History Student #6]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do these response options make sense for the question that is being asked? How would you answer this question if you studied a certain topic a lot for only a couple of weeks? The question below presents the same question with slightly different response options. Which of these two questions do you find easier to answer? Why?	Specific	Required
3	b. Can you describe, in your own words, what "themes" means to you?	Specific	Required
4	e. Can you provide an example of "how different events that happened at the same time are related to each other"? Would you have answered this question differently if another example were provided, or if no example were provided?	Specific	Required
5	f. Can you provide an example of "how historical events were affected by when and where it took place"? Would you have answered this question differently if another example were provided, or if no example were provided?	Specific	Required

[Grade 8 Social Sciences: U.S. History Student #7]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: U.S. History Student #8]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do these response options make sense for the question you are being asked? The question below presents the same question with slightly different response options. Which of these two questions do you find easier to answer? Why?	Specific	Required
3-5	Sub-items a. – c. Can you describe a recent assignment of this kind?	Specific	Required

[Grade 8 Social Sciences: U.S. History Student #9]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do these response options make sense for the question you are being asked? The question below presents the same question with slightly different response options. Which of these two questions do you find easier to answer? Why?	Specific	Required
3-4	Sub-items a. – b. Can you describe a recent assignment of this kind?	Specific	Required

[Grade 8 Social Sciences: U.S. History Student #10]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: U.S. History Student #11]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	e. Can you describe, in your own words, what "sources" means to you?	Specific	Required

[Grade 8 Social Sciences: U.S. History Student #12]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: U.S. History Student #13]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: U.S. History Student #14]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: U.S. History Student #15]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	When answering this question, did you consider school clubs to be an "outside of school" activity?	Specific	Required
3	Do these response options make sense for the question you are being asked? The questions below (Alternative Questions A and B) present the same question with slightly different response options. Which of these three questions do you find easiest to answer? Why?	Specific	Required
4	c. Can you describe, in your own words, what a video game about history is? Can you provide an example?	Specific	Required
5	f. What types of websites or online resources would you use to research historical topics?	Specific	Required

Alternative Question A

Alternative Question B

[Grade 8 Social Sciences: U.S. History Student #16]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: U.S. History Student #17]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: U.S. History Student #18]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: U.S. History Student #19]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: U.S. History Student #20]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: U.S. History Student #21]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Can you describe, in your own words, what "prepared" means to you? Do you include only schoolwork in this question, or are there other things you need to be prepared for class?	Specific	Required

[Grade 8 Social Sciences: U.S. History Student #22]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: U.S. History Student #23]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Compared to the previous question you answered [Ush30_000], which of these two questions do you find easier to think about and answer? Why? Would you still feel this way if both questions had the same number of statements (for example, if the current question had 6 statements to answer instead of 3)?	Specific	Required
3	The previous question [Ush30_000] asks you about understanding historical events. This question asks you about understanding other’s point of view. Do you find it easier or more difficult to answer questions about understanding history compared to answering questions about understanding other people? Or are they about the same?	Specific	Required

Teacher Items – Grade 8

[Grade 8 Social Sciences: Teacher #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required
2	How would you describe "social studies"? What types of information would a student learn in this type of class?	Specific	Required

[Grade 8 Social Sciences: Teacher #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #4]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/ Conditional

For generic probes for discrete items, see page 4

General

Required

Are there other bases for creating instructional groups that are missing from this item?

If yes: Can you provide examples?

[Grade 8 Social Sciences: Teacher #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	a. How would you describe "civics or United States government"? What types of information would a student learn in this type of class?	Specific	Required
3	b. How would you describe "geography"? What types of information would a student learn in this type of class?	Specific	Required
4	c. How would you describe "United States history"? What types of information would a student learn in this type of class?	Specific	Required

[Grade 8 Social Sciences: Teacher #6]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do you feel that you have the necessary information to answer this question? Do you think school administrators in your school could better answer this question?	Specific	Required

[Grade 8 Social Sciences: Teacher #7]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do you feel that you have the necessary information to answer this question? Do you think school administrators in your school could better answer this question?	Specific	Required

[Grade 8 Social Sciences: Teacher #8]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #9]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #10]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #11]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #12]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	d. Can you describe, in your own words, what "present" means to you? Would you answer this question differently if it were worded, "Lecture about social studies content to students"? Would the question be easier or more difficult to answer?	Specific	Required

[Grade 8 Social Sciences: Teacher #13]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #14]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #15]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #16]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #17]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #18]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #19]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #20]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	b. Can you describe, in your own words, what "theme" means to you?	Specific	Required
3	e. Can you provide an example of "how different events that happened at the same time are related to each other"? Would you have answered this question differently if another example were provided, or if no example were provided?	Specific	Required
4	f. Can you provide an example of "how historical events were affected by when and where it took place"? Would you have answered this question differently if another example were provided, or if no example were provided?	Specific	Required

[Grade 8 Social Sciences: Teacher #21]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #22]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/ Conditional

For generic probes for matrix items, see page 4

General

Required

Do these response options make sense for the question that is being asked? How would you answer this question if you taught a certain topic a lot for only a couple of weeks?

The question below presents the same question with slightly different response options. Which of these two questions do you find easier to answer? Why?

Specific

Required

[Grade 8 Social Sciences: Teacher #23]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/ Conditional

For generic probes for matrix items, see page 4

General

Required

Do these response options make sense for the question that is being asked? How would you answer this question if you taught a certain topic a lot for only a couple of weeks?

The question below presents the same question with slightly different response options. Which of these two questions do you find easier to answer? Why?

Specific

Required

[Grade 8 Social Sciences: Teacher #24]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #25]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #26]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #27]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #28]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #29]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #30]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #31]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #32]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #33]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #34]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #35]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #36]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #37]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #38]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #39]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required
2	Can you describe, in your own words, what "historical thinking skills" means to you?	Specific	Required

[Grade 8 Social Sciences: Teacher #40]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required
2	Can you describe, in your own words, what "civic participatory skills" means to you?	Specific	Required

[Grade 8 Social Sciences: Teacher #41]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for discrete items, see page 4	General	Required
2	Can you describe, in your own words, what “geographic skills" means to you?	Specific	Required

[Grade 8 Social Sciences: Teacher #42]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: Teacher #43]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

School Items – Grades 8

[Grade 8 Social Sciences: School #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	a. How would you describe “civics or United States government”? What types of information would a student learn in this type of class?	Specific	Required
3	b. How would you describe “geography”? What types of information would a student learn in this type of class?	Specific	Required
4	c. How would you describe “United States history”? What types of information would a student learn in this type of class?	Specific	Required
5	d. How would you describe “social studies”? What types of information would a student learn in this type of class?	Specific	Required

[Grade 8 Social Sciences: School #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do you feel that you have the necessary information to answer this question? Do you think teachers in your school could better answer this question?	Specific	Required

[Grade 8 Social Sciences: School #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do you feel that you have the necessary information to answer this question? Do you think teachers in your school could better answer this question?	Specific	Required

[Grade 8 Social Sciences: School #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do you feel that you have the necessary information to answer this question? Do you think teachers in your school could better answer this question?	Specific	Required

[Grade 8 Social Sciences: School #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required

[Grade 8 Social Sciences: School #6]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do you feel that you have the necessary information to answer this question? Do you think teachers in your school could better answer this question?	Specific	Required

[Grade 8 Social Sciences: School #7]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	Do you feel that you have the necessary information to answer this question? Do you think teachers in your school could better answer this question?	Specific	Required

[Grade 8 Social Sciences: School #8]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	How would you describe "social studies"? What types of information would a student learn in this type of class?	Specific	Required

[Grade 8 Social Sciences: School #9]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/ Conditional
1	For generic probes for matrix items, see page 4	General	Required
2	b. Can you describe, in your own words, what "service-learning" means? If yes response option is selected: Can you give an example of a class with a service-learning component that your school offers?	Specific	Required
3	c. Can you describe, in your own words, what "community service" means? If yes response option is selected: Can you give an example of a club with a community service component that your school offers?	Specific	Required

Part C. Mathematics and Reading Discrete Item Pretesting

Playtesting Protocol

Introduction to Study and Setup of Item Review Script

Text written in italics is to be read aloud by the staff facilitator.

NOTE: The facilitator should not read the script word-for-word, but should be familiar enough with its contents to conduct the playtesting sessions in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular items being studied.

Hello, my name is ______ and I work for [indicate organization]. I am here with my colleagues [introduce colleagues]. It’s nice to meet you, and thank you very much for helping us out today.

Create small talk to build rapport with the student(s) by asking a question, such as:

• What is your favorite subject in school?

When the student(s) responds, follow up with 2 or 3 questions to get the student used to talking, such as:

• That’s interesting—why do you enjoy <subject> so much?

• What are you studying in <subject> at the moment?

• And what’s the best thing you have studied in <subject> so far?

Good. Well, I think you’ll enjoy what we are going to be doing today.

We are developing a new set of tests in [mathematics OR reading] for a program funded by the federal government called the National Assessment of Educational Progress (NAEP). NAEP is a test taken by students in grades 4, 8, and 12 in the United States. You will help us develop better tests for other students. We are going to ask for your opinions about some interactive questions that we are creating for a NAEP [mathematics OR reading] test. Right now we have some early drafts of what these questions might look like. You are here to give us your thoughts about these questions. We’d like to hear about what you think is interesting, what you think is not interesting, and if you think any part of a question is confusing.

[For students working in pairs or groups]:

My colleagues and I will take notes on what you are saying about the questions. Please be as honest as possible, talk freely with each other, and don’t worry if you find something confusing. You are not being graded on anything; you are helping us by giving us your opinions and thoughts about these questions. There are no right or wrong thoughts.

[For students working on their own]:

My colleagues and I will take notes on what you are saying about the questions. Please be as honest as possible, talk freely with me, and don’t worry if you find something confusing. You are not being graded on anything; you are helping us by giving us your opinions and thoughts about these questions. There are no right or wrong thoughts.

What you say will not be used in any way that could identify you, except as required by law [if needed: Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573]. We will look at what you say later, but only so that we can understand how our test is working and how we can make it better. Overall, this session should take about [indicate correct length] minutes.

As you work through these questions, we want you to talk with each other or [for students working on their own] with me as you go along. We might also ask you a few questions now and then.

Do you have any questions?

After answering any questions and giving further explanation, continue with the activity. If a student is no longer interested in participating, thank the student for his/her time and end the activity for that student.

The facilitators will generally observe the students with little interruption and will listen to the conversation that occurs between group members or observations students make to the facilitator for insights into what the students are thinking about the questions and any associated stimulus. Intermittently, as appropriate, the facilitators can ask targeted questions such as those in this list:

Is this question passage interesting? Why?/Why not?
Are there any questions or words that seem confusing here? Did you understand that part?
How would you answer this question? [If present, ask different group members if their approaches would differ].
How could this question be improved? Could it be clearer, or more interesting, for example?
Were there parts of the question you did not like? Why?
This question asks you about [insert focus of question]. Have you learned about this before, either in school or in your own life?
Did you like working with the tablet? Why?
Did you find the directions for how to do the question/read the passage helpful or not? Why?

Facilitators will observe students, take notes, and pay attention to verbal answers to the questions above (and write them down). They will look for evidence of engagement, boredom, or confusion in what students say to each other (or to the facilitator) and in their facial expressions or body language. If appropriate, staff can use these moments to probe for more information.

Occasionally staff observing playtesting sessions will introduce a question to students talking about the items to get more information from students, particularly in cases when students talk about issues related to their interest (or non-interest) in the item or task, confusing language in the item/task, or how they might respond to a question on the task. In such cases observers might ask something like, “Can you tell me more about what you said?” Prior to each playtesting session, ETS staff may informally identify some key focus areas for each item/task. If students do not provide sufficient comments on targeted parts, a staff member may ask students if they had any thoughts about the particular sections, using questions such as those described above but focused on specific places or issues in the item or task.

At the end, students will be thanked for their participation and for helping us to improve our tests. Gift cards will be provided at the conclusion of each playtesting session (both for students and parents if they brought their child to the session).

Cognitive Interviews

I. Introduction to Study Script

NOTE: The interviewer should not read the script word-for-word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular items being studied. In general, protocols will be tweaked and customized in light of specific item and stimulus content.

Introduction:

Hello, my name is ______ and I work for [indicate organization]. I am here with my colleagues [introduce colleagues]. It’s nice to meet you, and thank you very much for helping us out today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?

When the student responds, follow up with 2 or 3 questions to get the student used to talking, such as:

That’s interesting—why do you enjoy [subject] so much?
What are you studying in [subject] at the moment?
And what’s the best thing you have studied in <subject> so far?

Good. Well, I think you’ll enjoy what we are going to be doing today. First, let me begin by explaining why I am here and what you are going to be doing. You are taking part in a special study looking at a new test questions to find out what students know in [mathematics OR reading]. It is part of the National Assessment of Educational Progress, or NAEP for short, a test that is funded by the U.S. Federal Government and run by a center in the Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will help us develop better tests for other students. Overall, this session should take about [indicate correct length] minutes.

It’s okay if you don’t know how to do any questions. I will not be grading your work today, and no one will know that it was you who did the questions. My goal is to learn how you react to the questions, so please try to do your best.

If at any time you decide you don’t want to go on, that is your choice, and you may stop.

This new test is on the tablet. The instructions about how to do the test are also on the tablet.

This interview is being recorded so we can review it later. We will be able to capture what you tap on and where you move your fingers.

[If eye tracking is being used]: We will also be using an eye-tracker, which is a camera attached to the computer to record where on the screen you are looking while you work.

What you say will not be told to anyone, or used in any way that could identify you, for any other purpose except as required by law [if needed: Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].We will look at what you say later, but only so that we can understand how our test is working and how we can make it better.

We will work together on [specify number of items/tasks].

Do you have any questions?

After answering any questions and giving further explanation, continue with the think-aloud training. If the student is no longer interested in participating, thank the student for his/her time and end the interview.

Okay, let’s move on. Before we look at the real questions, I want to make sure you have a good idea of what we’re going to be doing. So I am going to give you some practice questions. The practice questions should help you get used to what we will be doing during the real questions. They should help you understand how we want you to respond. Do you have any other questions before we start?

II. Think-Aloud

II a. Think-Aloud: Instructions and Modeling Script

NOTE: Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word-for-word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular items being studied.

If students are doing reading questions for which they must read a passage, students should only be prompted to think aloud based on this training for the questions, NOT while they are reading the passage. The modeling script below is for questions (items), NOT for reading extended passages.

To help us make our test better, we will ask you to complete some questions.

[If using Morae or other screen capture software]: While you are doing the questions, we will be recording everything that happens on the screen. The screen recording will capture all of your responses and movements on the tablet such as what you tap on and anything you put into text boxes.

I’m going to ask you to do the questions in a way that may be different from what you are used to. Instead of working quietly, I want you to tell me what you are thinking as you work through the questions. We call this “a think-aloud,” because we are asking you to say everything you are thinking out loud.

In a moment, I will give you an example of the think-aloud process. Then I will give you a chance to practice it. You won’t be graded on anything you say while you are thinking aloud. There are no incorrect thoughts, and everything you think and say is important to us.

Okay, now I’m going to show you how to think out loud—this will help you see how I want you to describe what you are thinking as you are working on the questions. When I am finished I’ll ask you to try it, so you can see how it works.

Think-Aloud Demonstration:

Since we can’t tell what is going on in your head, we need you to think aloud.

Let me give you an example [Place example question in front of student]. Look at this question. It asks me to look at the five animals and choose two that are the most similar. So I am going to do this task, and I will think aloud while I work on it.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar.

Beetle
Mouse
Crab
Dolphin
Cow

Okay, so I am reading the question…. It says I have to decide on the two animals that are the most similar. Hmm, I am just wondering, what do they mean by “similar?” That could mean anything. I’m thinking I’m not sure how I’m going to make that decision. Well, let’s look at the choices. So the choices are beetle, mouse, crab, dolphin, and cow. Well, as I am reading the choices I am already thinking that there are some things about them that are similar, but there are also differences. I’m thinking the beetle and crab are kind of similar, in a way—well, they both have lots of legs and they kind of move around fast. But one lives in the water and the other doesn’t, so that makes them different. I guess the mouse also moves around fast, but it’s not similar to either the beetle or the crab, except they are all small. Then there is the dolphin that lives in the water, too… but that’s about all it has in common with the crab, so I wouldn’t say they are very similar even though they’re both in the sea. Hmm, so the mouse and the cow are both mammals, they both have fur and they have live babies instead of laying eggs or whatever… oh right, but dolphins are mammals too, aren’t they? Yes, I think they are… but really they seem so different from mice and cows to me. I don’t know… this seems like a hard question… I’m not sure what to choose. Okay, I’m going to try to make a decision…umm…okay, I think I am going to go for… mouse and cow. Because they both have fur and have four legs and they both walk around on the land. That seems weird; they are not really similar! But when I think about all of the others they seem too different from each other in really basic ways, like where they live and their kind of body and that kind of thing… so, yes, even though it was really hard to decide, that seems like the best choice to me, because they are both mammals and they both live on the land. So those things seem like they are important, and that is how I made my decision.

Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do as you are thinking about the questions today. The point of the think-aloud is to get at whatever is in your head as you are doing the questions. Just say aloud the words and the thoughts that are in your head, as you are thinking about and working on the questions.

II b. Concurrent Think-Aloud: Student Practice Script I

Interviewers should place the practice question in front of the student so he or she can read it. Some students will be silent after reading the question. Students should be immediately encouraged to say whatever they are thinking.

It may be necessary to remind the student to talk aloud as he/she works through the questions. If necessary, interviewers should use the “think-aloud hints” given below to prompt the student, being careful not to lead the student. The interviewer needs to be familiar enough with the information to introduce the think-aloud in a natural, conversational manner.

Now you will try a think-aloud. You can use this example. Like last time, you have to pick two of these things that you think are the most similar. As you are reading the question and the choices, and as you are figuring out your decision, I want you to talk out loud to tell me all of the thoughts that are in your head at each moment.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I’ll be reminding you to think aloud if you get quiet because I need to hear all of your thoughts.

Okay, now you try. Go ahead and start working on this question and remember to think aloud as you are doing it.

Which two living things below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the questions? What questions would you like to ask me? [If the student says he or she feels okay and doesn’t have any questions, say: Good, then let’s begin our study]. [If the student expresses concerns, says he or she has questions, or appears to be hesitant or reluctant, ask him/her to say more about the concerns or questions, and try to address the concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him or her to stop].

II c. Concurrent Think-Aloud: Student Practice II (Optional)

Use only if you feel the student would benefit from another think-aloud practice before moving on to the actual questions. It may be beneficial to instead model thinking aloud again for the student rather than doing another practice [see think-aloud demonstration in IIa].

If the student struggles to think aloud, the interviewer should give the student another opportunity to practice. The interviewer should praise the student for the first attempt regardless of how good it was, for example: “Very good—let’s do another one before we start the real tasks. Are you ready? Here is the next practice question. Remember to think out loud as you begin to think about this question and all the way through—tell me what you are thinking as you work through it.” During the practice item, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, below).

Which two of the following objects have the most similar properties? Circle the two that are most similar.

Silver coin
Chocolate coin
Gold coin
Blue plastic coin
Brown plastic coin

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the questions? What questions would you like to ask me? [If the student says he or she feels OK and doesn’t have any questions: Good, then let’s begin our study]. [If the student expresses concerns, says he/she has questions, or appears to be hesitant or reluctant, ask him or her to say more about the concerns or questions, and try to address his/her concerns or uncertainties in a supportive way. If the student indicates he or she does not wish to continue or does not feel comfortable continuing, allow the student to stop].

II d. Concurrent Think-Aloud: Starting the Items

Now we will move on to the actual questions.

[If student is doing reading questions with an associated passage]: You will be reading a passage for doing some reading questions. You do not need to think aloud as you read the passage. However, when you are answering the questions, just like in our practice, I’d like you to say aloud everything that you’re thinking. I may remind you to do that if you are quiet.

[If student is NOT reading a passage]: Remember, as you do all the activities and answer all the questions, I’d like you to say aloud everything that you’re thinking, and I may remind you to do that if you are quiet.

The questions should take about [indicate correct length] minutes. Remember, you will not be graded on your responses to the questions and there is no right or wrong way to think aloud, as long as you keep telling me your thoughts. Your thoughts will help us make the questions better. I will also have a few questions after you have finished working.

Do you have any questions before we go on? [Answer any questions the student may ask].

Because the information you provide is so important to us, I am going to be taking notes while you think aloud and answer the questions.

Here are the questions. When you are ready, go ahead and start working.

II e. Concurrent Think-Aloud: Hints for the Interviewer

The goal of think-alouds is to capture all the student’s mental processes while working through items (questions). Interviewers must strive to have the student speak aloud all of his or her thoughts while doing the items. If a student is continually providing short utterances or not talking, interviewers should use “continuers” to encourage students to be more descriptive. The goal is to get students to verbalize thoughts without putting words in students’ mouths. Care should be taken not to ask questions that lead students’ responses in particular directions or make them rush or change their approach. Interviewers should be as objective and unbiased as possible.

In general, if the student is silent for approximately 5 to 10 seconds, interviewers should use the following as a guide for encouraging the student to describe his or her thoughts, or to help the student elaborate on his or her responses.

If the student is not verbalizing enough, interviewers should offer a verbal nudge to remind the students to keep talking, such as:

What’s going on in your head right now?
I see you’re looking at [screen/figure/chart/text]. What are you thinking?
You seem to be thinking hard—can you tell me what’s in your mind right now?
What are you thinking now?
Any other thoughts?

If the student says something about his or her thoughts but it is difficult to tell exactly what he or she means, or if the student begins to talk but does not say very much, interviewers can urge the student to elaborate. For example:

Tell me a bit more about that.
That’s interesting—can you say more about that?
Go on….
Okay, and what else are/were you thinking?

If more clarity or a more explicit description of a student’s thoughts is desirable:

Can you explain what you mean by that?
Can you say a bit more to help me understand what you mean/are thinking/were thinking?

It is important to be responsive and sensitive to each student’s behavior. If it becomes clear that students cannot tell any more about their thoughts, interviewers should not keep asking. Students should be encouraged to think aloud as much as they can, but their reactions should be closely observed and pushing them should be avoided if students seem frustrated or uncomfortable. While it is desirable that students articulate as much as they can, sometimes students will simply be unable to state what is in their mind beyond a basic reply such as “I am reading the question.” Interviewers must be sensitive to nonverbal signals that may indicate students cannot say any more than they have or that they need a few extra seconds to gather their thoughts to put them into words.

In addition, during observations if it seems a student is hung up on something, interviewers should note when this occurred.

II f. Notes on Student Actions and Behaviors during Cognitive Lab

As a student is working through items, , the interviewer should take notes regarding any of the student’s expressions or behaviors that may reflect the status of the student’s understanding, engagement, or reactions to items and/or stimulus (e.g., a reading passage). The following are examples of such behaviors for interviewers to note.

Does the student express signs of confusion, boredom, or excitement?
Does the student use tools provided by tapping on tabs or buttons?
Does the student rapidly move through the screen or take his or her time? Does the student spend a lot of time on a particular item or tool or reading passage?

These kinds of observations should be considered opportunities to prompt students to think aloud and to encourage them to express what is in their minds. Interviewers should also make a note of any reactions to or interactions with items, stimuli, or a reading passage that appear valuable for follow-up with some additional questions after work is completed (see section III).

III. Post-Think-Aloud Follow-Up Questions (Verbal Probing)

After completing the think-aloud process for items, interviewers will follow up with a brief period of focused retrospective questioning. The purpose of the additional post task questions is to capture more information on issues such as student actions during the course of responding to items, particular aspects of [mathematics or reading] targeted in the items, and general reflections about the items, stimulus, tools, and or/reading passage. The post-item questions will include targeted questions that are item-specific. These will be selected for by ETS staff prior to testing. Interviewers will receive the set of up to five questions for the items they are examining in each cognitive interview.

Interviewers will also be trained to generate post-item questions on the fly, for cases where they observe something that they judge to need follow-up probing. The following are some examples of questions that might be asked at this stage.

Did any words in the item(s)/stimulus/passage confuse you?
What ways could we change the item(s)/stimulus to make it better?
Did the [stimulus, tools, etc.] for the item(s) help you answer them?

IV. Eye Tracking

Eye tracking is designed to provide data about where students place their visual focus while taking an assessment. When eye tracking is used, students will be asked to do items on a tablet while a camera monitors where they look.

Apparatus

An eye-tracking system is video-based system utilizing a high speed camera and an LED infrared light that illuminates the face (no more powerful than typical sunlight). An Infrared camera is used to capture eye movements. The remote system requires no physical contact with the student; an infrared video camera acquires an image of the eyes and calculates gaze location in real time. To allow free (but limited) head and body movements while maintaining accuracy of localization, a reflective sticker is placed on the forehead of the student during the eye-tracking session. ETS eye-tracking experts have used this particular system with adults and with children as young as three years old. It takes a very short time (five minutes) to set up and calibrate the equipment for each participant.

Procedure

Students will be asked to work through items thinking aloud (save for while they are reading passages for reading items). As students work, their eye movements are unobtrusively recorded and all events on the screen are captured in real time.

The sequence to be followed for eye tracking is described below. Note that the eye-tracking calibration procedure may differ slightly for various hardware setups.

Setup and Calibration

The participating student is greeted by an ETS eye-tracking lab expert. The expert describes the study procedure and responds to questions or concerns the student may have. The student is then seated in front of the eye tracker and a sticker is placed on his or her forehead to assist the eye tracker in locating the eyes and to compensate for head movements. The system is then calibrated. The student is asked to watch a dot appearing at a number of different locations on the blank screen. The setup and calibration typically take about five minutes.

Eye-tracking Session

After a successful calibration, the student starts the assessment session, in which he or she goes through the assessment items on the tablet. Students are free to move their head and body during the assessment within the field of view of the eye-tracking camera. After the student completes the assessment task, eye tracking is stopped.

V. Debriefing and Thank You for Cognitive Interview

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the items?

Is there anything you would like to ask me about what we did today? [Answer student questions].

Thank you for helping us to improve our test.

Thank the student for his or her time. Provide a gift card (both for student and for the parent if the parent has brought the child to the session).

Small-Scale Tryout Protocol

Text written in italics are to be read aloud by the interviewer.

NOTE: The interviewer should not read the script word-for-word but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text in brackets is generic text that will be tailored based on task specifics.

Session Information

The interviewer should complete the information from rows 1-3 prior to the beginning of the tryout session. Rows 4-5 are to be completed at the end of the tryout session.

Ref#	Preliminary Information	Data
1	Name of interviewer
2	Date
3	Start time of interview
4	End time of interview
5	Special circumstances that may have affected the tryouts session [Do not ask test taker, just record observations]

Introduction Script:

Hello, my name is [name] and I work for EurekaFacts. I am here with my colleagues [introduce colleagues]. It’s nice to meet you, and thank you very much for helping us out today.

Let me begin by explaining why you’re here and what you’re going to be doing today. You are participating in a special study to try out new [mathematics OR reading] test questions. This is part of something called the National Assessment of Educational Progress, or NAEP for short, and is sponsored by the U.S. Department of Education. Today we’ll be asking you to try out some questions that are being developed for 4^th/8^th/12^th graders all over the United States.

If at any time you decide you do not want to go on, that is your choice and you may stop.

Your information and responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [if needed: Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].

Do you have any questions?

Interviewer: If a student is no longer interested in participating, thank the student for his or her time and end the tryout session for that one student only; continue with all others.

Instructions

Ok, now, I’m going to give you a set of questions to complete.

This is a timed activity, which will last for [indicate length of activity], and everyone will begin at the same time.

All of the instructions will be displayed on your computer screens.

The questions you are about to do is something that students, like you, will do on the computer. However, we aren’t finished making them yet, so there may be some parts that don’t work. Try to work through and complete the questions as best as you can.

You will not be graded on your answers to the questions, but please answer the questions as if it was a real test.

We will all work at the same time, first I would like you to read the instructions, once everyone has gone through the instructions, we will begin the questions together. I will start the timer and I will stop you once time is up.

If you finish early, please sit back and wait until the time is up. Wait until everyone is done and then we will submit your answers together.

Okay, I will come around and open the questions for each one of you, but please do not begin until I say that you may start.

Okay, now that everyone is ready, you may begin.

START THE TIMER NOW. STUDENTS HAVE 60 MINUTES TO COMPLETE THE QUESTIONS. STOP STUDENTS AT 60 MINUTES EVEN IF THEY HAVE NOT FINISHED.

Wait for all students to complete the questions within the allotted time (in the event some students complete early).

Once students have completed the questions, make sure that all students exit the test and submit their answers.

Shape1

Thank You

Thank participant(s) for their time and escort them to the front desk. Provide gift card(s), both for

students and parents if they brought their child to the session.

1 60 minutes for grade 4 playtesting and cognitive interviews and grades 4, 8, and 12 tryouts; 90 minutes for grades 8 and 12 playtesting and cognitive interviews; 90 minutes for teacher and school administrator cognitive interviews.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Science Volume II Cog Lab Protocols
Author	BlairJ
File Modified	0000-00-00
File Created	2021-01-27