National Assessment of Educational Progress (NAEP) 2022 Social Science and Economics Questionnaire Cognitive Interviews

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average 60 minutes, including the time to review instructions and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission, please write to: National Assessment of Educational Progress (NAEP), National Center for Education Statistics (NCES), Potomac Center Plaza, 550 12th St., SW, 4th floor, Washington, DC 20202.

Authorization and Confidentiality Assurance

National Center for Education Statistics (NCES) is authorized to conduct NAEP by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622).

All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C.§151).

OMB No. 1850-0803 Approval Expires 07/31/2019

The purpose of the cognitive interviews is to inform the development of the student, teacher, department head, and school questionnaires prior to pilot testing. Below are the probes that students, teachers, department heads, and school administrators will be asked during the cognitive interview. Please note that some probes may not be addressed if time does not permit¹.

Interviewer Welcome Script and Assent/Consent

The following script does not have to be read verbatim. You, as the interviewer, should be familiar enough with the script to introduce the participant to the cognitive interview process in a conversational manner. The text in italics is suggested content for you to become thoroughly familiar with in advance. You should project a warm and reassuring tone toward the participant in order to develop a friendly rapport. You should use conversational language throughout the interview.

After answering questions and giving further explanation, begin the interview with the first item.

Note: Students, teachers, department heads, and school administrators will be administered all items and probes, if time permits. The duration of each cognitive interview will be 60 minutes².

Student Welcome Script

Hello, my name is <name >.

Thank you for agreeing to participate in our study. Your help is very important to us. Let me tell you a bit more about what we are going to do today.

I have some questions that students in many different schools across the country will be asked to answer about themselves and their families. The questions will be part of the National Assessment of Educational Progress (NAEP).

Some of the questions may be difficult to understand or answer. Our goal is to make these questions better so that other students can easily understand them. That is why we need your feedback.

We want you to help us find out which questions are easy to understand and which are difficult to understand. There are no right or wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), which is part of the United States Department of Education, make these questions clearer for students like you. In the next hour we will try to get through as many questions as possible.

This interview is being audio recorded so that researchers can review the recordings later. No one else will hear the recording. Your participation is voluntary and all of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Do you have any questions?

If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

Teacher Welcome Script

Hello, my name is <name>.

Thank you for coming here today to help us.

I have some questions that teachers in many different schools will be asked to answer about themselves, their instructional practices, and their schools. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. There are no right or wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), part of the United States Department of Education, make these questions clearer for teachers like you. In the next hour we will try to get through as many questions as possible.

Do you have any questions?

If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

Department Head Welcome Script

Hello, my name is <name>.

Thank you for coming here today to help us.

I have some questions that department heads in many different schools will be asked to answer about themselves, the teachers in their schools, and instructional practices and resources used to teach economics. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. There are no right or wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), part of the United States Department of Education, make these questions clearer for teachers like you. In the next hour we will try to get through as many questions as possible.

Do you have any questions?

If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

School Administrator Welcome Script

Hello, my name is <name >.

Thank you for coming here today to help us.

I have some questions that school administrators in many different schools will be asked to answer about their school policies and characteristics. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. There are no right or wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), part of the United States Department of Education, make these questions clearer for school administrators like you. In the next hour we will try to get through as many questions as possible.

This interview is being audio recorded so that researchers can review the recordings later. No one else will hear the recording. Participant responses are voluntary, will be used only for statistical purposes, and will not be disclosed or used, in identifiable form, for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Do you have any questions?

If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

Instructions and Generic Probes

The interviewer should ask the participant to read and answer the questions, one at a time. Each question will be either one discrete question (multiple choice or free response) or one matrix question (i.e., an item stem with two or more multiple choice sub-items). The respondent is not supposed to read the question aloud, but should work on the question in the same fashion he/she would during a NAEP test administration or survey. If a respondent indicates they do not know a word or phrase, the interviewer can read words aloud and/or explain the meaning of words for the respondent.

For most items, generic probes for discrete and matrix items will include the following:

Generic Probes for Discrete Items (Multiple Choice and Free Response)–Full Set

No.	Probe	Instructions for Interviewer
1	Can you explain to me what the question is asking? Please try to explain without repeating the wording of the item.	Ask this probe for all discrete questions AFTER the respondent has answered the question.
2	Were there any words or parts of this question that were confusing?  Yes  No	Ask this probe for all discrete questions.
3	What did you find confusing? What could we do to make the question less confusing?	Ask both probes only if the respondent answered YES to the previous question.
4	Would you say it was very easy, easy, difficult, or very difficult to answer this question?  Very Easy  Easy  Difficult  Very Difficult	Ask this probe for all discrete questions.
5	How could we make it easier to answer this question?	Ask this probe only if the respondent answered DIFFICULT or VERY DIFFICULT to the previous question.
6	How sure are you about your answer to this item?  Very Unsure  Unsure  Sure  Very Sure	Ask this probe for all discrete questions.
7	Were you comfortable answering this item?  Yes  No	Only ask this probe for discrete questions inquiring about potentially sensitive topics.
8	How could we make the item more comfortable to answer?	Ask this probe only if the respondent answered NO to the previous probe. This probe should be skipped if the previous probe was not administered.

Generic Probes for Matrix Items – Full Set

No.	Probe	Instructions for Interviewer
1	Look back at the first part of the item where you were asked a question. Can you explain to me what this question is asking? Please try to explain without repeating the wording of the item.	Ask this probe AFTER the respondent has answered the entire matrix question. This probe is specifically for the “stem” of the item, not the entire matrix item or a specific part/sub-item.
2	Can you explain to me what [part a., b., c., etc.] means? Please try to explain without repeating the wording of the item.	Ask this probe for all parts/sub-items. Please note probes 2-6 should be asked together for one part/sub-item before moving on to the next part/sub-item.
3	Did you find any words or part of [part a., b., c., etc.] confusing?  Yes  No	Ask this probe for all parts/sub-items.
4	What did you find confusing? What could we do to make [part a., b., c., etc.] less confusing?	Ask both probes only if the respondent answered YES to the previous question.
5	Would you say answering [part a., b., c., etc.] was very easy, easy, difficult, or very difficult?  Very Easy  Easy  Difficult  Very Difficult	Ask this probe for all parts/sub-items.
6	How could we make it easier to answer [part a., b., c., etc.]?	Ask this probe only if the respondent answered DIFFICULT or VERY DIFFICULT to the previous question.
7	How sure are you about your answers to this item?  Very Unsure  Unsure  Sure  Very Sure	Ask this probe for all matrix items. Ask this probe once per matrix, after completing all other probes pertaining to parts/sub-items.
8	Were you comfortable answering this item?  Yes  No	Only ask this probe for matrix questions inquiring about potentially sensitive topics. Ask this probe after completing all other probes pertaining to parts/sub-items.
9	How could we make this item more comfortable to answer?	Ask this probe only if the respondent answered NO to the previous probe. This probe should be skipped if the previous probe was not administered.

Other existing items may only need to be administered in cog labs to ensure specific terminology or phrases are still relevant (e.g., “thesaurus”), or because the response options have been changed. In these cases, generic probes will not be administered. Probes specific to the item will be administered in relation to the terminology, phrase, or changed response options to ensure participants are able to provide the feedback.

Core Cognitive Interview Items and Probes

Student Items – Grade 8 Specific

[Grade 8 Core: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic	Required
2	How did you decide what answer to give to this question?	Specific	Required
3	[If selects options B, C, D, or E] Can you explain what the first response option means?	Specific	Conditional
4	[If selects options A, C, D, or E] Can you explain what the second response option means?	Specific	Conditional
5	Can you explain what “homeschooled” means to you?	Specific	Required

Student Items – Grade 12 Specific

[Grade 12 Core: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic	Required
2	How did you decide what answer to give to this question?	Specific	Required
3	[If selects options B, C, or D] Can you explain what the first response option means?	Specific	Conditional
4	[If selects options A, C, or D] Can you explain what the second response option means?	Specific	Conditional
5	Can you explain what “homeschooled” means to you?	Specific	Required

Teacher Items – Grade 8 Specific

[Grade 8 Core: Teacher #1]

Alternative version

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic	Required
2	How did you decide what answer to give to this question?	Specific	Required
3	Can you explain what devices qualify as a “computer”?	Specific	Required
4	Do you know whether your students have a high-speed or dial-up Internet connection at home?	Specific	Required
5	[Looking at all versions of the item] Please take a look at the alternative version of the item you just answered. What is the difference between the two versions?	Specific	Required
6	[Looking at all versions of the item] Would you answer the alternate version of the item differently? Why?	Specific	Required

[Grade 8 Core: Teacher #2]

Alternative version

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	a) Can you provide other examples of learning activities that require Internet access?	Specific	Required
3	d) Can you provide other examples of places to access to the Internet for free outside of school?	Specific	Required
4	e) and f) Can you explain what “affordable” means to you?	Specific	Required
5	Can you think of any other methods for accommodating students with limited Internet access at home?	Specific	Required
6	[Looking at all versions of the item] Please take a look at the alternative version of the item you just answered. The only difference is in the item stem. Would you answer any part of the alternate version of the item differently? Why?	Specific	Required

[Grade 8 Core: Teacher #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	[If responds “Yes, I participated because it was required” to a sub-item] Who required you to participate in this activity?	Specific	Conditional
3	a) Can you provide other examples of professional values?	Specific	Required
4	b) Can you provide some examples of reflective teaching practice?	Specific	Required
5	f) Can you provide some examples of student development knowledge?	Specific	Required
6	k) Can you provide some examples of student well-being awareness?	Specific	Required
7	What does “the last two school years” mean to you in this context?	Specific	Required
8	Do you think any of the professional development topics listed here are redundant?	Specific	Required
9	Which of these professional development topics do you think are the most important?	Specific	Required
10	Which of these professional development topics do you think are the least important?	Specific	Required
11	How do you know which professional development activities you are required to attend?	Specific	Required

Civics Cognitive Interview Items and Probes

Student Items – Grades 8 and 12 (Cross Grades)

[Grades 8 and 12 Civics: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	d) Can you explain what part d means to you?	Specific	Required
3	g) Can you explain what part g means to you?	Specific	Required
4	Can you explain the difference between Response Option A and Response B?	Specific	Required
5	Can you explain the difference between Response Option B and Response C?	Specific	Required
6	Can you explain the difference between Response Option C and Response D?	Specific	Required
7	Can you explain the difference between Response Option D and Response E?	Specific	Required

Please note that in addition to response option probes, only sub-items "d” and "g” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grades 8 and 12 Civics: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	b) Can you explain what part b means to you?	Specific	Required
3	c) Can you explain what part c means to you?	Specific	Required
4	e) Can you explain what part e means to you?	Specific	Required

Please note that only sub-items "b”, “c”, and "e” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grades 8 and 12 Civics: Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	d) Can you explain what part d means to you?	Specific	Required
3	e) Can you explain what part e means to you?	Specific	Required
4	f) Can you explain what part f means to you?	Specific	Required
5	f) Can you explain what it means to you to “participate” in an activity?	Specific	Required
6	f) Can you explain what “social activism” means to you?	Specific	Required

Please note that only sub-items "d”, "e”, and “f” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grades 8 and 12 Civics: Student #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	e) Can you explain what part e means to you?	Specific	Required
3	e) Do you think the examples provided in part e can be improved?	Specific	Required
4	e) If the student says that the examples provided in part e can be improved, then ask them: How can the examples in part e be improved?	Specific	Conditional

Please note that only sub-item "e” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grades 8 and 12 Civics: Student #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	e) Can you explain what part e means to you?	Specific	Required
3	e) Do you have difficulty answering part e?	Specific	Required
4	e) What does “community” mean to you in this context?	Specific	Required
5	e) What does “my community three years from now” mean to you in this context?	Specific	Required

[Grades 8 and 12 Civics: Student #6]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	f) Can you explain what part f means to you?	Specific	Required
3	f) Can you explain what “democratic process” means to you?	Specific	Required
4	f) Can you explain what it means to you to “participate in the democratic process”?	Specific	Required
5	g) Can you explain what part g means to you?	Specific	Required
6	h) Can you explain what part h means to you?	Specific	Required
7	h) Can you explain what “public policy” means to you?	Specific	Required
8	i) Can you explain what part i means to you?	Specific	Required
9	i) Can you explain what “serve your community” means to you?	Specific	Required

Please note that only sub-items "f”, "g”, "h” and "i” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grades 8 and 12 Civics: Student #7]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	c) Can you explain what part c means to you?	Specific	Required
3	c) Can you explain what “social causes” means to you?	Specific	Required
4	d) Can you explain what part d means to you?	Specific	Required

Please note that only sub-items "c” and "d” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

Student Items – Grade 12 Specific

[Grade 12 Civics: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	If student selects Response Option B for any of the sub-items, then ask these students: Please describe the class or course that you took that included some civics and/or United States government topics.	Specific	Conditional

Geography Cognitive Interview Items and Probes

Student Items – Grades 8 and 12 (Cross Grades)

[Grades 8 and 12 Geography: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	a) Can you explain what part a means to you?	Specific	Required
3	e) Can you explain what part e means to you?	Specific	Required
4	e) Can you explain what the phrase “interact across space” means to you?	Specific	Required

Please note that only sub-items "a” and "e” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grades 8 and 12 Geography: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	b) Can you explain what part b means to you?	Specific	Required
3	d) Can you explain what part d means to you?	Specific	Required
4	e) Can you explain what part e means to you?	Specific	Required
5	e) Can you explain the difference between part e and part b?	Specific	Required

Please note that only sub-items “b”, “d”, and "e” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grades 8 and 12 Geography: Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	c) Can you explain what part c means to you?	Specific	Required
3	d) Can you explain what part d means to you?	Specific	Required
4	e) Can you explain what part e means to you?	Specific	Required
5	e) Can you explain what “social activism” means to you?	Specific	Required

Please note that only sub-items “c”, "d”, and “e” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grades 8 and 12 Geography: Student #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	a) Can you explain what part a means to you?	Specific	Required
3	g) Can you explain what part g means to you?	Specific	Required
4	g) Do you have difficulty understanding part g?	Specific	Required
5	g) Have you learned about the topic mentioned in part g?	Specific	Required

Please note that only sub-item "a” and "g” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grades 8 and 12 Geography: Student #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	e) Can you explain what part e means to you?	Specific	Required
3	f) Can you explain what part f means to you?	Specific	Required

Please note that only sub-items “e” and "f” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grades 8 and 12 Geography: Student #6]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	a) Can you explain what part a means to you?	Specific	Required
3	f) Can you explain what part f means to you?	Specific	Required
4	h) Can you explain what part h means to you?	Specific	Required

Please note that only sub-item "a”, "f”, and "h” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grades 8 and 12 Geography: Student #7]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	c) Can you explain what part c means to you?	Specific	Required
3	c) Can you explain what “places outside your community” means to you in this context?	Specific	Required
4	c) Can you explain what “your community” means to you in this context?	Specific	Required
5	e) Can you explain what part e means to you?	Specific	Required
6	e) Can you explain what it means to you to “discuss” a topic on social media?	Specific	Required

Please note that only sub-items “c” and "e” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

Student Items – Grade 12 Specific

[Grade 12 Geography: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	If student selects Response Option B for any of the sub-items, ask students: Please describe the class or course that you took that included some geography topics.	Specific	Conditional

U.S. History Cognitive Interview Items and Probes

Student Items – Grades 8 and 12 (Cross Grades)

[Grades 8 and 12 U.S. History: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	b) Can you explain what part b means to you?	Specific	Required
3	e) Can you explain what “main idea presented in a source” means to you?	Specific	Required

Please note that only sub-item "b” and "e” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grades 8 and 12 U.S. History: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	d) Can you explain what part d means to you?	Specific	Required
3	e) Can you explain what part e means to you?	Specific	Required
4	f) Can you explain what part d means to you?	Specific	Required
5	f) Can you explain what “social activism” means to you?	Specific	Required

Please note that only sub-items "d”, “e”, and "f” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grades 8 and 12 U.S. History: Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	c) Can you explain what part c means to you?	Specific	Required
3	c) Can you explain what “main idea presented in a source” means to you?	Specific	Required
4	d) Can you explain what part d means to you?	Specific	Required

Please note that only sub-items "c” and "d” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grades 8 and 12 U.S. History: Student #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	c) Can you explain what part c means to you?	Specific	Required
3	c) Can you explain what “main idea presented in a source” means to you?	Specific	Required
4	d) Can you explain what part d means to you?	Specific	Required
5	h) Can you explain what part h means to you?	Specific	Required
6	h) Can you explain what “historical origins” means to you?	Specific	Required
7	h) Do you think that the example provided in part h can be improved?	Specific	Required
8	h) If the student answers that the example provided in part h can be improved, then ask the student: How can the example in part h be improved?	Specific	Conditional

Please note that only sub-item "c”, "d”, and "h” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

Student Items – Grade 12 Specific

[Grade 12 U.S. History: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	If student selects Response Option B for any of the sub-items, ask students: Please describe the class or course that you took that included some United States history topics.	Specific	Conditional

Social Studies Cognitive Interview Items and Probes

Teacher Items – Grade 8 Specific

[Grade 8 Social Studies: Teacher #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic	Required
2	Do you believe you know enough about instruction in non-Social Studies subjects to provide an accurate response to this question?	Specific	Required

[Grade 8 Social Studies: Teacher #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	a) Can you explain what part a means to you?	Specific	Required
3	b) Can you explain what part b means to you?	Specific	Required
4	e) Can you explain what part e means to you?	Specific	Required
5	f) Can you explain what part f means to you?	Specific	Required
6	f) If possible, please provide other examples of games that involve social studies concepts or content.	Specific	Required

Please note that only sub-items “a”, “b”, "e”, and "f” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grade 8 Social Studies: Teacher #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	c) Can you explain what part c means to you?	Specific	Required
3	d) Can you explain what part d means to you?	Specific	Required
4	e) Can you explain what part e means to you?	Specific	Required
5	e) Can you explain what “social activism” means to you?	Specific	Required

Please note that only sub-items “c”, "d”, and "e” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grade 8 Social Studies: Teacher #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required

[Grade 8 Social Studies: Teacher #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	a) Can you explain what part a means to you?	Specific	Required
3	b) Can you explain what part b means to you?	Specific	Required
4	c) Can you explain what part c means to you?	Specific	Required
5	d) Can you explain what part d means to you?	Specific	Required
6	g) Can you explain what part g means to you?	Specific	Required
7	Can you explain the difference between Response Option A and Response B?	Specific	Required
8	Can you explain the difference between Response Option B and Response C?	Specific	Required
9	Can you explain the difference between Response Option C and Response D?	Specific	Required
10	Can you explain the difference between Response Option D and Response E?	Specific	Required

Please note that only sub-items “a”, “b”, “c”, "d” and "g” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grade 8 Social Studies: Teacher #6]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	d) Can you explain what part d means to you?	Specific	Required
3	d) Can you explain what the phrase “interact across space” means to you in this context?	Specific	Required
4	e) Can you explain what part e means to you?	Specific	Required

Please note that only sub-items "d” and "e” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grade 8 Social Studies: Teacher #7]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required

[Grade 8 Social Studies: Teacher #8]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	d) Can you explain what part d means to you?	Specific	Required
3	h) Can you explain what part h means to you?	Specific	Required
4	h) Is the example provided in part h helpful?	Specific	Required
5	h) Can the example provided in part h be improved?	Specific	Required
6	h) If the student answers that the example provided in part h can be improved, then ask: How can the example provided in part h be improved?	Specific	Required

Please note that only sub-items “d” and "h” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grade 8 Social Studies: Teacher #9]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required

[Grade 8 Social Studies: Teacher #10]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic	Required
2	How did you decide what answer to give to this question?	Specific	Required
3	Can you explain what the first (A) response option means to you?	Specific	Required
4	Can you explain what the second (B) response option means to you?	Specific	Required

School Items – Grade 12 Specific

[Grade 12 Social Studies: School #1]

The following script should be read to the participant before they provide a response:

This item will only be administered to school administrators who answer “This course is not offered in my school” to an existing item. (INTERVIEWER: show the participant the item in the image below).

Existing Item:

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic	Required
2	Can you explain what you believe the difference is, if any, between a “class” primarily focused on social studies and a “course” primarily focused on social studies?	Specific	Required

[Grade 12 Social Studies: School #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	As an administrator, are you expected to be involved in decisions about curriculum in your school?	Specific	Required
3	Please describe your involvement, if any, in your school’s U.S. history curriculum.	Specific	Required
4	As an administrator, do you believe you can provide an accurate response to this question?	Specific	Required

[Grade 12 Social Studies: School #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	As an administrator, are you expected to be involved in decisions about curriculum in your school?	Specific	Required
3	Please describe your involvement, if any, in your school’s civics or government curriculum.	Specific	Required
4	As an administrator, do you believe you can provide an accurate response to this question?	Specific	Required

[Grade 12 Social Studies: School #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	As an administrator, are you expected to be involved in decisions about curriculum in your school?	Specific	Required
3	Please describe your involvement, if any, in your school’s geography curriculum.	Specific	Required
4	As an administrator, do you believe you can provide an accurate response to this question?	Specific	Required

Additional Social Studies Terminology Probes

Student Grades 8 and 12

No.	Probe	Type of Probe	Required/Conditional
1	What does the phrase “social studies” mean to you?	General	Required
2	What does the word “civics” mean to you?	General	Required
3	What does the word “geography” mean to you?	General	Required
4	What does the phrase “social science” mean to you?	General	Required

Teacher Grade 8

No.	Probe	Type of Probe	Required/Conditional
1	What does the phrase “social studies” mean to you?	General	Required
2	What does the word “civics” mean to you?	General	Required
3	What does the word “geography” mean to you?	General	Required
4	What does the phrase “social science” mean to you?	General	Required

School Grades 8 and 12

No.	Probe	Type of Probe	Required/Conditional
1	What does the phrase “social studies” mean to you?	General	Required
2	What does the word “civics” mean to you?	General	Required
3	What does the word “geography” mean to you?	General	Required
4	What does the phrase “social science” mean to you?	General	Required

Economics Cognitive Interview Items and Probes

Student Items – Grade 12 Specific

[Grade 12 Economics: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic	Required
2	When considering response option A, did you include the classes you are currently taking?	Specific	Conditional
3	What does ‘economics-related course’ mean to you?	Specific	Required

Please note that the item-specific probe will be asked to students who selected Response Option A.

[Grade 12 Economics: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	In which classes do you receive economics instruction?	General	Required
2	a) Can you describe what the difference is between a and f?	Specific	Conditional
3	c) Did you find any words or part of [c] confusing?	Specific	Conditional
4	h) Is ‘political science and economics course’ offered?	Specific	Conditional
5	i) What types of “any other course” did you think about when answering this item?	Specific	Conditional

Please note that only sub-items "a", “c”, “h”, “i” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion. Please note that the probes listed above will be asked on the condition that students select ‘taken/taking’ on the specific sub-items.

[Grade 12 Economics: Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic	Required

[Grade 12 Economics: Student #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	f) Can you explain what “political view” means to you?	Specific	Required

Please note that only sub-item "f" will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grade 12 Economics: Student #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see pages 7.	Generic	Required
2	Can you explain what “graphical analysis” means to you?	Specific	Required
3	Can you explain what kind of complex calculations are included in your economics course(s)/instruction?	Specific	Required

[Grade 12 Economics: Student #6]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see pages 8.	Generic	Required
2	c) What types of “podcasts” did you think about when answering this item?	Specific	Required
3	e) What types of “online tool” did you think about when answering this item?	Specific	Required
4	Do you use the Internet to pull/download data for use in economics class/coursework?	Specific	Required

[Grade 12 Economics: Student #7]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	a) What types of “newspapers and magazines (print or online)” did you think about when answering this item?	Specific	Required
2	d) Can you explain what “social media” means to you?	Specific	Required

Please note that only sub-items "a" and “d” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grade 12 Economics: Student #8]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	e) Can you explain what “finance challenge” means to you?	Specific	Required
2	f) Can you explain what “H&R Block Budget” means to you?	Specific	Required
3	g) What “other” types of economics-related activities did you think about when answering this item?	Specific	Required

Please note that only sub-items "e", “f”, “g” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grade 12 Economics: Student #9]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic	Required
2	When answering this question, what do you consider as work?	Specific	Required

[Grade 12 Economics: Student #10]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic	Required
2	What types of “economic concepts” did you think about when answering this item? (Please do not repeat the examples provided in the question.)	Specific	Required

Please note that this item will be administered to students who reported that they have not taken any economics-related courses in item #1 (VB595669).

[Grade 12 Economics: Student #11]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for free response items, see page 7.	Generic	Required

Please note that this item is administered only to students who select response option B in item EC_22_006.

The following script should be read to the participant before they provide a response:

This item will only be administered to students who answer “yes but I did not take it” to an existing item (INTERVIEWER: show the participant the item in the image below). If you would have answered “yes but I did not take it” to the existing item, explain why you did not take the course that covered economic concepts.

Existing item:

[Grade 12 Economics: Student #12]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	d) What types of “other course” did you think about when answering this item?	Specific	Required
3	Would you include Political Science in this list?	Specific	Required

[Grade 12 Economics: Student #13]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	b) Can you explain what "keeping a budget" means to you?	Specific	Required
3	g) Can you explain what "economic issues" means to you?	Specific	Required
4	h) Can you explain what "economic concepts" means to you?	Specific	Required

[Grade 12 Economics: Student #14]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	a) Can you explain what “monetary” means to you?	Specific	Required
3	b) Can you explain what “additional costs” means to you?	Specific	Required
4	b) Can you explain what “additional benefits” means to you?	Specific	Required
5	c) Can you explain what “e-book” means to you?	Specific	Required
6	(d) Can you explain what “consumers” means to you?	Specific	Required
7	g) Can you explain what “media report” means to you?	Specific	Required
8	g) Can you explain what “economic condition” means to you?	Specific	Required
9	h) Can you explain what “tax structures” means to you?	Specific	Required
10	h) Can you explain what “producers” means to you?	Specific	Required
11	j) Can you explain what “tariffs” means to you?	Specific	Required
12	j) Can you explain what “quotas” means to you?	Specific	Required
13	k) Can you explain what “specialization” means to you?	Specific	Required

[Grade 12 Economics: Student #15]

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	a) Did you find “and/or” confusing? If yes, why?	Specific	Required
3	b) Did you find “and/or” confusing? If yes, why?	Specific	Required

Item-Specific Probes:

Please note that item-specific probes will be raised only on sub-items "a" and “b” during cognitive interviews.

[Grade 12 Economics: Student #16]

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	a) Looking at the alternative version of the sub-item a, which version do you find easier to answer?	Specific	Required

Item-Specific Probes:

Alternative version for sub-item a:

“Make a money transaction (for example, send money to a friend)”

Department Head Items – Grade 12 Specific

[Grade 12 Economics: Department Head #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	a) Can you describe what the difference is between [a] and [f]?	Specific	Conditional
2	c) Did you find any words or part of [c] confusing?	Specific	Conditional
4	h) What types of “any other course” did you think about when answering this item?	Specific	Conditional
5	Do you think political science would be a natural fit to include in this list?	Specific	Conditional

Please note that only sub-items "a", “c”, “h” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion. Please also note that the probes listed above will be asked on the condition that department heads select ‘Offered’ on the specific sub-items.

[Grade 12 Economics: Department Head #2]

No.	Probe	Type of Probe	Required/Conditional
1	Did you find any words or part of the question confusing? If so, what did you find confusing?	Specific	Required

I tem-Specific Probes:

Please note that the probe only focuses on the item stem.

[Grade 12 Economics: Department Head #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	d) Did you find any words or part of [d] confusing? If so, what did you find confusing?	Specific	Required

Please note that only sub-item "d" will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grade 12 Economics: Department Head #4]

No.	Probe	Type of Probe	Required/Conditional
1	Did you find any words or part of this question confusing? If so, what did you find confusing?	Specific	Required

Item-Specific Probes:

Please note that the probe only focuses on the item stem.

[Grade 12 Economics: Department Head #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	b) Can you explain what “flipped classrooms” means to you?	Specific	Required
3	c) What types of “activity-based lessons” did you think about when answering this item?	Specific	Required
4	d) What types of “problem-based lessons” did you think about when answering this item?	Specific	Required
5	f) What “other” instructional strategies did you think about when answering this item?	Specific	Required

[Grade 12 Economics: Department Head #6]

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	c) Can you explain what “economics analysis and criticism” means to you?	Specific	Required
3	d) Can you explain what “internal economics” means to you?	Specific	Required

Item-Specific Probes:

[Grade 12 Economics: Department Head #7]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	Can you explain what “graphs in classroom presentations” means to you?	Specific	Required
3	b) Can you explain what “graphs in testing the students or student work” means to you?	Specific	Required
4	d) What types of “materials” did you think about when answering this item?	Specific	Required
5	e) What types of “materials on the internet” did you think about when answering this item?	Specific	Required

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	What types of “podcasts” did you think about when answering this item?	Specific	Required
3	What types of “visual clips and movie clips” did you think about when answering this item?	Specific	Required
4	What types of “simulations and games” did you think about when answering this item?	Specific	Required
5	What types of “phone applications” did you think about when answering this item?	Specific	Required

[Grade 12 Economics: Department Head #8]

Item-Specific Probes:

School Items – Grade 12 Specific

[Grade 12 Economics: School #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	a) Can you describe what the difference is between [a] and [f]?	Specific	Conditional
2	c) Did you find any words or part of [c] confusing?	Specific	Conditional
3	h) What types of “any other course” did you think about when answering this item?	Specific	Conditional

Please note that only sub-items "a", “c”, “h” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion. Please note that the probes listed above will be asked on the condition that the school administrator selects response options ‘B, C, D’ on the specific sub-items.

[Grade 12 Economics: School #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	a) Can you describe what the difference is between [a] and [f]?	Specific	Conditional
2	c) Did you find any words or part of [c] confusing?	Specific	Conditional
3	i) What types of “any other course” did you think about when answering this item?	Specific	Conditional

Please note that only sub-items "a", “c”, “f” “h”, “i” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion. Please note that the probes listed above will be asked on the condition that the school administrator selects response options ‘B, C, D, E, F’ on the specific sub-items.

[Grade 12 Economics: School #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	a) Can you describe what the difference is between [a] and [f]?	Specific	Conditional
2	c) Did you find any words or part of [c] confusing?	Specific	Conditional
3	e) What types of “any other course” did you think about when answering this item?	Specific	Conditional

Please note that only sub-items "a", “c”, “e” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion. Please note that the probes listed above will be asked on the condition that school administrators select response options ‘B, C’ on the specific sub-items.

[Grade 12 Economics: School #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
2	e) Can you explain what “Finance Challenge” means to you?	Specific	Required
3	f) Can you explain what “H&R Block Budget” means to you?	Specific	Required

Please note that only sub-items "e", “f” will be reviewed in cognitive interviews, but the entire matrix will be shown to avoid potential confusion.

[Grade 12 Economics: School #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	Generic	Required
2	What other school personnel did you think about when answering this item?	Specific	Required

[Grade 12 Economics: School #6]

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 7.	Generic	Required
2	What other reason did you think about when answering this item?	Specific	Required

Item-Specific Probes:

Additional Cross-subject Terminology Probes

Student Grades 8 and 12

No.	Probe	Type of Probe	Required/Conditional
1	You are currently using a Microsoft Surface Pro device to read and answer these items. Do you think of this device as a tablet or a laptop computer?	General	Required
2	Which phones do you think of when you hear the word “smartphone”? A “smartphone” is any phone that is able to connect to the Internet.	General	Required
3	Which tablets do you think of when you hear the word “tablet”? A “tablet” is bigger than a smartphone. It allows you to do many of the same things as a smartphone.	General	Required
4	Which social media apps or websites do you use most often?	General	Required
5	Can you explain in your own words what a “thesaurus” is?	General	Required
6	A thesaurus is a book that lists words that have exactly the same or nearly the same meaning in the same language. A thesaurus is not a dictionary. It does not provide definitions of words. Do you ever use a thesaurus when you work on assignments for school that require you to write several sentences or paragraphs?	General	Required
7	Can you explain in your own words what a “synonym” is?	General	Required
8	A synonym is a word that has exactly the same or nearly the same meaning as another word in the same language. Do you ever search for synonyms when you work on assignments for school that require you to write several sentences or paragraphs?	General	Required

Teacher Grade 8

No.	Probe	Type of Probe	Required/Conditional
1	You are currently using a Microsoft Surface Pro device to read and answer these items. Do you think of this device as a tablet or a laptop computer?	General	Required
2	Which phones do you think of when you hear the word “smartphone”? A “smartphone” is any phone that is able to connect to the Internet.	General	Required
3	Which tablets do you think of when you hear the word “tablet”? A “tablet” is bigger than a smartphone. It allows you to do many of the same things as a smartphone.	General	Required
4	Do you encourage students to use a thesaurus or to search for synonyms when working on assignments for school that require them to write several sentences or paragraphs?	General	Required
5	If yes, do you specifically use the words “thesaurus”, “synonym”, or do you use other language?	General	Required
6	If no, why don’t you encourage students to use a thesaurus or search for synonyms when working on assignments for school that require them to write several sentences or paragraphs?	General	Required

School Grades 8 and 12

No.	Probe	Type of Probe	Required/Conditional
1	You are currently using a Microsoft Surface Pro device to read and answer these items. Do you think of this device as a tablet or a laptop computer?	General	Required
2	Which phones do you think of when you hear the word “smartphone”? A “smartphone” is any phone that is able to connect to the Internet.	General	Required
3	Which tablets do you think of when you hear the word “tablet”? A “tablet” is bigger than a smartphone. It allows you to do many of the same things as a smartphone.	General	Required

General Debriefing and Thank You (For all student, teacher, and school administrator participants)

Before we finish, I’d like to hear any other thoughts you have about the survey questions we worked on today.

Is there anything else you would like to tell me about working on the survey questions?

Is there anything that you think could make these survey questions clearer?

Thank the participant for his/her time and provide gift card, as appropriate.

1 Please note that the order of probes will be reversed in approximately half of the interviews to ensure that each probe is administered.

2 Please note that the 60 minutes includes time for introductions, completing the interview, debriefing, and/or time for additional questions/feedback from the participants.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Qureshi, Farah
File Modified	0000-00-00
File Created	2021-01-20

National Assessment of Educational Progress (NAEP) 2022 Social Science and Economics Questionnaire Cognitive Interviews

NCES Cognitive, Pilot, and Field Test Studies System

Volume 2 NAEP 2022 SocSci & Econ Cog Labs