NAEP 2021 COVID-19 Educational Experiences Student, Teacher, and School Administrator Pretesting

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is OMB# 1850-0928. The time required to complete this information collection is estimated to average 45 minutes, including the time to review instructions and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission, please write to: National Assessment of Educational Progress (NAEP), National Center for Education Statistics (NCES), Potomac Center Plaza, 550 12th St., SW, 4th floor, Washington, DC 20202.

Authorization and Confidentiality Assurance

National Center for Education Statistics (NCES) is authorized to conduct NAEP by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622) and to collect students’ education records from education agencies or institutions for the purposes of evaluating federally supported education programs under the Family Educational Rights and Privacy Act (FERPA, 34 CFR §§ 99.31(a)(3)(iii) and 99.35). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C.§151).

OMB No. 1850-0928 Approval Expires 4/30/2023

The purpose of this pretesting activity is to inform the development of the student, teacher, and school questionnaires prior to use in the field. Below are the generic probes and example item-specific probes that students will be asked during the cognitive interviews and the general guiding questions teachers and school administrators will be asked to guide their review. Please note that some probes or guiding questions may not be addressed if time does not permit.

Student Interviewer Welcome Script and Assent/Consent

The following script does not have to be read verbatim. You, as the interviewer, should be familiar enough with the script to introduce the participant to the cognitive interview process in a conversational manner. The text in italics is suggested content for you to become thoroughly familiar with in advance. You should project a warm and reassuring tone toward the participant in order to develop a friendly rapport. You should use conversational language throughout the interview.

After answering questions and giving further explanation, begin the interview with the first item.

Note: Students will be administered all items and probes, if time permits. The duration of each cognitive interview will be 45-minutes¹.

Student Welcome Script

Hello, my name is <name >.

Thank you for agreeing to participate in our study. Your help is very important to us. Let me tell you a bit more about what we are going to do today.

I have some questions that students in many different schools across the country will be asked to answer about their learning experiences during school closures due to the coronavirus pandemic. The questions will be part of the National Assessment of Educational Progress (NAEP).

Some of the questions may be difficult to understand or answer. Our goal is to make these questions better so that other students can easily understand them. That is why we need your thoughts on the questions.

We want you to help us find out which questions are easy to understand and which are difficult to understand. There are no right or wrong answers, but please try to answer the questions the best you can. Your feedback will help the National Center for Education Statistics (NCES), which is part of the United States Department of Education, make these questions clearer for students like you. In the next 45-minutes, we will try to get through as many questions as possible.

This interview is being audio recorded so that researchers can review the recordings later. No one else will hear the recording. Your participation is voluntary and all of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Do you have any questions?

If, for any reason, the participant is no longer interested in participating, thank the participant for his or her or their time and end the interview.

Teacher and School Administrator Welcome Script and Assent/Consent

Teachers and school administrators will be asked to do an independent review of items on their own time, send feedback via email, and join a virtual meeting to discuss feedback with fellow committee/panel members or one-one-one with a researcher.

If teachers and school administrators opt to participate in a virtual meeting, the following script will be used. This script does not have to be read verbatim. You, as the facilitator, should be familiar enough with the script to introduce the participant to the review process in a conversational manner. The text in italics is suggested content for you to become thoroughly familiar with in advance. You should project a warm and reassuring tone toward the participant in order to develop a friendly rapport. You should use conversational language throughout the meeting.

After answering questions and giving further explanation, begin the meeting with the first item.

Note: Teachers and school administrators will be asked all guiding questions, if time permits. The duration of the virtual meeting will be 60-minutes².

Teacher Welcome Script

Hello, my name is <name>.

Thank you for agreeing to participate in our study. Let me tell you a bit more about what we are going to do today.

I have some questions that teachers in many different schools will be asked to answer about teaching and support of students as a result of the coronavirus pandemic health crisis that began during the 2019-2020 school year. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. There are no right or wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), which is part of the United States Department of Education, make these questions clearer for teachers like you. In the next hour, we will try to get through as many questions as possible.

Do you have any questions?

If, for any reason, the participant is no longer interested in participating, thank the participant for his or her or their time and allow them to sign off the meeting before continuing.

School Administrator Welcome Script

Hello, my name is <name >.

Thank you for agreeing to help with our study. Let me tell you a bit more about what we are going to do today.

I have some questions that school administrators in many different schools will be asked to answer about the school’s support of students and staff as a result of the coronavirus pandemic health crisis that began during the 2019-2020 school year. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. There are no right or wrong answers, but please try to answer the questions to the best of your ability. Your feedback will help the National Center for Education Statistics (NCES), which is part of the United States Department of Education, make these questions clearer for school administrators like you. In the next hour, we will try to get through as many questions as possible.

This interview is being audio recorded so that researchers can review the recordings later. No one else will hear the recording. Your participation is voluntary and all of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Do you have any questions?

If, for any reason, the participant is no longer interested in participating, thank the participant for his or her time and allow them to sign off the meeting before continuing.

Instructions and Generic Probes for Student Cognitive Interviews

Once the participant has joined the video call, the interviewer should share his or her video so that the participant can see the interviewer and share his or her own screen. The interviewer should ask the participant to read and answer the questions, one at a time. Each question will be either one discrete question (multiple choice or free response) or one matrix question (i.e., an item stem with two or more multiple choice sub-items). The respondent is not supposed to read the question aloud but should be asked to provide their response to the question aloud so it can be documented. Some items refer to future events, for example the 2020-2021 school year. For these items a student response to the question is not required. If a respondent indicates they do not know a word or phrase, the interviewer can read words aloud and/or explain the meaning of words for the respondent.

For most items, generic probes for discrete and matrix items below will be used (see Tables 1 and 2). In addition, Figures 1 and 2 show example discrete and matrix items from a previous NAEP 2018 cognitive interview activity. The structure of the items in this study will be similar (see Attachment 1).

Table 1. Generic Probes for Discrete Items (Multiple Choice and Free Response)–Full Set.

No.	Probe	Instructions for Interviewer
1	Can you explain to me what the question is asking? Please try to explain without repeating the wording of the item.	Ask this probe for all discrete questions AFTER the respondent has answered the question.
2	Were there any words or parts of this question that were confusing?  Yes  No	Ask this probe for all discrete questions.
3	What did you find confusing? What could we do to make the question less confusing?	Ask both probes only if the respondent answered YES to the previous question.
4	Would you say it was very easy, easy, difficult, or very difficult to answer this question?  Very Easy  Easy  Difficult  Very Difficult	Ask this probe for all discrete questions.
5	How could we make it easier to answer this question?	Ask this probe only if the respondent answered DIFFICULT or VERY DIFFICULT to the previous question.
6	How sure are you about your answer to this item?  Very Unsure  Unsure  Sure  Very Sure	Ask this probe for all discrete questions.
7	Were you comfortable answering this item?  Yes  No	Only ask this probe for discrete questions inquiring about potentially sensitive topics.

Figure 1. Example discrete item from 2018 cognitive interview.

Table 2. Generic Probes for Matrix Items – Full Set.

No.	Probe	Instructions for Interviewer
1	Look back at the first part of the item where you were asked a question. Can you explain to me what this question is asking? Please try to explain without repeating the wording of the item.	Ask this probe AFTER the respondent has answered the entire matrix question. This probe is specifically for the “stem” of the item, not the entire matrix item or a specific part/sub-item.
2	Can you explain to me what [part a., b., c., etc.] means? Please try to explain without repeating the wording of the item.	Ask this probe for all parts/sub-items. Please note that probes 2-6 should be asked together for one part/sub-item before moving on to the next part/sub-item.
3	Did you find any words or parts of this question confusing?  Yes  No	Ask this probe for the entire matrix.
4	What did you find confusing? What could we do to make the question less confusing?	Ask both probes only if the respondent answered YES to the previous question.
5	Would you say it was very easy, easy, difficult, or very difficult to answer this question?  Very Easy  Easy  Difficult  Very Difficult	Ask this probe for the entire matrix.
6	How could we make it easier to answer this question?	Ask this probe only if the respondent answered DIFFICULT or VERY DIFFICULT to the previous question.
7	How sure are you about your answers to this item?  Very Unsure  Unsure  Sure  Very Sure	Ask this probe for all matrix items. Ask this probe once per matrix, after completing all other probes pertaining to parts/sub-items.
8	Were you comfortable answering this item?  Yes  No	Only ask this probe for matrix questions inquiring about potentially sensitive topics. Ask this probe after completing all other probes pertaining to parts/sub-items.

Figure 2. Example matrix item from 2018 cognitive interview.

Additional Item-Specific and Cross-Item Terminology Probes for Student Cognitive Interviews

The generic probes listed above will be used for each question. In addition, there may be probes that reference terms, phrases, or high-level topics that are applicable to one or multiple items. Table 3 gives examples of these item-specific and cross-item probes.

Table 3. Example of item-specific and cross-item probes.

No.	Probe	Type of Probe	Required/Conditional
1	Can you explain what “coronavirus pandemic” means to you? If the student is able to provide a response, ask the following question: Can you think of an easier way to say “coronavirus pandemic”?	Generic (cross-item)	Required & Conditional
2	Can you explain what “online classes” means to you?	Specific	Required
3	Can you explain what “high-speed internet” means to you?	Specific	Required
4	Do you think it is hard to remember something that happened a year ago? For example, do you remember what you did last summer?	Generic (cross-item)	Required

Instructions and Guiding Questions for Teachers and School Administrators

Teacher and school administrator participants will be asked to share their general thoughts on the items developed (Attachments 2 and 3). These participants will be asked to conduct an independent review of the items and join a virtual meeting to discuss feedback of the items. To guide their feedback/facilitate the discussion, we will ask participants to focus on the following key questions:

How important do you consider each question for the NAEP coronavirus pandemic <teacher/school administrator> questionnaire?
Did you find any of the questions or response options confusing?
Are there any items that a <teacher/school administrator> respondent may not be able to accurately provide information for?
Are there any items that may be too burdensome for a <teacher/school administrator> to answer because the information is not readily available or easy to recall?

General Debriefing and Thank You (For all student cognitive interviews and teacher and school administrator virtual meetings)

Before we finish, I’d like to hear any other thoughts you have about the survey questions we worked on today.

Is there anything else you would like to tell me about working on the survey questions?

Is there anything that you think could make these survey questions clearer?

Thank the participant(s) for his or her or their time. Notify students that they will be sent their virtual gift card. Notify teachers and school administrators that their payment for the activity will be provided by the NAEP logistics contractor.

Attachment 1: Student Items

[Grade 4 and 8: Student #1]

[Grade 4 and 8: Student #2]

[Grade 8 only: Student #3]

[Grade 4 and 8: Student #4]

[Grade 4 and 8: Student #5]

[Grade 4 and 8: Student #6]

[Grade 4 and 8: Student #7]

[Grade 4 and 8: Student #8]

[Grade 4 and 8: Student #9]

[Grade 4 and 8: Student #10]

[Grade 4 and 8: Student #11]

[Grade 4 and 8: Student #12]

Attachment 2: Teacher Items

[Grade 4 and 8: Teacher #1]

[Grade 4 and 8: Teacher #2]

[Grade 4 and 8: Teacher #3]

[Grade 4 and 8: Teacher #4]

[Grade 4 and 8: Teacher #5]

[Grade 4 and 8: Teacher #6]

[Grade 4 and 8: Teacher #7]

[Grade 4 and 8: Teacher #8]

[Grade 4 and 8: Teacher #9]

[Grade 4 and 8: Teacher #10]

[Grade 4 and 8: Teacher #11]

[Grade 4 and 8: Teacher #12]

Attachment 3: School Administrator Items

[Grade 4 and 8: School #1]

[Grade 4 and 8: School #2]

[Grade 4 and 8: School #3]

[Grade 4 and 8: School #4]

[Grade 4 and 8: School #5]

[Grade 4 and 8: School #6]

[Grade 4 and 8: School #7]

[Grade 4 and 8: School #8]

[Grade 4 and 8: School #9]

[Grade 4 and 8: School #10]

[Grade 4 and 8: School #11]

1 Please note that the 45-minutes includes time for introductions, completing the interview, debriefing, and/or time for additional questions/feedback from the participants.

2 Please note that the 60-minutes includes time for introductions, completing the interview, debriefing, and/or time for additional questions/feedback from the participants.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Qureshi, Farah
File Modified	0000-00-00
File Created	2021-01-13

NAEP 2021 COVID-19 Educational Experiences Student, Teacher, and School Administrator Pretesting

NCES System Clearance for Cognitive, Pilot, and Field Test Studies 2019-2022

Volume II NAEP 2021 COVID19 SQ Pretesting