Change Request Memo

U NITED STATES DEPARTMENT OF EDUCATION

National Center for Education Statistics

September 15, 2020 (revised Oct. 1, 2020)

MEMORANDUM

To: Robert Sivinski, OMB

From: Lydia Malley, NCES

Through: Carrie Clarady, NCES

Re: Trends in International Mathematics and Science Study (TIMSS 2023) Cognitive Interviews (OMB# 1850-0803 v.273 – revised v.263)

The Trends in International Mathematics and Science Study (TIMSS) is an international assessment of fourth and eighth grade students’ achievement in mathematics and science. Since its inception in 1995, TIMSS has continued to assess students every 4 years, with the next TIMSS assessment, TIMSS 2023, being the eighth iteration of the study. On March 12, 2020 OMB approval (OMB# 1850-0803 v.263) was granted for the conduct of cognitive, in-person interviews of fourth- and eighth-grade mathematics and science assessment items that are being developed for the upcoming TIMSS 2023 assessment.

Due to the coronavirus pandemic, this cognitive interview study has been converted from an in-person to a virtual format to help limit the spread of the virus, reduce the burden on students and parents, and maintain the budget. The purpose of this request is to (a) change the mode of data collection from an in-person TIMSS cognitive interview format to virtual interview format using Adobe Connect software; (b) modify the data collection schedule to conduct the interviews in phases from October 2020 through March 2021; (c) modify the recruitment strategy to expand beyond the Washington, DC area and recruit students from across the United States; (d) modify the grade requirements to recruit students who have completed grade-level coursework due to the change in data collection dates from spring 2020 to fall 2020; and (e) modify the incentive format to Amazon gift cards, which will now be sent electronically to participants via e-mail.

The materials submitted have been revised to reflect the changes previously stated. The changes described in this document do not affect the total cost to the federal government for this study. There are no revisions to the estimated response burden, the total number of planned interviews, or the estimated number of interviews per phase.

The specific revisions made to the approved TIMSS 2023 clearance documents are detailed below. Text shown in red font denotes newly added text, and text that is crossed out and in red font denotes deleted text. Text shown in blue font denotes general updates to a section.

In response to the September OMB review comments, we have provided responses directly within the comment section of the document. In addition, text highlighted in yellow denotes revisions made in response to the September OMB review comments.

Revisions to Volume I

2) Background and Study Rationale

The TIMSS & PIRLS International Study Center at Boston College employs a collaborative process to develop the new achievement items needed for each TIMSS cycle. The development process is directed and managed by the staff of the TIMSS & PIRLS International Study Center, who have considerable experience in the measurement and assessment of mathematics and science achievement. They work collectively with the participating countries to develop the assessment frameworks and items and then to review and test the items. Prior to the Main Study, the items are evaluated in a field test. For additional item development testing, a pilot study is conducted prior to field testing. In the beginning stages of item development, cognitive interviews are employed to understand more about students’ experiences with the mathematics and science content being assessed in the items. More details on the item development process can be found at https://timss.bc.edu/timss2019/methods/chapter-1.html.

TIMSS transitioned to a digitally-based assessment, known as eTIMSS, in the 2019 administration. As part of the TIMSS 2023 item development process, specific new electronic assessment items will be pretested in a virtual cognitive laboratory setting on with a small number of students before they are administered to a larger sample through the field test and main study. During the cognitive labs, these assessment items will be in the form of presented in an online Adobe PDF format story boards on paper rather than in a fully interactive digital format. Information gathered from the cognitive interviews will be used to inform the development of each item’s content prior to creating them digitally for administration in the TIMSS field test.

In cognitive interviews (often referred to as a cognitive laboratory study or cog lab), an interviewer uses a structured protocol in a one-on-one interview (see Volume II for protocols). Cognitive interview studies are largely observational. Due to the coronavirus pandemic, this cognitive interview study has been converted from an in-person to a virtual format to help limit the spread of the virus, reduce the burden on students and parents, and maintain the budget. For this study students will use a computer in their home to join an online video virtual interview session using Adobe Connect. The students will answer a sample of TIMSS assessment questions (or items) displayed on their computer screen by the interviewer in an online Adobe PDF format and will use Adobe Connect tools to mark their responses shown as story boards as they normally would in a testing situation. After each item the interviewer will ask the student a few questions to gather information about student reactions to the item. The largely qualitative data collected will be mainly verbal reports in response to these questions, in addition to volunteered comments.

The objective of this cognitive interview study is to assess the clarity of the presentation of the assessment items and how well students understand what they have to do as they work through the items. The items will be a group of newly drafted higher complexity electronic items, called Problem Solving and Inquiry (PSI) items, in the form of online Adobe PDF story boards selected by the TIMSS & PIRLS International Study Center at Boston College. Each PSI item is estimated to take students either 10-15 minutes (short PSIs) or 20-30 minutes (long PSIs) to complete. The information gained from the study will be used by Boston College for potential revisions prior to creating the digital versions that will be tested during the TIMSS 2023 Field Test.

3) Recruitment and Data Collection, Sampling and Recruitment Plans

The TIMSS 2023 cognitive laboratory study will be conducted in a maximum of three phases. The first phase of cognitive interviews will be conducted in April October 2020 (or as soon as OMB clearance is approved), and the following two potential phases will be conducted in the winter or spring of 2021occur approximately 3 months after each previous phase. These additional phases will be administered as determined by Boston College’s needs and based on the availability of new PSI story boards to be tested. The implementation of this study in phases will allow interviews to be conducted with assessment items that are ready to be tested first, while others are still being prepared. As well, it will allow item developers at Boston College to make modifications to items tested in the initial phases to be tested again in later phases in order to identify whether improvements can be made.

Each phase of cognitive interviews will include Grade 4 and Grade 8 PSI items for testing. Each item will be administered to a total of 6 students in order to gain a range of perspectives on the quality of the item. Each cognitive interview session will test 1-2 PSIs, depending on the length of the PSI, and include time for debriefing questions, not to exceed a 1-hour session. Phase one of the cog labs, beginning in April October, will include up to 2 short Grade 4 PSI items administered to 6 students, and up to 3 Grade 8 PSI items (1 long and 2 short) rotated between up to 12 students, for a maximum total of 18 students in phase one. Subsequent phases may include less students, depending on the number of PSI items to be tested, but will not exceed a maximum of 56 students across all 3 phases.

Table 1: Estimated Sample Size by Phase

Cog Lab Phase	Number of Grade 4 Students	Number of Grade 8 Students	Max. Total Number of Students
Phase 1 (April October 2020)	6 (for 2 items)	12 (for 3 items)	18
Phase 2 & 3 combined (TBD 2020-21)	6-24 (for 2-6 items)	6-24 (for 2-6 items)	38
Max. Total			56*

*While number of students at each grade is to be determined and may not be equal across grades, the maximum number of students will not exceed 18 for Phase 1 and 38 for Phases 2 and 3 combined, and will not exceed a total of 56 students across all phases.

The cognitive labs will be conducted virtually by AIR staff in Washington, DC facilities using Adobe Connect software.

Students will be recruited from across the United States in the Washington, DC area by EurekaFacts. The EurekaFacts participant database includes tens of thousands of individuals nationally and is constantly refreshed and updated through the company’s independent participant outreach methods in both English and Spanish. These efforts ensure that the research participants recruited by EurekaFacts are not “professional respondents,” instead they are individuals with limited to no experience with qualitative research who can provide fresh and actionable information for stakeholders. As a result, the sample from the database will be representative of the survey’s population and should not impact testing results.

English-speaking students who demonstrate their comfort with talking to an interviewer during the initial recruitment screening step will be recruited from the following demographic populations in their databases of potential research participants for studies:

• A mix of race/ethnicity (Black, Asian, White, Hispanic, etc.);

• Mix of varied levels of mathematics ability;

• Mix of school types (public and private);

• A mix of socioeconomic background; and

• A mix of gender

EurekaFacts will collect demographic information during the recruitment screener and use what information they might already have in their databases to aim for a diverse sample of students. Although the sample will and should include a mix of student characteristics, The results will not explicitly measure differences by those characteristics, meaning that the sample need will not necessarily be representative of the grade 4 and 8 student populations.

Interested participants will be screened to ensure that students meet the criteria for participation in the study. See Appendix A for recruitment screener. When recruiting participants, staff will first speak to the parent/guardian of the interested minor before starting the screening process. During this communication, the parent/guardian will be informed about the objectives, purpose, and participation requirements of the data collection effort as well as the activities that it entails. After confirmation that participants are qualified, willing, and available to participate in the research project, they will receive a confirmation e-mail/letter (see Appendix B). Informed parental consent will be obtained for all respondents under the age of 18 who are interested in participating in the data collection efforts (see Appendix C). After the interview, the parent of the student who participated in the interview will receive a thank-you letter email and the electronic gift card incentives (see Appendix D). See Appendix A through D for all recruitment and communication materials.

3) Recruitment and Data Collection, Item Information

Items to be included in the study will be selected by the International Study Center at Boston College from the pool of Problem Solving and Inquiry (PSI) items being developed for the TIMSS 2023 Field Test. Problem Solving and Inquiry (PSI) items are intended to be administered digitally in the TIMSS Field Test and Main Study. The PSIs are scenario-based tasks with multiple questions that may take a student 10-15 minutes (short PSIs) or 20-30 minutes (long PSIs). For the purposes of these cognitive interviews, only paper-based online Adobe PDF storyboards of the PSIs will be presented to the students. The story boards will allow the students to work through the items on paper screen in a similar way as they would on an electronic device. The reason for using story boards during this stage of item development is to gather student input regarding the content of the items in order to inform development of PSIs prior to the labor-intensive process of developing the digital versions.

3) Recruitment and Data Collection, Calculator Use

Based on international guidelines set by IEA for calculator use in TIMSS assessments since 2003, calculators will not be permitted during fourth-grade cognitive interviews. However, the TIMSS policy on calculator use at the eighth grade is to give students the best opportunity to operate in settings that mirrors their classroom experiences. Thus, following the United States implementation of the calculator policy in the TIMSS main assessments since 2003, calculators will be permitted, but not required, during eighth-grade cognitive interviews. Because the TIMSS main assessments are conducted in schools, students in the U.S. have used the calculators they are accustomed to using and provided by their school. For this study eighth graders will be permitted, but not required, to bring use a calculator that they like to use for schoolwork. Additionally, AIR will provide basic calculators for eighth-grade students who do not bring their own calculator, should they wish to use a calculator.

3) Recruitment and Data Collection, Cognitive Interview Process

Participants will first be welcomed, introduced to the interviewer, and told they are there to help answer questions about how students answer assessment items. Participants will be reassured that their participation is voluntary and that their answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573 and 6 U.S.C. §151]. The interviewer will then review the technical logistics of Adobe Connect with the student and parent. At this time, participants will also be notified that the interview will be audio- and video recorded using the Adobe Connect software and shown the recorder and then asked for their verbal consent before the interview continues.

8. After the interview concludes, the interviewer will input their notes from the session within 24 hours and submit them and the Adobe Connect recording to the interview manager.

4) Consultations Outside the Agency

AIR is an established not-for-profit research organization. AIR offers facilities, tools, and staff to collect and analyze both qualitative and quantitative data. AIR will use facilities and their trained cognitive interviewer staff in Crystal City, VA and Rockville, MD to conduct the cognitive interviews virtually using Adobe Connect software.

EurekaFacts is a for-profit research firm that will be responsible for the student recruitment in the Washington, DC area.

5) Assurances of Confidentiality

A consent form that explains the purpose and duration of the interview—to be signed and returned before interviews are conducted—will be sent via e-mail to the parents or legal guardians of all students. Written consent will be obtained from legal guardians of all students before interviews are conducted. Interview The student will also be asked to provide verbal consent to participate at the start of the interview sessions, which will be audio recorded using Adobe Connect to assist with post-interview note-taking. Information about audio the interview recording will be included along with an opt-out box in the parent consent form (see Appendix C). Participants will be assigned a unique student identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files, secured for the duration of the study in a locked file cabinet restricted-access folder in AIR’s network, and will be destroyed after the final report is released. All e-mail exchanges with parents or legal guardians will be saved and archived in a restricted-access folder in AIR’s network and will be destroyed after the final report is released. Audio The Adobe Connect recordings will also be stored in a restricted-access folder in AIR’s network and will be deleted after the project concludes.

6) Estimate of Hourly Burden

Based on the proposed outreach and recruitment methods, we estimate initial respondent burden at 0.15 hours through phone calls. We are estimating a recruitment of 24 participants in phase one for the maximum 18 participants to show. In total across 3 phases of interviews we estimate a recruitment of 70 participants to achieve a total of 56 participants. Student interviews with items will be limited to 60 minutes for all students. The estimated burden for recruitment assumes attrition throughout the process. Interviews will be conducted virtually using the Adobe Connect software. take place at AIR facilities in Crystal City, VA and Rockville, MD.

7) Estimate of Costs for Paying Respondents

Each participating student will receive a $25 electronic Amazon gift card as a thank you for his or her time and effort. In addition, NCES is offering an electronic Amazon gift card of $25 for a will be given to the parent or legal guardian to thank him or her for the time involved in helping the child successfully connect to the Adobe Connect interview. and to help offset the travel/transportation costs of taking the participating student to and from the cognitive laboratory site. A generic gift card (e.g., Visa gift card) that can be used anywhere credit cards are accepted is the recommended incentive.

8) Costs to Federal Government

Table 3. Estimate of Costs

Activity	Provider	Estimated Cost
Design, preparation, and conduct of cognitive interviews (data collection, analysis, and reporting)	AIR	$84,000
Recruitment of students in Washington, DC area	EurekaFacts	$16,000
Total		$100,000

9) Schedule

Table 4. Schedule of Milestones and Deliverables

Activity	Dates
Expected OMB clearance	September 2020
Recruit participants for Phase 1 (subsequent to OMB clearance)	September - October 2020
Phase 1 data collection	October 2020
Phase 1 data analysis	October 2020
Phase 1 study report	Early November 2020
Potential Phase 2 recruit students	Tentative January 2021
Phase 2 data collection	Tentative January 2021
Phase 2 data analysis	Tentative January 2021
Phase 2 study report	Tentative February 2021
Potential Phase 3 recruit students	Tentative April 2021
Phase 3 data collection	Tentative May 2021
Phase 3 data analysis	Tentative May 2021
Phase 3 study report	Tentative June 2021

Revisions to Volume II

Overview

Each interview session will use a paper and pencil virtual format to test out either one longer or two shorter sets of questions that are referred to as a Problem Solving and Inquiry set, or PSI set, for short. The PSI sets will be presented on paper Adobe Connect software as PDF storyboards that show how the PSI set will appear to students on a digital device in the field test. As presented in Table 1 below, an interview session is composed of 6 or 8 parts and should last no more than 1 hour. Students administered two shorter PSI sets (expected to take 15 minutes each) will complete parts 1-8. Students administered one longer PSI set (expected to take 30 minutes) will complete parts 1-5 and 8. Each PSI set is administered in a one-on-one think-aloud session. During this part of the interview, the interviewer conducts quiet observations of the session, audio-records the session, and takes notes of the student’s verbal reporting.

Table 1. Overview of the Interview Session

Time (minutes) for short PSI session	Time (minutes) for long PSI session	Short PSI session	Long PSI session
10	10	1. Welcome and introduction	1. Welcome and introduction
		2. Think-aloud and Adobe Connect practice	2. Think-aloud and Adobe Connect practice
		3. Instructions for the test	3. Instructions for the test
15	30	4. Think-aloud session – PSI set 1	4. Think-aloud session – PSI set 1
10	20	5. Debriefing of the think-aloud session	5. Debriefing of the think-aloud session
15		6. Think-aloud session – PSI set 2
10		7. Debriefing of the think-aloud session
		8. Closing of the session	6. Closing of the session
Total: 60	Total: 60

Part 1. Welcome and Introduction

Hello, my name is <name of interviewer> and I work for the American Institutes for Research. It’s nice to meet you, and I thank you very much for joining online today. I have enabled your ability to use your microphone and webcam for this interview, so on the top menu you can use the drop-down arrows to select the microphone and webcam you would like to share during the interview. If you are having difficulties setting this up or if the volume is too low, I can help you troubleshoot. coming here to help us out today.

Once all three icons are green and active, then I should be able to see, hear, and speak with you.

To parents (if present at the time of the interview): I would like to let you know that you can stay in the room, but please sit off to the side during the interview to limit distractions. At the end of the interview, I will ask you to please return on camera so I can discuss the process for how you will receive the gift cards.

…

Over the next hour, I will be asking you to use software tools paper and a pencil to answer a group of the questions that may be included in the final version of the test, which will appear on a digital device. These items are confidential and are not to be shared. If you have a cell phone nearby, please place it to the side so you are not distracted. Please make sure to close all other windows on your computer, as other tasks may interfere with the software we are using. Let me know when you have done that. The questions you will be answering on paper are shown in the same way that they will appear on a digital device.

Some of the questions may be easy to answer and some may not be easy to answer or to understand. This is fine. Your answers will not be considered right or wrong and I will not be grading your answers, but we need you to answer the test questions the best you can and as if you are taking a real test.

After you finish a question, we will talk about what you thought of it. For example, I want to know what you think about the instructions and the tasks. With your help, we hope to make the test better and clearer for students your age. Because the information you provide is so important, I am going to be taking notes while you are working and when we are talking [INSERT IF THERE IS AN OBSERVER: and my colleague will also be observing our session today.] Also, this interview will be audio-recorded so researchers can review the tapes recorded video files later if they like necessary. All of your personally identifiable information, like your name or your school, will be kept confidential, or secret, and your responses will only be used for research.

…

Part 2. Think-Aloud Practice

This might feel a little strange at first, so let’s do a short “think-aloud” practice. I’m going to give you a simple question and ask you to think aloud as you decide on your answer. I may remind you throughout the interview to think-aloud as you are working, since it’s not usual for people to do this.[Open student “think-aloud question” document on the screen so child can view it.] [Hand student “think-aloud question” paper]

[Document Paper says: How many windows are there in the house or apartment where you live?]

[Some students may be silent. Encourage the student to say whatever he/she is thinking and probe as needed for detail. Let the child know it is okay to estimate the number of windows. If the student appears to need additional help with the concept, suggest that you demonstrate thinking aloud. Use the same question and demonstrate the process.]

…

[Interviewer will enable drawing privileges for the student by clicking on their name in Adobe Connect.]

Now, for the second practice task, I would like you to use the software tools on the left-hand side of your screen to draw the shape of the window in your home that is closest to you. You can do this by clicking the main square icon on the black bar on the left-hand side to then select either the square or line icons. [Model the steps of inserting a square and a line shape into the PDF being shared.]

[If the parent asks if they can help the child figure out how to use the Adobe drawing tools, please let them know that they can do so for the practice exercise period.]

[Document says: Please use the shape or drawing tools on the left to draw the window closest to you.]

[When student completes the practice:] Thank you for using the tools to draw the window closest to you.

[Document says: Please also use the text box feature on the left-hand side of your screen to click and type in your response to how many windows there are in the house or apartment you live in.

For your third and final practice task, please use the text box feature by selecting the “T” icon on the black bar on the left-hand side of your screen to click and type in your response to the question regarding how many windows there are in your house or apartment. Allow me to model this function first. [Insert a text box for the student.] Now you try.

[When student completes the practice:] Thank you for using the tools to insert the text box with your response to the first question.

After you have inserted a shape, line, or textbox, you can use the Undo and Redo functions (i.e., the curved arrows) on the bottom of the list to remove and restore the changes that you have recently made. Allow me to demonstrate this, then feel free to practice it if you would like. [Demonstrate the Undo/Redo functions for the student.]

Part 3. Instructions for the test

We will start the test shortly. As you answer the questions and think aloud, I will take some notes but not interrupt you, and we will discuss your thoughts after you are done with answering the questions. Also, I’m going to begin the recording now. place the recorder here, right next to you so that it will record your think-alouds.

8th Grade Only You may use a your own calculator for these questions. You may use your own calculator or I can provide you with one now. Do you need a calculator?

[If yes, give the student a calculator provided by AIR instruct them on how to access the calculator application on their computer. If no, proceed. If student asks for the calculator later, it is fine to give it to for them to access it at that time.]

…

So, dDo you have any questions before we start?

OK, let’s get started. [Turn on audio recorder Start the recording].

[For students taking two short PSIs] You will be presented with two 15-minute sets of questions, and we will discuss them after the first set and also at the end. [For all students] Read the questions and use the software tools that we practiced to mark your answers write your answers in the space provided. Please answer them as best as you can. Let me know when you have are finished all of the questions on a page, and I will turn to the next page. Once you have finished all of the questions, and we will begin talking about the questions you answered. Remember to think aloud as much as possible while you read and move through the questions. Let me know when you are finished, and we will begin talking about the questions you answered.

…

[Troubleshooting: The interviewer can assist throughout if the student forgets how to use the tools or asks a question about using the software. These are examples of situations when the interviewer can provide technical support to the student:

1. If the student experiences technical difficulty during the interview, the interviewer should intervene or help the student choose a software tool.

2. If the student forgets which button is used for a textbox, the interviewer can point the student to the tool.

3. If the student clicks on something he or she is not supposed to use, the interviewer should intervene or provide help if asked.]

Part 5. Probing and Debriefing Questions – First PSI

Below are examples of specific probes based on think-alouds and observations. Please show the specific items being probed to refresh the student’s memory. (Note to OMB: These are examples that have been used in previous TIMSS cognitive interviews. The wording highlighted in yellow would be changed to reflect the TIMSS 2023 PSI questions):

Part 7. Probing and Debriefing Questions – Second PSI

When the student has completed answering the second first PSI set or when approximately 15 minutes have passed, begin the discussion.

…

Below are examples of specific probing questions based on think-alouds and observations. Please show the specific items being probed to refresh the student’s memory. (Note to OMB: These are examples of probing questions that have been used in previous TIMSS cognitive interviews. The wording highlighted in yellow would be changed to reflect the TIMSS 2023 PSI questions):

Part 8. Thank You for Participating

[Ask if the child’s parent can join the video, and if so, confirm their e-mail address. Please send the Thank You E-mail along with the electronic gift card links via e-mail prior to ending the meeting with them and confirm they have received it. Once confirmed, thank them for their time and effort.

If the child’s parent is not available, please thank the child for their time and effort and let them know that you will be sending the Amazon gift card to their parent by email. After the interview, send the Thank You E-mail along with the electronic gift card links via e-mail to the email address the parent used when submitting their consent form.]

[Collect materials and escort student out of the interviewing room and back to his or her parent. Please give the parent a $25 gift card along with the thank you letter and give the student a $50 gift card to thank them for their time and effort.

Think-Aloud Practice Question Document (at end)

Think-Aloud Practice Question Document Handout (from Part 2, p. 6):

…

Please use the shape or drawing tools on the left-hand side of your screen to draw the window closest to you.

Please also use the text box feature on the left-hand side of your screen to type in your response to how many windows there are in your house or apartment.

Revisions to Appendix A. Recruitment Screener/Phone Script

Design: 56 total in-depth interviews with 4th- and 8th-graders

Specifications:

• Students must be in 4th or 8th grade for interviews conducted through June 2020 and must have just completed 4th grade or 8th grade in the prior school year for interviews starting in July - November October 2020

• Students must have varied mathematics abilities

• Students must have a mix of demographic characteristics: gender, race/ethnicity, socioeconomic status (of parents), and school type

• Students must have access to a computer with a webcam and audio, and an internet connection

We are looking for students to help us understand what they think about new interactive items for an international math and science study, called the Trends in International Mathematics and Science Study, or TIMSS [note: pronounced tims, like short for Timothy]. The study is conducted at regular intervals in the United States and many other countries around the world. TIMSS provides information that allows countries to understand the math and science knowledge of students globally. In the U.S., the National Center for Education Statistics (NCES), part of the Institute of Education Sciences in the Department of Education, sponsors this study in collaboration with the American Institutes for Research (AIR). The purpose of the study is to find out what students think about the new items in order to help us make the study better and more understandable for 4th- and 8th-graders. The study would take place with an AIR researcher through an online video interview at AIR offices in Crystal City, Virginia, or Rockville, Maryland, to discuss the new math and science questions for TIMSS. I’d like to ask you a few questions to see if you qualify for the study.

P2b. What grade is your child currently in?

RECORD GRADE

(April–June 2020: MUST be in 4th or 8th grade. OTHERWISE, THANK AND TERMINATE)

(July–November 2020: MUST have completed 4th grade or 8th grade in the prior school year. OTHERWISE, THANK AND TERMINATE)

P2c. To participate in an online interview, you would need access to a desktop computer or a laptop computer capable of connecting to a reliable and high-quality internet connection. Do you have such access?  

	SELECT ONE
Yes	1	®®(CONTINUE)
No	2	®®(THANK AND TERMINATE)

P2d. To participate in an online interview, you will also need access to speakers, a webcam, and a microphone on a desktop computer or a laptop computer. Do you have access to a webcam, speakers, and a microphone so that you can both see and hear AND be heard and seen? 

	SELECT ONE
Yes	1	®®(CONTINUE)
No	2	®®(THANK AND TERMINATE)

P4. We are conducting a study with students about new items for a study on math and science.………If your child qualifies, do you give your permission for us to invite him or her to participate in a 60-minute interview to talk about these new items? $5075 in total will be given at the end of the interview as a thank-you (gift cards of $25 will be given to the parent and $2550 to the student), and parking will be validated at the interview site.

P6. Before your child participates in the research study, you must sign and return a parental consent form. Your child can either bring the signed form to the interview or you can sign electronically, or you can scan or take a photograph of it and then e-mail it back. Without a signed form, your child cannot participate in the interview and receive the honorarium. May I e-mail you a parental consent form if your child chooses to participate?

(ASK P7. ONLY TO PARENTS WITH A CHILD WHO IS CURRENTLY IN GRADE 5)

P7. How well would you say your child typically does in his/her math class? (READ LIST)

				SELECT ONE
		Outstanding (letter grade A)		1		→(ATTEMPT TO RECRUIT A MIX)
		Very good (letter grade B)		2
		Satisfactory (letter grade C)		3
		Needs improvement (letter grade D)		4
		Not achieving (letter grade F)		5
(DO NOT READ)		Refused		6

(ASK P8. ONLY TO PARENTS WITH A CHILD WHO IS CURRENTLY IN GRADE 9)

P8. What best describes the math class your child is taking now? MARK ONE RESPONSE ONLY.

				SELECT ONE
		General mathematics		1
		Introduction to Algebra/Pre-algebra		2
		Algebra		3
		Integrated or sequential mathematics (course that includes high school algebra and high school geometry topics)		4
		Algebra II		5
		Geometry		6
(DO NOT READ)		Refused		7
	Don’t know		8

P9. Is your child male or female? (DO NOT READ OPTIONS)

		SELECT ONE
	Male	1	(ATTEMPT TO RECRUIT A MIX)
	Female	2

*After inserting the new question P9, the item number for later items in the section were re-numbered.*

As I explained to your parent, we are looking for your help in developing questions for a test about math and science. The test is called the Trends in International Mathematics and Science Study, or TIMSS, and is conducted all around the world. We are looking for students to join an online video virtual interview in order to tell us what they think about the new math and science questions so that we can make the test better. Right now, I’d like to talk to you to find out if you are able to participate in the study. There are no “right” or “wrong” answers to any of these questions, and whatever you say is OK. OK?

ASK THE FOLLOWING QUESTIONS OF THE STUDENT:

C1. What grade level are you currently in? (DO NOT READ LIST)

RECORD GRADE

(April–June 2020: MUST be in 4th or 8th grade. OTHERWISE, THANK AND TERMINATE)

(July–November 2020: MUST have completed 4th grade or 8th grade in the prior school year. OTHERWISE, THANK AND TERMINATE)

(ASK C3a. ONLY TO STUDENT WHO IS CURRENTLY IN GRADE 5)

C3a. How well would you say you typically do in your math class? (READ LIST)

		SELECT ONE
	Outstanding (letter grade A)	1
	Very good (letter grade B)	2
	Satisfactory (letter grade C)	3
	Needs improvement (letter grade D)	4
	Not achieving (letter grade F)	5
(DO NOT READ)	Refused	6

(ASK C3b. ONLY TO STUDENT WHO IS CURRENTLY IN GRADE 9)

C3b. What best describes the math class you are taking now? MARK ONE RESPONSE ONLY.

				SELECT ONE
		General mathematics		1
		Introduction to Algebra/Pre-algebra		2
		Algebra		3
		Integrated or sequential mathematics (course that includes high school algebra and high school geometry topics)		4
		Algebra II		5
		Geometry		6
(DO NOT READ)		Refused		7
	Don’t know		8

C4. How comfortable do you think you would feel talking online with a researcher you haven’t met before about how clear a set of questions and instructions are to you? Would you be: (READ LIST)

C6. RECORD GENDER OF STUDENT. DO NOT ASK.

		SELECT ONE
	Male	1	(ATTEMPT TO RECRUIT A MIX)
	Female	2

ASK TO SPEAK TO PARENT AGAIN:

NOTE TO INTERVIEWERS: IT IS EXTREMELY IMPORTANT THAT STUDENTS IN THIS STUDY BE ARTICULATE. IF STUDENT CANNOT OR WILL NOT EASILY GIVE A ONE- TO TWO-SENTENCE UNPROMPTED ANSWER IN WELL-UNDERSTOOD ENGLISH, PLEASE TERMINATE USING THE LANGUAGE BELOW (FOR STUDENTS WHO DO NOT QUALIFY).

(FOR STUDENTS WHO DO NOT QUALIFY): Thank you very much for your time, and we really appreciate your and your child’s interest, but unfortunately, your child does not qualify for this current project. Have a good morning/afternoon/evening.

(FOR NINTH-GRADERS WHO DO QUALIFY): For some items, your child can use a calculator during the interview. Do you have a calculator in the home that your child can use?

	SELECT ONE
Yes	1	(CONTINUE)
No	2	(THANK YOU, WE WILL SHOW YOUR CHILD HOW TO USE THE CALCULATOR ON YOUR COMPUTER INSTEAD)

SCHEDULING: Thank you so much for your willingness to help us. Your child qualifies to participate in an online video interview session with a researcher to discuss the new math and science questions for TIMSS. We are scheduling students for interview sessions between <DATE> and <DATE>. What day and time work best for you?

Enter date/time: ______________________________________________

ALTERNATE SCHEDULING: Thank you so much for your willingness to help us. Your child qualifies to participate in an online video interview session with a researcher to discuss the new math and science questions for TIMSS. We have sessions available at the following dates/times. Which of these dates/times would work best for you? 

• <DATE> and <TIME>.  

• <DATE> and <TIME>.  

• <DATE> and <TIME>.  

Thank you. As a reminder, the interviews will last approximately 60 minutes and we ask that you sign into the online session 10 minutes early to ensure a good connection. The interview is strictly for research purposes. Let me assure you that no one will attempt to sell or enroll you in anything. At the completion of the interview, we will provide via e-mail a $25 Amazon gift card for your child and a $25 Amazon gift card for the parent who helps the child join the video interview session.

So that we can send you a confirmation email about your participation in this project, will you please provide us with your contact information? 

RECORD FIRST NAME: ____________________________________ 

RECORD LAST NAME: _____________________________________ 

RECORD EMAIL ADDRESS: ________________________________ 

RECORD MOBILE PHONE NUMBER: _________________________________ 

RECORD HOME PHONE NUMBER: ___________________________________

Great, we will send you a confirmation email with the date, time, and connection instructions. Once again, thank you for your time.

INVITE TO SESSION

Your child qualifies to participate in an online video interview session with a researcher at AIR offices in Crystal City, Virginia, or Rockville, Maryland, to discuss the new math and science questions for TIMSS. The interview is scheduled for (INSERT DAY/DATE AND TIME). The session will last 60 minutes, and you should plan to join 10 minutes early to start the interview software. The interview is strictly for research purposes. Let me assure you that no one will attempt to sell or enroll you in anything. At the completion of the interview, we will provide via e-mail a $2550 Amazon gift card for your child and a $25 Amazon gift card for the parent who helps the child join the video interview session who provides the child with transportation to and from the interview.

Will you and [child’s name] be able to attend the interview?

	SELECT ONE
Yes	1	(CONTINUE)
No	2	(THANK AND TERMINATE)

In which AIR office do you prefer to have the interview take place?

	SELECT ONE
Crystal City, VA 1400 Crystal Drive 10th Floor Arlington, VA 22202	1
Rockville, MD 6003 Executive Blvd. 3rd Floor Rockville, MD 20852	2
	[THIS
THIS

To attend the session and receive your honorarium, you must bring with you the signed “PARENTAL CONSENT” form that we will e-mail you (or scan it and e-mail it back to us).

R evisions to Appendix B. Confirmation Letter E-mail

Confirmation E-mailLetter for the AIR Crystal City Office

TO: [INSERT NAME]

JOB#: XXXXXXX

Thank you for accepting our invitation to participate in a research interview. We will begin promptly at the scheduled time and it is important that you arrive 10 minutes early to check in plan to join by computer 10 minutes early to download, install, and open the Adobe Connect software prior to the interview. If you are late do not join the interview on time, we cannot guarantee that you will be interviewed or paid.

We will e-mail an invitation and detailed instructions on how to join the Adobe Connect interview within the next 48 hours.

LOCATION:

American Institutes for Research

1400 Crystal Drive, 10th Floor

Arlington, VA 22202

DATE:      

TIME:      

DURATION: 1 hour

TOPIC: Cognitive Laboratory Interviews for TIMSS 2023

GIFT CARD: Your child will receive a $2550 Amazon electronic gift card as a thank-you for their participation. and the parent or legal guardian who assists the child in joining the online interview will receive a $25 Amazon electronic gift card. If the parent or legal guardian brings the child to and from the interview, they will also receive a $25 gift card as a thank-you.

PLEASE BRING THE FOLLOWING ITEMS WITH YOU TO THE RESEARCH INTERVIEW:

• Parent’s or guardian’s driver’s license or valid photo ID

• Your child’s glasses, if needed for reading on paper

• Signed parental consent form

• (8th grade only): Calculator

PRIOR TO THE RESEARCH INTERVIEW. PLEASE COMPLETE THE PARENTAL CONSENT FORM BY SIGNING, SCANNING, OR TAKING A PHOTO OF IT AND E-MAILING IT BACK TO [ffonseca@air.org].

[FOR 8TH GRADE STUDENT LETTER] Please have a calculator with you when you join the video interview session.

Please plan to use a quiet space for the duration of the interview. Please make sure the computer you will be using for the interview has a webcam and microphone for audio. You may use a headset if distractions need to be limited.

The National Center for Education Statistics (NCES), within the U.S. Department of Education, conducts TIMSS in the United Status as authorized by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average 60 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or comments or concerns about the contents or the status of your individual submission of this questionnaire, please e-mail: Lydia.Malley@ed.gov, or write directly to: Lydia Malley, National Center for Education Statistics, Potomac Center Plaza, 550 12th Street, SW, Room 4014, Washington, DC 20202.

OMB No. 1850-0803, Approval Expires XX/XX/20XX.

* Please note that we deleted the directions to the different in-person interview locations from the revised OMB Appendix (previously on pages 11 – 16).*

Revisions to Appendix C. Parental Consent Document

Thank you for your interest in participating in the research project described below. This consent form explains the research study. Please read it carefully, and feel free to ask questions about anything you do not understand. Contact information is included under question 8, below. If you do not have questions now, you may ask them if they occur to you later. The collection of information in this study is authorized by 20 U.S. Code, Section 9543. The U.S. Office of Management and Budget has approved the data collection under the OMB control number XXXX-XXXX1850-0803 (expires XX/XX/XXXX).

1. What is this study about?

The National Center for Education Statistics (NCES), part of the Institute of Education Sciences in the U.S. Department of Education, in collaboration with the American Institutes for Research, is conducting interviews with students who are currently in or who have recently completed 4th or 8th grade about new test questions in an assessment called the Trends in International Mathematics and Science Study (TIMSS). Feedback about the questions will help NCES improve the questions and make sure they can be easily understood by students across the country.

For more information on TIMSS 2023, please view the TIMSS brochure at https://timssandpirls.bc.edu/timss2023/downloads/T2023_TIMSS_Brochure.pdf.

2. Where How will the research interviews take place?

The online video interviews will take place at the American Institutes for Research offices located at either 1400 Crystal Drive, 10th Floor, Arlington, VA 22202, or 6003 Executive Blvd, 3rd Floor, Rockville, MD 20852 virtually using a computer and the Adobe Connect software platform. Participants will receive an e-mail invitation from an AIR researcher within the next 48 hours with more information on how to join the meeting.

5. How long will the research interview last and when it will take place?

The research interview will last up to 60 minutes and will require only one visit to our office online video virtual session. Interviews will be held Monday through Saturday between 9 am and 8 pm, with the last interview of the day beginning at 7 pm.

By signing below, you agree that your child, _______________________________, may participate in this study. Your signature below will indicate that your questions have been answered satisfactorily and that you have read and understood the information provided above. Your signature also indicates that you understand that your child’s session will be audio-recorded unless you mark the opt-out box below.

Signature: ___________________________________________________Date:______________________

Audio r Recording opt-out

The National Center for Education Statistics (NCES), within the U.S. Department of Education, conducts TIMSS in the United Status as authorized by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average 60 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or comments or concerns about the contents or the status of your individual submission of this questionnaire, please e-mail: Lydia.Malley@ed.gov, or write directly to: Lydia Malley, National Center for Education Statistics, Potomac Center Plaza, 550 12th Street, SW, Room 4014, Washington, DC 20202.

OMB No. 1850-0803, Approval Expires XX/XX/20XX.

Revisions to Appendix D. Thank You E-mail/Letter

Thank You E-mail/Letter to Parents of Student Participants (Under Age 18)

As a thank you for your participation, we are providing a $25 Amazon electronic gift card to your child for his/her participation and a $25 Amazon electronic gift card to you for assisting your child with participation in the online interview. Below are the claim codes for the two Amazon electronic gift cards:

Gift card #1 ($25): XXXX-XXXXXX-XXXX

Gift card #2 ($25): XXXX-XXXXXX-XXXX

To redeem your Amazon gift card, you may enter the claim code during checkout when placing an order. Or, you may apply the gift card directly to your account by going to “Your Account” on the Amazon.com website, clicking “Apply a Gift Card to Your Account,” and entering your claim code and clicking “Apply to Your Balance.”

550 12^th Street, S.W., Washington, DC 20202

Our mission is to ensure equal access to education and to promote educational excellence throughout the Nation.