Trends in International Mathematics and Science Study (TIMSS 2023) Cognitive Interviews

Trends in International mathematics and science study (TIMSS 2023) Cognitive Interviews

Appendix

REQUEST FOR OMB Clearance

OMB# 1850-0803 V. 263

Submitted by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

October 2020

Table of Contents

A. Recruitment Screener/Phone Script 3

B. Confirmation E-mail 12

C. Parental Consent Document 14

D. Thank You E-mail 17

Appendix A. Recruitment Screener/Phone Script

Design: 56 total in-depth interviews with 4th- and 8th-graders

Specifications:

• Students must have completed 4th grade or 8th grade in the prior school year for interviews starting in October 2020

• Students must have varied mathematics abilities

• Students must have a mix of demographic characteristics: gender, race/ethnicity, socioeconomic status (of parents), and school type

• Students must have access to a computer with a webcam and audio, and an internet connection

STUDENT’S FIRST NAME:

PARENT’S FIRST NAME (SPOKEN TO / GIVES PERMISSION): _____________________________

DATE RECRUITED: RECRUITED BY:

DATE CONFIRMED: CONFIRMED BY:

ASK TO SPEAK TO THE HEAD OF THE HOUSEHOLD:

Hello, this is from EurekaFacts, an independent market research firm.

We are conducting a brief, but important study about education. Your participation is voluntary, and you will receive no sales pitch or follow-up calls based on your participation. I’d like to tell you a little bit about the study first; is now a good time? (YES or NO and call back later)

We are looking for students to help us understand what they think about new interactive items for an international math and science study, called the Trends in International Mathematics and Science Study, or TIMSS [note: pronounced tims, like short for Timothy]. The study is conducted at regular intervals in the United States and many other countries around the world. TIMSS provides information that allows countries to understand the math and science knowledge of students globally. In the U.S., the National Center for Education Statistics (NCES), part of the Institute of Education Sciences in the Department of Education, sponsors this study in collaboration with the American Institutes for Research (AIR). The purpose of the study is to find out what students think about the new items in order to help us make the study better and more understandable for 4th- and 8th-graders. The study would take place with an AIR researcher through an online video interview to discuss the new math and science questions for TIMSS. I’d like to ask you a few questions to see if you qualify for the study.

P1. Do you have a child in your household who is 18 years old or younger?

	SELECT ONE
Yes	1	®®(CONTINUE)
No	2	®®(THANK AND TERMINATE)

P2a. How old is this child?

RECORD AGE

P2b. What grade is your child currently in?

RECORD GRADE

(MUST have completed 4th grade or 8th grade in the prior school year. OTHERWISE, THANK AND TERMINATE)

P2c. To participate in an online interview, you would need access to a desktop computer or a laptop

computer capable of connecting to a reliable and high-quality internet connection. Do you have such access?  

	SELECT ONE
Yes	1	®®(CONTINUE)
No	2	®®(THANK AND TERMINATE)

P2d. To participate in an online interview, you will also need access to speakers, a webcam, and a

microphone on a desktop computer or a laptop computer. Do you have access to a webcam, speakers, and a microphone so that you can both see and hear AND be heard and seen? 

	SELECT ONE
Yes	1	®®(CONTINUE)
No	2	®®(THANK AND TERMINATE)

P3. What type of school does your child currently attend? (READ LIST)

	SELECT ONE
Public school	1	(ATTEMPT TO RECRUIT A MIX)
Private school	2
Homeschool Don’t know	3 4	(THANK AND TERMINATE)

P4. We are conducting a study with students about new items for a study on math and science.………If your child qualifies, do you give your permission for us to invite him or her to participate in a 60-minute interview to talk about these new items? $50 in total will be given at the end of the interview as a thank-you (gift cards of $25 will be given to the parent and $25 to the student).

	SELECT ONE
Yes	1	(CONTINUE)
No	2	(THANK AND TERMINATE)

P5. From your perspective, on a scale from 1 to 4 (where 1 means “not at all comfortable” and 4 means “very comfortable”), how comfortable would your child be expressing his or her thoughts to a researcher he/she has not met before about whether a set of study questions and instructions presented to him or her are clear and understandable? (READ LIST)

	SELECT ONE
Very Comfortable	4	(CONTINUE)
Somewhat Comfortable	3
Not Too Comfortable	2	(THANK AND TERMINATE)
Not At All Comfortable	1

P6. Before your child participates in the research study, you must sign and return a parental consent form. You can sign electronically, or you can scan or take a photograph of it, and then then e-mail it back. Without a signed form, your child cannot participate in the interview and receive the honorarium. May I e-mail you a parental consent form if your child chooses to participate?

	SELECT ONE
Yes	1	(CONTINUE)
No	2	(THANK AND TERMINATE)

The next few questions are for classification purposes only.

(ASK P7. ONLY TO PARENTS WITH A CHILD WHO IS CURRENTLY IN GRADE 5)

P7. How well would you say your child typically does in his/her math class? (READ LIST)

				SELECT ONE
		Outstanding (letter grade A)		1		→(ATTEMPT TO RECRUIT A MIX)
		Very good (letter grade B)		2
		Satisfactory (letter grade C)		3
		Needs improvement (letter grade D)		4
		Not achieving (letter grade F)		5
(DO NOT READ)		Refused		6

(ASK P8. ONLY TO PARENTS WITH A CHILD WHO IS CURRENTLY IN GRADE 9)

P8. What best describes the math class your child is taking now? MARK ONE RESPONSE ONLY.

				SELECT ONE
		General mathematics		1
		Introduction to Algebra/Pre-algebra		2
		Algebra		3
		Integrated or sequential mathematics (course that includes high school algebra and high school geometry topics)		4
		Algebra II		5
		Geometry		6
(DO NOT READ)		Refused		7
	Don’t know		8

P9. Is your child male or female? (DO NOT READ OPTIONS)

		SELECT ONE
	Male	1	(ATTEMPT TO RECRUIT A MIX)
	Female	2

P10. Is your child Hispanic or Latino? (DO NOT READ OPTIONS)

	SELECT ONE
Yes	1	→(ATTEMPT TO RECRUIT A MIX)
No	2
Prefer not to answer	3

P11. Which of the following categories best describes your child’s ethnic or racial background? (READ LIST)

	SELECT ALL THAT APPLY
American Indian or Alaska Native	1	(ATTEMPT TO RECRUIT A MIX)
Asian	2
Black or African American	3
Native Hawaiian or Other Pacific Islander	4
White	5
Other	6

P12. Is your total annual household income before taxes: (READ LIST)

		CIRCLE ONE
	Less than $15,000	1	(ATTEMPT TO RECRUIT A MIX)
	$15,000 to less than $30,000	2
	$30,000 to less than $45,000	3
	$45,000 to less than $60,000	4
	$60,000 to less than $75,000	5
	$75,000 to less than $100,000	6
OR	$100,000 or more	7
(DO NOT READ)	Refused	8

At this point, I’d like to be able to ask your child a few questions, during which time you are welcome to stay on the line. Is he/she available to speak with me now? (If not, get a re-call time and call back.) What is your child’s name? (RECORD NAME)

Again, you are welcome to stay on the line, but please allow your child to speak for him/herself. Thank you.

RECRUITER: Please determine how talkative and articulate the student is. It is critical that we speak with students who are comfortable expressing themselves with someone they are meeting for the first time. Please use your best judgment and recruit only students who are thoughtful and freely express themselves in a manner that is relevant to the topic being discussed.

All recruits MUST be articulate students who can answer open-ended questions and can provide support or explanation for their responses (for instance, “I like it because...” or “I don’t like it because...”).

DO NOT recruit any students who have difficulty answering these questions, whose parent answers FOR them, who seem hesitant or shy speaking with you (an adult they do not know), who answer “I don’t know,” or who CANNOT provide an explanation supporting their responses.

SAY TO STUDENT:

Hi (insert target student’s first name). My name is and I have been speaking with your parent for a few minutes.

As I explained to your parent, we are looking for your help in developing questions for a test about math and science. The test is called the Trends in International Mathematics and Science Study, or TIMSS, and is conducted all around the world. We are looking for students to join an online video interview in order to tell us what they think about the new math and science questions so that we can make the test better. Right now, I’d like to talk to you to find out if you are able to participate in the study. There are no “right” or “wrong” answers to any of these questions, and whatever you say is OK. OK?

Great.

ASK THE FOLLOWING QUESTIONS OF THE STUDENT:

C1. What grade level are you currently in? (DO NOT READ LIST)

RECORD GRADE

(MUST have completed 4th or 8th grade in the prior school year. OTHERWISE, THANK AND TERMINATE)

C2. What is the name of your school? (RECORD)

_________________________________________________________________________

(ASK C3a. ONLY TO STUDENT WHO IS CURRENTLY IN GRADE 5)

C3a. How well would you say you typically do in your math class? (READ LIST)

		SELECT ONE
	Outstanding (letter grade A)	1
	Very good (letter grade B)	2
	Satisfactory (letter grade C)	3
	Needs improvement (letter grade D)	4
	Not achieving (letter grade F)	5
(DO NOT READ)	Refused	6

(ASK C3b. ONLY TO STUDENT WHO IS CURRENTLY IN GRADE 9)

C3b. What best describes the math class you are taking now? MARK ONE RESPONSE ONLY.

				SELECT ONE
		General mathematics		1
		Introduction to Algebra/Pre-algebra		2
		Algebra		3
		Integrated or sequential mathematics (course that includes high school algebra and high school geometry topics)		4
		Algebra II		5
		Geometry		6
(DO NOT READ)		Refused		7
	Don’t know		8

C4. How comfortable do you think you would feel talking online with a researcher you haven’t met before about how clear a set of questions and instructions are to you? Would you be: (READ LIST)

	SELECT ONE
Very Comfortable	4	(CONTINUE)
Somewhat Comfortable	3
Not Too Comfortable	2	(THANK AND TERMINATE)
Not At All Comfortable	1

C5. Please tell me what your favorite subject is in school and why. (WRITE VERBATIM)

ASK TO SPEAK TO PARENT AGAIN:

NOTE TO INTERVIEWERS: IT IS EXTREMELY IMPORTANT THAT STUDENTS IN THIS STUDY BE ARTICULATE. IF STUDENT CANNOT OR WILL NOT EASILY GIVE A ONE- TO TWO-SENTENCE UNPROMPTED ANSWER IN WELL-UNDERSTOOD ENGLISH, PLEASE TERMINATE USING THE LANGUAGE BELOW (FOR STUDENTS WHO DO NOT QUALIFY).

(FOR STUDENTS WHO DO NOT QUALIFY): Thank you very much for your time, and we really appreciate your and your child’s interest, but unfortunately, your child does not qualify for this current project. Have a good morning/afternoon/evening.

(FOR NINTH-GRADERS WHO DO QUALIFY): For some items, your child can use a calculator during the interview. Do you have a calculator in the home that your child can use?

	SELECT ONE
Yes	1	(CONTINUE)
No	2	(THANK YOU, WE WILL SHOW YOUR CHILD HOW TO USE THE CALCULATOR ON YOUR COMPUTER INSTEAD)

SCHEDULING: Thank you so much for your willingness to help us. Your child qualifies to participate in an online video interview session with a researcher to discuss the new math and science questions for TIMSS. We are scheduling students for interview sessions between <DATE> and <DATE>. What day and time work best for you?

Enter date/time: ______________________________________________

ALTERNATE SCHEDULING: Thank you so much for your willingness to help us. Your child qualifies to participate in an online video interview session with a researcher to discuss the new math and science questions for TIMSS. We have sessions available at the following dates/times. Which of these dates/times would work best for you? 

• <DATE> and <TIME>.  

• <DATE> and <TIME>.  

• <DATE> and <TIME>.  

Thank you. As a reminder, the interviews will last approximately 60 minutes and we ask that you sign into the online session 10 minutes early to ensure a good connection. The interview is strictly for research purposes. Let me assure you that no one will attempt to sell or enroll you in anything. At the completion of the interview, we will provide via e-mail a $25 Amazon gift card for your child and a $25 Amazon gift card for the parent who helps the child join the video interview session.

So that we can send you a confirmation email about your participation in this project, will you please provide us with your contact information? 

RECORD FIRST NAME: ____________________________________ 

RECORD LAST NAME: _____________________________________ 

RECORD EMAIL ADDRESS: ________________________________ 

RECORD MOBILE PHONE NUMBER: _________________________________ 

RECORD HOME PHONE NUMBER: ___________________________________

Great, we will send you a confirmation email with the date, time, and connection instructions. Once again, thank you for your time.

A ppendix B. Confirmation E-mail

TO: [INSERT NAME]

JOB#: XXXXXXX

Thank you for accepting our invitation to participate in a research interview. We will begin promptly at the scheduled time and it is important that you plan to join by computer 10 minutes early to download, install, and open the Adobe Connect software prior to the interview. If you do not join the interview on time, we cannot guarantee that you will be interviewed or paid.

We will e-mail an invitation and detailed instructions on how to join the Adobe Connect interview within the next 48 hours.

DATE:      

TIME:      

DURATION: 1 hour

TOPIC: Cognitive Laboratory Interviews for TIMSS 2023

GIFT CARD: Your child will receive a $25 Amazon electronic gift card as a thank-you for their participation, and the parent or legal guardian who assists the child in joining the online interview will receive a $25 Amazon electronic gift card.

PRIOR TO THE RESEARCH INTERVIEW. PLEASE COMPLETE THE PARENTAL CONSENT FORM BY SIGNING, SCANNING, OR TAKING A PHOTO OF IT AND E-MAILING IT BACK TO [ffonseca@air.org].

[FOR 8^TH GRADE STUDENT LETTER] Please have a calculator with you when you join the video interview session.

Please plan to use a quiet space for the duration of the interview. Please make sure the computer you will be using for the interview has a webcam and microphone for audio. You may use a headset if distractions need to be limited.

We look forward to seeing you and would like to stress the importance of your participation in this study. If you find for any reason that you cannot fulfill your commitment, DO NOT invite anyone else to take your place. Instead, please call EurekaFacts at (240) 403-4800 so we can find a replacement and meet our obligation to our clients.

The National Center for Education Statistics (NCES), within the U.S. Department of Education, conducts TIMSS in the United Status as authorized by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average 60 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or comments or concerns about the contents or the status of your individual submission of this questionnaire, please e-mail: Lydia.Malley@ed.gov, or write directly to: Lydia Malley, National Center for Education Statistics, Potomac Center Plaza, 550 12th Street, SW, Room 4014, Washington, DC 20202.

OMB No. 1850-0803, Approval Expires XX/XX/20XX.

Appendix C. Parental Consent Document

PARENTAL CONSENT DOCUMENT

FOR PARTICIPATION IN RESEARCH

Title of Research Study: Cognitive Interviews in Support of the Trends in International Mathematics and Science Study (TIMSS) 2023

Thank you for your interest in participating in the research project described below. This consent form explains the research study. Please read it carefully, and feel free to ask questions about anything you do not understand. Contact information is included under question 8, below. If you do not have questions now, you may ask them if they occur to you later. The collection of information in this study is authorized by 20 U.S. Code, Section 9543. The U.S. Office of Management and Budget has approved the data collection under the OMB control number 1850-0803 (expires XX/XX/XXXX).

1. What is this study about?

The National Center for Education Statistics (NCES), part of the Institute of Education Sciences in the U.S. Department of Education, in collaboration with the American Institutes for Research, is conducting interviews with students who have recently completed 4th or 8th grade about new test questions in an assessment called the Trends in International Mathematics and Science Study (TIMSS). Feedback about the questions will help NCES improve the questions and make sure they can be easily understood by students across the country.

For more information on TIMSS 2023, please view the TIMSS brochure at https://timssandpirls.bc.edu/timss2023/downloads/T2023_TIMSS_Brochure.pdf.

2. How will the research interviews take place?

The online video interviews will take place virtually using a computer and the Adobe Connect software platform. Participants will receive an e-mail invitation from an AIR researcher within the next 48 hours with more information on how to join the meeting.

3. What will happen during the research interview?

Each participant will take a few multi-step math or science test items. An interviewer will ask your child to think aloud while they are completing the test items, and the interviewer will ask a few questions at the end of the interview about your child’s experience and thoughts on the test questions. Your child will not be graded. The purpose of this interview is to gain student feedback on the clarity and difficulty of test questions in order to improve the assessment.

4. Will the answers be kept confidential and the information private?

Your child will be assigned a unique student identifier, and at no time will your child’s name be linked to any of his/her answers. The interviewers will take notes that will only be used by the researchers. Also, all of your child’s answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

5. How long will the research interview last and when it will take place?

The research interview will last up to 60 minutes and will require only one online video session. Interviews will be held Monday through Saturday between 9 am and 8 pm, with the last interview of the day beginning at 7 pm.

6. Will the results go to my child’s school?

Your child’s answers will not be disclosed to his or her school and will not be linked to any personally identifiable information, such as the school’s name.

7. What are the possible risks of being in the study?

There are no known risks to participating in this study.

8. Who can I contact with questions or for further information?

Amy Rathbun is conducting this study at the American Institutes for Research on behalf of NCES. If you have any questions, please call her at (202) 403-6156 or e-mail her at arathbun@air.org. If Amy is not available, please call Frank Fonseca at (202) 403-5612 or e-mail him at ffonseca@air.org.

9. Can participation end early?

Your participation in this study is completely voluntary. Your child has the right to refuse to answer particular questions. You may elect to withdraw from this study at any time.

By signing below, you agree that your child, _______________________________, may participate in this study. Your signature below will indicate that your questions have been answered satisfactorily and that you have read and understood the information provided above. Your signature also indicates that you understand that your child’s session will be recorded unless you mark the opt-out box below.

Print Name: ________________________________________

Signature: ___________________________________________________Date:______________________

Recording opt-out

The National Center for Education Statistics (NCES), within the U.S. Department of Education, conducts TIMSS in the United Status as authorized by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average 60 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or comments or concerns about the contents or the status of your individual submission of this questionnaire, please e-mail: Lydia.Malley@ed.gov, or write directly to: Lydia Malley, National Center for Education Statistics, Potomac Center Plaza, 550 12th Street, SW, Room 4014, Washington, DC 20202.

OMB No. 1850-0803, Approval Expires XX/XX/20XX.

Appendix D. Thank You E-mail

Thank You E-mail to Parents of Student Participants (Under Age 18)

Subject: Thank You for Your Participation – TIMSS Research Study

<DATE>

Dear <NAME>,

On behalf of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences in the U.S. Department of Education, thank you for allowing <CHILD’S NAME> to participate in the Trends in International Mathematics and Science Study (TIMSS) research interview. <CHILD’S NAME>’s input helped us evaluate test questions for the TIMSS assessment. The assistance <he/she> provided, combined with input from other students, was incredibly useful and informative for improving the assessment for students internationally. We greatly appreciate your taking the time to bring <CHILD’S NAME> to and from the interview location to participate.

As a thank you for your participation, we are providing a $25 Amazon electronic gift card to your child for his/her participation and a $25 Amazon electronic gift card to you for assisting your child with participation in the online interview. Below are the claim codes for the two Amazon electronic gift cards:

Gift card #1 ($25): XXXX-XXXXXX-XXXX

Gift card #2 ($25): XXXX-XXXXXX-XXXX

To redeem your Amazon gift card, you may enter the claim code during checkout when placing an order. Or, you may apply the gift card directly to your account by going to “Your Account” on the Amazon.com website, clicking “Apply a Gift Card to Your Account,” and entering your claim code and clicking “Apply to Your Balance.”

For more information on TIMSS, please visit the NCES TIMSS website at http://nces.ed.gov/timss/index.asp.

Again, thank you for your assistance.

Sincerely,

Amy Rathbun

American Institutes for Research

(202) 403-6156