Trends in International Mathematics and Science Study (TIMSS 2023) Cognitive Interviews

Trends in International mathematics and science study (TIMSS 2023) Cognitive Interviews

Volume II

Protocols

REQUEST FOR OMB Clearance

OMB# 1850-0803 V.263

Submitted by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

February 2020

Table of Contents

TIMSS 2023 Cognitive Interview Protocol 3

Thank You and Closing Instructions 13

TIMSS 2023 Cognitive Interview Protocol

Overview

Each interview session will use a paper and pencil format to test out either one longer or two shorter sets of questions that are referred to as a Problem Solving and Inquiry set, or PSI set, for short. The PSI sets will be presented on paper as storyboards that show how the PSI set will appear to students on a digital device in the field test. As presented in Table 1 below, an interview session is composed of 6 or 8 parts and should last no more than 1 hour. Students administered two shorter PSI sets (expected to take 15 minutes each) will complete parts 1-8. Students administered one longer PSI set (expected to take 30 minutes) will complete parts 1-5 and 8. Each PSI set is administered in a one-on-one think-aloud session. During this part of the interview, the interviewer conducts quiet observations of the session, audio-records the session, and takes notes of the student’s verbal reporting.

Table 1. Overview of the Interview Session

Time (minutes) for short PSI session	Time (minutes) for long PSI session	Short PSI session	Long PSI session
10	10	1. Welcome and introduction
		2. Think-aloud practice
		3. Instructions for the test
15	30	4. Think-aloud session – PSI set 1
10	20	5. Debriefing of the think-aloud session
15		6. Think-aloud session – PSI set 2
10		7. Debriefing of the think-aloud session
		8. Closing of the session
Total: 60	Total: 60

Interviewer Instructions

The following scripts should not be read verbatim; rather, the interviewer should be familiar with the content in order to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation, as appropriate. Text in italics is suggested content with which the interviewer should be thoroughly familiar in advance. Text in brackets [ ] presents instructions for the interviewer. Text in carets < > means that the wording may change depending on context. The interviewer should project a warm and reassuring manner toward the student and use conversational language to develop a friendly rapport.

Please fill out Table 2 below before starting Part 1 of the interview.

Table 2. Background Information

Date
Name of interviewer
Student ID number
Student’s gender	 Male  Female
Student’s grade

Please keep in mind that your total interview time with the student should last no more than 1 hour. You are asked to record the time at the beginning of each part of the interview session. At the end of the interview session (either the short PSI session or the long PSI session), please fill out Table 3 below to record the beginning time of each part and to calculate how long each part took.

Table 3. Time Records for the Interview

Short PSI interview			Long PSI interview
Part	Begin time	Duration (minutes)	Part	Begin time	Duration (minutes)
1		Part 2-Part 1:	1		Part 2-Part 1:
2		Part 3-Part 2:	2		Part 3-Part 2:
3		Part 4-Part 3:	3		Part 4-Part 3:
4		Part 5-Part 4:	4		Part 5-Part 4:
5		Part 6-Part 5:	5		Part 8-Part 5:
6		Part 7-Part 6:
7		Part 8-Part 7:
8			8
		Total:			Total:

Part 1. Welcome and Introduction

***[Please note the time: ________ ]***

Hello, my name is <name of interviewer> and I work for the American Institutes for Research. It’s nice to meet you, and I thank you very much for coming here to help us out today.

[Make small talk to build rapport with the student by asking a question, such as:]

• What is your favorite subject in school?

• What have you been learning in school this week?

[When the student responds, follow up with one question to get the student used to talking. For instance:]

• That’s interesting—why do you enjoy <subject> so much?

Well, I think you’ll enjoy what we are going to be doing today. Let me explain.

You are here today for a special study for the National Center for Education Statistics, part of the Institute of Education Sciences in the U.S. Department of Education. This study is called the Trends in International Mathematics and Science Study, or TIMSS for short. TIMSS is a test of mathematics and science, which will be given on a digital device to students like you in different schools in the United States and in many countries around the world. The American Institutes for Research is conducting interviews with students to ask for your help in making the test questions better. 

Over the next hour, I will be asking you to use paper and a pencil to answer a group of the questions that may be included in the final version of the test, which will appear on a digital device. The questions you will be answering on paper are shown in the same way that they will appear on a digital device. Some of the questions may be easy to answer and some may not be easy to answer or to understand. This is fine. Your answers will not be considered right or wrong and I will not be grading your answers, but we need you to answer the test questions the best you can and as if you are taking a real test.

After you finish a question, we will talk about what you thought of it. For example, I want to know what you think about the instructions and the tasks. With your help, we hope to make the test better and clearer for students your age. Because the information you provide is so important, I am going to be taking notes while you are working and when we are talking [INSERT IF THERE IS AN OBSERVER: and my colleague will also be observing our session today.] Also, this interview will be audio-recorded so researchers can review the tapes later if they like. All of your personally identifiable information, like your name or your school, will be kept confidential, or secret, and your responses will only be used for research.

Remember, I am here to get information from you, and I will not be grading your work. Would you like to be a part of this study?

[If student agrees, continue; if not, end the interview. Also make sure you have collected the completed Parental Consent form by this time.]

If at any time you want to stop, just let me know.

[If, for any reason, the student is no longer interested in participating, thank the student for his/her time and end the interview.]

Before we continue, do you have any questions about what I just said? [Answer any questions the student may ask.]

Part 2. Think-Aloud Practice

***[Please note the time: ________ ]***

As you start answering some mathematics and science test questions, we want to learn what you think about these questions and how they are explained. One way we do this is to ask you to “think aloud” throughout what you are doing. All I mean by “think aloud” is for you to tell me what you’re thinking as you go through the questions and how you think you would answer them.

This might feel a little strange at first, so let’s do a short “think-aloud” practice. I’m going to give you a simple question and ask you to think aloud as you decide on your answer. [Hand student “think-aloud question” paper]

[Paper says: How many windows are there in the house or apartment where you live?]

[Some students may be silent. Encourage the student to say whatever he/she is thinking and probe as needed for detail. If the student appears to need additional help with the concept, suggest that you demonstrate thinking aloud. Use the same question and demonstrate the process.]

[When student completes the practice:] Thank you; that’s the kind of detail I’m looking for when you talk to me while reading and answering the questions.

Part 3. Instructions for the test

***[Please note the time: ________ ]***

As we just practiced, there are no right or wrong ways to think aloud. What I want to know is what you’re thinking about as you try to understand and answer a question. You do not need to say everything out loud, just the main points of how you go about answering a question: what you find confusing, what questions you have about what is being asked.

We will start the test shortly. As you answer the questions and think aloud, I will take some notes but not interrupt you, and we will discuss your thoughts after you are done with answering the questions. Also, I’m going to place the recorder here, right next to you so that it will record your think-alouds.

8th Grade Only You may use a calculator for these questions. You may use your own calculator or I can provide you with one now. Do you need a calculator?

[If yes, give the student a calculator provided by AIR. If no, proceed. If student asks for the calculator later, it is fine to give it to them at that time.]

So, do you have any questions before we start?

[If yes, answer any questions the student may have and proceed. If no, proceed.]

OK, let’s get started. [Turn on audio recorder].

[For students taking two short PSIs] You will be presented with two 15-minute sets of questions, and we will discuss them after the first set and also at the end. [For all students] Read the questions and write your answers in the space provided. Please answer them as best as you can. Let me know when you are finished, and we will begin talking about the questions you answered. [For students taking two short PSIs] If you are not finished in about 15 minutes, I will stop you to begin our first discussion. [For students taking one long PSI] If you are not finished in about 30 minutes, I will stop you to begin our discussion. You may begin now.

[Give the student no more than 15 minutes (or 30 minutes for longer PSI sets) to complete to each PSI set. Take notes in Part 4—and if applicable, Part 6 below—based on student think-alouds, paying particular attention to places where the student found problems or seemed confused.]

[Observations to Focus On:

1. Student perception of degree of challenge/difficulty (listen to think-alouds, as well as observe what students do in responding to a task)

2. Student engagement in the PSI (listen to think-alouds, as well as observe student’s ease or frustration with questions)]

Part 4. Think-Aloud Notes & Interviewer Observations – First PSI

***[Please note the time: ________ ]***

Insert Storyboard Description Here for Interviewer Notes

Part 5. Probing and Debriefing Questions – First PSI

[When the student has completed answering the PSI set or when approximately 15 minutes (for a short PSI) or 30 minutes (for a long PSI) have passed, begin the discussion.]

***[Please note the time: ________ ]***

Thank you for working on those <insert PSI name> questions. Now, I have several things to ask you about your work.

[If the student does not complete the test question set, ask the following question.].

Do you think you would have been able to complete the questions if you had been given more time?

[As a priority, you should ask specific probing questions below based on issues that came up in the think-alouds. You can include the notes for specific probing questions with the think-aloud notes you took for each question in Part 4.]

Probing Questions

Below are examples of specific probes based on think-alouds and observations (Note to OMB: These are examples that have been used in previous TIMSS cognitive interviews. The wording highlighted in yellow would be changed to reflect the TIMSS 2023 PSI questions):

• Can you tell me more about what you were thinking when you were answering the question <creating a six-sided garden for Lily>?

• What did you do to help you answer the [question]?

• You said, “I don’t know what I’m supposed to do” (or something similar) when you were looking at [the question]. Can you tell me what you found confusing?

• Have you seen the word <perimeter> [or other words or concepts that appeared difficult for the student] in other books you have read?

• What do you do if you read a word [or there is some new information] and you are not sure of its meaning?

• Can you tell me how you answered this question? Did you use <approach A> or <approach B>?

If you have time after the probing questions, the debriefing questions are prioritized so that you should ask the questions below in the order in which they are listed.

Debriefing Questions

1. Overall, do you think this set of test questions was easy, medium, or hard or a mix? [Follow up with] Why? [If appropriate] If you chose a mix, can you explain which parts were easier or harder?

2. Were there any test questions that were confusing? [Follow up with] Which question(s) were confusing and how were those questions confusing? [Follow up with, if appropriate] Do you have any suggestions for making that part of the test better or less confusing?

3. Did you feel that you were easily able to understand what each of the questions were asking you to do? Across the set of questions, was any part more difficult to understand? [Follow up by asking which part of the question(s) was more difficult to understand and why]

4. What was your favorite part of this set of questions? [Follow up with] Why?

5. Do you have any other comments?

Thank you. [If interview is for two short PSIs: Let’s move to the next and final set of questions.]

[If interview is for one longer PSI, skip to Part 8]

Part 6. Think- Aloud Notes & Interviewer Observations – Second PSI

***[Please note the time: ________ ]***

Insert Storyboard Description Here for Interviewer Notes

Part 7. Probing and Debriefing Questions – Second PSI

When the student has completed answering the first PSI set or when approximately 15 minutes have passed, begin the discussion.

***[Please note the time: ________ ]***

Thank you for working on those <insert PSI name> questions. Now, I have several things to ask you about your work.

[If the student does not complete the test question set, ask the following question.]

Do you think you would have been able to complete the set of test questions if you had been given more time?

[As a priority, you should ask specific probing questions below based on issues that came up in the think-alouds. You can include the notes for specific probing questions with the think-aloud notes you took for each question in Part 6.]

Probing Questions

Below are examples of specific probing questions based on think-alouds and observations. (Note to OMB: These are examples of probing questions that have been used in previous TIMSS cognitive interviews. The wording highlighted in yellow would be changed to reflect the TIMSS 2023 PSI questions):

• Can you tell me more about what you were thinking when you were answering the question <creating a six-sided garden for Lily>?

• What did you do to help you answer the [question]?

• You said, “I don’t know what I’m supposed to do” (or something similar) when you were looking at [the question]. Can you tell me what you found confusing?

• Have you seen the word <perimeter> [or other words or concepts that appeared difficult for the student] in other books you have read?

• What do you do if you read a word [or there is some new information] and you are not sure of its meaning?

• Can you tell me how you answered this question? Did you use <approach A> or <approach B>?

If you have time after the probing questions, the debriefing questions are prioritized so that you should ask the questions below in the order in which they are listed.

Debriefing Questions

1. Overall, do you think this set of test questions was easy, medium, or hard or a mix? [Follow up with] Why? [If appropriate] If you chose a mix, can you explain which parts were easier or harder?

2. Were there any questions that were confusing? [Follow up with] Which question(s) were confusing and how were those questions confusing? [Follow up with, if appropriate] Do you have any suggestions for making that part of the test better or less confusing?

3. Did you feel that you were easily able to understand what each of the questions were asking you to do? Across the set of questions, was any part more difficult to understand? [Follow up by asking which part of the question(s) was more difficult to understand and why]

4. What was your favorite part of this set of questions? [Follow up with why?]

5. Do you have any other comments?

Thank you.

[After the cognitive interview is complete or 60 minutes have passed, the interviewer should proceed to Part 8.]

Thank You and Closing Instructions

Part 8. Thank You for Participating

***[Please note the time: ________ ]***

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making the TIMSS test questions better.

[Collect materials and escort student out of the interviewing room and back to his or her parent. Please give the parent a $25 gift card along with the thank you letter and give the student a $50 gift card to thank them for their time and effort.

After the interview: Please go back to page 2 and fill out Table 3 to record the beginning time for each part and calculate how much time each part took.]

Think-Aloud Practice Question Handout (from Part 2, p. 6):

How many windows are there in the house or apartment where you live?