Trends in International mathematics and science study (TIMSS 2015) Cognitive Interviews

Trends in International mathematics and science study (TIMSS 2015) Cognitive Interviews

Volume II

Cognitive Interview Protocols

REQUEST FOR OMB Clearance

OMB# 1850-0803 v.86

Submitted by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

September 2013

Table of Contents

Cognitive Laboratory Script: Welcome, Introduction and Assent/Consent 3

Example Generic Test Question Script: Grade 4 and Grade 8 5

Example Generic Test Question Script: TIMSS Advanced – Grade 12 10

Thank You and Closing Instructions 14

Cognitive Laboratory Script: Welcome, Introduction and Assent/Consent

The following script should not be read word for word, but rather the interviewer should be familiar with its content to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. Text written in italics is to be read aloud by the interviewer. Text in <brackets> is generic and should be tailored to fit the grade level and subject test questions being studied.

Part 1. Welcome and Introduction Script

Hello, my name is <name of interviewer> and I work for the American Institutes for Research in <city, state>. It’s nice to meet you and thank you very much for coming here to help us out today. Before we get started I’d like to learn about what you have been doing in school recently.

Create small talk to build rapport with the student by asking a question, such as:

• What is your favorite subject in school?

• What have you been learning in school this week?

When the student responds, follow up with 2 or 3 questions to get the student used to talking, such as:

• That’s interesting—why do you enjoy <subject> so much?

• What are you studying in <subject> at the moment?

• And what’s one of the most interesting things you have studied in <subject> so far?

Good. Well I think you’ll enjoy what we are going to be doing today. Let me begin by explaining why I am here and what you are going to be doing.

For TIMSS 2015 interviews:

You are participating in a special study sponsored by the National Center for Education Statistics which is a part of the U.S. Department of Education. The study is looking at sample <subject> test questions from the Trends in International Mathematics and Science Study, or TIMSS for short. TIMSS is a test of mathematics and science which will be given to <insert grade 4 or grade 8> students like you in different schools in the United States and in many countries around the world. The goal of this study is to improve the test questions. We are asking for your help in making the questions better.

For TIMSS Advanced interviews:

You are participating in a special study sponsored by the National Center for Education Statistics which is a part of the U.S. Department of Education. The study is looking at sample <subject> test questions from the Advanced Trends in International Mathematics and Science Study, or TIMSS Advanced for short. TIMSS Advanced is an international assessment in mathematics and science given to students in grade 12 who have taken or are taking calculus or advanced physics across many different countries, including the United States. The goal of this study is to improve the test questions and we are asking for your help in making the questions better.

For both TIMSS 2015 and TIMSS Advanced interviews:

Over the next hour, I will be asking you to answer a small group of the questions that may be included on the final version of the test. Some of the test questions may be easy to answer and some may not be easy to answer or to understand. Your answers will not be considered right or wrong, but we need you to answer the test questions the best you can. It’s okay if you do not know how to answer a question. I will not be grading your work today or giving you the answers. Only group results will be shared so no one will know that it was you who answered the question or how you specifically answered the question. To help us make the test better we need to you to answer each question like it was a real test.

Remember, I am not here to give you a test and I will not be grading your work. I am here to find out how students like you react to test questions like these, and to hear your suggestions for how the questions can be made better. Your answers and our discussions will help make the questions clearer for students like you. Because the information you provide is so important, I am going to be taking notes while you are working and when we are talking. Your feedback will be used only for statistical purposes and may not be used for any other purpose except as required by law.

If at any time you decide you do not want continue, that is your choice and you may stop.

If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

Before we continue, do you have any questions about what I just said? [Answer any questions the student may ask.]

Okay, let’s begin.

Example Generic Test Question Script

Grade 4 and Grade 8

Part 2. Interview Instructions

Students will be given a series of 4 or more mathematics test questions or 4 or more science test questions one question at a time. The student will be asked to complete each test question as they normally would under classroom testing conditions, stopping after answering each test question to discuss their response and to respond to a series of general and question-specific follow-up probes from the interviewer.

The actual TIMSS questions to be used in the study will be similar to questions previously used in TIMSS which have been released. Example TIMSS grade 4 and 8 mathematics and science test questions are provided on the following pages. Additional example test questions can be found here: http://nces.ed.gov/timss/educators.asp.

Example Grade 4 and 8 Test Questions:

Grade 4

Grade 8

Test questions may be multiple-choice or they may ask students to construct their own response. Interviewers will be provided with answer keys but will not provide students with the correct answers to the questions.

Individual test questions may be written in the format of a single question or a set of questions. If the test question has multiple parts, the student may be asked to work and discuss each part separately. When this happens, the student will be able to refer back to the parts of the question he or she has already completed as they complete other parts of the question.

Interviewers are not allowed to read any portion of the test question to students or provide any definitions.

Students are not allowed to use calculators in responding to the grade 4 mathematics or science questions.

Students are allowed to use calculators in responding to the grade 8 mathematics or science questions. If the student does not bring their own calculator, one will be provided to them.

Part 3. Introduction of Test Questions Script

You are going to be answering a small group of <subject> test questions just as you normally would on a test, except I will give you one question at a time. After you finish each question, we will stop and I will ask you several questions about it. When we are finished talking, I will give you the next question to work on. You will have about 3 to 5 minutes to work on each question before we talk about it. Some questions have multiple parts and I might ask you to work on one part at a time. If you are not finished when I ask you to stop that is OK. If you are finished before the time is up just let me know and we will begin talking about the question. Remember we need you to answer each question the best you can.

(8^TH GRADE ONLY) You may use a calculator for these test questions if you would like to at any time. You may use your own calculator or I can provide you with one now. Do you need a calculator? [If yes, give the student a calculator provided by AIR, answer any questions the student may have and proceed. If no, answer any questions the student may have and proceed.]

Are you ready to begin? [If yes, give the student the first test question. If no, answer any questions the student may have. Be sure all of their questions are answered before continuing.]

Part 4. Individual Test Question Script

Script: Here is question #X. Read the test question carefully and write your answer in the space provided. Please answer it as best as you can. Let me know when you are finished and we will begin talking about the test question. If you are not finished in about <insert 3, 4, or 5> minutes, I will stop you to begin our discussion. You may begin now.

Instructions: Give the student no more than 3 to 5 minutes, depending on the item, as determined by AIR prior to the interview, to respond to the question. Take notes while the student is working on the item. If you see areas where the students may be having difficulty ask the student about those during your discussion.

When the student has completed answering the question or when approximately five minutes has passed begin the discussion.

Thank you for working on that <insert subject> question. Now, I have several things to ask you about your work and the test question itself. Remember, I will be taking notes as we talk.

If the item is multiple-choice the interviewer should note whether the student answered the test question correctly. If the item is constructed-response, the interviewer should note if the answer is correct, if possible. If the interviewer is unsure, scoring should occur after the interview is completed.

If the student does not complete the test question, ask the following question and then move to the general questions. Ask additional follow-up probes as appropriate depending on the type of test question, the student’s responses to the general questions or interviewer observations and time remaining.

Do you think you would have been able to complete the question if you would have been given more time? [Ask this question if you asked the student to stop working after the allotted amount of time has passed.]

The interviewer will ask the first general question to all students to help identify clarity of understanding issues. Follow-up questions should be asked as needed for further clarification in an area. Specific questions may be included depending on the test question contents. Text placed in brackets is generic text that will be tailored to suit the particular test question being studied. The interviewer will have the list of all interviewing questions (general, follow-up, and specific) for each test question.

General questions:

• Was there anything in the question that was not clear or you did not understand? [If the student answers yes, ask the student to identify which parts of the question were unclear and why.]

If the interviewer noted that the student answered the question correctly and found it to be clear, the interviewer should ask the remaining 3 general questions as total interview time permits and then move to Part 5 of the script.

• Describe to me how you went about solving the problem.

• Did you think this problem was easy, medium or hard? Why do you say that?

• Have you answered questions like this in school or on other tests? [If yes, ask the students when.]

Additional probes:

• General follow-up questions

• What do you think <insert appropriate part of the question> means?

• Describe what you think the question is asking you to do.

• How do you think this <insert appropriate part of the question> helps you answer the question?

• Were there any words in the questions that you did not understand?

• Use your own words to tell me what you think <insert word> means?

• For test questions which have been broken into parts and the student is asking the second or third part of the question

• What additional information from the <insert appropriate part of the original stimulus> did you need to use to solve this particular question?

• Did you use the information from other parts of the question to solve this problem? If yes, what information did you use and how?

• For test questions with graphs/diagrams/tables

• What do you think this <table, chart, graph> shows?

• Did you use the information in <table, chart, graph> to solve the problem? If yes, what information did you use and how?

• For multiple-choice test questions

• How did you decide what answer was best?

• Why did you rule out the options you did not choose?

• Did more than one choice seem correct to you? Why?

• For constructed response test questions

• A complete answer to the test question includes <insert appropriate part of correct response>. Could you tell this would be required in your answer from the way the test question is worded? Why or why not?

Part 5. Test Question Transition Script

Thank you. Let’s move to the next question.

Interviewer moves to next part of multi-part test question, or next test question, as appropriate and repeats Part 4 and 5 until the cognitive interview is completed or approximately 60 minutes has passed.

After the cognitive interview is complete or 60 minutes have passed, the interviewer should proceed to Part 6.

Example Generic Test Question Script

TIMSS Advanced – Grade 12

Part 2. Interview Instructions

Students will be given a series of 4 or more mathematics test questions or 4 or more physics test questions one question at a time. The student will be asked to complete each test question as they normally would under classroom testing conditions, stopping after answering each test question to discuss their response and to respond to a series of general and question-specific follow-up probes from the interviewer.

The actual TIMSS questions to be used in the study will be similar to questions previously used in TIMSS which have been released. Example TIMSS Advanced mathematics and science test questions are provided below.

Example Grade 12 Test Questions:

Additional example test questions can be found in the TIMSS 2008 Advanced International Database (http://timssandpirls.bc.edu/timss_advanced/idb.html).

Test questions may be multiple-choice or they may ask students to construct their own response. Interviewers will be provided with answer keys but will not provide students with the correct answers to the questions.

Individual test questions may be written in the format of a single question or a set of questions. If the test question has multiple parts, the student may be asked to work and discuss each part separately. When this happens, the student will be able to refer back to the parts of the question he or she has already completed as they complete other parts of the question.

Interviewers are not allowed to read any portion of the test question to students or provide any definitions.

Students are allowed to use graphing calculators in responding to the mathematics or science questions. If the student does not bring their own calculator, one will be provided to them.

Part 3. Introduction of Test Questions Script

You are going to be answering a small group of <subject> test questions just as you normally would on a test, except I will give you one question at a time. After you finish a question we will stop and I will ask you several questions about it. When we are finishing talking, I will give you the next question to work on. You will have about 5 to 8 minutes to work on each question before we talk about it. If the question has multiple parts, I might ask you to work on and discuss one part at a time. If you are not finished with a question when I ask you to stop that is OK. If you are finished before the time is up just let me know and we will begin talking about the question. Remember we need you to answer each question the best you can.

You may use a graphing calculator for these test questions if you would like to at any time. You may use your own graphing calculator or I can provide you with one now. Do you need a calculator? [If yes, give the student a calculator provided by AIR, answer any questions the student may have and proceed. If no, answer any questions the student may have and proceed.]

Are you ready to begin? [If yes, give the student the first question. If no, answer any questions the student may have. Be sure all of their questions are answered before continuing.]

Part 4. Individual Test Question Script

Script: Here is question #X. Read the question carefully and write your answer in the space provided. Please answer it as best as you can. Let me know when you are finished and we will begin talking about the question. If you are not finished in about <insert 5, 6, 7, or 8> minutes, I will stop you to begin our discussion. You may begin now.

Instructions: Give the student no more than 5 to 8 minutes, depending on the item, as determined by AIR prior to the interview, to respond to the test question. Take notes while the student is working on the test question. If you see areas where the students may be having difficulty ask the student about those during your discussion.

When the student has completed answering the test question or when approximately five minutes has passed begin the discussion.

Thank you for working on that <insert subject> question. Now, I have several things to ask you about your work and the test question itself.

If the item is multiple-choice the interviewer should note whether the student answered the test question correctly. If the item is constructed-response, the interviewer should note if the answer is correct, if possible. If the interviewer is unsure, scoring should occur after the interview is completed.

If the student does not complete the test question, ask the following and then move to the general questions. Ask additional follow-up probes as appropriate depending on the type of test question, the student’s responses to the general questions or interviewer observations and time remaining.

Do you think you would have been able to complete the question if you would have been given more time? [Ask this question if you asked the student to stop working after the allotted amount of time has passed.]

The interviewer will ask the first general question to all students to help identify clarity of understanding issues. Follow-up questions should be asked as needed for further clarification in an area. Specific questions may be included depending on the item contents. Text placed in brackets is generic text that will be tailored to suit the particular test question being studied. The interviewer will have the list of all questions (general, follow-up, and specific) for each test question.

General questions:

• Was there anything in the question that was not clear or you did not understand? [If the student answers yes, ask the student to identify which parts of the question were unclear and why.]

If the interviewer noted that the student answered the test question correctly and found it to be clear, the interviewer should ask the remaining 3 general questions as total interview time permits and then move to Part 5 of the script.

• Describe to me how you went about solving the problem.

• Did you think this problem was easy, medium or hard? Why do you say that?

• Have you answered test questions like this in school or on other tests? [If yes, ask the students when.]

Additional probes:

• General follow-up questions

• What do you think <insert appropriate part of the question> means?

• Describe what you think the question is asking you to do.

• How do you think this <insert appropriate part of the question> helps you answer the question?

• Were there any <subject> terminology in the questions that you did not understand?

• Use your own words to tell me what you think <insert word> means?

• For test questions which have been broken into parts and the student is asking the second or third part of the question

• What additional information from the <insert appropriate part of the original stimulus> did you need to use to solve this particular question?

• Did you use the information from other parts of the question to solve this problem? If yes, what information did you use and how?

• For test questions with graphs/diagrams/tables

• What do you think this <table, chart, graph> shows?

• Did you use the information in <table, chart, graph> to solve the problem? If yes, what information did you use and how?

• For multiple-choice test questions

• How did you decide what answer was best?

• Why did you rule out the options you did not choose?

• Did more than one response option seem correct to you? Why?

• For constructed-response test questions

• A complete answer to the test question includes <insert appropriate part of correct response>. Could you tell this would be required in your answer from the way the test question is worded? Why or why not?

Part 5. Test Question Transition Script

Thank you. Let’s move to the next test question.

Interviewer moves to next part of multi-part item, or next item, as appropriate and repeats Parts 4 and 5 until the cognitive interview is completed or approximately 60 minutes has passed.

After the cognitive interview is complete or 60 minutes have passed, the interviewer should proceed to Part 6.

Thank You and Closing Instructions

For Every Participant

Grades 4, 8, and 12

Part 6. Thank You for Participating

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making these test questions better.

All materials will be collected by the interviewer and student will be escorted out of the interviewing room. Students will be given a $25 gift card to thank them for their time and effort. In addition, if the participating students’ parent or legal guardian brings them to and from the interview site, the parent or guardian will also receive a $25 gift card to thank him or her for the time and effort involved and to offset transportation costs.

Volume II: Protocols 2