High School and Beyond 2020 (HS&B:20) Base-Year Cognitive and Usability Testing Round 2

National Center for Education Statistics (NCES)

Attachments XI-XII

Assessment Questions with Cognitive and Usability Interview Probes

High School and Beyond 2020 (HS&B:20) Base-Year
Cognitive and Usability Testing Round 2

OMB# 1850-0803 v.252

August 2019

Table of Contents

Attachment XI: Cognitive and Usability Interview Protocol: English Language Screener 3

Attachment XII: Cognitive and Usability Interview Protocol: Math Assessment 8

Attachment XI:
Cognitive and Usability Interview Protocol: English Language Screener

HS&B:20 EL Screener Cognitive Testing Items, Probes, and Interview Flow

Direction Screen 1	Now you will be completing…	In your own words, can you please provide a brief summary of the information presented on this screen? How clear or unclear did you think the directions were? (Very clear, somewhat clear, somewhat unclear, very unclear) -If unclear, what did you find unclear? What could we do to make it more clear?
Direction Screen 2	Select your answer…	In your own words, can you please provide a brief summary of the information presented on this screen? How clear or unclear did you think the directions were? (Very clear, somewhat clear, somewhat unclear, very unclear) -If unclear, what did you find unclear? What could we do to make it more clear?
HSB20ESL01	Did you…	Optional: Ask once at the end unless issues are noticed while the student is working through the items, or unless interviewer notices the student is moving on from an item without responding. Did you have any difficulty seeing any words or pictures? Please describe. Did moving from question to question work as you expected? If not, please describe. Why did you skip this question? Or did you skip any questions? If so, why? How easy or difficult was it to answer this question. (very easy, somewhat easy, somewhat difficult, very difficult) If difficult, what can we do to make it easier to answer this question? Did you find any parts of the question confusing? If yes, what could we do to make the question less confusing?
HSB20ESL02	Don’t [blank] to do…
HSB20ESL03	I found my…
HSB20ESL04	Mary and Donna…
HSB20ESL05	The students made…
HSB20ESL06	The subject of…
HSB20ESL07	It’s going to...
HSB20ESL08	We were walking…
HSB20ESL09	Lions and tigers…
HSB20ESL10	Why did you…
HSB20ESL11	Marco had not…
HSB20ESL12	Mr. Johnson chose…
HSB20ESL13	Joshua and Elena…
HSB20ESL14	The two buildings…
HSB20ESL15	When David handed…
HSB20ESL16	The [blank] score…
HSB20ESL17	The [blank] age…
HSB20ESL18	The [blank] between…

HSB20ESL19	The bill for…	How did you select [or mark] your response on the computer? [Did student know how to use drop-down menu?] Can you describe in your own words what you think this item is asking? Please try changing your response to this item. Was it clear to you how to change your answer? Did it work as you expected? Why or why not? Go ahead and change it back to your original response. Please tell me about the process you used to figure out the answer to this item.
HSB20ESL20	I cannot solve…	How did you select [or mark] your response on the computer? [Did student know how to use drop-down menu?] Can you describe in your own words what you think this item is asking? Optional if student did not understand from HSB20ESL19: Please try changing your response to this item. Was it clear to you how to change your answer? Did it work as you expected? Why or why not? Go ahead and change it back to your original response. Please tell me about the process you used to figure out the answer to this item.
HSB20ESL21	If it rains…	How did you select [or mark] your response on the computer? [Did student know how to use drop-down menu?] Can you describe in your own words what you think this item is asking? Optional if student did not understand from HSB20ESL20: Please try changing your response to this item. Was it clear to you how to change your answer? Did it work as you expected? Why or why not? Go ahead and change it back to your original response. Optional: Please tell me about the process you used to figure out the answer to this item.
Direction Screen 3	Next, you will see…	In your own words, can you please provide a brief summary of the information presented on this screen? How clear or unclear did you think the directions were? (Very clear, somewhat clear, somewhat unclear, very unclear) -If unclear, what did you find unclear? What could we do to make it more clear?
HSB20ESL22	There are three…	Optional: Ask once at the end unless issues are noticed while the student is working through the items, or unless interviewer notices the student is moving on from an item without responding. Did you have any difficulty seeing any words or pictures? Please describe. Did moving from question to question work as you expected? If not, please describe. Why did you skip this question? Or did you skip any questions? If so, why? How easy or difficult was it to answer this question. (very easy, somewhat easy, somewhat difficult, very difficult) If difficult, what can we do to make it easier to answer this question? Did you find any parts of the question confusing? If yes, what could we do to make the question less confusing?
HSB20ESL23	The teacher set up…
HSB20ESL24	Kate has exactly…
HSB20ESL25	The white square…
HSB20ESL26	The books are…
HSB20ESL27	The lamp is…
HSB20ESL28	The top line…	How did you select [or mark] your response on the computer? [Did student know how to use radio buttons?] Can you describe in your own words what you think this item is asking? Can you describe in your own words what is in each of the pictures? Please try changing your response to this item. Was it clear to you how to change your answer? Did it work as you expected? Why or why not? Go ahead and change it back to your original response. Please tell me about the process you used to figure out the answer to this item.
HSB20ESL29	Half of the circles…	How did you select [or mark] your response on the computer? [Did student know how to use radio buttons?] Can you describe in your own words what you think this item is asking? Can you describe in your own words what is in each of the pictures? Optional if student did not understand from HSB20ESL28: Please try changing your response to this item. Was it clear to you how to change your answer? Did it work as you expected? Why or why not? Go ahead and change it back to your original response. Please tell me about the process you used to figure out the answer to this item.

HSB20ESL30	The dog with…	How did you select [or mark] your response on the computer? [Did student know how to use radio buttons?] Can you describe in your own words what you think this item is asking? Can you describe in your own words what is in each of the pictures? Optional if student did not understand from HSB20ESL29: Please try changing your response to this item. Was it clear to you how to change your answer? Did it work as you expected? Why or why not? Go ahead and change it back to your original response. Optional: Please tell me about the process you used to figure out the answer to this item.
Post Screener	Ask these follow up questions unless they have already been addressed during the item probes. Did you have any difficulty seeing any words or pictures? Please describe. Did moving from question to question work as you expected? If not, please describe. If not addressed previously: Did you skip any questions? If so, why?
General/Debrief	Is there anything else you would like to tell me about working on these items? Is there anything that you think could make [this/these] item(s) or directions clearer? Did answering these questions on the [computer/tablet/device] work the way you expected? Before we finish, I’d like to hear [any/other] thoughts about any of the things you did during this session.

Attachment XII:
Cognitive and Usability Interview Protocol: Math Assessment

HS&B:20 Math Cognitive Testing Items, Probes, and Interview Flow

Direction Screens 1-7

Students will go through all seven direction screens and straight to the first math question. After reading through or completing the first math question, the student will be asked the general and tailored probes related to the question. The student will then work through all the rest of the math questions and probes.

If the interviewer notices that a student had difficulty entering an answer, using the calculator, etc., the interviewer will ask the probes regarding the directions, tools, or navigation features that could have contributed to the difficulty. Otherwise, the direction and usability probes will be asked after all the math questions and probes have been completed.

Direction Screen 1	You will now answer some math questions…	Notes to Interviewer: Notate whether students are reading through or just skipping the direction screens. Notate whether the way students respond to the items indicates they did not read or understand the directions.
Direction Screen 2	To enter an integer…
Direction Screen 3	Jane bought a pen…
Direction Screen 4	[subtraction involving fractions] …
Direction Screen 5	Indicate whether each of the following…
Direction Screen 6	A scientific calculator…
Direction Screen 7	Additional information…

Item ID: HSB039 (Multiple Choice – Select One) Moderately Easy 45-60

Source: Newly Written Domain: Data, Statistics, and Probability

[ITEM: In the xy-plane…]

General Questions:

• Did the student answer the question? Y/N

  • If N, probe: Why did you skip this item?

  • If Y, probe:

• How easy or difficult was it to understand what the question was asking you to do? (Very easy, somewhat easy, somewhat hard, very hard)

  • If difficult, what can we do to make it easier to understand?

• How easy or difficult was it to determine which of the answer choices to select? (Very easy, somewhat easy, somewhat hard, very hard, I did not answer the question)

  • If difficult, what can we do to make it easier to understand?

• Is there one clear answer to this problem? Or are the answer choices confusing?

• How sure are you that your answer is correct? (Very sure, somewhat sure, somewhat unsure, very unsure). Follow up with “Why are you unsure?” based on student response.

• Briefly describe how you determined the answer to the question.

• Did the student seem confused about how to provide a response? Y/N

  • If Y, probe: Is it clear how to enter your response? Why or why not?

• Did the student use any of the available tools/aids? Y/N?

  • If Y, probe:

    • Did [X] tool work as you expected?

    • Did you find it helpful? Why or why not?

  • If N, ask follow-up question at the end of the interview about why the student didn’t use any tools

Tailored Questions:

• This question involved x- and y-intercepts of a line. How familiar or unfamiliar are you with x- and y-intercepts of a line? (Very familiar, somewhat familiar, somewhat unfamiliar, very unfamiliar)

  • If “Very familiar” or “somewhat familiar” probe: How easy or difficult do you generally find it to answer questions about the x- and y-intercepts of a line? (Very easy, somewhat easy, somewhat hard, very hard)

Optional Usability and Direction Probes: can be asked once at the end unless the interviewer notices something specific and deems it appropriate to probe about usability as the students work through the items.

Item ID: HSB357 (Numeric Entry Decimal) Moderately Hard 45-60

Source: Newly Written Domain: Data, Statistics and Probability

[ITEM: a gram of fat contains…]

General Questions:

• Did the student answer the question? Y/N

  • If N, probe: Why did you skip this item?

  • If Y, probe:

• How easy or difficult was it to understand what the question was asking you to do? (Very easy, somewhat easy, somewhat hard, very hard)

  • If difficult, what can we do to make it easier to understand?

• How easy or difficult was it to determine the answer to the question? (Very easy, somewhat easy, somewhat hard, very hard, I did not answer the question)

  • If difficult, what can we do to make it easier to understand?

• Is there one clear answer to this problem?

• How sure are you that your answer is correct? (Very sure, somewhat sure, somewhat unsure, very unsure). Follow up with “Why are you unsure?” based on student response.

• Briefly describe how you determined the answer to the question.

• Did the student seem confused about how to provide a response? Y/N

  • If Y, probe: Ask any appropriate usability or direction probes or ask, “Is it clear how to enter your response? Why or why not?”

• Did the student use any of the available tools/aids? Y/N

  • If Y, probe:

    • Did [X] tool work as you expected?

    • Did you find it helpful? Why or why not?

  • If N, ask follow-up question at the end of the interview about why the student didn’t use any tools

Tailored Questions:

• This question involved using information in a chart to solve a problem involving percent in a real-life setting.

  • How familiar or unfamiliar are you with using information from a chart to solve a percent problem? (Very familiar, somewhat familiar, somewhat unfamiliar, very unfamiliar)

    • If “Very familiar” or “somewhat familiar” probe: How easy or difficult do you generally find it to use information in a chart to solve a problem? (Very easy, somewhat easy, somewhat hard, very hard)

  • Have you ever had to solve a problem like this outside of school? If yes, please describe.

  • How easy or difficult do you generally find it to use information from a chart to solve problems in a real-life setting? (Very easy, somewhat easy, somewhat hard, very hard)

  • How easy or difficult do you generally find it to solve percent problems in a real-life setting? (Very easy, somewhat easy, somewhat hard, very hard)

• This question involved rounding your answer to the nearest whole percent. How familiar or unfamiliar are you with rounding a number to the nearest whole percent? (Very familiar, somewhat familiar, somewhat unfamiliar, very unfamiliar)

  • If “Very familiar” or “somewhat familiar” probe: How easy or difficult do you generally find it to round your answer to the nearest whole percent? (Very easy, somewhat easy, somewhat hard, very hard)

• How easy or difficult do you generally find it to round a number to a specified degree of accuracy other than whole numbers? For example, if you were asked to round the number 1.723 to the nearest tenth, or you were asked to round the number 23.8762 to the nearest hundredth, would you know what to do? (Very easy, somewhat easy, somewhat hard, very hard)

  • Note to Interviewer: follow up with participant, asking them to do the rounding from one or both examples as needed and notate responses.

Optional Usability and Direction Probes: can be asked once at the end unless the interviewer notices something specific and deems it appropriate to probe about usability as the students work through the items.

Item ID: HSB113 (Numeric Entry Fraction) Hard 45-60

Source: Newly written Domain: Number Systems and Arithmetic

[ITEM: multiplying and dividing fractions]

General Questions:

• Did the student answer the question? Y/N

  • If N, probe: Why did you skip this item?

  • If Y, probe:

• How easy or difficult was it to understand what the question was asking you to do? (Very easy, somewhat easy, somewhat hard, very hard)

  • If difficult, what can we do to make it easier to understand?

• How easy or difficult was it to determine the answer to the question? (Very easy, somewhat easy, somewhat hard, very hard, I did not answer the question)

  • If difficult, what can we do to make it easier to understand?

• Is there one clear answer to this problem?

• How sure are you that your answer is correct? (Very sure, somewhat sure, somewhat unsure, very unsure). Follow up with “Why are you unsure?” based on student response.

• Briefly describe how you determined the answer to the question.

• Did the student seem confused about how to provide a response? Y/N

  • If Y, probe: Ask any appropriate usability or direction probes or ask, “Is it clear how to enter your response? Why or why not?”

• Did the student use any of the available tools/aids? Y/N

  • If Y, probe:

    • Did [X] tool work as you expected?

    • Did you find it helpful? Why or why not?

  • If N, ask follow-up question at the end of the interview about why the student didn’t use any tools

Tailored Questions:

• This question involved multiplying and dividing fractions. How familiar or unfamiliar are you with multiplying and dividing fractions? (Very familiar, somewhat familiar, somewhat unfamiliar, very unfamiliar)

  • If “Very familiar” or “somewhat familiar” probe: How easy or difficult do you generally find it to work with arithmetic expressions that involve multiplying and dividing fractions? (Very easy, somewhat easy, somewhat hard, very hard)

Optional Usability and Direction Probes: can be asked once at the end unless the interviewer notices something specific and deems it appropriate to probe about usability as the students work through the items.

Item ID: HSB084 (Numeric Entry Decimal) Moderately Hard Over 120

Source: MGLS:2017 Domain: Algebra and Functions

[ITEM: The table contains some…]

General Questions:

• Did the student answer the question? Y/N

  • If N, probe: Why did you skip this item?

  • If Y, probe:

• How easy or difficult was it to understand what the question was asking you to do? (Very easy, somewhat easy, somewhat hard, very hard)

  • If difficult, what can we do to make it easier to understand?

• How easy or difficult was it to determine the answer to the question? (Very easy, somewhat easy, somewhat hard, very hard, I did not answer the question)

  • If difficult, what can we do to make it easier to understand?

• Is there one clear answer to this problem?

• How sure are you that your answer is correct? (Very sure, somewhat sure, somewhat unsure, very unsure). Follow up with “Why are you unsure?” based on student response.

• Briefly describe how you determined the answer to the question.

• Did the student seem confused about how to provide a response? Y/N

  • If Y, probe: Ask any appropriate usability or direction probes or ask, “Is it clear how to enter your response? Why or why not?” [Note: ask the probe about entering negative numbers here as appropriate.]

• Did the student use any of the available tools/aids? Y/N

  • If Y, probe:

    • Did [X] tool work as you expected?

    • Did you find it helpful? Why or why not?

  • If N, ask follow-up question at the end of the interview about why the student didn’t use any tools

Tailored Questions:

• This question involved using information in a table to determine a function. How familiar or unfamiliar are you with this topic? (Very familiar, somewhat familiar, somewhat unfamiliar, very unfamiliar)

  • If “Very familiar” or “somewhat familiar” probe: How easy or difficult do you generally find it to use information in a table to solve these types of problems? (Very easy, somewhat easy, somewhat hard, very hard)

• How easy or difficult do you generally find it to determine functions? (Very easy, somewhat easy, somewhat hard, very hard)

Optional Usability and Direction Probes: can be asked once at the end unless the interviewer notices something specific and deems it appropriate to probe about usability as the students work through the items.

Item ID: HSB358 (Table Grid, Single-Select Row) Hard 60-90

Source: Newly Written Domain: Data, Statistics and Probability

[ITEM: Students in classes…]

General Questions:

• Did the student answer the question? Y/N

  • If N, probe: Why did you skip this item?

  • If Y, probe:

• How easy or difficult was it to understand what the question was asking you to do? (Very easy, somewhat easy, somewhat hard, very hard)

  • If difficult, what can we do to make it easier to understand?

• How easy or difficult was it to determine the answer to the question? (Very easy, somewhat easy, somewhat hard, very hard, I did not answer the question)

  • If difficult, what can we do to make it easier to understand?

• Is there one clear answer to this problem?

• How sure are you that your answer is correct? (Very sure, somewhat sure, somewhat unsure, very unsure). Follow up with “Why are you unsure?” based on student response.

• Briefly describe how you determined the answer to the question.

• Did the student seem confused about how to provide a response? Y/N

  • If Y, probe: Ask any appropriate usability or direction probes or ask, “Is it clear how to enter your response? Why or why not?”

• Did the student use any of the available tools/aids? Y/N

  • If Y, probe:

    • Did [X] tool work as you expected?

    • Did you find it helpful? Why or why not?

  • If N, ask follow-up question at the end of the interview about why the student didn’t use any tools

Tailored Questions:

• This question involved using information in tables to solve a problem about data representation in a real life setting.

  • How familiar or unfamiliar are you with using information in tables to solve a problem about data representation? (Very familiar, somewhat familiar, somewhat unfamiliar, very unfamiliar)

    • If “Very familiar” or “somewhat familiar” probe: How easy or difficult do you generally find it to use information in a table to solve a problem? (Very easy, somewhat easy, somewhat hard, very hard)

  • Have you ever had to solve a problem like this outside of school? If yes, please describe.

  • How easy or difficult do you generally find it to use information in tables to solve problems in a real-life setting? (Very easy, somewhat easy, somewhat hard, very hard)

  • How easy or difficult do you generally find it to solve problems about data representation in a real-life setting? (Very easy, somewhat easy, somewhat hard, very hard)

Optional Usability and Direction Probes: can be asked once at the end unless the interviewer notices something specific and deems it appropriate to probe about usability as the students work through the items.

Item ID: HSB355 (Multiple Choice -Select One) Medium 45-60

Source: Newly Written Domain: Data, Statistics, and Probability

[ITEM: The chart shows…]

General Questions:

• Did the student answer the question? Y/N

  • If N, probe: Why did you skip this item?

  • If Y, probe:

• How easy or difficult was it to understand what the question was asking you to do? (Very easy, somewhat easy, somewhat hard, very hard)

  • If difficult, what can we do to make it easier to understand?

• How easy or difficult was it to determine which of the answer choices to select? (Very easy, somewhat easy, somewhat hard, very hard, I did not answer the question)

  • If difficult, what can we do to make it easier to understand?

• Is there one clear answer to this problem? Or are the answer choices confusing?

• How sure are you that your answer is correct? (Very sure, somewhat sure, somewhat unsure, very unsure). Follow up with “Why are you unsure?” based on student response.

• Briefly describe how you determined the answer to the question.

• Did the student seem confused about how to provide a response? Y/N

  • If Y, probe: Is it clear how to enter your response? Why or why not?

• Did the student use any of the available tools/aids? Y/N

  • If Y, probe:

    • Did [X] tool work as you expected?

    • Did you find it helpful? Why or why not?

  • If N, ask follow-up question at the end of the interview about why the student didn’t use any tools

Tailored Questions:

• This question involved finding specific information in a chart. How familiar or unfamiliar are you with using information from charts to answer math questions? (Very familiar, somewhat familiar, somewhat unfamiliar, very unfamiliar)

  • If “Very familiar” or “somewhat familiar” probe: How easy or difficult do you generally find it to find specific information in a chart to answer a question? (Very easy, somewhat easy, somewhat hard, very hard)

Optional Usability and Direction Probes: can be asked once at the end unless the interviewer notices something specific and deems it appropriate to probe about usability as the students work through the items.

Item ID: HSB136 (Multiple Choice -Select One) Medium 45-60

Source: Newly Written Domain: Number Systems and Arithmetic

[ITEM: Which of the following expressions…]

General Questions:

• Did the student answer the question? Y/N

  • If N, probe: Why did you skip this item?

  • If Y, probe:

• How easy or difficult was it to understand what the question was asking you to do? (Very easy, somewhat easy, somewhat hard, very hard)

  • If difficult, what can we do to make it easier to understand?

• How easy or difficult was it to determine which of the answer choices to select? (Very easy, somewhat easy, somewhat hard, very hard, I did not answer the question)

  • If difficult, what can we do to make it easier to understand?

• Is there one clear answer to this problem? Or are the answer choices confusing?

• How sure are you that your answer is correct? (Very sure, somewhat sure, somewhat unsure, very unsure). Follow up with “Why are you unsure?” based on student response.

• Briefly describe how you determined the answer to the question.

• Did the student seem confused about how to provide a response? Y/N

  • If Y, probe: Is it clear how to enter your response? Why or why not?

• Did the student use any of the available tools/aids? Y/N

  • If Y, probe:

    • Did [X] tool work as you expected?

    • Did you find it helpful? Why or why not?

  • If N, ask follow-up question at the end of the interview about why the student didn’t use any tools

Tailored Questions:

• This question involved manipulating an expression involving multiplication, subtraction, and parentheses to get an equivalent expression. How familiar or unfamiliar are you with this topic? (Very familiar, somewhat familiar, somewhat unfamiliar, very unfamiliar)

  • If “Very familiar” or “somewhat familiar” probe: How easy or difficult do you generally find it to manipulate an expression involving multiplication, subtraction, and parentheses to get an equivalent expression? (Very easy, somewhat easy, somewhat hard, very hard)

Optional Usability and Direction Probes: can be asked once at the end unless the interviewer notices something specific and deems it appropriate to probe about usability as the students work through the items.

Item ID: HSB027 (Multiple-Choice- Select one) Hard 60-90

Source HSLSQ051S Domain: Algebra and Functions

[ITEM: Which of the following systems of equations…]

General Questions:

• Did the student answer the question? Y/N

  • If N, probe: Why did you skip this item?

  • If Y, probe:

• How easy or difficult was it to understand what the question was asking you to do? (Very easy, somewhat easy, somewhat hard, very hard)

  • If difficult, what can we do to make it easier to understand?

• How easy or difficult was it to determine which of the answer choices to select? (Very easy, somewhat easy, somewhat hard, very hard, I did not answer the question)

  • If difficult, what can we do to make it easier to understand?

• Is there one clear answer to this problem? Or are the answer choices confusing?

• How sure are you that your answer is correct? (Very sure, somewhat sure, somewhat unsure, very unsure). Follow up with “Why are you unsure?” based on student response.

• Briefly describe how you determined the answer to the question.

• Did the student seem confused about how to provide a response? Y/N

  • If Y, probe: Is it clear how to enter your response? Why or why not?

• Did the student use any of the available tools/aids? Y/N

  • If Y, probe:

    • Did [X] tool work as you expected?

    • Did you find it helpful? Why or why not?

  • If N, ask follow-up question at the end of the interview about why the student didn’t use any tools

Tailored Questions:

• This question involved a graph and a system of equations. How familiar or unfamiliar are you with graphing lines or systems of equations? (Very familiar, somewhat familiar, somewhat unfamiliar, very unfamiliar)

  • If “Very familiar” or “somewhat familiar” probe: How easy or difficult do you generally find it to graph lines or systems of equations? (Very easy, somewhat easy, somewhat hard, very hard)

Optional Usability and Direction Probes: can be asked once at the end unless the interviewer notices something specific and deems it appropriate to probe about usability as the students work through the items.

Item ID: HSB033 (Multiple Choice – Select one) Hard 45-60

Source HSLS Q162S Domain: Algebra and Functions

[ITEM: The formula can be used…]

General Questions:

• Did the student answer the question? Y/N

  • If N, probe: Why did you skip this item?

  • If Y, probe:

• How easy or difficult was it to understand what the question was asking you to do? (Very easy, somewhat easy, somewhat hard, very hard)

  • If difficult, what can we do to make it easier to understand?

• How easy or difficult was it to determine which of the answer choices to select? (Very easy, somewhat easy, somewhat hard, very hard, I did not answer the question)

  • If difficult, what can we do to make it easier to understand?

• Is there one clear answer to this problem? Or are the answer choices confusing?

• How sure are you that your answer is correct? (Very sure, somewhat sure, somewhat unsure, very unsure). Follow up with “Why are you unsure?” based on student response.

• Briefly describe how you determined the answer to the question.

• Did the student seem confused about how to provide a response? Y/N

  • If Y, probe: Is it clear how to enter your response? Why or why not?

• Did the student use any of the available tools/aids? Y/N

  • If Y, probe:

    • Did [X] tool work as you expected?

    • Did you find it helpful? Why or why not?

  • If N, ask follow-up question at the end of the interview about why the student didn’t use any tools

Tailored Questions:

• This question involved using an algebraic formula to answer a question in a real-life setting.

  • How familiar or unfamiliar are you with using formulas to answer math questions? (Very familiar, somewhat familiar, somewhat unfamiliar, very unfamiliar)

    • If “Very familiar” or “somewhat familiar” probe: How easy or difficult do you generally find it to use an algebraic formula to answer a question in a real-life setting? (Very easy, somewhat easy, somewhat hard, very hard)

  • Have you ever had to solve a problem like this outside of school? If yes, please describe.

Optional Usability and Direction Probes: can be asked once at the end unless the interviewer notices something specific and deems it appropriate to probe about usability as the students work through the items.

Item ID: HSB353 (Multiple Choice – Select One) Medium 45-60

Source: Newly Written Domain: Data, Statistics, and Probability

[ITEM: Five students at a certain high school…]

General Questions:

• Did the student answer the question? Y/N

  • If N, probe: Why did you skip this item?

  • If Y, probe:

• How easy or difficult was it to understand what the question was asking you to do? (Very easy, somewhat easy, somewhat hard, very hard)

  • If difficult, what can we do to make it easier to understand?

• How easy or difficult was it to determine which of the answer choices to select? (Very easy, somewhat easy, somewhat hard, very hard, I did not answer the question)

  • If difficult, what can we do to make it easier to understand?

• Is there one clear answer to this problem? Or are the answer choices confusing?

• How sure are you that your answer is correct? (Very sure, somewhat sure, somewhat unsure, very unsure). Follow up with “Why are you unsure?” based on student response.

• Briefly describe how you determined the answer to the question.

• Did the student seem confused about how to provide a response? Y/N

  • If Y, probe: Is it clear how to enter your response? Why or why not?

• Did the student use any of the available tools/aids? Y/N

  • If Y, probe:

    • Did [X] tool work as you expected?

    • Did you find it helpful? Why or why not?

  • If N, ask follow-up question at the end of the interview about why the student didn’t use any tools

Tailored Questions:

• This question involved using a trend line in a scatterplot. How familiar or unfamiliar are you with using a trend line in a scatterplot? (Very familiar, somewhat familiar, somewhat unfamiliar, very unfamiliar)

  • If “Very familiar” or “somewhat familiar” probe: How easy or difficult do you generally find it to use a trend line in a scatterplot? (Very easy, somewhat easy, somewhat hard, very hard)

• This question involved using a trendline to estimate a data value. How familiar or unfamiliar are you with using a trend line to estimate a data value? (Very familiar, somewhat familiar, somewhat unfamiliar, very unfamiliar)

  • If “Very familiar” or “somewhat familiar” probe: How easy or difficult do you generally find it to use a trend line to estimate a data value? (Very easy, somewhat easy, somewhat hard, very hard)

Optional Usability and Direction Probes: can be asked once at the end unless the interviewer notices something specific and deems it appropriate to probe about usability as the students work through the items.

Usability and Direction Probes:

Usability and direction probes can be asked once at the end of the cognitive testing interview, after the student has responded to all items, with the caveat that the interviewer should probe about usability as the students work through the items if they notice something specific.

1. Describe how the direction screens helped or didn’t help you understand what to expect in the math questions.

   1. How clear and understandable were they?

   2. How could they be improved?

2. Describe how the direction screens helped or didn’t help you understand how to enter responses for the various item types.

   1. Did you encounter challenges with entering your answer?

   2. Did you know how to indicate a negative number for your response?

   3. Did you know how to provide a response with a decimal or fraction?

3. Describe how the direction screens helped or didn’t help you understand what to expect for the calculator, read aloud player, or scratch paper provided.

4. Describe your experiences with using any of the available tools and aids such as the calculator, the read aloud player, or the scratch paper provided. [If the interviewee has not yet used the read aloud or the calculator it might be good to ask them to access each to get some feedback on the tools.]

5. Did you have any difficulty seeing any words or pictures? Please describe.

6. Did moving from question to question work as you expected? If not, please describe.

Debrief Probes:

Once students have finished working through all items and responded to the corresponding probes, these debrief probes can be given.

1. Is there anything else you would like to tell me about working on these items?

2. Is there anything that you think could make [this/these] item(s) or directions clearer?

3. Did answering these math questions on the [computer/tablet/device] work the way you expected?

4. Before we finish, I’d like to hear [any/other] thoughts about any of the things you did during this session.

3