National Assessment of Educational Progress (NAEP) Engagement Augmentation Study

National Center for Education Statistics

National Assessment of Educational Progress

National Assessment of Educational Progress (NAEP) Engagement Augmentation Study

Volume II

Protocols

OMB# 1850-0803 v.279

December 2020

Overview

Each virtual cognitive interview session will use a laptop computer to present one or two digitally-delivered NAEP Science Interactive Computer Tasks: the original pilot task (Task A) will be presented using Unity through Internet Explorer (hosted on the local drive of the laptop and made available to a student through Zoom remote control feature that allows the student to take control of the interviewers’ screen in a meeting); and/or the augmented task (Task B) will be presented using a similar interface through PowerPoint (hosted on the same laptop and presented to students via Zoom remote control feature).

ACT will recruit a sample of 87 students in Grade 8 for virtual cognitive interviews (see Volume I of this submittal for further details on recruitment, as well as the Appendix to inspect the recruitment and communication materials).

Students will be compensated with a $25 electronic gift card for their participation. Participation will take no more than 1 hour, with an additional 15-minutes scheduled for explanation of the task and time to answer questions (scheduled for 75-minutes with 15-minutes minimum in between).

A signed consent form for each participating student will be provided via email by a parent/guardian prior to the participation in the virtual cognitive interviews. Each student will join the Zoom session with a link provided via email.

The purpose of the cognitive interviews is to gauge students’ engagement in a selected NAEP Science pilot ICT, provided by NCES, for comparison with an augmented version of the task, created by ACT randomly assigned across participants as indicated in Volume I. All students will work through the original and enhanced tasks (Experimental Groups AB and BA) or through enhanced task only (Control Group) in a think-aloud setting and respond to questions based on the protocol below.

NAEP Science ICT Cooperative Agreement Cognitive Interview Protocol

Student ID# ___

Assigned Group#: AB / BA / B___

[Make sure you have verified on the Participants List the completed Parental Consent form sent via email. If, for any reason, the student is no longer interested in participating, thank the student for his/her time and end the interview.]

Welcome and Introduction

Before we start, may I kindly ask you to turn on your microphone and turn off your video camera.

[Make sure participant’s microphone is on and video camera is off.]

Hello, my name is <name of interviewer> and I work for ACT. It’s nice to meet you, and I thank you very much for coming here to help us out today. Before we begin, I should tell you that this project has been reviewed by the Office of Management and Budget, and the OMB control number for this project is 1850-0803.

To help us with development of assessments that measure what students know and can do, we are working with 8th-graders to learn what they think about different types of activities and items. Over the next hour, you will work on two versions of a science task [Group B: “one science task”], and afterward we will talk about your experience while completing each task [Group B: “the task”]. These tasks are part of the National Assessment of Educational Progress, which is a test that students across the country take every two-to-three years. The test does not affect your grade or your school in any way. The scores from the test are not given to students or teachers but are used to understand the performance of all 8th-graders in the US. It is important to emphasize that we are NOT TESTING YOU or your performance in any way. We are interested in using your input and feedback to make the tasks better. We will not include your name or any information about you in the reports we make, the reports will group all of the students’ ideas together to report them anonymously.

We want you to complete this task as realistically as possible, so imagine that you are at school and are told to complete the following science assessment.

We are recording audio and screen during this session to help us keep a record of your suggestions and feedback, we are not using it to evaluate you– you are helping us evaluate the assessment tasks.

There will be no video recording of your face. Please keep your video camera off for the duration of this interview.

No one outside of our research group is going to view this recording.

If you are not talking loudly enough for the microphone to pick up your voice, I will remind you to please speak louder.

As you start answering science assessment questions, I want to learn what you think about these questions and how they are explained. One way we do this is to ask you to “think aloud” throughout what you are doing. All I mean by “think aloud” is for you to tell me what you’re thinking as you go through the questions and how you think you would answer them. You do not need to say everything out loud, just the main points of how you go about answering a question: what you find confusing, what questions you have about what is being asked.

Remember, you are here to help us with gaining insights on the science assessment. We are NOT TESTING YOU or your performance in any way. Now that you understand more about this study do you have any questions, and would you like to be a part of this study? [Answer any questions the student may ask.]

May I kindly ask your parent/guardian to confirm verbally your participation in the study? Do you [parent/guardian] have any questions, and would you like to grant permission to your child to participate in the study? [Answer any questions the parent/guardian may ask.]

[If student AND parent/guardian agree, indicate verbal consent on the Participants List and continue; if not, end the interview.]

If at any time you want to stop, just let me know.

[If, for any reason, the student OR parent/guardian is no longer interested in participating, thank the student and the parent/guardian for their time and end the interview.]

[Show the screen with the selected task to start with.]

I will now turn on the audio and screen capturing recording. [Turn on audio and screen capturing recorder.]

OK, let’s get started. I would like to give you remote control of my screen to enable full interaction with the task.

[While screen sharing via Zoom, click Remote Control and select the participant you want to give control to.]

Click inside the screen share to start controlling my screen.

[To regain control, click anywhere on your screen. The participant can still restart remote control by clicking on their screen. You can also click Stop Share to regain control and not allow the other participant to start remote control again.]

[Make sure the student is in control of the screen. Repeat the instructions, if needed. In outstanding cases of technical difficulties with screen control, thank the student for his/her attempts and shift to the Alternative Option: The student will see the interviewer’s screen with the assessment task, will share explanation/selections verbally with the interviewer who will enter student’s response directly to the designated space in the task/item].

You will be presented with two 15-minute tasks [Group B: “One task”]. We will discuss each task [Group B: “The task”] after you complete it. Read the questions in the task and provide your answers in the space provided [Option B only: “Alternatively, share your answer or selections with the interviewer”]. Please answer them as best as you can. Let me know when you are finished, and we will begin talking about the questions you answered. If you are not finished in about 15-minutes, I will stop you to begin our first discussion.

[Make sure to have the first assigned task already on screen, ready for student to complete.]

You may begin the task now. As a reminder, we encourage you to think out loud as you puzzle through this task.

[Give the student no more than 15-minutes to complete the task. Take notes based on student think-alouds, paying particular attention to places where the student seemed disengaged with questions, frustrated or confused.]

[When the student has completed the task or when approximately 15-minutes have passed, begin the discussion.]

Thank you for working on this task. I have several things to ask you about your work.

[Below are examples of specific probes based on study objectives. You can include the notes for specific probing questions with the think-aloud notes you took for each question. Regain screen control as needed.]

• Do you understand the task and what is required from you?

• Can you tell me more about what you were thinking when you were answering the questions?

• Overall, do you think this set of questions was easy, medium, or hard or a mix? Why?

• Which parts were like things you’ve done before? Which parts weren’t?

• Were there any test questions that were less interesting or were less inviting to make an effort to work on the question? Which question(s) were the least interesting or inviting an effort and why?

• Do you have any suggestions for making the task better or more engaging? What would you change?

Thank you. Let’s move to the next and final task [Group B: Skip to Thank You section].

[Regain screen control as needed.]

You will be presented with an alternative version of the task you just completed. Read the questions and provide your answers in the space provided [Option B only: Alternatively, share your answer or selections with the interviewer]. Please answer them as best as you can, and remember to try to talk through your thought process. Let me know when you are finished, and we will begin talking about the questions you answered. If you are not finished in about 15-minutes, I will stop you to begin our next discussion. You may begin now.

[Give the student no more than 15-minutes to complete the second task. Take notes based on student think-alouds, paying particular attention to places where the student seemed disengaged with questions, frustrated or confused.]

[When the student has completed the second task or when approximately 15-minutes have passed, begin the discussion. Regain screen control as needed.]

Thank you for working on this task. Now, I have several things to ask you about your work.

[Below are examples of specific probes based on study objectives. You can include the notes for specific probing questions with the think-aloud notes you took for each question.]

• Do you understand the task and what is required from you?

• Can you tell me more about what you were thinking when you were answering the questions?

• Were there any test questions that were disengaging? Which question(s) were the least engaging and why were they the least engaging?

• Do you find this version of the task less engaging, engaging at the same level, or more engaging compared to the previous task? Do you have any suggestions for making that alternative version of the task better or more engaging? What would you change?

[Thank you section]

We are finished! On behalf of the U.S. Department of Education’s National Center for Education Statistics (NCES), we would like to thank you for helping us. The information you’ve provided will be very helpful for making the task better.

We will be pleased to provide you with a $25 e-gift card that will be sent to the email listed in your screening survey. Again, thank you for your time and effort.

I will now regain screen control, and turn off the audio and screen capturing recording.

[Regain screen control, and turn off audio and screen capturing recording.]