National Assessment of Educational Progress (NAEP) Writing Tasks Cognitive Interviews

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average 90 minutes, including the time to review instructions and complete and review the information collection. If you have suggestions for improving this collection or any comments or concerns regarding the accuracy of the time estimate or the status of your individual submission of this form, please write to: National Assessment of Educational Progress, National Center for Education Statistics, 1990 K Street, NW, Washington, DC 20006.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

OMB No. 1850-0803

I. Cognitive Laboratory Script - Writing

Text written in italics is to be read aloud by the interviewer.

NOTE: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate.

Introduction:

Hello, my name is ______ and I work for <Company Name>. It’s nice to meet you and thank you very much for helping us out today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?
- If student responds with “language arts/reading/writing,” follow up with: Good, then I think you’ll enjoy what we are going to do today.
- If student responds with another subject, respond accordingly (e.g., if the student responds with “science,” ask them what they are studying in their science class).

Let me begin by explaining why I am here and what you are going to be doing. You are participating in a special study about grade <8, 12> writing tasks, which is part of the National Assessment of Educational Progress, or NAEP for short, overseen by the National Center on Education Statistics, of the U.S. Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will be doing writing tasks.

It’s okay if you do not know how to do the writing task. I will not be grading your work today, and no one will know that it was you who did the task. My goal is to learn how you react to the task, so please try to do your best.

If at any time you decide you do not want to go on, that is your choice and you may stop.

This new test is on the tablet. The instructions about how to do the test are also on the tablet.

This interview is being recorded so we can review the recordings later. What you say may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573).

We’ll be going through the instructions first. After you do this, you will do a writing task. We will work together for about an hour.

Do you have any questions?

Ask the participant if he/she has any questions. After answering questions and giving further explanation, continue. If the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

OK, let’s begin.

II. Retrospective Think-Aloud

IIa. Retrospective Think-Aloud: Instructions and Modeling Script

Using the retrospective (versus concurrent) think-aloud approach, students are instructed to complete a task as they would under normal testing conditions (silently, without interruption). Then, after the task is completed, they will be asked to describe what they were thinking while reviewing the task on the computer. A replay of the task (e.g., using software like Camtasia® or Morae®)¹ will help cue the student’s recall of the thought processes that occurred during the task as the student progressed through it.

Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, interviewers should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that can be tailored to suit the particular writing task being studied.

To help us make our test better, we will ask you to complete a writing task. While you are doing the task, we will be recording everything that happens on the screen. The screen recording will capture all of your writing and movements on the tablet.

First you’ll do the writing task you will see on the screen, working at your own pace. After you have finished, I’m going to ask you to go back over the task and your writing with me. As we go back over these, I will ask you to tell me what you were thinking as you were reading the task and working on your writing. We call this “thinking aloud,” because we are asking you to say out loud everything you were thinking while you worked. To help you remember what you were thinking as you worked, we will look at the recording we made of you doing the task. This way, you will be able to see your screen actions and what you wrote. As we watch the recording, I will be asking you to tell me what you were thinking at different points. Does that make sense? Do you understand what we’ll be doing?

[Answer any questions and clarify the sequence of steps, if needed.]

In a moment, I will give you an example of the think-aloud process. Then I will give you a chance to practice it. You won’t be graded on anything. There are no incorrect thoughts, and everything you think and say is important to us.

Okay, now I’m going to show you how to think out loud. When I’m finished, I’ll ask you to try it.

Think-Aloud Demonstration:

Since we can’t tell what is going on in your head, we need you to “Think-Aloud.”

Let me give you an example. [Place example question in front of student.] Look at this question. It asks me to choose which two animals are the most similar. I’m going to do this question just as I would for a test. After I have finished, I will try to remember what I was thinking as I was working on the question so I can tell you.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar:

Beetle
Mouse
Crab
Dolphin
Cow

[The interviewer ‘works’ silently for about a minute on the question. The interviewer should enact some behaviors that suggest that he/she is thinking hard about the question and carefully considering the possible answers; for example, pointing with a pencil tip at each option in turn; moving a pencil or fingertip between answer options; returning to the question; giving nonverbal signals indicating consideration, uncertainty, etc. Finally the interviewer circles two answers, B and E, and then begins the retrospective think-aloud.]

Okay, so here I was reading the question…. It says I have to decide on the two animals that are the most similar. So at first I was wondering what do they mean by “similar?” That could mean anything. And I was thinking I wasn't sure how I’m going to make that decision. So..... the choices..... beetle, mouse, crab, dolphin, and cow. Yeah, well, as I'm reading the choices I'm already thinking there are some things about them that are similar, but there are also differences. I was thinking the beetle and crab are kind of similar, in a way—well, they both have lots of legs and they kind of move around fast. But one lives in the water and the other doesn’t, so that makes them different. And the mouse also moves around fast, but it’s not similar to either the beetle or the crab, except they are all small. Oh yeah, and then there is the dolphin that lives in the water, too… and I wondered about that, but that’s about all it has in common with the crab, so I couldn’t say they're very similar even though they’re both in the sea. So in the end I thought the mouse and the cow are both mammals, they both have fur and they have live babies instead of laying eggs or whatever… but then I thought, dolphins are mammals too, right? I think they are… anyway they seem so different from mice and cows to me. So I wasn't sure… it seemed like a hard question… I wasn't sure what to choose, but in the end I had to make a decision, so I decided to go for mouse and cow. Because they both have fur and four legs and walk around on land. It’s weird because they’re really not that similar. But, when I think of all the others, they seem really different from each other like in where they live, and their bodies. So even though it was really hard to decide, that seemed like the best choice to me, because they both live on land and they’re both mammals. So those things seemed [pretty important and that’s how I made my decision.

Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do as you are thinking about your writing task today. The point of the think-aloud is to get at whatever was in your head as you were doing the task. Just say aloud the words and the thoughts that were in your head, as you were thinking and working.

Ok, let’s have you give it a try.

IIb. Retrospective Think-Aloud: Student Practice Script I

Interviewers should place the practice question in front of the student so he/she can read it. Allow the student as much time as needed to answer the question. When he/she indicates he/she has finished, ask the student to begin thinking aloud. Some students will be silent after reading the question and indicating they are finished. Such students should be asked to say whatever they are thinking. In general, it may be necessary to remind students to talk aloud during the retrospective think-alouds.”. If necessary, interviewers should use the “Think-Aloud Hints” shown below to prompt students, being careful not to lead students. The interviewer needs to be familiar enough with the script to encourage the think-aloud in a natural, conversational manner.

Now you will try a think-aloud. I will give you a question like the one I just did. You will read and answer the question and then let me know when you’re done.

Once you are done answering, I will ask you to look back at the question and the answer you chose and try to say out loud the thoughts that were in your head at each moment. If I don’t hear you speaking I’ll ask you to keep talking. I’m telling you that so you won’t think I am criticizing the way you are thinking. I’ll be reminding you to think-aloud if you get quiet because I need to hear all of the thoughts you had while reading and answering the question.

[Hand students the question at this point.]

Okay, now go ahead and start working on the question. Let me know when you are finished.

Which two organisms below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

[When students indicate they are finished, make sure they have circled two answers, and then ask them to begin saying their thoughts out loud. As necessary, include the appropriate prompting questions, such as the following:]

Okay, so tell me what you were thinking as you were reading and answering this question. What did you think at the beginning, when you started to read the question? … And then what were you thinking? … And what else did you think about? … Anything else? … Can you tell me more about that? … etc.

IIc. Retrospective Think-Aloud: Student Practice II (Optional)

(Use only if you feel the student would benefit from another think-aloud practice before moving on to the actual task. It may be beneficial to instead model thinking aloud again for the student rather than doing another practice [see think-aloud demonstration in Ia])

If the student struggles to think-aloud, the interviewer should give the student another opportunity to practice. The interviewer should praise the student for the first attempt regardless of how good it was, for example: “Very good—let’s do another one before we start the real task. Are you ready? Here is the next practice question. [Place example question in front of student.] Remember that after you have finished answering this question I will ask you to talk out loud to tell me what you were thinking while you were working. Go ahead and do the question, and let me know when you have finished.”

As before, during the post-task think-aloud phase, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, section IIb).

Which two of the following objects have the most similar properties? Circle the two that are most similar

Silver coin

B. Chocolate coin

C. Gold coin

D. Blue plastic coin

E. Brown plastic coin

After the think-aloud phase is finished:

Now that you have practiced, do you feel that you understand how you should talk aloud about what you were thinking while you were doing the task? Is this something you feel okay about doing? [If students say yes:] Good, then let’s begin our study.[If students say no or appear to be hesitant or reluctant, ask them to say more about any reservations they have, and try to address their concerns or uncertainties in a supportive way. If students indicate they do not wish to continue or do not feel comfortable continuing, allow them to stop.]

IId. Retrospective Think-Aloud: Starting the Tasks

Now we will move on to the actual writing task. Remember, after you have finished, I will ask you to say aloud everything that you remember thinking while you were working on the task, and I may remind you to do that if you are quiet. This task should take you about 30 minutes, but let me know when you feel you are done working. Remember, you will not be graded on what you do during the task. There is no right or wrong way to think-aloud, as long as you keep telling me your thoughts. Your thoughts will help us make the task better. I will also have a few questions after we have finished.

Do you have any questions before we go on? [Answer any questions the student may ask.]

Because the information you provide is so important to us, I am going to be taking notes while you are doing the task and while you are thinking aloud afterwards.

The writing task is here on the screen. Here is a piece of paper and a pencil you may use if you wish to plan your writing. And, here is a brochure to help you plan and revise your writing if you wish.

When you are ready, go ahead and start working.

IIe. Retrospective Think-Aloud: Prompts and Questions After Completion of Task

Now I would like you to tell me what you were thinking as we review what you did for the task. We will be recording what you are saying as you think-aloud. Feel free to stop at any time. What you say will not be used to grade you. The information will only be used to help improve the task. I will play the task and your writing back from the beginning now, and I want you to talk about the thoughts you had as you were working. From time to time, I might pause the video, to give you enough time to explain what you were thinking at that point or if I need to ask you any extra questions. I will let you know when I am pausing the playback.

The interviewer plays and watches the recording of the task and student writing in real time with the student, and asks the student to describe aloud what he/she was thinking at various points. Here are suggested points for asking students to think aloud if the student is silent or does not address an action you saw happening (e.g., use of the brochure or planning paper):

After the student has spent time reading or viewing the task
If student used the brochure or planning paper
At point when student began writing
At points when student paused while reading/viewing the task or while composing
At points when students deleted substantially (several lines) or used keys to make edits such as cut/copy/paste
At points when student looked back at task or replayed any stimulus
Midpoint of writing
Conclusion of writing

The interviewer should pause the recording whenever more time is needed, making sure to say “I’ll just pause this for a moment” or “I am pausing the task here,” so that the student understands that the playback has been paused.

If the student is not verbalizing for 15 seconds, interviewers should offer a verbal “nudge” to remind the student to keep talking, such as:

What was going on in your head at this time?
I see you were looking at the task/stimulus again here. What were you thinking?
You seemed to be thinking hard at this point in your writing—can you tell me what was in your mind right then?
I noticed you made notes on the planning paper here. What were you thinking?

If the student says something about his/her thoughts but it is difficult to tell exactly what the student means, or if the student begins to talk but does not say very much, interviewers can urge the student to elaborate, for example:

Tell me a bit more about that.
That’s interesting—can you say more about that?
Go on….
Uh huh, okay, and what else were you thinking?

If more clarity or a more explicit description of a student’s thoughts is desirable:

Can you explain what you mean by that?
Can you say a bit more, to help me understand what you were thinking?

IIf. Notes on Student Actions and Behaviors During Tasks

As a student is proceeding through the task, a screen-capture system will record evidence of his or her actions on the task (such as where a student places the cursor, how he or she moves through the writing, and any text entered into the writing space). Use of the planning page and brochure should always be noted with the approximate time they were used.

During the session, the interviewer should also take notes of any of the student’s expressions or behaviors that may reflect the status of his/her understanding, engagement, or reaction to the task. The following are examples of such behaviors for interviewers to note:

Does the student express signs of confusion, boredom, or excitement?
Does the student get stuck on viewing/reading the task or on use of the planning paper?
Does the student generate brief amounts of text and then sit with seemingly nothing to add?
Does the student seem discouraged?

These kinds of observations should be considered opportunities to prompt students once they are done working as to what they were thinking at the time. In general, interviewers should make notes of any places during the course of the student’s working that appear valuable for follow-up after the task is completed (see section II).

III. Post-Think-Aloud Follow-Up Questions (Verbal Probing)

After completing the think-aloud process for a task, interviewers may follow up with a brief period of focused retrospective questioning assuming the think-aloud did not reveal the answers to the questions. For example, if it is desirable to know how a student conceptualized the audience while working on a task, a specific verbal probe that asks students this directly will be used. In addition to these pre-specified task-dependent questions, interviewers will be trained to generate post-task questions on the fly, if they observe something that they judge to need follow-up probing. Although probes will vary with tasks and students, a list of example probes has been included below. Any given student can be asked a maximum of five verbal probes, though a maximum of three is more desirable.

Who did you think you were supposed to be writing for in this task?
What do you think the audience for this task is like?
Did knowing that you had to write an essay help you or make the task harder?
Did the [image, video, audio, etc.] for the task help you come up with ideas to write about?”
Did any words in the task confuse you?
What ways could we change the task to make it better?

IV. Standardized Question for All Tasks: Task-Specific Prior Knowledge

Have you studied anything related to this task in school, or have you learned about or come across these topics in your own life? [If yes:] Tell me about what you have learned or studied or experienced that is related to this task.

V. Debriefing and Thank You for Cognitive Interview

Thank student for his/her time. Provide a gift card.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the task?

Is there anything you would like to ask me about what we did today? [Answer student questions]

Thank you for helping us to improve our test.

1 Morae® software (by TechSmith) can capture a student’s interactive behaviors as they happen, while one or more observers can simultaneously record comments that are time‐locked to the student actions and to the video recording. Adding Morae® Observer software allows observers to be located in a remote location. This is both a convenience for observers and a potential means of reducing student stress or distraction.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Hilary Persky
File Modified	0000-00-00
File Created	2021-01-27

National Assessment of Educational Progress (NAEP) Writing Tasks Cognitive Interviews

NCES Cognitive, Pilot, and Field Test Studies System

Vol 2 NAEP Writing Tasks Cog Labs