National Assessment of Educational Progress (NAEP) Pretesting Visual Representations in 4th Grade Writing Tasks

National Center for Education Statistics

National Assessment of Educational Progress

Volume II

Protocols & Materials

National Assessment of Educational Progress (NAEP) Pretesting Visual Representations in 4^th Grade Writing Tasks

OMB# 1850-0803 v. 196

May 2017

Table of Contents

Part A: Paperwork Burden Statement 1

Part B: Grade 4 Writing Study Interviewer Guide and Protocol – Cognitive Interview 2

Part C: Grade 4 Writing Study Interviewer Guide and Protocol – Small Scale Tryout 16

Part D: Survey Questionnaire 19

Part E: Writing Brochure 20

Part A: Paperwork Burden Statement

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to be average xx¹ minutes, including the time to review instructions and participate in the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any other concerns, please write to: National Assessment of Educational Progress (NAEP), National Center for Education Statistics, Potomac Center Plaza, 550 12^th St., SW, 4th floor, Washington, DC 20202.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

OMB No. 1850-0803 Approval Expires 07/31/2019

Part B: Grade 4 Writing Study Interviewer Guide and Protocol – Cognitive Interview

Writing Study Cognitive Interview Guide and Protocol

Student ID:		Date of Interview:
Start time:		End time:
Interviewer
Task name/ version

Purpose of this cognitive lab study

This cognitive lab study is designed to explore specific multimedia features that may inhibit student performance. In particular, we are looking for evidence regarding students’ interactions with the following possible inhibitors.

• Competition among different types of simultaneously presented information

• Distracting visual details, including inconsistent fonts

• Potentially misleading information

• Fast pace of onscreen text

• Different wording between narrated statement of writing task and prompt text

We will also look for evidence related to a small number of facilitators:

• Opportunities for recognition such as having their letter (hypothetically) read on TV if it is among the most convincing.

• Engaging features such as color, animation, narration, or sound effects

You need to be familiar with the specific locations of these inhibitors and facilitators in each writing task and probe accordingly and strategically. Therefore, carefully review the relevant sections of the document “Screen by Screen Explication of Cog Lab Writing Tasks.”

The think aloud and probing may also uncover other features that appear to affect students’ understanding and task performance, and these should be highlighted in your notes as well.

Types of evidence could include specific statements from the student indicating confusion when certain features are present, but evidence could also come from observing the student’s actions or from the student’s performance—for example, the amount of detail that the student remembers after viewing the video.

Overall structure of the cog lab

This cognitive lab has three sections:

1. Introduction, including think aloud demonstration and practice (10 minutes)

2. Writing task, with interspersed think aloud and probing (35 minutes)

During this segment the student completes one version of a writing task. Opportunities for think aloud and probing are built in at multiple points along the way, including some required probes. After the student finishes the task, you may ask additional probes, based on what was observed, to clarify the student’s interactions with the task, particularly as these relate to potential inhibitors identified in the study.

3. Additional required probes, including those that ask the student to compare different versions of the same task (10 minutes)

This section includes a series of structured probes designed to further explore the potential inhibitors in two ways: first by having the student refocus on aspects of the task stimulus and second by having the student explicitly compare (verbally) the version of the task that they completed with an alternative version of the same task.

4. Post-interview survey questionnaire (5 minutes)

General Approach and Note Taking

Before the student enters the room, test the recording devices to determine where they should be located for optimal sound quality and for optimal video capture of the student’s actions.

When you turn on the recording devices after the student has given his/her consent, be sure that they are actually recording and are still properly positioned.

At designated points during the time when the student is working on the writing task, ask the student to think out loud or respond to required probes as specified in the protocol. If the student cannot/does not think aloud, even with encouragement, indicate this in your notes and move on to the required probes. The focus of the required probes is on digging deeper into evidence regarding specific inhibitors (or their absence). We don’t, however, want to lead the student, so most of the probes are phrased in more general terms (e.g., was anything confusing? What was confusing?). If it appears that a specific inhibitor may be implicated (e.g., that the student may have been confused at a certain point because he/she had trouble keeping up with reading the on-screen text), you should follow up to clarify/elaborate.

Once the student actually begins to write, hold your questions until the student is finished. You can make notes to yourself about points you want to follow up with later. For example, you might say: I noticed that while you were writing, you went back and watched the video again. Can you tell me why you did that? Were you looking for something specific? Be sure to write down what you asked as well as what the student responded

Refocus on the task stimulus/compare to an alternative version of the task. As directed in the protocol, have the student refocus on designated aspects of the task stimulus, then compare the task stimulus to an alternative version. Use the required probes plus follow up probes, as necessary to clarify/elaborate the student’s perceptions, particularly as they relate to the specific inhibitors.

NOTE: the interviewer should manipulate the replay of the video and alternative video as this will go faster than if the student has to figure it out.

Your notes should include detailed information about:

• What the student says in the think alouds.

• What the student responds to each of the required prompts included in the protocol.

• What “interviewer-constructed” probes you ask the student and what the student says in response.

• Observations about the student’s actions and affect. For example, while writing his/her response, does the student go back to reread the prompt, look at the writing brochure, replay the video? For how long does the student write? Did the student appear focused or distracted while writing?

Take what notes you can as you go, but go back to either the audio or video recording later and augment your final notes with information from the tape. It is highly unlikely that you can capture sufficient detail in the notes you take during the interview.

Think-Aloud Hints for the Interviewer

If the student is silent for 5 or more seconds or is continually providing short responses or not answering, use “continuers” to encourage him/her to be more descriptive. The goal is to get the student to verbalize his/her thoughts without “putting words in his/her mouth.” You have to be as objective and unbiased as possible, but you may offer a verbal “nudge,” such as:

• What are you thinking now?

• Any other thoughts?

• I can’t hear what you’re saying…

• Could you explain to me how you came up with that answer?

• And that’s because…

• Is anything coming to your mind?

• Can you say more?

In addition, use reinforcers, such as the following, frequently during the interview.

• You’re doing a great job at thinking out loud.

• This is really helpful. We are learning so much.

Introduction

Text written in italics is to be read aloud.

You should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate.

The script includes suggestions for statements that can be used to reassure students who are anxious about their performance in the cog lab. You should use your discretion in deciding how many of these reassuring statements (and at what point in the proceedings) are appropriate for a given student.

If the student did not bring the consent form (and it was not sent to the recruitment firm), you must have the parent or accompanying guardian sign the consent form before beginning the interview.

Hello, my name is ______ and I work for EurekaFacts. It’s nice to meet you and thank you very much for coming today.

Chat briefly to build rapport with the student while you are getting him/her settled in the interview room. Get the student talking by asking a question, such as:

• What is your favorite subject in school?

  • If the student responds with “English” (or a related subject), ask them what they are studying and follow up with: Good, then I think you’ll enjoy what we are going to do today.

  • If the student responds with another subject, respond accordingly (e.g., if the student responds with “Social studies,” ask them what they are studying in their social studies class or why they enjoy it).

I want to talk a little bit with you about what we are doing today. Our job today is to help the people who write tests for the National Assessment of Educational Progress, or NAEP. Students all over the country take the NAEP tests, and we want to make sure that the tests are easy to understand and work the way they are supposed to. Today we will be looking at a task from a writing test. Overall our interview will take about one hour.

Here’s what we’re going to do: I’m going to ask you to work on the writing task and I will ask you some questions to help me understand how you are thinking about the task. You will read the writing task directions [and watch a video] on the computer, but then you will write your response using paper and pencil. You might think that the writing task is hard or that you aren’t giving a good answer. That’s OK. This is not a test of YOU. We’re mostly interested in how the writing task works; how students think about the writing task. But we need you to try hard to write your response—like you would do in a real test. The information we get from you will help us make tests that are fair and accurate for other students like you.

I am going to record this interview with a video recorder and an audio recorder so that, later on, researchers who are part of the study can review what we talked about. What you say and write will be used only to help us improve these writing tasks. What we learn from you and other participating students will be summarized in a report where no student names will be listed. After the session, we will use what you wrote to help us understand how our test is working and how we can make it better.

Do you have any questions about this?

I am going to go ahead and record the interview, ok?

[If student says ok, TURN ON VIDEO AND AUDIO RECORDERS NOW.]

If the student is willing to be audio recorded only, it is OK to go ahead with the interview, but set up a second audio recorder as a backup.

If the student does not want to be recorded at all, thank him/her for his/her time and end the interview. Provide the student and the parent their incentives.

I also want to note that your participation is voluntary. If at any time you want to stop participating you can. Just let me know.

If, at any point, the student is no longer interested in participating, thank the student for his/her time and end the interview. Provide the student and the parent their incentives.

Introduction to Concurrent Think Aloud

OK. Now we’ve come to the most important part of our work together. As I said, I will be asking you some questions to help me understand how you are thinking about the writing task. When you answer me, please do your best to tell me everything you are thinking about. We call this “thinking out loud.”

This “thinking out loud” is the important information we are looking for. We will practice this before you start.

Do you have any questions before we begin? If you have any questions about anything during this interview, please feel free to ask. For example, if a question I ask is not clear to you, don’t hesitate to ask me to repeat it or explain further. But, if you ask a question about the test, I may not be able to answer you because we want you to try and figure it out on your own.

OK, let’s begin.

Concurrent Think-Aloud Modeling Script

First I’m going to show you how to think out loud. When I’m finished, I’ll ask you to try it.

Remember: the italicized text should not be read word for word. You should be familiar enough with its contents to demonstrate a think aloud in a natural and conversational manner. The main point is to demonstrate talking about how you are thinking AS you read the prompt, so pause after each sentence or two to interpret what you are reading.

Give the student a copy of the writing task so they can follow along.

[NOTE: In order to model and practice think-aloud using materials that are consistent with the writing tasks being evaluated in this study, it was necessary to base these activities on (other) secure writing tasks. Consequently, we have not included the text that will be included in the actual script to give the interviewer an example on which to base his or her model think aloud. The general pattern is to read each sentence of the writing prompt out loud, then pause to interpret and comment on that sentence before proceding to the next.

Student Practice Script

The point of the think-aloud is to describe what you are doing and thinking. Basically, just say out loud the words and thoughts that are in your head.

Ok, let’s have you give it a try. I’m going to give you a writing task to practice with. [Hand the student a copy of the first practice task] and you think aloud as you read the task and make sense of what the instructions are asking you to do. Remember to say out loud whatever you are thinking.

You may need to remind the student to talk aloud as he/she works through the task. If necessary, prompt the student, but be careful not to lead the student.

If the student is not thinking aloud you may want to administer a second practice item.

I’m going to ask you to do another think aloud practice. [Hand the student a copy of the second practice task] Remember to think aloud and tell me everything that is going through your mind as you read the task and make sense of what the instructions are asking you to do.

Transition to Writing Task

Okay, let’s move on to the real writing task. Do you have any questions before we continue? [Answer any questions the student may ask.]

I want you to work on the writing task just like you would on a real test. But I will interrupt you here and there to ask you questions. When you answer my questions, try to “think out loud” so I can understand how you are thinking about the writing task. Because the information you provide is so important to us, I am going to be taking notes while you do that.

Remember, you are not getting a grade on the writing assignment you are doing today, but I hope you will try hard so that we get a good idea of how you would think about this writing assignment if you WERE taking a real test. You are an important part of this research study, and we will work together to understand how students think about writing tasks.

Okay, let’s begin.

Do you have any questions before you start?

Writing Task Protocol and Notes

Open the computer to the page that has the general writing task instructions. Give the student a minute to read the page, then, one by one, hand the student the paper answer sheet and a couple of pencils, the planning paper, and the writing brochure (see Part E).

Say: This is the answer booklet where you will write your response. This is some extra paper you can use for planning, if you want. This is the writing brochure. It has two parts. The first part gives you ideas for planning your writing, [Point to first part.] The second part gives you ideas for reviewing what you have written. [Point to second part.] Take a few minutes to look over the brochure. You don’t have to think aloud for this part; just begin thinking aloud when we actually start the writing task. But you can keep the brochure with you and refer back to it as much as you want.

Give the student up to 3 minutes to look at the writing brochure. If the student is still looking at the brochure after 3 minutes, move him/her on to the actual task statement. To move to the task statement, the student should press the “next” button, but if he/she doesn’t, it’s OK to help with this. The cog lab is NOT focused on evaluating the writing brochure or the general navigation.

When the student is on the next page say: OK. Here is your writing task. Please read the task out loud, like we did in the practice, and tell me what you think it is asking you to do.

Student should read the task statement out loud. Encourage him/her to think aloud while doing so.

Notes from think-aloud of task statement

Ask the following probes if the answers are not fully evident from the think-aloud.

1. PROBE: Can you tell me in your own words what you are being asked to do?

2. PROBE: Do you think that the instructions were clear? Or were they confusing? In what way?

[IF THE WRITING TASK PROMPT INCLUDES A VIDEO] Now let’s watch the video. Remember to talk to me about what you think when you look at the video.

Notes from think-aloud of video

Ask the following probes if the answers are not fully evident from the think-aloud.

3. PROBE: Can you tell me in your own words what you saw in the video? Anything else?

4. PROBE: Were there certain things about the video that you thought were confusing? What things? Why were they confusing?

5. PROBE: Did you think there were any places where the video went too fast? Where? Were you reading along or just listening to the voice? Was it hard to read fast enough to keep up with the video?

OK. Now you have read the assignment and watched the video. Please spend a few minutes thinking about what you will write. You can use the planning paper if you want.

If you can, please talk out loud while you are thinking so that I can understand how you decide what to write. Remember, you can watch the video again or look back at the brochure any time you want.

If the student doesn’t think out loud concurrently, ask him/her to review his/her thinking retrospectively. Be very neutral in probing and only probe for clarification. Don’t, for example, ask the student if he/she will include information from the video in his/her response unless the student says that this is what he/she plans to do.

Notes from planning think-aloud

OK. It’s time to write your reesponse. Please try hard to write a good response, just like you would if this were a real test. I will give approximately 20 minutes. If you are still writing or reviewing your work after 15 minutes, I’ll let you know that you have 5 minutes to wrap up.

After the student is done writing, ask:

6. PROBE: How did it go? Did you write your response the way you planned? Or did you get some new ideas while you were writing?

7. PROBE: Is there anything else you want to say about working on this writing task?

Additional required probes

1. Refocus on the task stimulus.

[IF THE WRITING TASK PROMPT INCLUDES A VIDEO] OK, you’re doing a great job. Now I’d like to replay the video and ask you a few more questions.

8. PROBE: Did you think that it was important to have the video? Do you think that watching the video helped you write your letter? In what way?

This section continues with probes that direct the student’s attention to specific parts of the video that contain the hypothesized inhibitors and facilitators and explores the student’s perceptions. For example, “What did you notice [in this section of the video]?” and “Did you feel like you had enough time to look at everything?”

2. Compare to an alternative version of the task.

Now I am going to show you a very similar task assignment [and video]. It is only slightly different from the one you used to write your assignment, but I would like to know what you think about the differences.

This section includes probes specific to a particular pair of writing tasks and the inhibitors and facilitators identified in those tasks. The kinds of questions include: “Did you find it easier to watch and understand one version of the graphs compared to the other? Why or why not?” and “Do you think that is a good change? Would it have helped you to plan your letter? Why or why not?”

Survey Questionnaire

After completing all of the required probes, administer the survey questionnaire.

Okay, this is the last activity and should only take a few more minutes, but feel free to take your time. Please answer the questions on this sheet of paper to the best of your knowledge. You can circle the answers that you choose.

Hand the student the piece of paper containing the Survey Questions.

Please let me know when you have finished.

Concluding the interview

After completing the survey questionnaire (see Part D), conclude the interview

We’re finished. This has been very, very helpful. Thank you again for coming today and helping us make a better writing test.

Give the gift cards to the student and to the parent (if he/she brought their child to and from the interview), and have each sign the acknowledgement form.

Be sure to turn off all of the recording devices and collect the materials on your way out.

Part C: Grade 4 Writing Study Interviewer Guide and Protocol – Small Scale Tryouts

Small Scale Tryout Guide and Protocol

Text written in italics are to be read aloud by the session proctor.

Session Information:

The session proctor should complete the information from rows one to three prior to the beginning of the tryout session. Rows four to five are to be completed at the end of the tryout session.

Ref#	Preliminary Information	Data
1	Name of session proctor
2	Date
3	Start time of ttryout session
4	End time of tryout session
5	Special circumstances that may have affected the tryouts session (Do not ask test takers, just record observations.)

Materials list:

Incentives:

• Student –$35 gift card

• Parent –$25 gift card

Paper Materials:

• Incentive receipt acknowledgment form (2 forms – student/parent)

• Two copies of consent form (in case parent has not signed one prior to arrival at EurekaFacts); One copy is for the parent if he/she wants a copy

• Student answer booklet, including survey questions (stapled and printed double-sided)

• Planning sheet

• Writing brochures (folded)

Other:

• Pencils/Pens (for student and your note-taking)

Introduction Script:

Hello, my name is _______ and I work for EurekaFacts. It’s nice to meet you, and thank you very much for helping us out today.

Let me begin by explaining why you’re here and what you’re going to be doing today. You are participating in a special study to help the people who write tests for the National Assessment of Educational Progress, or NAEP. Students all over the country take the NAEP tests, and we want to make sure that the tests are easy to understand and work the way they are supposed to. Today you will try out two tasks from a writing test and answer a short survey. These tasks are being developed for fourth graders, just like yourselves, all over the United States.

If at any time you decide you do not want to go on, that is your choice and you may stop.

What you say and write will be used only to help us improve these writing tasks. What we learn from you and other participating students will be summarized in a report where no student names will be listed. After the session, we will use what you wrote to help us understand how our test is working and how we can make it better. Overall, this session should take about 1 hour and 30 minutes.

Do you have any questions?

Interviewer: If a student is no longer interested in participating, thank the student for his or her time and end the tryout session for that one student only (hand them their incentive); continue with all others.

Task Instructions:

Distribute anwer booklets that have been pre-coded with student’s study ID and the ID for the task form that has been assigned to that student.

All of the instructions will be displayed on your computer screens, but you will write your answers in the answer booklet that I just gave you. You will have 30 minutes to complete each of the writing tasks. When you have written your answer to the first writing task, you can review your work, but do not move on to the second writing task until you are instructed to do so. For each writing task, I’ll give you a warning after 25 minutes so you know that you have 5 more minutes to wrap up.

Now please read the general instructions on the computer.

Make sure each student can see the page on the computer that has the general writing task instructions. Once the students have had a minute to read the instructions, hand out the planning paper and then the writing brochure (see Part E).

You already have the answer booklet where you will write your responses.

This is some extra paper you can use for planning, if you want.

This is the writing brochure. It has two parts. The first part gives you ideas for planning your writing, [Point to first part.] The second part gives you ideas for reviewing what you have written. [Point to second part.] Take a few minutes to look over the brochure. You don’t have to use it if you don’t want to, but it may help you write a better response. You can keep the brochure with you and refer back to it as much as you want.

Give the students a minute to look at the writing brochure.

Okay, we are now ready to begin. Please press the “Next” button on the screen. You should now all see the first writing task and can begin. You have 30 minutes to write your response, starting now.

If you have any questions about how to work the computer or the headphones, please raise your hand and I will come to help you.

Start the timer now. Students have 30 minutes to complete the first writng task. After 25 minutes tell them: “You have 5 more minutes to complete the first writing task. ”Now we will move on to the second writing task. Once again, you will have 30 minutes to write your answer, and I will give you a warning when you have 5 minutes left to finish up. Press the “Next” button on the screen to view the second writing task.

Make sure that everyone has successfully moved to the second writng task on the computer.

After you call time on the second writing task say: “Please turn to the last page in your answer booklet, and answer the questions printed on the page (see Part D). Raise your hand when you are finished.”

Concluding the interview:

We’re finished. This has been very, very helpful. Thank you again for coming today and helping us make a better writing test.

Give the gift cards to the student and to the parent (if he/she brought their child to and from the interview), and have each sign the acknowledgement form.

Part D: Survey Questions

Instructions: Please circle the most appropriate answer choice.

1. This school year, how often have you used a computer to watch a video for school?

Skip to question 2

1. Never

   

2. A few times

   

   Go to question 1.1

3. Once every few weeks

   

4. About once a week

5. More than once a week

1. How often did you use information from a video to write a paper for school?

1. Never

2. A few times

3. Once every few weeks

4. About once a week

5. More than once a week

Continue to questions 2 and 3

2. This school year, how often have you used a computer to write a paper for school that was longer than a page?

1. Never

2. A few times

3. Once every few weeks

4. About once a week

5. More than once a week

3. On a weekday, about how many hours do you use a computer for doing schoolwork, including homework?

1. None

2. Less than an hour

3. 1 to 2 hours

4. 2 to 3 hours

5. 3 to 4 hours

6. More than 4 hours

Part E: Writing Brochure

1 60 minutes for cognitive interview and 90 minutes for small scale tryouts