Justification

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB# 1850-0803), which allows for NCES to conduct various procedures (such as cognitive interviews) to test new methodologies, question types, or delivery methods to improve survey instruments and procedures. This request is to conduct cognitive interviews to test new writing tasks for upcoming assessments.

2)Background and Study Rationale

The National Assessment of Educational Progress (NAEP) is a federally authorized survey of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is administered by NCES, part of the Institute for Education Sciences, in the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the various subject areas and to collect survey questionnaire (i.e., non-cognitive) data from students, teachers, and principals to provide context for the reporting and interpretation of assessment results.

In 2017, the NAEP writing assessment for grades 8 and 12 will transition from a laptop-delivered assessment to a tablet-delivered assessment. In 2016, replacement tasks for grades 8 and 12 will be piloted on the tablet-based eNAEP^¹ writing interface. This submission describes cognitive interview testing (also referred to as cog labs) of the new tablet-based writing tasks developed for grade 8 and 12 students, to precede the pilot-testing phase. Cognitive testing of items before using them in large‐scale formal pilot assessments helps to identify and eliminate potential problems with the tasks, resulting in fewer challenges in scoring and analysis durig the pilot, higher pilot item survival rates, imorved assessments, and time efficiencies gained in operationalizing items.

Cognitive testing uses a structured protocol in a one-on-one interview drawing on methods from cognitive science. The objective is to explore how participants are thinking and what reasoning processes they are using to work through tasks. For these writing cognitive interviews, retrospective think-aloud and verbal probing techniques will be employed to elicit student feedback. The focus will be on the degree to which new grades 8 and 12 replacement writing tasks successfully measure targeted cognitive skills and abilities, including whether task stimuli support eliciting targeted skills. The interviews will help answer questions such as, "Does this task measure the construct as intended? Do task stimuli function as intended? Are there any barriers to students completing this task?” The writing cognitive interviews will also utilize process-tracing methodologies, which are designed to reflect the cognitive process as it occurs in real time (e.g., interactive behaviors such as key presses show how the student was interacting with the task and generating text). In addition, immediately after completion of a task, a screen capture of the task as it is being completed will be played, overlaid with a student’s gaze pattern; this will be used to help the student describe what he or she was thinking while engaging in the writing process. Overall, the writing cognitive interviews will allow the gathering action (process) data; a screen capture video of the whole session showing students’ on-screen actions, which can be used to support students’ think-alouds; and a recorded retrospective think-aloud overlaid onto the screen-capture video of the student’s process.

3)Sampling and Recruitment Plan

NCES has contracted Educational Testing Service (ETS), the NAEP cognitive item developer to carry out the cognitive interview activity described in this package. ETS and EurekaFacts, one of ETS’ sub-contractors, will recruit participants and conduct the cognitive interviews. ETS will recruit participants from districts that are located near the ETS Princeton, New Jersey, campus using existing contacts, such as staff at schools and afterschool programs for students. EurekaFacts will recruit from the greater Washington, DC/Baltimore metropolitan area using various outreach methods including over the phone recruitment based on targeted mailing lists of residential addresses and land line telephone listings, newspaper/Internet ads, outreach to community organizations (e.g., Boys and Girls Clubs, Parent‐Teacher Associations), and mass media recruiting (such as postings on the EurekaFacts website). Students will be sampled so as to achive the following participant criteria:

A mix of race/ethnicity (Black, Asian, White, Hispanic),
A mix of socioeconomic background, and
A mix of urban/suburban/rural.

Although the sample will include a mix of student characteristics, results will not explicitly measure differences by these characteristics. Up to 84 students from grades 8 and 12 will participate in the interviews to assure 5-7 students per task and up to 6 tasks per grade. Because some students will complete more than one task, fewer students may be needed.

Interested participants will be screened to ensure that they meet the criteria for participation (e.g., their parents/guardians have given consent and they are from the targeted demographic groups outlined above). When recruiting participants, ETS or EurekaFacts staff will first communicate with the parent/guardian of the interested minor before starting the screening process. The parent/guardian will be informed about the objectives, purpose, and participation requirements of the study and the activities it entails. After confirming that a student is qualified and available to participate, a confirmation e‐mail/letter will be sent and informed parental/guardian consent for the minor’s participation will be obtained. Appendices A-KK provide sample recruitment materials that will be used by ETS and EurekaFacts^².

Recruitment efforts and cognitive interviews will occur on an ongoing basis throughout the study and some cognitive interviews may be conducted simultaneously for different tasks at different stages of development. NCES and contractor staff will discuss key findings and implications for task development thrgouhout the project. Additionally, a summary report will be produced after cognitive interviews are complete, which will include a description of participant characterstics, detected problems or sources of difficulties with tasks, and plans for revising and improving tasks for piloting if necessary.

4)Data Collection Process

Cognitive interviews will take place at ETS, EurekaFacts, or another suitable venue (e.g., a school library or after-school office). Participants will first be welcomed, introduced to the interviewer and the observer (if an in‐room observer is present), and told that they are there to help answer questions about how students respond to writing tasks. Students will be reassured that their participation is voluntary and that their answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573]. Interviewers will explain the think‐aloud process and conduct a practice session with a sample question (see Volume II for the welcome script, think‐aloud instructions, hints for the interviewers, and verbal probes).

Next, students will work through a writing task in a naturalistic way, without interruption. During this phase their actions will be recorded and all events on the screen captured in real time. This is a technique that has been used in previous cognitive lab studies. It involves two software programs: a screen capture recorder such as Morae® and a media player such as the Windows Media Player, which is capable of full screen replay. While participants are working on the writing task, Morae® will record the screen into a video file. During the post-task retrospective think-aloud phase, the screen recording video will be played full screen using the media player, while Morae® will again record the screen plus audio input from a microphone the participant will wear. The resulting video will “overlay” the retrospective think-aloud with the record of the student’s writing process recorded initially. During the think-aloud, the student will be able to see what he or she was doing. Seeing their actions is meant to act as a prompt and help students to reconstruct their thinking at each point in the task.

The protocols for the think‐aloud sections will contain largely generic prompts to be applied flexibly by the interviewer to facilitate and encourage students to verbalize their thoughts. For example: “I see you’re looking at your last paragraph. What are you thinking?”

On completion of a task, the interviewer will proceed with follow‐up questions to collect more information about the strategies and reasoning that the student employed when working through the task. In this verbal probing component, the interviewer will ask the student targeted questions about specific aspects of skill or ability that the task is attempting to measure (e.g., persuasive writing addressed to an authority figure). Interviewers will also be encouraged to raise additional issues that became evident during the course of the interview. For example, if a student pauses for a long time over a particular stimulus, appears to be frustrated at any point, or indicates an ‘aha’ moment, the interviewer might probe the student to find out what caused their reaction. To minimize the burden on the student, efforts will be made to limit the number of verbal probes that can be used in any one session or in relation to any one task. Video screen‐capture software will record all interactions and responses to be replayed for later analysis of how a given student progressed through a task.

During cognitive interviews, participants will work through tasks while data are gathered. Data will then be synthesized in the form of lessons learned about inferred student cognitive processes, observed student behaviors, and the performance of tasks with a focus on whether a task appears to be eliciting the construct of interest (e.g., explanatory writing to a given audience). These lessons will then inform ongoing task development.

Analysis Plan

The results will be compiled to identify patterns of responses for tasks, including patterns of responses to probes or debriefing questions, or types of actions observed from students at specific points in composing a response to a task. This approach will help to ensure that the data are analyzed in a way that is thorough and systematic, and enhance identification of problems with tasks and developing recommendations for addressing these problems.

5)Consultations Outside the Agency

ETS is working with NCES to develop cognitive and survey items for NAEP assessments and is responsible for the carrying out the writing cognitive interview study. Its sub-contractor, EurekaFacts, is a research and consulting firm in Rockville, Maryland that offers facilities, tools, and staff to collect and analyze both qualitative and quantitative data. EurekaFacts will be involved in recruitment and the conduct of cognitive interviews.

6)Assurance of Confidentiality

Students taking part in the cognitive interviews will be notified that their participation is voluntary and that their answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (20 U.S.C. §9573)]. Written consent will be obtained from participants and from parents or legal guardians of students below age 18. Participants will be assigned a unique identifier (ID), which will be created solely for data file management and to keep all participant materials together, and will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the interview files, secured for the duration of the study, and destroyed after the final report is completed. The interviews will be recorded. The only identification included on the files will be the participant ID. The recorded files will be secured for the duration of the study and destroyed after the final report is submitted.

7)Justification for Sensitive Questions

This study does not include sensitive questions.

8)Estimate of Hourly Burden

ETS and Eureka Facts have different attrition rates and the burden estimates for the number of respondents and responses reflect this difference (each cell in Table 1 shows the estimates for Eureka Facts + estimates for ETS). The estimated burden for recruitment assumes attrition throughout the process. Assumptions for approximate attrition rates for direct participant recruitment from initial contact to follow-up are 40 percent for EurecaFacts and 75 percent for ETS, and from follow-up to confirmation 50 percent for EurecaFacts and 20 percent for ETS. All cognitive interview sessions will be scheduled for no more than 90 minutes.

Table 1. Burden for Writing Cognitive interviews (EurekaFacts Conducted + ETS Conducted)

Respondent	Hours per respondent	Number of respondents	Number of responses	Total hours (rounded up)
Schools and Organizations
Initial contact	0.05	100 + 40	100 + 40	7
Follow-up contact	0.15	60* + 10*	60 + 10	11
Confirmation	0.05	30* + 8*	30 + 8	2
Sub-Total		140	248	20
Parent or Legal Guardian for Student Recruitment
Initial contact	0.05	250 + 140	250 + 140	20
Follow-up contact	0.15	150* + 35*	150 + 35	28
Consent form completion and return	0.13	75* + 28*	75 + 28	14
Confirmation	0.05	75* + 28*	75 + 28	6
Sub-Total		390	781	68
Participation (Cognitive Interviews)
Students	1.5	64^a + 20^a	64 + 20	126
Sub-Total		64^a + 20^a	64 + 20	126
Total Burden		614	1,113	214

* Subset of initial contact group, not double counted in the total number of respondents.

^a Estimated number of actual participants will be somewhat less than confirmation numbers.

9)Recruitment Costs

To encourage participation and thank them for their time and effort, a $25 credit card gift card will be offered to each participating student. If a parent or legal guardian brings their student to and from the testing site they will also receive a $25 gift card along with a thank you letter for allowing their child to participate in the study.

10)Costs to Federal Government

The estimated cost to federal government for the writing cognitive interview activities is $606,960.

Table 2. Estimate of Costs to Federal Government

Activity	Provider	Cost
Design, prepare, and conduct writing cognitive interviews (including recruitment, allocation of incentive costs, data collection, analysis, and reporting)	ETS	$ 175,804
Prepare and conduct writing cognitive interviews (including recruitment, allocation of incentive costs, data collection, analysis, and reporting)	EurekaFacts	$ 431,156
Total Estimate		$ 606,960

11)Schedule

The following high‐level schedule assumes a pilot test in 2016 for NAEP 2017 grades 8 and 12 assessment.

Table 4. Timeline for Cognitive Interviews for NAEP Writing, Grades 8 and 12

Activity	Dates
Recruit participants	January 2015 ─ March 2015
Data collection, preparation, and coding	February 2015 ─ April 2015
Data analysis of cognitive interview results	April 2015 ─ May 2015
Cognitive interview reports	June 2015

1 eNAEP is the assessment delivery system used to deliver the NAEP technology-based assessments.

2 Note: If appropriate, relevant appendices (i.e., parental screening calls) may be translated to facilitate communication.

File Type	application/msword
Author	George P Barrett
Last Modified By	Kashka Kubzdela
File Modified	2015-01-22
File Created	2015-01-15