Justification

National Center for Education Statistics

National Assessment of Educational Progress

Volume I

Supporting Statement

National Assessment of Educational Progress (NAEP) Science Questionnaire Cognitive Interviews 2017

OMB# 1850-0803 v.175

October 2016

Table of Contents

1) Submittal-Related Information 3

2) Background and Study Rationale 3

3) Sampling and Recruitment Plans 4

4) Data Collection Process 6

5) Consultations Outside the Agency 6

6) Assurance of Confidentiality 6

7) Justification for Sensitive Questions 7

8) Estimate of Hourly Burden 7

9) Recruitment Costs 8

10) Costs to Federal Government 8

11) Schedule 8

1. Submittal-Related Information

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803), which provides for NCES to conduct various procedures (such as field tests, cognitive interviews, usability studies) to test new methodologies, question types, or delivery methods to improve assessment instruments and data collection methodologies and expand on its educational data analyses that can be provided to the public. This request is to test new questionnaire content through cognitive interviews.

2. Background and Study Rationale

The National Assessment of Educational Progress (NAEP) is a federally authorized survey of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is administered by NCES, which is part of the Institute of Education Sciences in the U.S. Department of Education. NAEP also collects survey questionnaire (i.e., noncognitive) data from students, teachers, and school administrators¹ to provide context for the reporting and interpretation of NAEP results.

The questionnaires aim to capture data related to important subject-specific (i.e., science) and nonsubject-specific (core) contextual factors for student achievement. Table 1 contains the possible areas of focus for the questionnaire development for upcoming NAEP questionnaires.

Table 1. Core Modules and Science Issues

	Core	Science
Module 1/Issue 1	Socio-Economic Status (SES)	Resources for Learning and Instruction
Module 2/Issue 2	Technology Use	Organization and Instruction
Module 3/Issue 3	Grit	Teacher Preparation
Module 4/Issue 4	Desire for Learning	Student Factors
Module 5/Issue 5	School Climate	n/a

New items related to the issues for core and science have been developed for potential inclusion in the NAEP questionnaires for students, teachers, and school administrators, and for pre-testing in cognitive interviews.

In cognitive interviews (often referred to as a cognitive laboratory study or cog labs), an interviewer uses a structured protocol in a one-on-one interview drawing on methods from cognitive science². The objective is to explore how participants are thinking and what reasoning processes they are using to work through tasks. A verbal probing technique will be used for this cognitive interview activity. With verbal probing techniques, the interviewer asks questions, as necessary, to explore additional issues that have been identified a priori as being of particular interest. This interview technique has proven to be productive in previous NAEP pretesting and will be the primary approach in the NAEP cognitive interviews under this submission.

Cognitive interview studies produce largely qualitative data in the form of verbalizations made by participants in response to the interviewer probes. Some informal observations of behavior are also gathered by a second observer who will be present in addition to the interviewer. Behavioral observations may include things such as nonverbal indicators of affect (i.e., emotional states such as frustration or engagement), and interactions with the task (e.g., ineffectual or repeated actions suggesting misunderstanding or usability issues).

Cognitive interviews are instrumental in helping to identify potential problems with items, and they inform item improvement. The main purpose of the proposed here cognitive interview research is to:

1. Identify potential problems with questionnaire items, ensuring each item is understood by all participants and is not sensitive in nature or makes the participant uncomfortable; and

2. Find ways to improve wording of existing items where possible.

The results from this study will also be used to inform which items should be administered during the 2018 core and pilot science test.

Volume I of this submittal contains descriptions along with design, sampling, burden, cost, and schedule information for the study. Volume II contains the student probes, while the appendices contain sample scripts and notification documents.

3. Sampling and Recruitment Plans

ETS has been contracted by NCES to develop NAEP questionnaires and to manage the cognitive interview activity described in this package. EurekaFacts, subcontractor to ETS, will administer the interviews (see Section 5).

Various resources will be employed to recruit participants. For students³, these will include:

• existing participant databases;

• targeted telephone and mail contact lists (i.e., lists that consist of individuals meeting basic criteria such as age or school grade);

• school system research/assessment directors;

• community organizations (e.g., Boys/Girls clubs, Parent-Teacher Associations, and limited on-site location-based and mass media recruiting); and

• outreach/contact methods and resources (e.g., internet ads, flyers/bookmarks, canvassing, and having representatives available to talk to parents, educators, and community members at appropriate local community events, school fairs, etc.).

Teachers and school administrators will be recruited using the following recruitment resources in addition to those mentioned above:

• national organizations’ databases of administrators and faculty;

• NCES school database (e.g., Common Core of Data and Private School Universe Survey);

• contacts within organizations and groups that can serve as recruitment partners (e.g., Horton’s Kids, Housing Authority of the City of Frederick); and if needed

• targeted contact lists.

EurekaFacts will recruit fourth, eighth, and twelfth grade students (i.e., a mix of gender, race/ethnicity, urban/suburban/rural, English language learner (ELL) status⁴, and socioeconomic (SES) background⁵), teachers (i.e., a mix of school sizes and school SES demographics), and school administrators (i.e., a mix of school sizes and school SES demographics) so that a diverse sample is achieved. Please note, SES will be given a higher priority than other respondent characteristics when recruiting, but we will maintain a sufficient balance of other criteria (i.e., a sample includes participants from both genders, both ELL status subgroups, all neighborhood types, school sizes, and all racial subgroups). Additionally, it should be noted that the sample is not large enough to support subgroup analyses. Table 2 summarizes the numbers and types of respondents planned for the cognitive interviews. A minimum number of five respondents per respondent group per grade is recommended to identify major problems with an item, and to conduct a meaningful analysis of data from exploratory cognitive interviews to test the usability of developed prototype questions⁶. We will interview a larger number of students to better ensure identification of confusion or sensitivity issues.

Table 2. Sample Size for Cognitive Interviews

Respondent Group	Grade 4	Grade 8	Grade 12	Total
Students	15	15	15	45
Teachers	5	5	n/a	10
School Administrators	5	5	5	15
Overall Total	25	25	20	70

EurekaFacts will recruit potential participants in urban areas such Washington, D.C., and Baltimore, MD, as well as suburban and rural areas in Maryland and Virginia. EurekaFacts may also recruit teachers and school administrators in other states represented in their database. No more than three students will be recruited per school. No more than one teacher or school administrator will be recruited per school.

To minimize the travel burden of students, parents/guardians, teachers, and school administrators, cognitive interviews will be conducted in nearby venues that are convenient for the participants, such as EurekaFacts offices in Rockville, MD, community centers, facilities of community-based organizations, and school building sites (after school only). Before administering any interviews in school building sites, ETS and the school principal will be notified to confirm approval.

Cognitive interviews will be conducted with students, teachers, and school administrators (specifically principals), and will take approximately 90 minutes each, including time for introductions (maximum 15 minutes), conducting the interview (60 minutes), and debriefing and/or time for additional questions/feedback from the participants (maximum 15 minutes). All student cognitive interviews and the majority of teacher and school administrator cognitive interviews will be conducted in-person. A limited number of teacher/administrator interviews may be conducted via phone or WebEx, if needed.

Recruitment Process

The recruitment process is outlined below:

1. EurekaFacts sends an email of introduction about the cognitive interview research to (a) various middle school and high school principals, (b) individuals from the subcontractors’ existing databases, (c) community centers, organizations, and research/assessment directors, (d) targeted individuals from telephone and mail contact lists, (e) parents/guardians, and (f) teachers and principals (Appendices A-E). The email of introduction will include flyers (Appendices F), an information brochure (Appendix G), and informational bookmarks (Appendix H)⁷.

2. EurekaFacts will only discuss recruitment with those community organizations that contact EurekaFacts upon receiving the email of introduction, flyers, information brochure, and informational bookmarks.

3. After receiving a contact of interest, a EurekaFacts staff member will follow up with the parent/guardian, student age 18 or older, teacher, and school administrator via phone (Appendices I-K), and ask them to provide demographic information to ensure that a diverse sample is selected as per the aforementioned criteria.

4. If the parent/guardian allows their student to participate, EurekaFacts will follow up to confirm participation and the date and time of the cognitive interview session (Appendices O-Q). Similarly, if students age 18 or older, teachers, and school administrators agree to participate, EurekaFacts will follow up to confirm participation and the date and time of the cognitive interview session (Appendices O-Q).

Parents/guardians (on behalf of the students under 18), students age 18 or older, teachers, and school administrators will be required to sign informed consent forms prior to the cognitive interview session (Appendices L-N).

Students, teachers, and school administrators with a signed consent will be asked to participate in a cognitive interview that may last up to 90 minutes. After participating in the cognitive interview, students, parents/guardians (only if they provided transportation to and from the cognitive interview), teachers, and school administrators will receive their incentive (see Section 9) and be sent a thank you letter/email (Appendices R-T).

4. Data Collection Process

Prior to the cognitive interview, participants will be contacted to confirm participation and make logistical arrangements. EurekaFacts staff with experience in cognitive interviews will administer the interviews. Participants will first be welcomed, introduced to the interviewer and the observer (if an in-room observer is present), and told that they are there to ensure that students/teachers/administrators understand the newly developed core and science items (see Volume II). Participants will be reassured that their participation is voluntary and that their responses will be used for research purposes only (see Section 6). As part of the introduction process, the interviewer will explain to participants that their responses will be audio recorded. For the phone/web-based teacher and school administrator cognitive interviews, the interviewer will explain the technology and describe the tools the participants may use, such as muting their phone and asking questions.

The interviewer will be tasked with keeping participants engaged by asking probe questions (see Volume II), soliciting responses from less talkative participants, and asking follow-up questions where appropriate (e.g., “That’s interesting, could you tell me a little bit more about that?”) Interviewers may also take additional notes during the in-person cognitive interviews, including behaviors (e.g., the participant’s facial expressions indicating they are confused) and whether extra time was needed to answer certain questions. Please refer to Volume II for the specific protocols and item probes for the various survey questions being pretested.

Analysis Plan

After the session, the notes and audio recording will be summarized to report main findings and provide illustrative statements that will be analyzed by the NAEP questionnaire development team. The cognitive interview results will be used to improve question wording and to inform which items should be administered during the 2018 core and science pilot test.

5. Consultations Outside the Agency

Educational Testing Service (ETS) serves as the Item Development contractor on the NAEP project, developing cognitive and survey items for NAEP assessments. As such, ETS will manage the activities described in this submission. EurekaFacts, a subcontractor to ETS, is a small, established research and consulting firm in Rockville, Maryland, that offers facilities, tools, and staff to collect and analyze both qualitative and quantitative data. EurekaFacts will recruit participants and administer the cognitive interviews.

6. Assurance of Confidentiality

Participants will be notified that their participation is voluntary and that their answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (20 U.S.C. §9573)].

Written consent will be obtained from participants and from parents or legal guardians of students. Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files and secured for the duration of the study. They will be destroyed after the final report is completed.

The interviews will be audio recorded. The only identification included on the files will be the participant ID. The recorded files will be secured for the duration of the study and will be destroyed after the final report is completed.

7. Justification for Sensitive Questions

Throughout the item, task, and interview protocols development processes, effort has been made to avoid asking for information that might be considered sensitive or offensive. Reviewers have attempted to identify and minimize potential bias in questions.

8. Estimate of Hourly Burden

The estimated burden for recruitment assumes attrition throughout the process. Assumptions for approximate attrition rates for direct participant recruitment are 33 percent from initial contact to follow-up, and 50 percent from follow-up to confirmation. All cognitive interviews will be scheduled for no more than 90 minutes. Table 3 details the estimated burden for the cognitive interview processes.

Table 3. Hourly Burden for Students, Teachers, and School Administrators

Respondent	Hours per respondent	Number of respondents	Number of responses	Total burden hours
Principal/School Administrator or Point Person for Community Organizations for Student Recruitment
Initial contact	0.05	20	20	1
Follow-up & identify students	1.0	13*	13	13
Sub-Total		20	33	14
Parent or Legal Guardian for Student Recruitment
Initial contact	0.05	210	210	10.5
Follow-up via phone	0.15	140*	140	21
Consent & confirmation	0.15	70*	70	10.5
Sub-Total		210	420	42
Teacher and School Administrator Recruitment
Initial contact	0.05	105	105	5.25
Follow-up via phone or e-mail	0.15	70*	70	10.5
Consent & confirmation	0.15	35*	35	5.25
Sub-Total		105	210	21
Participation (Cognitive Interviews)
Students	1.5	45**	45	67.5
Teachers	1.5	10**	10	15
School administrators	1.5	15**	15	22.5
Sub-Total		70**	70	105
Total Burden		335	733	182

* Subset of initial contact group.

** Estimated number of actual participants will be somewhat less than confirmation numbers.

9. Recruitment Costs

To encourage participation and thank participants for their time and effort, a $25 VISA gift card will be offered to each participating student. If a parent or legal guardian brings their student to and from the testing site, they will also receive a $25 VISA gift card to thank them for their time and effort. Additionally, they will receive a thank you letter (see Appendix R) for allowing the student to participate in the study. Teacher and school administrator participants will be offered a $40 VISA gift card and a thank you letter (see Appendix T) for taking part in the study.

10. Costs to Federal Government

The estimated cost for the questionnaire cognitive interview activities described in this submittal is $277,000 as delineated in Table 4.

Table 4. Estimate of Costs

Activity	Provider	Estimated Cost
Design, preparation, analysis, and reporting for questionnaire cognitive interviews	ETS	$ 31,000
Preparation and administration of questionnaire cognitive interviews (including recruitment, incentive costs, and data collection and documentation)	EurekaFacts	$ 246,000

11. Schedule

Table 5 depicts the high-level schedule for the core and science questionnaire cognitive interview activities.

Table 5. High-Level Schedule for the NAEP Questionnaire Cognitive Interview Activities

Dates	Activity
November 2016 – January 2017	Recruitment activities for all questionnaire cognitive interviews. Administer questionnaire cognitive interviews.
January – February 2017	Compile data and findings from questionnaire cognitive interviews.
March 2017	Finalize cognitive interview report.

1 Please note that when “school administrator” is mentioned, we are referring specifically to principals. In NAEP, the questionnaire is referred to as the “school administrator survey questionnaire,” however the directions specify that the questionnaire should be completed by a principal.

2 Blair, J. and Presser, S. (1993). Survey procedures from conducting cognitive interviews to pretest questionnaires: A review of theory and practice. In JSM proceedings, Survey Research Methods Section. Alexandria, VA: American Statistical Association. 370-375.

3 For students under age 18, parents/guardians will receive the various contact information.

4 Based on the spoken language at home.

5 Based on annual income, participation in the National School Lunch Program, and parental education.

6 Roach, A. T., & Sato, E. (2009). White paper: Cognitive interview methods in reading test design and development for alternate assessments based on modified academic achievement standards (AA-MAS). Dover, NH: Measured Progress and Menlo Park, CA: SRI International.

7 Representative flyers and brochures are included in this submission. Finalized versions will be used for actual distribution by subcontractors.