NAEP Questionnaires Cog Interview Supporting Statement

National Center for Education Statistics

National Assessment of Educational Progress

Volume I

Supporting Statement

NAEP Pretesting Survey Questions for Core, Reading, Writing, and Mathematics – Cognitive Interviews

OMB# 1850-0803 v.117

October 31, 2014

Cognitive Interviews - Core, Reading, Writing, and Mathematics

Table of Contents

1) Submittal-Related Information 1

2) Background and Study Rationale 1

3) Sampling and Recruitment Plans 2

4) Data Collection Process 4

5) Consultations Outside the Agency 5

6) Assurance of Confidentiality 5

7) Justification for Sensitive Questions 6

8) Estimate of Hourly Burden 6

9) Recruitment Costs 6

10) Costs to Federal Government 7

11) Schedule 7

1. Submittal-Related Information

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB# 1850-0803), which allows for NCES to conduct various procedures (such as cognitive interviews) to test new methodologies, question types, or delivery methods to improve survey instruments and procedures.

2. Background and Study Rationale

The National Assessment of Educational Progress (NAEP) is a federally authorized survey of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is administered by NCES, part of the Institute for Education Sciences, in the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the various subject areas and to also collect survey questionnaire (i.e., non-cognitive) data from students, teachers, and principals to provide context for the reporting and interpretation of assessment results.

The NAEP Core¹, Reading, Writing, and Mathematics Survey Questionnaires aim to capture data related to important subject-specific (i.e., reading, writing, and mathematics) and nonsubject-specific (Core) contextual factors for student achievement. Table 1 contains the areas of focus for the questionnaire development for upcoming NAEP survey questionnaires.

Table 1. Areas of Focus for Questionnaire Development.

Core	Reading	Writing	Mathematics
Socio-Economic Status (SES)	Resources for Learning and Instruction	Availability and Use of Instructional Resources	Resources for Learning and Instruction
Technology Use	Organization and Instruction	Organization of Writing Instruction	Organization and Instruction
Grit2	Teacher Preparation	Teacher Preparation	Teacher Preparation
Desire for Learning	Student Factors	Role of Technology in Writing	Student Factors
School Climate	n/a	Student Engagement with Writing	n/a

New items related to questionnaire content for core and subject areas have been developed for pre-testing in cognitive interviews and for potential inclusion in the NAEP Core, Reading, Writing, and Mathematics Survey Questionnaires.

In cognitive interviews (often referred to as a cognitive laboratory study or cog lab), an interviewer uses a structured protocol in a one-on-one interview drawing on methods from cognitive science. The objective is to explore how participants are thinking and what reasoning processes they are using to work through tasks. A verbal probing technique will be used for this cognitive interview activity. With verbal probing techniques, the interviewer asks probing questions, as necessary, or to explore additional issues that have been identified a priori as being of particular interest. This interview technique has proven to be productive in previous NAEP pretesting and will be the primary approach in the NAEP cognitive interviews under this package.

Cognitive interview studies produce largely qualitative data in the form of verbalizations made by participants in response to the interviewer probes. Some informal observations of behavior are also gathered, since typically a second observer is involved, in addition to the interviewer. Behavioral observations may include such things as nonverbal indicators of affect, suggesting emotional states such as frustration or engagement, and interactions with the task, such as ineffectual or repeated actions suggesting misunderstanding or usability issues.

Cognitive interviews are important given that they help to identify potential problems with items, as well as help inform how to improve items.

The main purposes of the proposed cognitive interview research are to:

1. Identify problems with the items (i.e., ensure the item is understood by the participant at all grade levels, and confirm items are not sensitive in nature or make the participant uncomfortable);

2. Explore ways to improve examples used within items;

3. Find ways to simplify wording in items where possible, particularly for the grade 4 student questionnaires;

4. Explore whether matrix items can be included in grade 4 student questionnaires³;

5. Compare alternative versions of items to identify appropriate version(s) for NAEP;

6. Compare different response options in order to identify appropriate sets of response options and replace vague response options by more quantifiable and specific response options if feasible; and

7. Compare response scales with different number of scale points (e.g., a 3 point scale or 5 point scale) in order to identify the most appropriate approach for NAEP.

The results from this study will be used to help inform which items should be administered during upcoming NAEP administrations (2016 pilot and 2017 operational). Volume I of this submittal contains descriptions of the design and sampling, as well as burden, cost, and schedule information for the study. Volume II contains the protocols, items, and probes for the core, reading, writing, and mathematics cognitive interviews. The appendices contain notifications, consent forms, screening checklists, informational flyers, phone scripts, and thank you documents.

3. Sampling and Recruitment Plans

NCES contracted ETS to develop NAEP survey questionnaires and to carry out the cognitive interview activity described in this package. EurekaFacts and CRP, Inc., subcontractors to ETS, will conduct the cognitive interviews.

Various resources will be employed to recruit participants. For students⁴, these will include:

• existing participant databases;

• targeted telephone and mail contact lists;

• school system research/assessment directors;

• NAEP State Coordinators (see section 5) when possible to recruit in schools;

• community resources (e.g., Boys/Girls clubs, Parent-Teacher Associations, and limited on-site location-based and mass media recruiting); and

• out-reach/contact methods and resources (e.g., internet ads, flyers/bookmarks, canvassing, and having representatives available to talk to parents, educators, and community organizers throughout the community at appropriate local events, school fairs, etc.).

Teachers and school administrators will be recruited using the following recruitment resources in addition to those mentioned above:

• national organizations’ databases of administrators and faculty;

• NCES school database;

• contacts within organizations and groups that can serve as recruitment partners; and if needed

• targeted contact lists.

The contractors will recruit 4^th, 8^th, and 12^th grade students (i.e., a mix of gender, race/ethnicity, urban/suburban/rural, and socioeconomic background), teachers (i.e., a mix of school sizes, and a mix of school socioeconomic demographics), and school administrators (i.e., a mix of school sizes, and a mix of school socioeconomic demographics) so that a diverse sample is achieved. Please note SES (socio-economic status) characteristics will be given a higher priority than other respondent characteristics when recruiting while also ensuring sufficient balance of other criteria. Table 2 summarizes the numbers and types of cognitive interviews that are planned for these pretesting activities.

Table 2. Sample Size for Core, Reading, Writing, and Mathematics Cognitive Interviews

Respondent Group	Grade 4	Grade 8	Grade 12	Total
Core Students	20	20	10	50
Core Teachers	8	8	n/a	16
Core School Administrators	8	8	5	21
Core Sub-total	36	36	15	87
Reading Students	15	15	n/a	30
Reading Teacher	5	5	n/a	10
Reading School Administrators	5	5	n/a	10
Reading Sub-total	25	25	n/a	50
Writing Students	n/a	15	15	30
Writing Teachers	n/a	5	n/a	5
Writing School Administrators	n/a	5	5	10
Writing Sub-Total	n/a	25	20	45
Mathematics Students	15	15	n/a	30
Mathematics Teachers	5	5	n/a	10
Mathematics School Administrators	5	5	n/a	10
Mathematics Sub-total	25	25	n/a	50
Overall Total	86	111	35	232

The total number of planned cog interviews per category are: students-140; teachers-41; school administrators-51.

Both subcontractors will recruit potential participants in urban areas such Washington, D.C.; and Baltimore, MD, as well as suburban and rural areas in Maryland and Virginia. In addition to the aforementioned areas, CRP will also recruit teachers and school administrators in up to 12 other states represented in their database.

To minimize the travel burden of students, parents/guardians, teachers, and school administrators cognitive interviews will be conducted in nearby venues that are convenient for the participants, such as CRP and EurekaFacts offices in Silver Spring, MD and Rockville, MD, community centers, facilities of community-based organizations, and school building sites (after school only). Before conducting any interviews in school building sites, ETS, the school principal, and the NAEP State Coordinators will be notified to confirm approval.

Each subcontractor will follow a similar protocol for the cognitive interview activity (i.e., recruit participants, follow up with participants, confirm participation, and attain consent), but may follow slightly different recruitment strategies given their available resources and expertise. One and a half hours (90 minutes)⁵ cognitive interviews will be conducted with students, teachers, and school administrators (specifically principals). All student cognitive interviews and the majority of teacher and school administrator cognitive interviews will be conducted in-person⁶.

Recruitment Process

The recruitment process includes:

1. Contractors send an email of introduction about the cognitive interview research to (a) various elementary, middle school, and high school principals, (b) individuals in the subcontractors’ existing databases, (c) community centers and research/assessment directors, (d) targeted telephone and mail contact lists, (e) parents/guardians, and (f) teachers and school administrators (Appendices A-E). The email of introduction will include flyers (Appendices F-J), an information brochure (Appendix K), and informational bookmarks (Appendix L)⁷.

2. CRP will follow up with the principal/school administrator or point person at the community centers/youth centers to confirm their willingness to allow CRP to recruit students in their school or facilities (Appendix M). Principals/school administrators and point persons at the community centers/youth centers that agree to participate will be asked to disseminate the flyers. EurekaFacts will only discuss recruitment with those community centers/youth centers that contact EurekaFacts upon receiving the email of introduction, flyers, information brochure, and informational bookmarks.

3. After receiving a contact of interest, a CRP/EurekaFacts staff member will follow up with the parent/guardian, teacher, and school administrator via phone (Appendices N-P), and ask them to provide demographic information to ensure that a diverse sample is selected as per the aforementioned criteria.

4. If the parent/guardian allows their student to participate, and the teacher and school administrator agree to participate, the subcontractors will follow up to confirm participation and the date and time of the cognitive interview session (Appendices T-V).

5. Parents/guardians (on behalf of the students under 18), students age 18 or older, teachers, and school administrators will be required to sign informed consent forms prior to the cognitive interview session (Appendices Q-S).

6. Students, teachers, and school administrators with a signed consent will be asked to participate in cognitive interviews that may last up to 90 minutes. After participating in the cognitive interview, students, parents/guardians (only if they provided transportation to and from the cognitive interview), teachers, and school administrators will receive their incentive (see Section 9) and be sent a thank you letter/email (Appendices W-Y).

4. Data Collection Process

Participants will first be welcomed, introduced to the interviewer and the observer (if an in-room observer is present), and told that they are there to help ensure that students/teachers/administrators like them understand the newly developed core, reading, writing, and mathematics items (see Volume II). Participants will be reassured that their participation is voluntary and that their responses will be used for research purposes only (see Section 6). As part of the introduction process, the interviewer will explain to participants that their responses will be audio recorded. For the phone/web-based student, teacher, and school administrator cognitive interviews, the interviewer will explain the technology and describe the tools the participants may use, such as muting their phone and asking questions.

The interviewer will be tasked with keeping participants engaged by asking probe questions (see Volume II), soliciting responses from less talkative participants, and asking follow-up questions where appropriate (e.g., “that’s interesting, could you tell me a little bit more about that”). Interviewers may also take additional notes during the in-person cognitive interviews, including behaviors (e.g., the participant’s facial expressions indicated they are confused) and if extra time was needed to answer certain questions. Please refer to Volume II for the specific protocols and item probes for the various survey questions being pretested.

Analysis Plan

After the session, the notes and audio recording will be summarized to report main findings and illustrative statements which will be analyzed by the NAEP questionnaire development team. The cognitive interview results will be used to help inform which items should be administered during the 2016-2017 NAEP administrations.

5. Consultations Outside the Agency

Educational Testing Service (ETS) serves as the Item Development (ID) contractor to NCES on the NAEP project, working with NCES to develop cognitive and survey items for NAEP assessments. ETS is responsible for the management of all activities described in this package.

NAEP State Coordinators serve as liaisons between the state education agency and NAEP staff, coordinating NAEP activities in his or her state. Through their interactions with local education agencies and public schools, the state coordinators can provide leads (through the NAEP State Service Center contractor (Westat)) for potential participants for this study.

CRP, Incorporated (CRP) is a professional services and management consulting firm located in Silver Spring, Maryland that provides technical consulting primarily in the fields of education and behavioral health. CRP is a small business and serves as a subcontractor to ETS on the survey item development work.

EurekaFacts is a research and consulting firm in Rockville, Maryland that offers facilities, tools, and staff to collect and analyze both qualitative and quantitative data. EurekaFacts is working as a subcontractor for ETS on survey questionnaire development projects, and will recruit and conduct the cognitive interviews.

6. Assurance of Confidentiality

Participants will be notified that their participation is voluntary and that their answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (20 U.S.C. §9573)].

Written consent will be obtained from participants and from parents or legal guardians of students. Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files and secured for the duration of the study and will be destroyed after the final report is completed. The interviews will be audio recorded. The only identification included on the files will be the participant ID. The recorded files will be secured for the duration of the study and will be destroyed after the final report is submitted.

7. Justification for Sensitive Questions

Throughout the item, task, and interview protocols development processes, effort has been made to avoid asking for information that might be considered sensitive or offensive. Reviewers have attempted to identify and minimize potential bias in questions.

8. Estimate of Hourly Burden

The estimated burden for recruitment assumes attrition throughout the process.⁸ All cognitive interviews will be scheduled for no more than 90 minutes. Table 3 details the estimated burden for the survey questionnaire cognitive interview processes.

Table 3. Hourly Burden for Students, Teachers, and School Administrators for Core, Reading, Writing, and Mathematics Cognitive Interviews

Respondent	Hours per respondent	Number of respondents	Number of responses	Total hours
Principal/School Administrator or Point person for community centers/youth centers for Student Recruitment
Initial contact	0.05	75	75	4
Follow-up and assistance	0.05	50*	50	3
Sub-Total		75	125	7
Parent or Legal Guardian for Student Recruitment
Initial contact	0.05	540	540	27
Follow-up contact	0.15	360*	360	54
Consent form completion and return	0.13	180*	180	24
Confirmation	0.05	180*	180	9
Sub-Total		540	1,260	114
Teacher and School Administrator Recruitment
Initial contact	0.05	315	315	16
Follow-up contact	0.15	210*	210	32
Consent form completion and return	0.13	105*	105	14
Confirmation	0.05	105*	105	6
Sub-Total		315	735	68
Participation (Cognitive Interviews)
Students	1.5	140*a	140	210
Teachers	1.5	41*a	41	62
School Administrators	1.5	51*	51	77
Sub-Total		232*a	232	349
Total Burden		930	2,352	538

* Subset of initial contact group, not double counted in the total number of respondents.

^a Estimated number of actual participants will be somewhat less than confirmation numbers.

9. Recruitment Costs

To encourage participation and thank participants for their time and effort, a $25 VISA gift card will be offered to each participating student. If a parent or legal guardian brings their student to and from the testing site they will also receive a $25 VISA gift card along with a thank you letter (see Appendix W) for allowing the child to participate in the study. Teacher and school administrator participants will be offered a $25 VISA gift card for interviews conducted remotely (via telephone or WebEx) or a $40 VISA gift card for interviews conducted in person, and a thank you letter (see Appendix Y) for taking part in the study.

10. Costs to Federal Government

The estimated cost for the survey questionnaire cognitive interview activities in this submittal is $684,000 as delineated in Table 4.

Table 4. Estimate of Costs

Activity	Provider	Estimated Cost
Design, preparation, and analysis for Core, Reading, Writing, and Mathematics survey questionnaire cognitive interviews	ETS	$58,000
Preparation and conduct of Core survey questionnaire cognitive interviews (including recruitment, incentive costs, data collection, analysis, and reporting)	CRP	$149,000
Preparation and conduct of Reading, Writing, and Mathematics survey questionnaire cognitive interviews (including recruitment, incentive costs, data collection, analysis, and reporting)	EurekaFacts	$477,000
Total Estimate		$684,000

11. Schedule

Table 5 depicts the high-level schedule for the core, reading, writing, and mathematics cognitive interview activities. The commencement of activities is contingent upon OMB approval.

Table 5. High-level schedule for the core and subject-specific cognitive interview activities.

Dates	Activity
November 2014 –February 2015	On-going recruitment activities for all cognitive interviews for core and subject-specific questionnaires. Conducting core and subject-specific cognitive interviews.
Early March 2015– mid March 2015	Compilation of data and findings from cognitive interviews.
March 2015	ETS ID submits cognitive interview report to NCES.

1 NCES collects student question data, referred to as core questions, that are required by law (20 U.S.C. § 9622; i.e., race, ethnicity, and socioeconomic status) and provide a context for reporting student performance.

2 Grit is defined as perseverance and passion for long-term goals (Duckworth, Peterson, Matthews, and Kelly, 2007).

3 This point is not applicable for writing survey questionnaires which will only be administered at grades 8 and 12 in 2017.

4 For students under age 18, parents/guardians will receive the various contact information.

5 Please note that the 90 minutes includes time for introductions (maximum 15 minutes), conducting the interview (60 minutes), and debriefing and/or time for additional questions/feedback from the participants (maximum 15 minutes).

6 A limited number of teacher/administrator interviews may be conducted via phone or WebEx if needed.

7 Representative flyers and brochures are included in this submission. Finalized versions will be used for actual distribution by contractors.

8 Assumptions for approximate attrition rates for direct participant recruitment are 33 percent from initial contact to follow-up, 50 percent from follow-up to confirmation.

i