Volume I -NAEP TEL and eNAEP Pretesting Study 2017

National Center for Education Statistics

National Assessment of Educational Progress

Volume I

Supporting Statement

National Assessment of Educational Progress (NAEP)

Technology and Engineering Literacy (TEL) and eNAEP

Pretesting Study

OMB# 1850-0803 v. 199

June 2017

Table of Contents

1) Submittal-Related Information 3

2) Background and Study Rationale 3

3) Recruitment and Data Collection 3

4) Consultations outside the agency 5

5) Justification for Sensitive Questions 5

6) Paying Respondents 5

7) Assurance of Confidentiality 6

8) Estimate of Hourly Burden 6

9) Cost to federal government 6

10) Project Schedule 7

1. Submittal-Related Information

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB# 1850-0803), which provides for NCES to conduct various procedures (such as pilot tests, cognitive interviews, and usability studies) to test new methodologies, question types, or delivery methods to improve survey and assessment instruments and procedures.

2. Background and Study Rationale

The National Assessment of Educational Progress (NAEP) is a federally authorized survey (the National Assessment of Educational Progress Authorization Act; 20 U.S.C. §9622) of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is conducted by NCES, which is part of the Institute of Education Sciences, within the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the different subject areas and collect survey questionnaire (i.e., non-cognitive) data from students, teachers, and principals to provide context for the reporting and interpretation of assessment results.

Over the last few years, NAEP has been transitioning to digitally based assessments (DBA) that are administered on laptops and tablets using a test delivery system developed for NAEP (known as eNAEP). The eNAEP system was successfully used in the 2015 and 2016 pilot assessments.¹ The first operational use of the eNAEP system was in conjunction with the 2017 NAEP assessments. Enhancements have been made after each administration of eNAEP to address issues identified in the field, to make the system more user-friendly, and to allow for the assessment of additional content and scenario-basted item types.

The request is to conduct a real-world test of the Technology and Engineering Literacy (TEL) and eNAEP systems with students, allowing the systems to be tested in the manner that will be used in the national study to help identify system issues early in the software development process. The rationale for this study is based on lessons learned and issues encountered by students in the field during the 1) 2017 assessments and 2) the 2014 TEL administration during normal testing. It is believed that students use and interact with the systems differently than adult QC testers. Therefore, including students as part of the pretesting and QC process should allow for issues to be identified and addressed prior to the operational use of the system.

The pretesting will be conducted with students from grade 8 for TEL and grades 4, 8, and 12 for eNAEP², and will be held in a simulated classroom after each preliminary/draft version of TEL and eNAEP (referred to as “builds”) is produced. Up to 10 pretesting events will take place over the course of the TEL and eNAEP development, refinement, and quality control with event sessions lasting approximately 100 minutes each.

3. Recruitment and Data Collection

Recruitment and Sample Characteristics

An NCES subcontractor for NAEP, EurekaFacts, will recruit up to 30 students for each session of TEL pretesting. Each TEL session will be exclusively composed of 8^th graders, and there will be a maximum of three sessions. For each round of eNAEP testing (up to seven rounds), EurekaFacts will recruit a maximum of 102 students (42 students in grade 4, 30 students in grade 8, and 30 students in grade 12). Each session will be held at the EurekaFacts facility in Rockville, Maryland, most likely on a Saturday.

EurekaFacts will recruit participants for the pretesting study from the District of Columbia, Maryland, Virginia, and West Virginia. Although the sample will include a mix of student characteristics, the results will not explicitly measure differences by those characteristics. Students will be recruited to obtain the following criteria:

• A mix of race/ethnicity (Black, Asian, White, Hispanic);

• A mix of socioeconomic background;

• A mix of urban/suburban/rural areas; and

• A mix of students requiring accommodations.

While EurekaFacts will use various outreach methods (see Appendices A-H) to recruit students to participate, the bulk of the recruitment will be conducted by telephone and be based on their acquisition of targeted mailing lists containing residential addresses and landline telephone listings. EurekaFacts will also use a participant recruitment strategy that integrates multiple outreach methods and resources such as newspaper and internet ads, community organizations (e.g., Boys and Girls Clubs, Parent-Teacher Associations), and mass media recruitment (e.g., postings on the EurekaFacts website).

Interested students will be screened (see Appendix F) to ensure that they meet the criteria for participation in the pretesting study (i.e., the students are from the targeted demographic groups outlined above and their parents/legal guardians have given consent). When recruiting participants, EurekaFacts staff will speak to the parent/legal guardian of the interested minor before starting the screening process. During this communication, the parent/legal guardian will be informed about the objectives, purpose, and participation requirements of the data collection effort as well as the activities that it entails. After confirming that a participant is qualified, willing, and available to participate in this study, he or she will receive a confirmation email/letter and phone call. Written informed parental consent (see Appendix J) will be obtained for all respondents who are interested in participating in the data collection efforts. Shortly after each of the preliminary/draft TEL and eNAEP builds are released, a pretesting event with students will be held in a simulated classroom.

All events will have one session lasting approximately 100 minutes each, and it will be structured as follows:

• During each session, each student will be asked to take the assessment under standard NAEP assessment conditions (approximately 90 minutes). Westat will administer the session using standard procedures. Students will take the full assessment, including the tutorial, cognitive items and tasks³, and the survey questionnaires.⁴

• A group debrief (up to 10 minutes) will be conducted to solicit feedback from the students. See Volume II for the debriefing script.

As part of the assessment administration in all event sessions, students will take a set of survey questionnaires. The maximum time for the survey questionnaire component is 15 minutes (included in the 100-minute time estimation for each session). Students will take a “core” section regarding general student and contextual information and a subject-specific section. Volume II includes the library of possible student survey items to be administered.⁵ Not all of the items presented in Volume II will be administered in this TEL and eNAEP pretesting study. The number of items selected for each student will be appropriate to the time allocated. As the items for the 2018 administration are finalized throughout the development process, the final subset will be included in the eNAEP system for pretesting. As such, the earlier builds may include different items selected from the library in Volume II than the final build.

Data Collection Process

Normal data collection will be enabled by the TEL and eNAEP systems, and any errors generated will be collected automatically by the system. Note that student responses will not be scored. In addition to the TEL and eNAEP systems recording information, administrators and observers from NCES, Westat, Fulcrum, ETS, and/or EurekaFacts will monitor the assessments and record notes detailing any issues encountered by the students, as well as what the students were doing at the time each issue occurred. In addition, observers may ask individual students for clarification of the actions he or she took prior to an issue or error occurring. For example, observers may ask questions such as, “What is the error?”; “What was the last thing you saw before the error?”; “What were you expecting to happen?”; or “What did you do right before the error happened?”. Understanding and documenting what caused the system error is necessary in order to have enough information for staff to replicate the error and develop a fix for it.

The sessions will be audio and/or video recorded to capture information regarding any student actions that resulted in system errors or issues.

4. Consultations outside the agency

Westat is the Sampling and Data Collection contractor for NAEP. Westat will provide the laptops and tablets for the students’ use and carry out the pretesting study.

Fulcrum is the NAEP contractor responsible for the development and ongoing support of NAEP DBAs for NCES, including the system to be used for the TEL and eNAEP pretesting study. Fulcrum will be onsite to assist Westat in the administration of the study.

ETS serves as the Planning and Coordination, Item Development, Design, Analysis, and Reporting contractor for NAEP. ETS staff may assist in administering and/or observing some sessions.

EurekaFacts is located in Rockville, Maryland. It is an established for-profit research and consulting firm, offering facilities, tools, and staff to collect and analyze both qualitative and quantitative data. EurekaFacts is working as a subcontractor for ETS to recruit participants and provide the facilities to be used for the study. In addition, EurekaFacts staff may assist in administering and/or observing some sessions.

5. Justification for Sensitive Questions

Throughout the item and debriefing question development processes, effort has been made to avoid asking for information that might be considered sensitive or offensive.

6. Paying Respondents

To encourage participation and thank students for their time and effort, a $25 gift card from a major credit card will be offered to each participating student. If a parent or legal guardian brings their student to and from the testing site, they will also receive a $25 gift card to thank them for their time and effort in transporting their child.

7. Assurance of Confidentiality

The study will not retain any personally identifiable information. Prior to the start of the study, students will be notified that their participation is voluntary. As part of the study, students will be notified that the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Before each pretesting session is administered, for each participant under 18 years of age a written consent will be obtained from their parent/legal guardian and directly from students over 18 years of age. Participants will be assigned a unique student identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files, secured for the duration of the study, and will be destroyed after the final report is released. Pretesting activities may be recorded using audio and/or screen capture technology. The only identification included on the files will be the participant ID. The recorded files will be secured for the duration of the study and will be destroyed after the final report is completed.

8. Estimate of Hourly Burden

The estimated burden for recruitment assumes attrition throughout the process.⁶ In all events, each student will participate in one session for a total of 100 minutes. Table 1 details the estimated burden.

Table 1. Estimate of Hourly Burden

Respondent	Number of respondents	Number of responses	Hours per respondent	Total hours
Parent or Legal Guardian for Student Recruitment
Initial contact	3,572	3,572	0.05	179
Follow-up via phone	1,786*	1,786	0.15	268
Consent & confirmation	893*	893	0.15	134
Recruitment Totals	3,572	6,251	-	581
Participation (Pretesting)
Students in TEL Events	90	90	1.67	150
Students in eNAEP Events	714	714	1.67	1,192
Participation Totals	804	804	-	1,342
Total	4,376	7,055	-	1,923

* Subset of initial contact group

Note: numbers have been rounded and therefore may affect totals

9. Cost to federal government

The total cost of the study is $523,656 as detailed in Table 2.

Table 2: Estimate of Costs

Activity	Provider	Estimated Cost
Recruiting students and providing facilities for the study	EurekaFacts	$384,268
Administering the study	Westat	$139,388
Total		$523,656

10. Project Schedule

The schedule for this study including all activities, will begin in July 2017 through December 2017 as detailed in Table 3.

Table 3: Project Schedule

Activity	Dates
Recruitment, pretesting, data collection, analysis and final report	July–December 2017

1 More information about NAEP DBAs can be found at http://nces.ed.gov/nationsreportcard/dba/default.aspx.

2 For sessions occurring during the summer, students entering or who have completed grades 4, 8, and 12 will be included in recruiting procedures.

3 Technology and engineering literacy, mathematics, science, reading, U.S. history, civics, and geography items and tasks will be administered as part of the eNAEP pretesting study. Within a session, each student will receive items from only one subject area.

4 Draft content may be used in the earlier builds.

5 The final items will consist of those selected for NAEP 2018 administration (currently under OMB review: OMB #1850-0928 v. 5). The questionnaire components in Volume II are a subset of the questionnaires provided in the 1850-0928 v.5 submittal.

6 Assumptions for approximate attrition rates are 50 percent from initial contact to follow-up, 50 percent from follow-up to confirmation, and 90 percent from confirmation to participation.