Justification

National Center for Education Statistics

National Assessment of Educational Progress

Volume I

Supporting Statement

The National Assessment of Educational Progress (NAEP)

2018 Field Trial

OMB# 1850-0803 v.209

September 2017

Table of Contents

1 Submittal-Related Information 3

2 Background and Study Rationale 3

3 Recruitment and Sample Characteristics 4

4 Study design and data collection 4

5 Consultations outside the agency 5

6 Justification for Sensitive Questions 5

7 Paying Respondents 5

8 Assurance of Confidentiality 6

9 Estimate of Hourly Burden 6

10 Cost to the Federal Government 7

11 Project Schedule 7

1. Submittal-Related Information

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB# 1850-0803), which provides for NCES to conduct various procedures (e.g., focus groups, cognitive interviews, usability testing) to test new methodologies, question types, or delivery methods to improve survey and assessment instruments and study procedures, in order to improve the resulting data quality, utility, and study participant experience.

2. Background and Study Rationale

The National Assessment of Educational Progress (NAEP) is a federally authorized survey (the National Assessment of Educational Progress Authorization Act; 20 U.S.C. §9622) of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is conducted by NCES, which is part of the Institute of Education Sciences, within the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the different subject areas and collect survey questionnaire (i.e., non-cognitive) data from students, teachers, and principals to provide context for the reporting and interpretation of assessment results.

As NAEP continues to develop and improve its digitally based data collection instruments, the assessment and questionnaire digital platform also continues to be developed. For each new administration of digital NAEP, the platform is updated based on lessons learned from previous years of NAEP administrations. For example, tools may be added or updated, login procedures may change, the browser version being used to access the assessment may be updated, and so on. As such, items that were piloted in previous years will, in the end, be nationally administered in a somewhat different version of the assessment platform and under somewhat different administrative procedures.

While NAEP has successfully conducted digitally-based administrations over the last few years, unforeseen issues have arisen during the first week of the administration, when field staff were administering the assessment in schools and uploading the data according to NAEP procedures. For example, in the first week of the 2017 administration, issues were encountered with the consistency of the field staff’s administrator laptops connecting to the NAEP-provided router during the assessment and with the field staff’s home Wi-Fi when in the evening they were to upload the assessment data. While the encountered issues did not impact the success of the NAEP administration, such issues are best detected and resolved before the start of the national data collection. Depending on the issues encountered, their solutions may require changes to administrative procedures for the field staff, updates to the software/hardware used by the field staff, and/or an update to the student testing application (known as eNAEP).

This request is to conduct a NAEP 2018 Field Trial to identify and address any issues with the NAEP 2018 digital platform and its accompanying procedures. The NAEP 2018 operational assessment and questionnaire items were tested during the prior year pilot tests (using those years’ versions of the platform and procedures), and now will be used (along with the digital content to be pilot tested on tablets in 2018) in the NAEP 2018 Field Trial to test the 2018 version of the platform and procedures. The purpose of NAEP pilot testing is to try-out new content and item types, gather data on their performance, and refine them before they are operationalized. The Field Trial, on the other hand, is designed to test the updated platform and administrative procedures to identify any logistical challenges and address them prior to the 2018 NAEP administration. The procedures to be tested in the Field Trial include:

• Conducting pre-assessment activities such as contacting the school to confirm logistics;

• Arriving on the day of the assessment with all necessary equipment;

• Setting up, administering, and breaking down the assessment following standard NAEP procedures; and

• Transferring student data to NAEP servers and checking data.

The Field Trial will be conducted in schools by NAEP administration field staff, replicating the actual testing conditions to fullest extent possible. It will cover the cognitive blocks and questionnaires of all NAEP 2018 digitally-based subjects delivered on tablets¹ among a sample of fourth, eighth, and twelfth-grade students. Teacher and school survey questionnaires will not be included as part of this study.

Results from this study will not be publicly released, but they will be used to identify issues and generate solutions or workarounds in advance of the main NAEP 2018 administration.

3. Recruitment and Sample Characteristics

States will be asked to participate in the study on a voluntary basis. NAEP State Coordinators in volunteer states will recruit schools in their state that are not part of the main NAEP 2018 sample (see Appendix A for a sample letter from a State Coordinator to a school principal). State Coordinators (see section 5) will leverage relationships within the state to contact schools and identify those willing to participate in the study. The NAEP State Coordinator will forward the contact information for participating schools to Westat, the NAEP data collection contractor (see section 5). Westat field administration staff will contact each school to make arrangements for students from the school to participate (see Appendix C for a sample school contact script).

A total of 10 schools will participate in the study. While the study will seek participation from schools with various demographic characteristics, including a mix of urban/suburban/rural locations and a mix of students’ race/ethnicity and socioeconomic backgrounds, detailed sampling requirements will not be targeted because of the study’s small sample size for school. Additionally, schools that are within a relatively close proximity in a state will be selected to ensure that one field administration staff member is able to conduct the study for all schools in the state. Private schools will not be recruited for this study.

Students will be selected to participate in the Field Trial either by random sampling or by selection of a classroom within the school, whichever is the school’s preference. If students are randomly selected, all students have an equal opportunity to be sampled including students with disabilities and English language learners. If intact classrooms are selected, students with disabilities and English language learners that are part of the regular classroom will be included in the study. The school coordinator will complete a worksheet for students with disabilities and English language learners (see Appendix D) using the guidelines provided in the inclusion policies (see Appendices E-J). Student with disabilities and English language learners will receive the NAEP allowed accommodations that are listed in the NAEP Inclusion Policies. Student lists will be collected from the school by field staff on the day of the assessment (see Appendix K).

4. Study design and data collection

Prior to the study, Westat field administration staff will contact cooperating schools to confirm student sampling and make logistical arrangements (see Appendix C for the contact script). Westat field administration staff who are familiar with digitally-based administration will conduct the study. They will bring all necessary materials, including the tablets, earbuds, and/or headphones with microphones to the schools on assessment days. The study will be conducted in 10 schools and will cover subjects that are being administered as part of the main NAEP 2018 administration. Table 1 provides a summary of the schools, grades, and subjects being assessed.

Students in all subjects will be provided a tutorial on the eNAEP test delivery system. For all subjects except Oral Reading Fluency (ORF), students will complete two 30-minute cognitive blocks and a 15-minute survey questionnaire. For the ORF study, students will complete two 30-minute cognitive blocks of NAEP reading, one 15-minute ORF module, and a 15-minute survey questionnaire. For all subjects except ORF, the study will require approximately 90 minutes (15 minutes for getting students situated and logged on to the NAEP tablet and 75 minutes of assessment time). For students taking ORF, the study will require approximately 105 minutes (15 minutes for getting students situated and logged on to the NAEP tablets and 90 minutes of assessment time)².

NCES and its contractors will evaluate feedback received during the study from the field staff and HelpDesk calls to determine whether changes should be made to the procedures and/or directions given to the field staff during NAEP 2018. In addition, ETS will evaluate student response data by conducting frequency distribution analysis on student responses to ensure proper and reasonable data capture, and will adjust the data uploading process, if necessary.

Table 1. Schools, Grades, and Subjects Included in Data Collection^*

Grade/Subject	Number of Schools	Number of Students	Session Description
Grade 4 Pilot (Science Pilot and Reading SBT Study)	2	100	Each school will have 2 sessions of 25 students.
Grade 8 Pilot (Science Pilot and Reading SBT Study)	2	100
Grade 12 Pilot (Science Pilot, Reading SBT Study, Math Pilot, Reading Pilot)	2	100
Grade 4 Pilot and ORF Study	2	74	Each school will have 2 sessions, 1 with 25 students (pilot assessments) and 1 with 12 students (ORF study).
Grade 8 Operational (Social Sciences)	2	100	Each school will have 2 sessions of 25 students.
Total	10	474

*The distribution across grades and subjects is tentative and may be adjusted, but the number of schools will be limited to 10.

5. Consultations outside the agency

Each NAEP State Coordinator will serve as the liaison between his/her state education agency and NAEP, will work with schools to identify participating schools for the Field Trial, and will coordinate NAEP activities in the state. Westat is the Sampling and Data Collection contractor for NAEP. Westat will administer the NAEP 2018 Field Trial. Fulcrum IT Services, LLC is the NAEP contractor for the development and ongoing support of NAEP digitally-based assessments. ETS serves as the Item Development contractor as well as the Design, Analysis, and Reporting contractor. ETS staff will be involved in item development, scoring, and analysis activities for this study.

6. Justification for Sensitive Questions

Throughout the item development process, efforts have been made to avoid asking for information that might be considered sensitive or offensive.

7. Paying Respondents

Schools will receive a $200 gift card to an office supply store (e.g., Staples or Office Depot) to encourage their participation and to thank them for their time and effort. The study will take place during regular school hours, and thus there will not be any monetary incentive for the student participants. However, students will be permitted to keep the pencils and earbuds used during the study. Note that the ORF study uses more sophisticated headphones which students will not be permitted to keep, but they will still be given the type of earbuds used in other subjects.

8. Assurance of Confidentiality

The study will not retain any personally identifiable information. Prior to the start of the study, students will be notified that their participation is voluntary. As part of the study, students will be notified that all of the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Written notification will be sent to parents/legal guardians of students before the study is conducted (see Appendix B). Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form.

9. Estimate of Hourly Burden

The school principal burden is estimated at 20 minutes for initial contact communications. The school coordinator burden is estimated at two hours including scheduling the assessment, identifying the sample of participating students, completing the SD and ELL worksheet, completing the Field Trial Administration form, sending a parental notification letter, and coordinating assessment logistics.

Parents/legal guardians of participating students will receive a letter explaining the study, for which the parent’s burden is estimated at three minutes. An additional burden of 15 minutes is estimated for a small portion of parents (up to 25) who may write to refuse approval for their child or may research information related to the study.

Approximately 474 students from 10 schools will participate in the study. Student burden is calculated based on 15 minutes for setup and reviewing the tutorial and 15 minutes to respond to survey questionnaire (30 minutes in total) from a total study session time of 90 minutes. ³ An additional 15 minutes of burden is included for the 24 students who will take the ORF module.

Estimated hourly burden for the participants is provided in Table 2.

Table 2. Estimate of Hourly Burden

Person	Task	Number of Respondents*	Number of Responses	Hours per Respondent	Total Burden (in hours)
School principal	Initial Contact	10	10	0.33	4
School coordinator	Scheduling and Logistics	10	10	2.00	20
Parents/Legal guardians	Initial Notification	474	474	0.05	24
Parents/Legal guardians*	Refusals or Additional Research	25	25	0.25	6
Students	NAEP 2018 Field Trial (all subjects)	474	474	0.50	237
Students**	Assessment	474	474	1.00	474
Students**	NAEP 2018 Field Trial (ORF)	24	24	0.25	6
Total		968	1,017	-	297

* These parents are a subset of those who were initially notified.

**These are a subset of the total number of students.

Note: Burden estimated for the field trial assessments, which are not subject to PRA, are demarcated in gray font, presented here for information only purposes, and not included in burden totals

10. Cost to the Federal Government

The total estimated cost for this study, including data collection activities, development and support of the tested NAEP digital platform, and data analysis is $25,000.

11. Project Schedule

Table 3 provides the overall schedule.

Table 3: Schedule

Date	Event
September-November 2017	Recruitment
December 2017	Data Collection
December 2017-January 2018	Evaluate Data

1 Technology and Engineering Literacy (TEL) will be part of the NAEP 2018 main administration but will be delivered on laptops rather than tablets and thus is not included in the NAEP 2018 Field Trail study.

2 Communications to schools and parents for all subjects except ORF indicate 120 minutes to allow for transition time to and from the study classroom. Communications to schools and parents for ORF indicate 120-135 minutes to allow for transition time to and from the study classroom.

3 Similar to main NAEP assessments, the cognitive item portions of the study are not included in the burden totals because they are not subject to the Paperwork Reduction Act (PRA).

6