Justification

National Center for Education Statistics

National Assessment of Educational Progress

Volume I

Supporting Statement

The National Assessment of Educational Progress (NAEP)

Reading Scenario-Based Task (SBT) Tryouts

OMB# 1850-0803 v.222

January 2018

Table of Contents

1. Submittal-Related Information 3

2. Background And Study Rationale 3

3. Sampling And Recruitment Plans 3

4. Data Collection Process 5

5. Consultations Outside The Agency 5

6. Assurance Of Confidentiality 5

7. Justification For Sensitive Questions 5

8. Paying Respondents 5

9. Estimate Of Hourly Burden 6

10. Cost To Federal Government 6

11. Project Schedule 6

1. Submittal-Related Information

This material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803), which provides for NCES to conduct various procedures (e.g., focus groups, cognitive interviews, usability tests, experiments, etc.) to develop and test study materials and methodologies so as to improve future data quality, utility, and study participant experience.

2. Background and Study Rationale

The National Assessment of Educational Progress (NAEP) is a federally authorized survey (by the National Assessment of Educational Progress Authorization Act; 20 U.S.C. §9622) of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is conducted by NCES, part of the Institute of Education Sciences, within the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in different subject areas and to collect questionnaire (i.e., non-cognitive) data from students, teachers, and school administrators¹ to provide context for the reporting and interpretation of assessment results.

This request is to pretest, as part of the NAEP assessment development process, scenario-based tasks (SBTs) for the grade 4 reading assessment. This pretesting will include large-scale tryouts intended to identify and eliminate potential issues with new digitally-enhanced NAEP items, tasks, and stimuli. Subsequently, the results should minimize challenges in item scoring and analysis and lead to better item performance. These large-scale tryouts will have a particular focus on collecting timing data for the grade 4 reading SBTs.

The SBTs included in the tryouts were previously piloted as part of the 2017 main NAEP administration (as approved in OMB #1850-0928 v.3). The grade 4 reading SBTs were found to be speeded, meaning that students could not complete all required items in the allotted time. Revisions have been made to shorten the SBTs included in the tryout so that students should be reasonably able to complete the tasks within the time provided. The goal of the tryouts is to test these revised SBTs under standard NAEP administration conditions to collect data on timing and student performance.

In tryouts, students will work uninterrupted through selected sets of draft items/tasks. Tryouts provide a snapshot of the range of responses and actions that items elicit, which can be gathered much earlier in the assessment development process and with fewer resource implications than pilot testing. Tryouts allow for pretesting of a wide range of content and the collection of robust data on ranges of student responses, item difficulty, assessment timing, and item functionality. The larger samples and timed testing conditions of tryouts are especially useful for gathering quantitative data about items, investigating the possible effects of different item features on student performance, and learning how long it takes students to complete items. Tryout samples used to date in NAEP have ranged from smaller (50 students per item/task) to considerably larger (several hundred or more students per item/task), depending on the nature of the items/tasks. This will be a large- scale tryout with a particular focus on collecting timing data for the SBTs.

3. Sampling and Recruitment Plans

EurekaFacts will recruit participants for this study in several locations throughout the United States to target a diverse sample of students. Five metropolitan areas will be selected and specific locations representing different levels of urbanicity (urban, rural, and suburban) within and surrounding these metropolitan areas will be identified, along with testing sites in rural areas and small towns in close vicinity to the metropolitan areas.

For tryouts planned in each metropolitan area, setup of a recruitment plan and campaign will be customized to each site based on the location, quota, target demographics, and the makeup of the population specific to the region. Following location-specific recruitment setup, direct outreach and recruitment efforts will be implemented to recruit, schedule, and retain participants through an established and effective standardized process of email, call, letter confirmations, and reminder calls/texts. The recruitment process is an ongoing and continuous effort involving direct and personal communication with participants, established to create integrity, rapport, and an informed, positive experience for parents and students.

While EurekaFacts will use various outreach methods to recruit students to participate, the bulk of the recruitment will be administered by telephone. Various resources will be employed to recruit participants. For parents/legal guardians of 4^th grade students these will include:

• existing participant databases;

• targeted telephone and mail contact lists (of individuals meeting basic criteria such as age or school grade);

• school system research/assessment directors;

• NAEP State Coordinators, when possible, to recruit in schools;

• community organizations (e.g., boys/girls clubs, parent-teacher associations, and limited on-site location-based and mass media recruiting); and

• other outreach/contact methods and resources (e.g., internet ads, social media, flyers/bookmarks, canvassing, and having representatives available to talk to parents, educators, and community members at appropriate local community events, school fairs, etc.).

The following is a general overview of the recruitment process for these tryouts:

1. Place advertisement/study information in highly selective/targeted social media venues, including Facebook and Google ads restricted to specific locations where the testing will be conducted (Appendices V and W).

2. Send an email of introduction about the tryouts study to: (a) various elementary school principals (to recruit students); (b) individuals in EurekaFacts’ existing databases; (c) community centers/organizations and research/assessment directors; (d) targeted telephone and mail/email contact lists; and (e) parents/legal guardians. The email of introduction will include an informational brochure (Appendix Q).

3. Discuss additional recruitment materials, such as flyers and informational bookmarks (Appendices P and R), with contacts such as community organizations with which EurekaFacts has already an established relationship and with community organizations that contact EurekaFacts upon receiving the email of introduction.

4. A EurekaFacts staff member will follow up with any interested parent/legal guardian of a 4^th grade student via phone and ask them to provide demographic information to ensure that a diverse sample is selected (Appendices M and N).

5. If the parent/legal guardian allows his or her student to participate, and the student has expressed interest and has agreed to participate, EurekaFacts will follow up to confirm participation and the date and time of the session (Appendices H and I).

6. Parents/legal guardians will be required to sign consent forms prior to the tryout session (Appendix S) and to sign a receipt after their child’s participation to acknowledge receiving the incentive (Appendix T and U).

To minimize travel burden, tryouts will be conducted in venues convenient for the participants, such as facilities of community-based organizations, school buildings (after school only), and conference centers. The target is to conduct the tryouts with a sample of 500 grade 4 students (see Table 1 for target student sample characteristics).

Table 1. Targeted Student Sample Characteristics

	Subgroup	Targeted Sample Percentages
Gender	Male	51
	Female	49
Race/Ethnicity	White	48
	Black	15
	Hispanic	26
	Asian/Pacific Islander	6
	American Indian/Alaska Native	1
	Other	4
Type of Location	City	31
	Suburban	40
	Town	11
	Rural	18
IEP status	Yes	13
	No	87
LEP status	Yes	11
	No	89

\

4. Data Collection Process

The tryouts are intended to simulate standard NAEP administration conditions. As such, Westat field administration staff who are familiar with digitally-based NAEP administration will administer the tryouts. Additionally, EurekaFacts staff will be on-site to oversee administrative tasks such as registration, ensuring consent has been obtained from parents/legal guardians, and incentive disbursement.

Students will complete the tryouts under standard NAEP administration procedures. They will be provided a tutorial on the eNAEP test delivery system, then complete two 30-minute SBTs and a 10-minute survey questionnaire (see Volume 2). A tryout session will take approximately 85 minutes (15 minutes for getting students situated and logged on to the NAEP tablet and 70 minutes of assessment time).

The data collected from the tryouts will be transferred to ETS to complete analysis of machine scored items and process timing data. Pearson will complete the scoring of constructed responses. Once all scoring is complete, ETS will conduct analyses, including standard item analysis and timing data, and prepare a summary of results.

5. Consultations Outside the Agency

ETS serves as the NAEP Item Development, Data Analysis, and Reporting contractor for NAEP cognitive and questionnaire items. EurekaFacts, a subcontractor to ETS, will conduct tryout recruitment activities and provide field support for administration. Westat is the NAEP data collection contractor and will administer the tryout sessions. Pearson is the NAEP scoring contractor and will complete scoring of constructed response items. Fulcrum IT Services, LLC, is the contractor for the development and ongoing support of NAEP digitally-based assessments.

6. Assurance of Confidentiality

The study will not retain any personally identifiable information. Prior to the start of the study, participants will be notified that their participation is voluntary. As part of the study, participants will be notified that all of the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Written consent will be obtained from parents or legal guardians of each participating student. Participants will be assigned a unique identifier (ID) at the time of recruitment, which will be created solely for data file management and used to keep all participant materials together. The participant ID will be separated from the participant’s name before the report from each stage of the study is finalized. The signed consent form, which includes the participant name, will be separated from the participant files. All consent forms will be secured for the duration of the study and will be destroyed after the final report is completed.

7. Justification for Sensitive Questions

Throughout the item/task development processes, effort has been made to avoid asking for information that might be considered sensitive or offensive.

8. Paying Respondents

To encourage participation, a $25 gift card from a major credit card company will be offered to each student who participates in a tryout session as a thank you for his/her time and effort. Each student’s parent or legal guardian will also be offered a $25 gift card from a major credit card company to thank them for bringing their participating student to and from the study site.

9. Estimate of Hourly burden

The estimated burden for recruitment assumes attrition throughout the process. Tryouts are expected to take 85 minutes in all cases.

Table 2. Estimate of Hourly Burden for Tryouts

Respondent	Number of respondents	Number of responses	Hours per respondent	Total hours
Student Recruitment via Teachers and Staff
Initial contact with staff: email, flyer distribution, and planning	67	67	0.33	23
Parent or Legal Guardian for Student Recruitment
Initial contact	1,334	1,334	0.05	67
Follow-up via phone	667*	667*	0.15	101
Consent and confirmation	500*	500*	0.15	75
Recruitment Totals	1,401	2,568		266
Student
Grade 4 reading SBT tryout	500	500	1.4	700
Tryout Totals	500	500	-	700
Total Burden	1,901	3,068	-	966

* Subset of initial contact group

Numbers have been rounded which may affect totals.

10. Cost to Federal Government

Table 3. Total Estimated of Costs of Pretesting

Activity	Provider	Estimated cost
Recruitment and administrative support (including incentive costs)	EurekaFacts	$280,000
Study design, assessment content and analysis, and report preparation	ETS	$70,000
Integrating tasks into and monitoring of the test delivery system	Fulcrum	$50,000
Scoring of constructed response items	Pearson	$300,000
Administration of tryouts	Westat	$40,000
	Total	$740,000

11. Project Schedule

Recruitment for tryouts will begin in January 2018, upon OMB approval. Data collection and analyses for this study are scheduled to end by May 2018.

1 Please note that in this submission “school administrator” refers to the principal or assistant/vice principal. In NAEP main study administrations, other individuals who are not the head principal are allowed to complete the school administrator questionnaire.

6