Justification

National Center for Education Statistics (NCES)

Volume I

Supporting Statement

High School and Beyond 2020 (HS&B:20) Base-Year Field Test Cognitive Testing Round 1

OMB# 1850-0803 v.240

September 2018

Attachments:

Attachment I – Recruitment Procedures and Contacting Materials

Attachment II – Eligibility Screening Questions

Attachment III – Consent to Participate in Research

Attachment IV – Interview Protocol

Attachment V – Selected Student Survey Questions with Probes

Attachment VI – Selected Parent Survey Questions with Probes

Attachment VII – Selected Teacher Survey Questions with Probes

Attachment VIII – Selected Administrator Survey Questions with Probes

Attachment IX – Selected Counselor Survey Questions with Probes

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803), which provides NCES the capability to improve data collection instruments by conducting testing, such as usability tests, focus groups, and cognitive interviews, to improve methodologies, survey and test questions, and/or delivery methods.

The High School and Beyond 2020 study (HS&B:20) will be the sixth in a series of longitudinal studies at the high school level conducted by NCES. HS&B:20 will follow a nationally-representative sample of ninth grade students from the start of high school in the fall of 2020 to the spring of 2024 when most will be in twelfth grade. The study sample will be freshened in 2024 to create a nationally representative sample of twelfth-graders. A high school transcript collection and additional follow-up data collections beyond high school are also planned. The NCES secondary longitudinal studies examine issues such as students’ readiness for high school; the risk factors associated with dropping out of high school; high school completion; the transition into postsecondary education and access/choice of institution; the shift from school to work; and the pipeline into science, technology, engineering, and mathematics (STEM). They inform education policy by tracking long-term trends and elucidating relationships among student, family, and school characteristics and experiences. HS&B:20 will follow the Middle Grades Longitudinal Study of 2017/18 (MGLS:2017) which followed the Early Childhood Longitudinal Study, Kindergarten Cohort of 2011 (ECLS-K:2011), thereby allowing for the study of all transitions from elementary school through high school and into higher education and/or the workforce. HS&B:20 will include surveys of students, parents, students’ math teachers, counselors, and administrators. Students will also receive assessments in mathematics and reading, and be given a 2-minute vision test and a 10-minute hearing test. The request to conduct, beginning in January 2019, state, school district, school, and parent recruitment activities, including collection of student rosters and selection of the base-year field test sample in preparation for the HS&B:20 base-year field test data collection scheduled to begin in September 2019, is currently in a 60-day public comment period (OMB# 1850-new v.1).

The overarching purpose of HS&B:20 is to provide a rich variety of data to answer questions about how students’ backgrounds and school experiences affect education, employment, and other life outcomes. HS&B:20 represents a new and compelling evolution of NCES’s secondary longitudinal studies program. While maintaining the hallmarks of prior study designs—longitudinal data collection; cognitive assessments measuring student achievement growth; contextual information from parents, teachers, and administrators; and administrative records collection—HS&B:20 also incorporates exciting innovations and further improvements. Many survey and test items planned for the upcoming HS&B:20 base-year field test data collection have been previously tested or were included in prior NCES secondary longitudinal studies or other NCES studies.

This request is to conduct cognitive testing starting in October 2018 in preparation for the HS&B:20 base-year field test, allowing NCES to evaluate survey items that are either new or have been revised from items used in prior NCES longitudinal studies. Items to be evaluated include questions selected for each of the five HS&B:20 surveys: student, parent, administrator, teacher, and counselor. Examples of items to be tested include a section asking students and parents about the parent’s education and employment; select items asking administrators about public school choice; and a section asking counselors for information on students’ math course sequences and placement. The full array of items to be tested is presented in Attachments V through IX of this submission. Ultimately, HS&B:20 will be administered in multiple modes to include web, Computer-Assisted Telephone Interviewing (CATI), and Computer-Assisted Personal-Interviewing (CAPI). However, the current cognitive testing will exclusively use paper forms, as the instrument content is still being developed and the automated instruments have not yet been programmed. For this reason, the current testing will not include usability testing. The results will guide potential survey modifications, which will be used to refine the base-year field test items. These refined items will be submitted to OMB for review in early 2019 as part of the HS&B:20 base-year field test data collection request. RTI International has been contracted by NCES to collect HS&B:20 data; and EurekaFacts, LLC (hereafter referred to as EurekaFacts) serves as RTI’s subcontractor for aspects of HS&B:20 cognitive testing.

Background

The cognitive testing for which this clearance is requested will be used to refine the survey questions, maximize the quality of data collected, and provide information on issues with important implications for the survey design, such as the following:

• Identify whether respondents can provide accurate data;

• Evaluate the extent to which terms in questions are comprehended;

• Update and add terminology when necessary;

• Examine the thought processes used to arrive at answers to survey questions;

• Determine appropriate response categories to questions;

• Identify sources of burden and respondent stress; and

• Evaluate the appropriate group (e.g., students vs. parents) to ask about topics.

Design and Context

The purpose of this cognitive testing is to evaluate the wording, language, difficulty-level, and content of select survey items, so they are clear and can be easily understood for the HS&B:20 base-year field test instruments. Testing will be conducted using a subset of items proposed for inclusion in the final base-year field test instruments, which will be administerested to 9th-grade students, parents, math teachers, school administrators, and school counselors.

EurekaFacts staff have extensive experience in cognitive testing methodologies and will be responsible for recruiting participants, conducting the interviews, compiling interview audio recordings, and summarizing the findings. Interviewer probing during testing will encourage respondents to elaborate about their understanding of the HS&B:20 survey questions and how they formulate their answers. Participant responses to probes will be used to evaluate and revise question wording as needed. Interviewers will use a combination of generic probes and item specific probes to elicit information on the mental steps respondents took to arrive at an answer. Generic probes will be particularly useful when the participant exhibits signs of confusion or expresses difficulty with the question. Specific probes will be used to evaluate potential issues that are pre-determined (see Attachments V through IX for the cognitive test items and the associated generic and specific probes). Probing also ensures that respondents, especially teenagers, remain engaged in the process as they work on the items.

All student/parent pair participants will complete their session at the EurekaFacts office. School staff participants will have the option to complete their session in-person or remotely via telephone, which provides the needed flexibility for accommodating this population. Remote participation will require that respondents have access to view the survey materials on a computer or in print and have a strong and reliable phone service connection to speak with the interviewer. The EurekaFacts office can accommodate testing observers. For remote interviews, testing observers will be able to listen to the interview through a Skype meeting invitation, and they will be able to communicate with each other and the interviewer via a chat room. All interviews will be recorded, and the recordings will be available to NCES and RTI’s HS&B:20 staff for review. EurekaFacts will compile and report their observations once half of the interviews have been conducted and again at the end of testing.

The sample for testing will contain a diverse pool of 9th-grade students and their parents, math teachers, school administrators, and school counselors (as identified in the eligibility screener, see Attachment II for specific eligibility screener questions). Specifically, this sample will include:

• 9th-grade students and one of their parents/legal guardians,

• High school principals,

• High school counselors, and

• Math teachers currently teaching 9th-grade students.

Recruitment will continue until a total of 95 respondents have participated in testing. The student sample will include a mix of demographic characteristics, including race and ethnicity, gender, type of school attended, socio-economic status, urbanicity, grade level, and English-Language Learner (ELL) status. See Table 1 for the expected number of testing participants by respondent type. Student participants will be identified using EurekaFacts’ database of potential research respondents in the Greater Washington, DC - Baltimore, MD Metropolitan area as well as referrals, advertisements in student newspapers and online forums, social media postings, flyers, and through outreach to community-based organizations and after-school programs serving students and their parents.

The school staff sample will include math teachers, school administrators, and school counselors. Participants will be selected from different types of schools (e.g., public, charter, and private) across geographic regions. They will also be selected to represent a range of years working in the field of education. School staff participants will be identified using EurekaFacts’ database of potential research respondents across the United States, as well as referrals, advertisements in educational publications and online forums, social media postings, flyers, and through outreach to educational and professional organizations and unions serving educational professionals. Screening of all potential respondents will be conducted using an online survey or in a telephone interview comprised of eligibility questions specific to the needs of this study to ensure that potential participants qualify (see Attachment II).

Table 1: Screening and participant numbers by respondent type

Respondent type	Expected to be screened	Testing participants
9th-grade student/parent pairs	333	50
High school principals	225	15
High school counselors	225	15
9th-grade math teachers	225	15
Total	1,008	95

Attachments I through IV present EurekaFacts’ interview protocol and materials, including recruitment materials, eligibility screening questions, consent forms, and scripts. Attachments V through IX include all survey cognitive interview items with generic probes and specific probes planned for each.

Estimated Respondent Burden

To yield 95 completed interviews, we anticipate screening up to 1,008 individuals to ensure that we are achieving the desired distribution of respondent types. The screening process, on average, is estimated to take about 4 minutes per person (see Attachment II). Testing sessions will last between 40 and 60 minutes. Table 2 summarizes the anticipated burden, by length of interview.

Table 2: Estimated respondent burden

Activity	Number of respondents	Number of responses	Minutes per respondent	Maximum total burden hours
Screening	1,008	1,008	4	68
Cognitive interview
9th-grade students	25*	25	60	25
Parents of 9th-grade students	25*	25	40	17
Math teachers	15*	15	40	10
School administrators	15*	15	40	10
School counselors	15*	15	40	10
Study Total	1,008	1,103	N/A	140

* Subset of the screened group.

Estimate of Costs for Recruiting and Paying Respondents

In order to be able to recruit a representative range of respondents, and to thank them for their time and participation, we will offer prospective participants incentives for completing the cognitive interview. The incentive amount and interview duration vary based on respondent type, with incentives ranging from $25 to $50 and interview times lasting between 40 and 60 minutes. Table 3 details the different incentive amounts and time commitments.

Table 3: Incentive details by respondent type

Respondent Type	Number of respondents	Length of time (minutes)	Incentive Dollar Value	Total Incentive Cost
9th-grade students	25	60	$25	$625
9th-grade parents	25	40	$50	$1,250
Math teachers	15	40	$50	$750
School administrators	15	40	$50	$750
School counselors	15	40	$50	$750
Study Total	95	N/A	N/A	$4,125

Estimate of Cost Burden

Except for the potential cost for transportation to the testing facilities, there are no direct costs for respondents.

Cost to Federal Government

The cost to the federal government for conducting these cognitive interviews will be $341,500 under the EurekaFacts subcontract to RTI. This cost includes recruitment, conducting interviews, analyses, report writing, and participant incentives.

Assurance of Confidentiality

Cognitive testing respondents will be informed that their participation is voluntary and that:

EurekaFacts, LLC is carrying out this research for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

All respondents will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all materials for each respondent together. The respondent ID will not be linked to the respondent’s name. Respondents will be sent a consent form (Attachment III) via email, which they will need to sign, scan, and send back to EurekaFacts’ office in order to confirm their participation. The signed consent forms will be kept separately from the interview files for the duration of the study and will be destroyed three years after the final report is released.

Schedule for HS&B:20 Cognitive Testing

EurekaFacts will begin recruiting and screening potential participants upon receiving OMB clearance. Testing is scheduled to begin no later than October 15, 2018. Informed by the testing, a modified draft of the survey and student assessment instrument will be used in the HS&B:20 base-year field test with approximately 5,110 sampled students, scheduled to begin in September 2019.

Activity	Dates
Recruit cognitive testing participants	September – November 2018
Conduct cognitive testing	October – December 2018
Finalize revisions to item wording	December 2018 – February 2019