2nd Follow-up Field Test of High School Longitudinal Study of 2009 (HSLS:09) Cognitive Interviews

National Center for Education Statistics

Second Follow-up Field Test of the High School Longitudinal Study of 2009 (HSLS:09) Cognitive Interviews

OMB# 1850-0803 v.118

Volume I

November 10, 2014

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803 v.118) that allows NCES to improve the methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, and cognitive interviews. The request for approval described in this memorandum is to conduct cognitive interviews in preparation for the Second Follow-Up Field Test of the High School Longitudinal Study of 2009 (HSLS:09). Procedures and burden for recruiting were approved on 10/29/2014 (OMB# 1850-0803 v.114). The data collection for this study is being carried out for NCES by RTI International – with Research Support Services (RSS) as a subcontractor – under contract to the U.S. Department of Education (Contract number ED-IES-14-R-0335).

The HSLS:09 Second Follow-Up will focus on respondents who have been out of high school for 2 and 3 years (including both high school graduates and students who dropped out). The survey will help to inform researchers, policymakers, and other stakeholders about issues of transition from high school to postsecondary education and the labor force, with a particular emphasis on sample members who are pursuing postsecondary education and careers in science, technology, engineering, and mathematics (STEM) fields. Cognitive interviews will help us to better understand respondent perspectives on a variety of issues and will be used to refine the survey questions to maximize the quality of data collected as part of the field test.

Design and Context

This request is to conduct cognitive interviews planned to begin in November 2014. The cognitive research report based on the results will be available by March 2015. Cognitive testing protocols and probes are provided in Volume II.

RSS (located in Evanston, Illinois) is recruiting in the greater Chicago area for 30 cognitive interviews. The sample will include a mix of respondents from the following five categories – those who: are enrolled in a 4-year baccalaureate program; have completed or enrolled in a 2-year post high school program (associate’s degree); are employed; are not enrolled in any post high school program; and who left high school without a diploma. Respondents are also being selected to ensure a mix of racial and ethnic groups.

NCES is considering the inclusion of questionnaire items related to Lesbian, Gay, Bisexual, Transgender, or Questioning (LGBTQ) topics¹ in the HSLS:09 Second Follow-Up, and RSS is attempting to recruit approximately ten participants who identify as LGBTQ. RSS has experience with recruiting sexual minorities, and their inclusion in the cognitive interviews will help to inform the potential use of these items in the field test and full-scale study. Table 1, below, provides a distribution of the desired minimum number of respondents by type, although a respondent may fit more than one category.

Table 1: Desired minimum number of respondents, by respondent type

Respondent type	Desired minimum number of respondents
Never enrolled in postsecondary education	5
Currently enrolled in postsecondary education Enrolled in a 2-year program Enrolled in a 4-year program	5 5
Working while enrolled	5
Working but not enrolled	5
Left high school without diploma or has GED	5
Identifies as LGBTQ	10

Trained RSS staff will conduct the cognitive interviews in rented conference rooms, with the goal of completing an average of five cases per day. The cognitive testing will involve intensive one-on-one interviews. Each interview will last a maximum of 60 minutes, and all participants will be offered an incentive of $40 for their participation. Audio recording of each interview will be available to NCES for review.

The organizing objective of the cognitive testing approach will be to identify the processes by which respondents answer draft survey questions and to pinpoint potential sources of question misinterpretation in their responses. For example, respondents will be asked to “think aloud” as they answer questions. Concurrent and retrospective protocols can provide a valuable source of evidence about the organization of information in memory, comprehension of the questions, strategies used in retrieving information, judgment processes that come into play, and other processes affecting the final answers to survey items. To elicit relevant responses, respondents may be asked to point out unfamiliar terms, to paraphrase the question or its accompanying instructions, to define a term, and to make judgments regarding the confidence they place in their answers. Typical probes – e.g., “How certain are you of your answer?” or “How easy or difficult was it to answer this question?” – seek to verify respondent interpretations, investigate the meaning of specific potentially ambiguous phrases, or elicit notions that the respondent thought were critically relevant to but absent from the question.

Immediately following the conclusion of each interview, methodologists will review the interview recordings and notes, and highlight potential themes that may have arisen. Following each interview, the digital audio recording will be archived for qualitative analysis. RSS will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and that their responses may be used only to help inform the survey design and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573). Participants will be assigned a unique student identifier (ID) that will be created solely for file management and used to keep all interview materials together. Participant names will not be connected to interview responses except by participant ID. The crosswalk that enables comparison of names to participant IDs is necessary for consent form management, and will be kept only on paper, not electronically. It will be stored in a locked cabinet in a secure room and will be destroyed once the final report is completed. The informed consent forms signed by participants will be kept separate from the interview files and notes in a locked cabinet in a secure room for the duration of the study, and will also be destroyed after the final report is completed.

Schedule for HSLS:09 Second Follow-Up Cognitive Testing Activities

RSS is in the process of recruiting for the cognitive testing. Interviews are scheduled to begin in November 2014. This timeline allows the project team to use the results from the qualitative research to improve the survey questions administered in the spring 2015 field test, and seek their clearance beforehand.

Estimate of Respondent Burden

To yield 30 completed interviews, we estimated needing to interview 120 individuals to screen them for eligibility and to ensure that we are achieving the target goals in terms of respondent types. The screening process, on average, is estimated to take about 5 minutes per person.

Table 2: Estimate respondent burden

Activity	Number of respondents	Number of responses	Hours per respondent	Maximum total burden hours
Screening	120	120	0.083	10
Cognitive interview	30	30	1.0	30
Study Total*	120	150		40

* The total for this study is shown for information purposes only. Respondent burden for recruitment and screening were approved in October 2014 (OMB# 1850-0803 v.114). This request is for the cognitive interview portion only.

Estimate of Costs for Recruiting and Paying Respondents

In order to encourage participation by recent high school students in this study, and to thank them for their time and effort spent traveling to the interview facility and partaking in the interview, each respondent will be offered $40 for their participation.

Estimate of Cost Burden

Participants in the cognitive interviews must bear direct costs associated with travel by car or use of Chicago Transit Authority services to RSS’s rented conference rooms in Chicago.

Cost to Federal Government

The cost to federal governent for conducting these cognitive interviews will be $58,680, under the RSS subcontract to RTI, including recruitment, interviewing, analysis, report writing, and participant incentives.

1 The wording of these items were suggested by a panel of experts on LBGTQ research, including members of GLSEN, The Fenway Institute, and the Gender Identity in US Surveillance (GeniUSS) group. The items came from a variety of sources, including the CDC's Youth Risk Behavior Survey (YRBS), the MA Department of Public Health's Youth Health Survey, and the NICHD's National Longitudinal Survey of Adolescent Health, among others. HSLS has a particular need to test these items given the necessity to select only a very few well-performing items due to limited space on the questionnaire.

2