OMB83 Change Memo HSLS 2009 2nd Followup MS Baseline Incentive

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: April 29, 2016

TO: Robert Sivinski, OMB

THROUGH: Kashka Kubzdela, OMB Liaison, NCES

FROM: Elise Christopher, HSLS:09 Project Officer, NCES

SUBJECT: High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study Baseline Incentive Change Request (OMB# 1850-0852 v.19) - Calibration Sample Results: Phases 1-2

The High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study and 2018 Panel Maintenance request was approved by OMB in December 2015 with an update approved in March 2016 (OMB# 1850-0852 v.17 and v.18 respectively). This memorandum provides data collection results from the first two calibration samples recruitment phases, and requests approval for recommended baseline incentive plans for the main sample.

For reference, an excerpt from Part B of the second follow-up main study submission that describes main study responsive design plans is presented in Attachment 1 below. In the HSLS:09 second follow-up main study, there are three subgroups of special interest.

1. Subgroup A¹ (high school late/alternative/non-completers) is the subset of sample members who, as of the 2013 Update, had not completed high school, were still enrolled in high school, received an alternative credential, completed high school late, or experienced a dropout episode with unknown completion status.

2. Subgroup B (ultra-cooperative respondents) includes sample members who participated in the base year, first follow-up, and 2013 Update without an incentive offer. These cases were also early web respondents in the 2013 Update and, by definition, are high school completers.

3. Subgroup C (high school completers and unknown high school completion status) includes cases that, as of the 2013 Update, were known to be on-time or early regular diploma completers (and not identified as ultra-cooperative) and cases with unknown high school completion status who were not previously identified as ever having a dropout episode.

To determine optimal incentive amounts, a calibration subsample has been selected from each of the aforementioned subgroups to begin data collection ahead of the main sample. The experimental subsamples are fielded about six weeks prior to the main sample to allow time to analyze the results and consult with OMB to determine the baseline incentive amounts to be implemented for each subgroup in the main sample.

Calibration sample results from phases 1 and 2. As described in Attachment 1 below, results from the calibration samples were presented to OMB prior to main study implementation. Attachment 2 provides the detailed results of the calibration experiments and evaluations. To summarize, key findings are presented below.

Summary of calibration sample data collection results as of 4/28/2016:

• Subgroup A (high school late/alternative/non-completers) – Because of the analytical importance of this subgroup and the relative difficulty in gaining their cooperation, incentive amounts are higher than for the other groups. Subgroup A was randomly assigned to receive a baseline incentive in one of three amounts: $30, $40, or $50. Among cases in Subgroup A, response rates were highest among cases assigned a baseline incentive of $40. The $40 response rate is more than six percentage points higher than the $30 rate.

• Subgroup B (ultra-cooperative respondents) – Subgroup B cases were not offered a baseline incentive, yet this group of “ultra-cooperative” sample members had a response rate of 64 percent. The Subgroup B calibration sample responded, with no incentive offer, at a rate similar to that seen among BPS:12/14 sample members with high predicted response propensity and with a $40 incentive.

• Subgroup C (high school completers and unknown high school completion status) – Within Subgroup C, the response rate is highest among cases assigned the $30 incentive at 43 percent.

Additional evaluations to broaden the set of cases defined as “ultra-cooperative.” The original set of “ultra-cooperative” cases included 1,027 high school completers who participated in the base year and the first follow-up, and completed the 2013 Update in early web period, with no incentive. As noted above, the response rate for the ultra-cooperative calibration sample at the end of phase 2 was very high at 64 percent. To investigate the possibility that this classification can successfully be broadened to include more cases, and therefore obtain a larger yield of complete interviews without offering incentives, the definition was revised to require that the following conditions be met:

• Must be in Subgroup C, and

• Must have predicted response propensity > 0.90, and

• Must have been a 2013 respondent, and

  • the respondent must have been the student not the parent, and

  • must have responded in the first twelve weeks, and

  • must not have received $5 prepay, and

• Must have been an early/on-time high school completer, and

• Must not have ever dropped out of high school.

According to these criteria, 3,481 cases have been identified in the main sample.

Recommended baseline incentives for main sample

• Subgroup A (high school late/alternative/non-completers) – Based on the experimental comparison of response rates by incentive levels, we recommend that $40 be offered to this subgroup.

• Subgroup B (ultra-cooperative respondents) – Given the high response rate among this subgroup based upon the more restrictive definition, we recommend expanding the set of cases to be treated as “ultra-cooperative” to include the 3,481 cases; this subgroup will not be offered a baseline incentive.

• Subgroup C (high school completers and unknown high school completion status) – Based on the experimental comparison of response rates by incentive levels, we recommend that $30 be offered to this subgroup.

NCES thanks OMB for considering these changes. Data collection for the main sample is scheduled to begin on May 9, 2016.

Attachment 1 – Description of main study responsive design plans (excerpt from Supporting Statement Part B approved on 12/15/2015)

B.4.d Main Study Plans

NCES and RTI are working closely together to design a data collection approach that makes use of evaluations from prior interventions that were used to improve sample representativeness by ensuring that the responding sample is as similar as possible to the total sample. In previous rounds of HSLS:09 and in other NCES studies (such as BPS:12/14, B&B:08/12, and ELS:2002 third follow-up), responsive designs have been used to improve sample representativeness in key survey variables. The proposed main study data collection plan has been designed to maximize data quality through a responsive design approach in which variance between the responding sample and the overall sample is estimated at several points during data collection. An advantage of the proposed responsive design is that it allows us to determine, during data collection, how representative the responding sample is of the total sample, so that we can focus efforts and resources on bringing in the cases that are most needed to achieve balance in the responding sample.

Plans for the HSLS:09 second follow-up main study are based upon 1) results of incentive experiments and responsive design modeling simulations from the HSLS:09 second follow-up field test, 2) results from related longitudinal studies, and 3) prior experience with the HSLS:09 cohort. This section describes plans for responsive design in the main study data collection. In particular, there are three subgroups of interest that will be handled differently. This section describes the phases of data collection and how and when interventions will be implemented and evaluated. Finally, we discuss the development of the response likelihood and bias likelihood models that will be used to identify cases for targeted treatments.

Sample subgroup classification. In the HSLS:09 second follow-up main study, there will be three subgroups of special interest.

1. Subgroup 1 (high school late/alternative/non-completers) will be the subset of sample members who, as of the 2013 Update, had not completed high school, were still enrolled in high school, received an alternative credential, completed high school late, or experienced a dropout episode with unknown completion status.

2. Subgroup 2 (ultra-cooperative respondents) includes sample members who participated in the base year, first follow-up, and 2013 Update without an incentive offer. These cases were also early web respondents in 2013 Update and, by definition, are high school completers.

3. Subgroup 3 (high school completers and unknown high school completion status) will include cases that, as of the 2013 Update, were known on-time or early regular diploma completers (and not identified as ultra-cooperative) and cases with unknown high school completion status who were not previously identified as ever having a dropout episode.

Calibration subsamples. To determine optimal incentive amounts, a calibration subsample will be selected from each of the aforementioned subgroups to begin data collection ahead of the main sample. A similar approach was used successfully in BPS:12/14, where approximately 10 percent of that sample (3,700 cases) was selected and fielded seven weeks prior to the rest of the BPS:12/14 sample. The experimental subsample was treated in advance of the remaining cases, and after analyzing the results for the experimental sample and consultation with OMB, the successful treatment was implemented with the remaining sample. In the HSLS:09 second follow-up main study, a similar approach is proposed with the HSLS:09 calibration subsamples fielded six weeks prior to the rest of the HSLS:09 sample. Exhibit B-7 shows the estimated size of each subgroup, the percentage of cases to be selected for the calibration subsample, and the estimated number of cases in the calibration sample.

Exhibit B-7. Calibration Sample Sizes, by Subgroup

Subgroup Number	Subgroup Description	Main Sample	Calibration Sample	Calibration Percent
1	High School Late/Alternative/ Non-Completers Non-completers, late completers, still enrolled, and alternative credential as of the 2013 Update as well as ever dropouts with no completion status,	2,545	509	20%
2	Ultra-Cooperative Respondents High school completers who participated in base year and the first follow-up, and completed the 2013 Update in early web period, with no incentive	1,027	154	15%
3	All Other High School Completers and Unknown Cases HS Diploma completed early/on-time unknown or unknown completion status with no known dropout episode	19,747	1,975	10%

Data collection phases, treatments, and evaluations. For the second follow-up main study, the data collection plan includes a phased responsive design strategy specifically aimed at improving sample representativeness in the final survey participants. Exhibit B-8 presents the schedule for the planned phases of data collection for both the calibration samples and the main samples. Exhibit B-9 summarizes the baseline and boost incentives to be tested for each subgroup. The phases will proceed as follows:

Baseline incentive (phase 1). During this beginning phase of data collection, the survey will be open exclusively for self-administered interviews via the web. Web response will remain open throughout the entire data collection. As described above, the calibration samples will allow for testing of incentive amounts on a subset of cases, and the results will inform the implementation plan for the main samples. Prior to the start of the main sample data collection for phase 1, calibration sample response rates will be evaluated. An ANOVA-based model will be used to perform pairwise contrasts between the different incentive amounts offered to the treatment and control groups in each phase. NCES and OMB will meet to review the results of the calibration experiment and determine the optimal incentive amount for each of the subgroups.

• Subgroup 1 (high school late/alternative/non-completers) will be offered 3 different baseline incentive amounts ($30, $40, or $50). The optimal amount (to be determined in consultation with OMB) will be offered to all cases in the subgroup 1 main sample.

• Subgroup 2 (ultra-cooperative respondents) will not be offered a baseline incentive. The subgroup 2 calibration sample response rate will be evaluated against early response rates for other cohorts (such as BPS:12/14 and ELS:2002 third follow-up) to estimate a “successful” response benchmark for HSLS:09. If it is determined that the subgroup 2 calibration sample response rate is not successful, we will discuss with OMB the possibility of offering a baseline incentive (amount to be determined in consultation with OMB) to the subgroup 2 main sample.

• Subgroup 3 (high school completers and unknowns) will be offered 6 different incentive amounts, ranging from $15 to $40 ($15, $20, $25, $30, $35, or $40). The $15 starting point for this baseline incentive calibration experiment is based on the results of the HSLS:09 second follow-up field test experiment. The optimal amount (to be determined in consultation with OMB) will be offered to all cases in the subgroup 3 main sample.

Exhibit B-8. Data Collection Schedule and Phases.

Outbound CATI prompting (phase 2). After phase 1 data collection which is self-administered via the web (except for instances when sample members call in to the help desk), phase 2 will initiate another mode of data collection. Telephone interviewers will begin making outbound calls to prompt for self-administration or to conduct telephone interviews. No additional incentives will be offered during phase 2.

• Subgroup 1 will begin outbound CATI earlier than the other subgroups, to allow additional time for telephone interviewers to work these high priority cases.

Incentive boosts (phases 3 and 4). Phases 3 and 4 introduce the use of responsive design with the bias likelihood model. Targeted cases will be offered an incentive boost in addition to the baseline incentive offer. The calibration samples will allow for testing of incentive boost amounts on a subset of the remaining nonrespondents in phases 3 and 4, and the results will inform the incentive boost implementation plan for the main samples. Prior to the start of the main sample data collection for phases 3 and 4, calibration sample response rates will be evaluated. An ANOVA-based model will be used to perform pairwise contrasts between the different incentive boost amounts offered to the treatment and control groups in each phase. NCES and OMB will meet to review the results of the calibration experiment and determine the optimal incentive boost amount for each of the subgroups.

• Subgroup 1 (high school late/alternative/non-completers) will be offered an incentive boost of either $15 or $25, on top of the baseline incentive they were offered in phase 1. The optimal amount (to be determined in consultation with OMB) based on the calibration sample results, will be offered to all remaining nonrespondents in subgroup 1.

• The subset of subgroup 2 (ultra-cooperative respondents) cases that are targeted for intervention, based on bias likelihood modeling, will be offered an incentive boost of either $10 or $20, and the optimal amount (to be determined in consultation with OMB) will be offered only to targeted cases among the remaining subgroup 2 nonrespondents.

• The subset of subgroup 3 (high school completers and unknowns) cases that are targeted for intervention, based on bias likelihood modeling, will be offered an incentive boost of either $10 or $20, and the optimal amount (to be determined in consultation with OMB) will be offered only to targeted cases among the remaining subgroup 3 nonrespondents.

Exhibit B-9. Main study baseline and incentive boost experiments

	Incentive Phase	Amount	Total Cumulative Incentives Offered	Estimated Number of Cases to be Worked
High School Late/Alternative/Non-Completers	Base Incentive	$30	$30 to $50	170
	(all calibration sample cases)	$40		170
		$50		169
	Boost 1 (all remaining calibration sample nonrespondents)	$15	$45 to $75	158
		$25		158
	Boost 2 (all remaining calibration sample nonrespondents)	$10	$55 to $95	102
		$20		102
Ultra-Cooperative Respondents	Base Incentive (all calibration sample cases)	$0	$0	154
	Boost 1 (for targeted cases only: combined with subsample 3)	$10	$10 to $20 targeted; $0 otherwise	(very few if any cases expected to be selected)
		$20
	Boost 2 (for targeted cases only: combined with subsample 3)	$10	$10 to $40 targeted; $0 to $20 otherwise	(very few if any cases expected to be selected)
		$20
High School Completers and Unknowns	Base Incentive	$15	$15 to $40	330
	(all calibration sample cases)	$20		329
		$25		329
		$30		329
		$35		329
		$40		329
	Boost 1 (for targeted cases: 1/2 of non-respondents)	$10	$25 to $60 targeted; $15 to $40 otherwise	250
		$20		250
	Boost 2 (for targeted cases: 1/2 of non-respondents)	$10	$25 to $80 targeted; $15 to $60 otherwise	175
		$20		175

Additional treatments for targeted cases. In addition to the monetary interventions described above, the HSLS:09 second follow-up main study design includes non-monetary treatments to be used with targeted cases.

Field interviewing (phase 5). Field interviewing will be conducted for all targeted nonrespondents at the same time; there will be no time lag between the calibration and main samples. Cases identified for targeted treatment (all high school late/alternative/non-completers, and sample members with high bias likelihood scores) will be considered for field interviewing. The decision to conduct field interviewing for a case may also be determined by other factors, such as the location of a case and its proximity to other likely field cases. Nontargeted cases may potentially be included in field interviewing if it is cost effective to do so. Conversely, given the expense of field interviewing, cases with a very low response likelihood may not be pursued.

Extended data collection (phase 6). Cases identified for targeted treatment (all high school late/alternative/non-completers, and sample members with high bias likelihood scores) will be part of an extended data collection period. During this period (the last month of data collection), only targeted cases will be actively prompted to participate. Data collection will remain open for all other cases if they choose to participate, but effort to pursue those cases will be suspended.

Model development. A critical element of any responsive design is the method used to identify cases that will receive special treatment. As described above, the primary goal of this approach is to improve sample representativeness. The bias likelihood model will help determine which cases are most needed to balance the responding sample, and the response likelihood model will help determine which cases may not be optimal for pursuing with targeted interventions so that project resources can be most effectively allocated. In this section, we describe our modeling approach and the variables to be considered for use as predictor variables for both the bias likelihood and the response likelihood models. Variables will be drawn from data obtained in prior waves of data collection with this cohort (base-year, first follow-up, and 2013 Update survey data; high school transcripts; school characteristics; sampling frame information; and paradata). The models for the HSLS:09 second follow-up main study have been developed and will be refined from models for previous rounds of HSLS:09, ELS:2002, and other NCES studies, including BPS:12/14.

Response Likelihood Model. The response likelihood model will be run only once, before data collection begins. Using data obtained in prior waves that are correlated with response outcome (primarily paradata variables), we will fit a model predicting response outcome in the 2013 Update. We will then use the coefficients associated with the significant predictors to estimate the likelihood of response in the second follow-up main study, and each sample member will be assigned a likelihood score prior to the start of data collection. Exhibit B-10 lists the universe of predictor variables that will be considered for the response likelihood model.

During data collection, the response likelihood scores will be used as a “filter” to assist in determining intervention resource allocation. For example, cases that have a very high likelihood of participation may not be offered an incentive boost, since they are likely to participate without it. The response likelihood score can also be used to exclude cases with very low likelihood from the field interviewing intervention. We will also consider using the response likelihood score to adjust the classification of cases in the subgroups. For example, cases with very high response likelihood scores could potentially be treated as “ultra-cooperative” cases. The primary objective of the response likelihood model is to provide information that will inform decisions about inclusion or exclusion of targeted cases for interventions, thereby controlling costs.

Bias Likelihood Model. The bias likelihood model will be used to identify cases that are most unlike the set of sample members that have responded. As was done in the responsive design approach for the 2013 Update, the bias likelihood model will use only key survey and frame variables as predictors to identify nonrespondents most likely to reduce bias in key survey variables if converted to respondents. To calculate bias likelihood, we will run a logistic regression with the second follow-up response outcome as the dependent variable. The bias likelihood model will be run at the beginning of phases 3, 4, 5, and 6 for the calibration samples and at the beginning of phases 3, 4, 5, and 6 for the rest of the cases. (Modeling will be done on the combined sample [calibration cases and rest of cases] prior to phases 5 and 6.) We will then use the coefficients associated with the significant predictors to assign a bias likelihood score for each case. Because the set of respondents and nonrespondents is dynamic, the bias likelihood score for an individual case may change across the phases. The universe of candidate predictor variables have been selected due to their analytic importance for the study, and are presented in Exhibit B-11.

Exhibit B-10. Candidate Variables for the Main Study Response Likelihood Model

Data collection wave	Variables
Base year	Response outcome Response mode Early phase response status
First follow-up	Response outcome Response mode Early phase response status
Panel Maintenance and Address Updates	Panel maintenance response status Address update response status
2013 Update Survey	2013 Update response by student (not parent) Early phase response status Response mode Incentive amount (to control for the effect of incentives on response outcomes) Ever called in to the help desk Ever refused (sample member) Ever refused (other contact) Ever agreed to complete web interview Dual language speaker HS completion status indicator Gender
	Count of email addresses Count of phone numbers Count of addresses

Exhibit B-11. Candidate Variables for the Main Study Bias Likelihood Model

Data collection wave	Variables
Sampling frame	Race
	Gender
	School type
	Metropolitan area
	Geographic region
Base Year	Student’s educational expectations Parent’s education expectations Taking a Fall 09 math course Taking a Fall 09 science course Base year math assessment score
First Follow up	When Algebra 1 was taken Grade in Algebra 1 Student’s educational expectations Parent’s education expectations Grade in 2011-12 Location Dual language indicator Socioeconomic status indicator Repeated a grade? F1 math assessment score Attended a job fair? Toured a college? Taken a college class? Completed an internship? Performed work in job related to career goals? Searched internet/college guides for college options? Talked to HS counselor about after high school options? Talked with college admission counselor? Taken a college entrance exam prep course? Taking math classes in spring 2012?
2013 Update Survey and High School Transcript Collection	Sample member has high school credential Date of high school credential School characteristics of last-attended high school Dual-enrollment status/information
	Taking postsecondary classes as of Nov 1 2013 Sector of postsecondary institution as of Nov 1 2013
	Apprenticing as of Nov 1 2013
	Working for pay as of Nov 1 2013
	Serving in the military as of Nov 1 2013
	Starting family/taking care of children as of Nov 1 2013
	Attending high school or homeschool as of Nov 1 2013
	In a course to prepare for GED as of Nov 1 2013 Number of postsecondary institutions applied to
	Completed a FAFSA for teenager's education
	Did not complete FAFSA because did not want to go into debt
	Did not complete FAFSA because can afford college without financial aid
	Did not complete FAFSA because thought ineligible or unqualified
	Did not complete FAFSA because did not know how
	Did not complete FAFSA because forms were too time-consuming/too much work
	Did not complete FAFSA because did not know could
	Did not complete FAFSA because teen does not plan to continue education
	Currently working for pay
	Number of high schools attended
	Attended CTE center (flag)
	English language learner status
	GPA: overall GPA: English
	GPA: mathematics
	GPA: science
	Total credits earned
	Credits earned in academic courses Ever had a dropout episode

The goal of the bias likelihood model is not to accurately predict response, but to classify sample members’ current response rates along the dimensions represented by the predictor variables. As such, statistical significance should not be a determining factor in which variables are included in the model, rather the criterion should be that variable’s importance for HSLS:09. The threshold for identifying cases for targeted treatment will be based on an assessment of the bias likelihood score, the response likelihood score, and available project resources.

Evaluation of responsive design approach.

There are three elements to be evaluated in the proposed responsive design approach: (1) that sample cases that contribute to sample representativeness can be identified at the beginning of the third and subsequent data collection phases, (2) that interventions used during each phase of the data collection design are effective in increasing participation, and (3) that increasing response rates among the targeted cases will improve sample representativeness. We intend to examine these three aspects of the responsive design and its implementation for the HSLS:09 second follow up as follows:

1. Evaluate the bias likelihood model used to identify targeted cases. To assess whether the bias likelihood model successfully identifies nonresponding cases that are underrepresented on key survey variables, we will compare estimates within the categories of each model variable for respondents and nonrespondents at each phase. This comparison will highlight the model variables that exhibit bias at each phase and the relative size of the imbalance that remains to be reduced through the intervention.

2. Evaluate the effectiveness of each intervention in increasing survey participation. The second key component of this responsive design is the effectiveness of the targeted treatments in increasing participation. Experiments conducted with the calibration samples will allow us to assess the efficacy of the various treatments.

3. Evaluate the ability to increase sample representativeness, by identifying cases for targeted treatment. We will measure sample representativeness by comparing estimates on key variables for respondents and nonrespondents, at each phase of data collection and at the end of data collection. We will then be able to assess whether sample representativeness is improved over the course of data collection through the use of the targeted interventions for cases identified with the bias likelihood model.

References

Rosen, J. A., Murphy, J. J., Peytchev, A., Holder, T. E., Dever, J. A., Herget, D. R., & Pratt, D. J. (2014). Prioritizing low-propensity sample members in a survey: Implications for nonresponse bias. Survey Practice, 7(1), 1–8.

Pratt, D. J. (Invited Speaker). (2014, March). What is adaptive design in practice? Approaches, experiences, and perspectives. Presented at FedCASIC 2014 Workshop Plenary Panel Session, Washington, DC.

Pratt, D. J. (2013, December). Modeling, prioritization, and phased interventions to reduce potential nonresponse bias. Presented at Workshop on Advances in Adaptive and Responsive Survey Design, Heerlen, Netherlands.

Attachment 2 – Calibration sample results for phases 1 and 2, as of 4/28/2016

HSLS:09 calibration study analysis: Phase 1 (baseline incentive) and 2 (CATI)

Baseline incentive (Phase 1 and 2). To assess the efficacy of the baseline incentive within the three subgroups defined below, chi-square tests were used to perform pairwise contrasts between the incentive amounts offered to cases selected for the calibration study. Note that partially complete cases are excluded from this analysis. Included in this analysis are the following:

• Subgroup A (High School Late/Alternative/ Non-Completers) cases randomly assigned to baseline incentive amounts of $30, $40, or $50.

• Subgroup B (Ultra-Cooperative Respondents) cases not offered a baseline incentive. While not an experimental comparison, the Subgroup B response rate is presented along with response rates from other studies. We compare with BPS:12/14 because the BPS:12/14 and HSLS:09 second follow-up samples are similar in age. We also include results from the B&B:08/12, another highly cooperative population.

• Subgroup C (All Other High School Completers and Unknown Cases) cases randomly assigned to six different incentive amounts, ranging from $15 to $40 ($15, $20, $25, $30, $35, or $40). The $15 starting point for this baseline incentive calibration experiment is based on the results of the HSLS:09 second follow-up field test experiment.

Subgroup A (High School Late/Alternative/ Non-Completers). Exhibit 1 displays Subgroup A response rates by incentive level; Exhibit 2 displays the results of the pair-wise chi-square tests. Exhibit 1 shows that the highest response rate was achieved with the $40 incentive. Exhibit 2 presents statistical tests of differences; among cases in Subgroup A, response rates were highest among cases assigned a baseline incentive of $40. The $40 response rate is about 6.5 percentage points higher than the $30 rate, though the response rate associated with the $40 incentive is not significantly higher than with the $30 incentive at the 0.05 level, (χ2 (1, N = 1) = 1.84, p = 0.17). There is virtually no difference in response rates between the $40 incentive group and the $50 group (χ2 (1, N = 1) = 0.01, p = 0.93). Given the size of the observed difference between $30 and $40, we recommend a baseline incentive of $40 be offered to all cases in the Subgroup A main sample.

Exhibit 1. Subgroup A (High School Late/Alternative/ Non-Completers) response rates by baseline incentive amount

Baseline incentive		Sample members (n)	Phase 1 and 2 response (n)	Phase 1 and 2 response rate (percent)
$30		170	39	22.9
$40		170	50	29.4
$50		169	49	29.0
Total		509	138	27.1

SOURCE: U.S. Department of Education, National Center for Education Statistics. High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study.

Exhibit 2. Subgroup A response rate chi-square tests and p-values

Baseline incentive		Comparison group
		$30	$40	$50
$30	Chi-Square	†	1.84	1.62
	Probability	†	p =0.1748	p = 0.2037
$40	Chi-Square	1.84	†	0.0072
	Probability	p =0.1748	†	p = 0.9326
$50	Chi-Square	1.62	0.0072	†
	Probability	p = 0.2037	p = 0.9326	†

 Not applicable.
SOURCE: U.S. Department of Education, National Center for Education Statistics. High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study.

Subgroup B (Ultra-Cooperative Respondents). Subgroup B cases were not offered a baseline incentive, yet this group of “ultra-cooperative” sample members had a response rate of 63.6 percent (98 completes out of 154 sampled.) For context, exhibit 3 provides the Subgroup B response rates as well as response rates for selected other studies (BPS:12/14 and B&B:08/12.) The Subgroup B calibration response rate performs above what might be expected given other comparable cohorts. These results show that the HSLS:09 second follow-up ultra-cooperative calibration sample responded, with no incentive offer, at a rate similar to that seen among BPS:14/14 sample members with high predicted response propensity and with a $20 incentive. The HSLS:09 second follow-up ultra-cooperative calibration sample response rate of 50 percent is also similar to that seen among B&B:08/12 sample members who had responded to the two earlier waves of the study with at least a $20 incentive offer. Given the strong response rate for this group, our recommendation is to maintain a $0 baseline incentive among Subgroup B cases in the main sample.

Exhibit 3. Comparison of Subgroup B response rates with response rates from selected studies

Study	Response rate (percent)
HSLS:09 second follow-up calibration sample
Subgroup B Phase 1 and 2 (Ultra-cooperative, $0)	63.6
BPS:12/14 calibration sample
High response propensity (>0.9), $0 incentive	23.0
High response propensity (>0.9), $10 incentive	30.0
High response propensity (>0.9), $20 incentive	44.0
High response propensity (>0.9), $30 incentive	59.0
High response propensity (>0.9), $40 incentive	62.0
High response propensity (>0.9), $50 incentive	66.0
B&B:08/12 final response rates by subgroup (baseline incentives of $20, $45, or $55)
B&B:08/09 respondents	47.3
NPSAS:08 & B&B:08/09 respondents	48.1
B&B:08/09 early respondents	64.5
NPSAS:08 & B&B:08/09 early respondents	69.9

SOURCE: U.S. Department of Education, National Center for Education Statistics. High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study; U.S. Department of Education, National Center for Education Statistics, 2012/14 Beginning Postsecondary Students Longitudinal Study (BPS:12/14); U.S. Department of Education, National Center for Education Statistics, 2008/12 Baccalaureate and Beyond Longitudinal Study (B&B:08/12).

Subgroup C (All Other High School Completers and Unknown Cases). Exhibit 4 displays Subgroup C response rates by incentive level; Exhibit 5 displays the results of pair-wise chi-square tests. Within Subgroup C, the response rate is highest among cases assigned the $30 incentive at 43 percent. The $30 response rate is not significantly lower than $35 (χ2 (1, N = 1) = 0.90, p = 0.34) or $40 (χ2 (1, N = 1) = 0.01, p = 0.75). Response rates for the $30 incentive group were significantly higher than the $15 and $20 incentives (χ2 (1, N = 1) = 17.28, p < 0.0001 and χ2 (1, N = 1) = 6.59, p = 0.0103 respectively).

Note that the comparison of response rates between $30 and $25 (43.2 and 37.1 percent, respectively) is not statistically significant (χ2 (1, N = 1) = 2.53, p = 0.1117). However, we are concerned that there is not enough power to detect a statistical difference of 6 percent with the available sample size. Experience in many other studies of similar populations has shown that, among college-age young adults, at least a $30 incentive is needed to achieve an adequate response rate. Subgroup C constitutes the largest subgroup in the sample (nearly 18,000 sample members), and a 6 percent response rate difference for this group is non-trivial. While the statistical results indicate that a minimum of $25 is needed, we recommend that a baseline incentive of $30 be offered to all Subgroup C main sample cases.

Exhibit 4. Subgroup C (All Other High School Completers and Unknown Cases) response rates by baseline incentive amount

Baseline incentive	Sample members (n)	Phase 1 and 2 response (n)	Phase 1 and 2 response rate (percent)
$15	329	91	27.7
$20	329	110	33.4
$25	329	122	37.1
$30	329	142	43.2
$35	329	130	39.5
$40	329	138	41.9
Total	1974	733	37.1

SOURCE: U.S. Department of Education, National Center for Education Statistics. High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study.

Exhibit 5. Subgroup C (All Other High School Completers and Unknown Cases) response rate chi-square tests and p-values

Baseline incentive		Comparison group
		$15	$20	$25	$35	$40
$30	Chi-Square	17.28	6.59	2.53	0.90	0.10
	Probability	<0.0001	0.0103	0.1117	0.3421	0.7525

SOURCE: U.S. Department of Education, National Center for Education Statistics. High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study.

1 Note that in Attachment 1, the subgroups are labeled as 1, 2, and 3 rather than A, B, and C.