OMB83 C Change Memo

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: September 22, 2016

TO: Robert Sivinski, OMB

THROUGH: Kashka Kubzdela, OMB Liaison, NCES

FROM: Elise Christopher, HSLS:09 Project Officer, NCES

SUBJECT: High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study Incentive Boost and Institutional Collection Change Request (OMB# 1850-0852 v.23) – Increased Incentive Boost Amount for High School Non-Completer Cases and Targeted Cases among Other High School Completers/Unknowns; Revisions for Transcript and Student Records Collection Materials

The High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study and 2018 Panel Maintenance request was approved by OMB in December 2015 (OMB# 1850-0852 v.17) with updates in March, May, June, and July 2016 (OMB# 1850-0852 v.18-22). This request is to: 1) increase an additional incentive boost 2b offer for Subgroup A (high school late/alternative/non-completer) main sample cases and boost-2-targeted Subgroup C (other high school completer and unknown cases) main sample cases, 2) increase the incentive offer for the originally selected but misclassified Subgroup A (high school late/alternative/non-completer cases) calibration cases, and 3) revise transcript and student records collection materials.

1. Additional incentive boost 2 offer

In the last change memo (OMB# 1850-0852 v.22), we recommended the $10 amount for incentive boost 2 for targeted cases (i.e., all of Subgroup A – High School Non-Completer (HSNC) cases – as well as targeted cases in Subgroup B (Ultra-Cooperative [UC]) and targeted cases in Subgroup C (all other high school completer and unknown cases), based on the 4 week results of the calibration sample experiment that were available at that time.

While response rates between the $10 and $20 boost 2 incentive groups were equivalent then for each of the subgroups, the differences have since become large and statistically significant for subgroups A and C (subgroup B had very small numbers and no observed difference). For reference, those results (from 7/21/2016) are presented in Tables 1, 2, and 3 (for Subgroups A, B, and C respectively) along with the results as of 9/7/2016.

Based on these results, we propose an adaptive component to our responsive design: offering an additional boost (incentive boost 2b) of $10 to Subgroup A (HSNC) and Subgroup C (all other high school completer and unknown) main sample cases. The additional time for the calibration sample cases in phase 4 has revealed an effect that was not evident at the end of the first 4 weeks of phase 4. In the intervening weeks, we have increased locating, prompting, and case review efforts for all pending cases (regardless of incentive amount assignment), and this has likely resulted in the location of cases that had not been previously located. In light of the most recent calibration sample results, we request approval to offer the additional boost of $10 to the remaining nonrespondents in the Subgroup A (HSNC) main sample (for a total incentive offer of $75 to these cases), given the importance of these cases and the effectiveness of the increased amount. We also request approval to offer the additional boost (2b) of $10 to the targeted boost 2 cases in subgroup C (all other high school completer and unknown) main sample. We do not recommend modifying the boost amount for the subgroup B targeted cases.

By treating this incentive boost 2b as an unscheduled adaptive boost for our responsive design, we can preserve the main sample results for the purpose of evaluating the boost as planned.

Table 1. Subgroup A (HSNC) Calibration Results, at 4 and 11 weeks, by boost 2 incentive amount

Boost 2 incentive amount	Sample members (n)	Boost 2 results after 4 weeks (as of 7/21/2016)			Boost 2 results after 11 weeks (as of as of 9/7/2016)
		Number of respondents	Response rate (%)		Number of respondents	Response rate (%)
$10	152	8	5.3		12	7.9
$20	156	9	5.8		28	17.9	***
Total	308	17	5.5		40	13.0

*** p < .01

SOURCE: U.S. Department of Education, National Center for Education Statistics. High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study.

Table 2. Subgroup B Calibration Results, at 4 and 11 weeks, by boost 2 incentive amount

Boost 2 incentive amount	Sample members (n)	Boost 2 results after 4 weeks (as of 7/21/2016)			Boost 2 results after 11 weeks (as of as of 9/7/2016)
		Number of respondents	Response rate (%)		Number of respondents	Response rate (%)
$10	7	1	14.3		2	28.6
$20	6	1	16.7		2	33.3
Total	13	2	15.4		4	30.8

SOURCE: U.S. Department of Education, National Center for Education Statistics. High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study.

Table 3. Subgroup C Calibration Results, at 4 and 11 weeks, by boost 2 incentive amount

Boost 2 incentive amount	Sample members (n)	Boost 2 results after 4 weeks (as of 7/21/2016)			Boost 2 results after 11 weeks (as of as of 9/7/2016)
		Number of respondents	Response rate (%)		Number of respondents	Response rate (%)
$10	285	17	6.0		30	10.5
$20	288	19	6.6		49	17.0	***
Total	573	36	6.3		79	13.8

*** p < .05

SOURCE: U.S. Department of Education, National Center for Education Statistics. High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study.

2. Increased incentive offer for Subgroup A Calibration Cases that Did Not Receive Baseline Incentive Offer

The HSLS:09 calibration High School Non-Completer (HSNC) subsample was intended to receive a baseline incentive offer ($30, $40, or $50) whereas Ultra-Cooperative (UC) calibration cases were intended not to be offered a baseline incentive. In the original selection of calibration cases, the HSNC cases and UC cases were misclassified such that 154 HSNC cases were not offered a baseline incentive while 509 UC cases were offered a baseline incentive ($30, $40, or $50). Upon discovery of this error, 509 additional HSNC and 154 additional UC cases were redrawn for the calibration sample and have been the basis for calibration sample analysis and reporting. Although the incorrectly-drawn HSNC and UC cases continued to be worked during the calibration study through phase 2 and through the boosts in phases 3 and 4 in their correct groups, their results were not included in the reports of the results from the calibration study. The misclassified HSNC calibration cases should have received the $40 baseline incentive for HSNC cases at the start of Phase 2 in the calibration study (even though they were no longer included in the calibration results). Now, since the $40 HSNC baseline incentive was never offered to the incorrectly-assigned HSNC cases and 111 (72.1 percent) of them have not yet responded, we request approval to add the HSNC baseline incentive to the incentive offer being made to these 111 cases. Both HSNC calibration groups have been offered two incentive boosts as part of responsive design phases 3 and 4. However, the total incentive offer for the incorrect group is less than the total incentive offered to the correct group ($25-$45 vs. $55-$95). After approximately 11 weeks of production since the start of phase 4 (implementation of the second incentive boost), the response rate for the calibration HSNC cases that did not receive a baseline incentive offer (27.9 percent) continues to lag far behind the calibration HSNC cases that did (47.4 percent) and the main sample HSNC cases (42.9 percent). Table 4 illustrates the response rates for HSNCs by subgroup. We propose to increase the incentive offered to the pending calibration HSNC cases that did not receive a baseline incentive offer (n=111) by $40 to match the baseline incentive being offered to the main sample HSNCs. The rationale for increasing the incentive at this time is to make up for the “missed” baseline incentive offer, to encourage participation, and to reduce the significant response rate gap for the HSNC group that was not offered an incentive at baseline.

Table 4. HSLS:09 High School Non-Completer (HSNC) Calibration and Main Samples: Data Collection Results as of 9/7/2016, by Baseline Incentive Offer

Group	Baseline incentive offer	Boost 1 offer start	Boost 2 offer start	Total current incentive offer	Response rate (as of 9/7/16)	Number of pending cases (as of 9/7/16)
Calibration HSNC no baseline incentive	$0	5/4/16	6/15/16	$25-$45	27.9%	111
Calibration HSNC baseline incentive	$30-50	5/4/16	6/15/16	$55-$95	47.4%	268
Main Sample HSNC	$40	6/20/16	8/1/16	$651	42.9%	1,074

SOURCE: U.S. Department of Education, National Center for Education Statistics. High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study.

3. Revisions to burden estimate, contacting materials for student records and transcript collections, and student records facsimile

As described in the last change memo (OMB# 1850-0852 v.22), HSLS:09 and BPS:12/17 are combining efforts for transcript and student records collections. The HSLS:09 second follow-up package has been revised to incorporate changes needed to be consistent with BPS:12/17.

Changes to estimated burden time rate. Part A.12 has been modified to reflect the updated burden time rate for postsecondary institution personnel ($40.47) and associated source citation.

Changes to contacting materials. The contacting materials, shown in Appendix F-G, have been revised to add the PRA statement to the letters on pages F-12, F-14, and F-16, and the abbreviated PRA reference text to the emails on page F-34, G-10, and G-11. Additionally, a document entitled “Student Records Item Overview Handout” has been added to the website text beginning on page G-18.

Changes to the student records facsimile. Appendix H contains the student records facsimile and reflects updates made based on the results of the BPS:12/17 qualitative evaluation, and is consistent with the student records instrument submitted as part of the BPS:12/17 full-scale package (OMB #1850-0631 v.10), which is currently in the 60-day public comment period (due to be submitted to OMB on October 25, 2016). Table 1 within Appendix H provides a summary of the changes to the content of the instrument when compared to the instrument originally cleared for HSLS:09. The table includes color coding to indicate whether items have remained the same (black), were revised (purple), removed (red), or added (green). Changes to the previously approved instrument are intended to reduce respondent burden and improve data quality. The instrument was also revised to include data elements for the 2011-2012 academic year; these items are exact duplicates of the parallel items from academic years 2012-2013 through 2016-2017. This academic year was added for three reasons: first, to make the HSLS:09 and BPS instruments completely consistent; second, to reduce potential confusion for institutions that are participating in HSLS:09 and BPS student record collections simultaneously; and third, to allow for the collection of postsecondary data for HSLS:09 cohort members that began their postsecondary education earlier than most other students in their cohort.

1 The current approved total incentive offer for the HSNC main sample is $65, however it will be $75 if request 1 to increase boost 2 to $20 for remaining Subgroup A pending nonrespondents is approved.