Field Test Results

March 19, 2013

High School Longitudinal Study of 2009 (HSLS:09)

College Update and Transcript Main Study (2013)

Appendix 5

2012 Update Field Test Results

Request for OMB Review
OMB# 1850-0852 v.11

Submitted by

National Center for Education Statistics

U.S. Department of Education

The 2012 Update field test was designed to be a brief survey conducted between the first and second follow-ups to learn about students’ plans after high school. In this summary of the field test, results are provided for data collection, the use of the Mahalanobis distance function to minimize nonresponse bias, and the validation study conducted to test the reliability of parent and student responses to the questionnaire. Field test results will be included as an appendix in the 2013 Update/Transcript Data File Documentation (DFD).

Data Collection Results

The purpose of the 2012 Update field test was to test procedures for having either the student or parent complete a 20-minute questionnaire about the student’s plans for the fall of 2012 and beyond. Data collection was conducted in three phases:

• Phase 1 consisted of a two week web-only period.

• Phase 2 was a three week period (weeks 3 through 5) with telephone prompting and outbound interviewing added to web data collection.

• Phase 3 comprised the remainder of data collection (weeks 6 through 15) with the introduction of incentives for high-distance cases based on a Mahalanobis distance score calculation at the start of the phase.

For 69 percent of nonresponding cases at the start of Phase 3, a $5 pre-paid incentive was mailed to the sample member with the promise of an additional $10 incentive for completing the questionnaire. The remaining cases received no incentive.

Response rates and mode of response for the 2012 Update are provided in Table 5.1. Overall, 68 percent of sample members responded to the 2012 Update. The students provided 57 percent of the responses compared to 43 percent parent responses. Both students and parents were more likely to participate via Web than telephone. Table 5.2 shows response by enrollment status as of the first follow-up. Of the students still enrolled at the base-year school as of the first follow-up, 74 percent participated. Students or parents of transfer students had a response rate of 49 percent. Due to small sample sizes, students who had dropped out of school, graduated early, were home-schooled, or had an unknown status were grouped together and 65 percent participated.

Table 5.1. 2012 Update response rates by sample member , first follow-up enrollment status, and mode of completion

	Number  of sample members	Percentage of sample members
Total Sample	754	100
Overall response	514	68.2
Student	292	56.8
Web	159	54.5
CATI	133	45.5
Parent	222	43.2
Web	141	63.5
CATI	81	36.5
Mode
Web	300	58.4
CATI	214	41.6

Table 5.2. 2012 Update response rates by first follow-up enrollment status
	Total sample members	Number responding	Percent responding
Total	754	514	68.2
Enrolled at base-year school	537	398	74.1
Transferred	157	77	49.0
Other/Unknown	60	39	65.0

Mahalanobis Distance Function

The 2012 Update field test utilized a responsive design methodology to strategically target nonresponse cases that could potentially contribute to bias if they remained nonrespondents. The methodology centered on identifying nonrespondent cases most unlike respondent cases, and targeting these nonresponding cases in a manner that was intended to increase the likelihood that they would become respondents. A Mahalanobis distance function score was used to rank nonresponding cases in terms of their overall difference from existing respondents.

A combination of survey variables, sample frame variables, and paradata were used in building the model to select target cases. Both student- and parent-level variables were considered for use in the Mahalanobis distance calculations. The variables used to calculate Mahalanobis distance are shown in Table 5.3.

Additional survey variables were considered (performance on the assessment, educational expectations, etc…) for the models, but many of the candidate variables had a large number of unknown values for cases in the field test sample. Imputing missing values for survey variables was ruled out due to the high level of missing values. At the time of calculating the Mahalanobis distance, 210 parent cases had responded, leaving 544 pending cases that were available for consideration for nonresponse follow up period. The 375 highest-distance nonrespondents were selected as the target cases for Phase 3. These 375 targeted cases had a mean Mahalanobis value of 8.33. Non-targeted non responding cases had a mean Mahalanobis value of 5.75. Phase 3 targeted cases and non-targeted cases had response rates (54 percent and 59 percent, respectively) that were not statistically different (x² = 1.08, p = .2996).  This may suggest that the phase 3 pre-paid $5 incentive with the offer of $10 more upon completion may have had some effect in encouraging participation from the targeted group of cases, given that they were likely more challenging.  Table 5.4 shows the response by phase of data collection.

Table 5.3. Variables used for calculation of Mahalanobis Distance

Source	Variables
Survey variables	Enrollment status Gender
Sample frame variables	School type Metro area Race
Paradata	Whether sample member contacted the help desk Whether sample member logged in but did not complete the College Update questionnaire Number of contact attempts in the early data collection period Whether sample member made an appointment to complete the interview Whether sample member told interviewer they would do the web interview Student base year and first follow-up response outcomes Parent base year and first follow-up response outcomes Parent response in the panel maintenance update Student enrollment status at first follow-up Reason for prior student nonresponse (refusal, absent) if applicable Call counts in base year and first follow-up

Table 5.4. 2012 Update field test response by data collection phase

	Total	Participated	Percent
Total	754	514	68.2
Phase 1	754	81	10.7
Phase 2	673	129	17.1
Phase 3	544	304	55.9
High Distance	375	204	54.4
Low distance	169	100	59.2

The primary goal of offering an incentive to cases with high Mahalanobis distance scores is to reduce the potential for nonresponse bias in key survey estimates. Key survey estimates were examined for indications of reduced nonresponse bias resulting from the third phase of data collection. To do this, estimates produced from the combined set of respondents including phase 1, phase 2 and the non-targeted phase 3 respondents were compared against the full respondent set that also includes the phase 3 respondents. Non-targeted and targeted cases were also compared to determine if differences existed between these sets of cases. Five key variables were analyzed: 1) earned a high school diploma, 2) taking classes at a college or university, 3) applied to college, 4) completed a FAFSA, and 5) currently working. Differences in estimates would suggest that the incentives offered in Phase 3 were effective in lowering the potential for bias by capturing responses from sample members who would otherwise have not participated. The estimates are shown in Table 5.4.

An examination of the estimates shows that the targeted cases who responded were less likely than the non-targeted cases to have earned a high school diploma, less likely to be taking college or university classes, less likely to have applied to a postsecondary institution, less likely to have completed a FAFSA, and more likely to be working. All these differences in point estimates were significant at the .05 level. The Mahalanobis distance function identified cases that were different, but importantly, these identified and targeted cases appear different in their survey responses. Examining these estimates suggests that the high-distance cases were a good choice of nonresponding cases to target. While the estimates of the targeted cases that participated look different from the overall set of respondent estimates, the non-targeted phase-3 cases that participated look more similar to the overall respondent set. Targeting the nonresponding non-targeted set of cases would likely have brought into the respondent pool cases that look similar to those who had already been interviewed, a nonresponse follow-up scenario that is not advisable (Schouten, Cobben, and Bethlehem, 2009).

Table 5.5. Survey estimates with and without phase 3 respondents

Variable	Phase 3 targeted cases (n = 201)	Phase 3  non targeted cases (n=84)	Phase 1, phase 2 and phase 3 non targeted cases (without phase 3 targeted cases, n = 313)	Overall estimate (n=514)
Earned a high school diploma	84.6	94.0	96.5	91.8
Did not earn a high school diploma	15.4	6.0	3.5	8.2
Taking classes at a postsecondary institution	68.7	85.7	87.5	80.2
Applied to postsecondary institution(s)	36.3	54.8	61.7	51.8
Completed a FAFSA	62.7	81.0	77.3	71.6
Currently working	56.2	42.7	47.6	50.1

Validation Study

In addition to the primary data collection, a validation study was conducted to determine the reliability between student and parent responses on the same items. The 2012 Update interview was designed to be administered to either the teenage sample member or one of his/her parents. When selecting items for the instrument, preference was given to factual questions that could be answered by either the teenager or the parent and had the highest likelihood of consistent teenager-parent responses. However, some important questions that were subjective in nature were included as well.

To evaluate consistency of responses, the complete interview was conducted with both the student and one of his/her parents for 112 pairs. At the beginning of the data collection cycle, student and parent respondents who reached the end of the interview were asked to provide contact information for the other (i.e., students were asked to provide parent contact information and parents were asked to provide student contact information) so that RTI could follow up for the validation study. Student-parent pairs were recruited regardless of the mode of the first interview, self-administered web interview or CATI. RTI then attempted to contact and interview the other or encourage completion by web. When at least 100 pairs of completed interviews had been achieved, participants were no longer recruited.

For analysis of the results, before the percentage agreement was calculated for a variable, cases with a “don’t know” response or a nonresponse to the question were eliminated from the analysis. In other words, both the student and the parent had to have a response other than “don’t know” to be considered a valid pair for comparison. Additionally, continuous variables were categorized and job earnings were placed on the same scale (annual earnings). When questions pertained to a particular institution named by the respondent, care was taken to only compare responses when the institutions named by the student and the parent were the same. When the two respondents listed the same schools but in a different order, the responses were matched up by institution before comparison.

Many of the items in the 2012 Update provided an explicit “don’t know” option for students and parents. However, students and parents who did not know may have also left the question unanswered. Therefore, for the comparison of the percentage of “don’t know” responses, both explicit and implicit forms of “don’t know” were counted. Cases where an item was legitimately skipped based on instrument routing were excluded from the analysis.

Results from the validation study analysis are presented in table 5.6. The table presents the number of valid student-parent pairs, the percent agreement within valid pairs, and the percent of students and parents who either answered with a “don’t know” category, or left the item missing. In summary, 108 items had sufficient and appropriate data to conduct an analysis. Among the 108 items, 45 items (41.7 percent) had at least 85 percent of valid pairs giving the same answer, 36 items (33.3 percent) had 70 to 84.9 percent of valid pairs in agreement, and 27 items (25 percent) had less than 70 percent of pairs in agreement. Furthermore, 78 items (72.2 percent) had less than 10 percent of students answering “don’t know” or skipping the item, while 30 items (27.8 percent) had 10 percent or more of respondents answering “don’t know” or skipping the item. Among parents, 66 items (61.1 percent) had less than 10 percent of respondents who answered “don’t know” or skipped the item, and 42 items (38.9 percent) that had 10 percent or more “don’t know” or nonresponse.

Results from the analysis were presented to the Technical Review Panel. Along with the results from the analysis, additional measures of distribution, and expert knowledge of the content, the Technical Review Panel reviewed the instrument and made recommendations for the 2013 Update instrument, which have been incorporated into the main study design.

Table 5.6. Percent agreement of valid pairs and percent of students and parents who did not answer item

Variable name	Variable label	Number of valid pairs	Percent agreement of valid pairs	Percent of respondent who answered don’t know or did not answer
				Student	Parent
CUHSCRED	Earned high school credential and credential type	112	97.3	0.0	0.0
HSCREDDATE	Combined month and year high school credential awarded (recode)	108	88.9	2.7	0.9
CUENROLLHS12	High school enrollment status	#	#	#	#
CULASTHSMO	Month last attended high school	#	#	#	#
CULASTHSYR	Year last attended high school	#	#	#	#
CULASTHS	Last attended BY school, F1 school or another school	111	100.0	0.9	0.0
CULASTHSNAME	Name of high school last attended	#	#	#	#
CUOTHHS	Has attended any other high school besides BY school & most recent	110	100.0	0.0	1.8
CUOTHHSNAME	Name of other high school attended	#	#	#	#
CUOTHERHS	Attended any other high schools	#	#	#	#
CUDUALMATH	Has taken a math course for college credit	91	81.3	8.9	13.4
CUDUALSCIENCE	Has taken a science course for college credit	90	83.3	9.8	13.4
CUDUALOTHER	Has taken a course in another subject for college credit	105	84.8	1.8	4.5
CUCLGUNIV	Taking classes at a college or university in fall 2012	107	97.2	4.5	0.0
CUOCCSCHOOL	Taking classes at a school for occupational training in fall 2012	91	84.6	10.7	9.8
CUCERTLIC	Studying for an industry certification or license in fall 2012	91	92.3	12.5	6.3
CUAPPRENTICE	Participating in an apprenticeship program in fall 2012	91	98.9	11.6	9.8
CUOTHTRAIN	Receiving another form of training in fall 2012	93	93.6	8.0	8.9
CUWORK	Working in fall 2012	83	69.9	16.1	10.7
CUMILITARY	Serving in the military fall 2012	100	99.0	1.8	8.9
CUFAMILY	Starting a family or taking care of children in fall 2012	103	99.0	0.9	7.1
CUHS	Attending high school in fall 2012	#	#	#	#
CUGEDCOURSE	Attending a GED completion course in fall 2012	#	#	#	#
CUFOCUS	Teenager's main focus in fall 2012	27	92.6	0.0	4.5
CUBACHELOR	Enrolling in Bachelor's degree program	100	92.0	0.0	1.0
CUAABA	Enrolling in Associate's degree program-plans to transfer to BA/BS	100	91.0	0.0	1.0
CUAANOBA	Enrolling in Associate's degree program-no plans to transfer to BA/BS	100	99.0	0.0	1.0
CUCERTPROG	Enrolling in certificate/diploma program at school providing occupational training	100	99.0	0.0	1.0
CUNOPROG	Not enrolling in program, just taking classes	100	96.0	0.0	1.0
CUOTHPROG	Enrolling in another type of program	100	100.0	0.0	1.0
CUDK	Don't know what type of program will enroll in	100	97.0	0.0	1.0
CUCLGFT	Enrolling full-time or part-time	95	96.8	6.7	1.0
CUCLGIPEDS	Fall 2012 postsecondary institution	91	98.9	1.0	2.9
CUMAJORGEN01	Major in Fall 2012 postsecondary institution	82	73.2	12.6	9.6
CUWORKFT	Working full-time	24	83.3	19.2	2.3
CUACTDUTY	On active duty	#	#	#	#
CUAPPCLG	Applied to any (other) colleges	112	76.8	0.0	0.0
CUCLGAPPNUM	Number of colleges applied to	58	70.7	0.0	1.4
CUAPP1IPEDS	(Other) college applied to - 1	57	80.7	1.4	0.0
CUAPP2IPEDS	(Other) college applied to - 2	45	57.8	8.1	5.4
CUCHOICEAPP	First choice of schools applied to, not considering cost	50	72.0	0.9	1.8
CUAPP1STATUS	Status of application at (other) college applied to - 1	46	97.8	1.4	2.9
CUAPP2STATUS	Status of application at (other) college applied to - 2	26	100.0	0.0	0.0
CUCHOICEACC	First choice of schools accepted to, not considering cost	46	91.3	0.9	0.9
CUAPPFAFSA	Completed a FAFSA	96	96.9	10.7	4.5
CUNODEBT	Did not complete FAFSA because didn't want debt	9	55.6	6.7	13.6
CUCANAFFORD	Did not complete FAFSA because can afford college/school without it	11	81.8	0.0	4.5
CUINELIGIBLE	Did not complete FAFSA because thought ineligible/unqualified	11	63.6	0.0	9.1
CUDKHOW	Did not complete FAFSA because didn't have information on how to	10	70.0	0.0	13.6
CUFORMWORK	Did not complete FAFSA because too much work or time	11	81.8	0.0	9.1
CUDKCOULD	Did not complete FAFSA because didn't know could	10	80.0	0.0	13.6
CUNOPOSTSEC	Did not complete FAFSA because don't plan to continue education	9	100.0	0.0	18.2
CUNOQUALFAM	Thought would not qualify because another family member didn't qualify	5	80.0	0.0	7.1
CUNOQUALCRED	Thought would not qualify because of credit score	5	100.0	0.0	14.3
CUNOQUALINC	Thought would not qualify because income is too high	5	60.0	0.0	7.1
CUNOQUALTEST	Thought would not qualify because grades or test scores too low	5	80.0	0.0	14.3
CUNOQUALPT	Thought would not qualify because will attend part-time	5	80.0	0.0	14.3
CUNOQUALOTH	Thought would not qualify for another reason	4	75.0	16.7	21.4
CUAPPOTHAID	Completed financial aid applications besides FAFSA	75	86.7	21.4	13.4
CUFLSTAFFORD	Fall 2012 college offered Stafford loan for first academic year	41	80.5	36.6	22.1
CUFLOTHLOAN	Fall 2012 college offered other loan for first academic year	46	63.0	23.2	22.1
CUFLWKSTD	Fall 2012 college offered work-study for first academic year	46	84.8	17.1	22.1
CUFLPELL	Fall 2012 college offered Pell grant for first academic year	37	86.5	30.5	25.6
CUFLOTHGRNT	Fall 2012 college offered other grant for first academic year	59	79.7	11.0	12.8
CUFLOTHAID	Fall 2012 college offered other financial aid for first academic year	38	73.7	25.6	32.6
CUFLNOAID	Fall 2012 college offered no financial aid for first academic year	33	69.7	30.5	36.0
CUCHSTAFFORD	First choice accepted college offered Stafford loan for 1st academic yr	2	100.0	63.6	36.4
CUCHOTHLOAN	First choice accepted college offered other loan for 1st academic yr	3	100.0	18.2	36.4
CUCHWKSTD	First choice accepted college offered work-study for 1st academic yr	3	100.0	18.2	36.4
CUCHPELL	First choice accepted college offered Pell grant for 1st academic yr	1	100.0	45.5	45.5
CUCHOTHGRNT	First choice accepted college offered other grant for 1st academic yr	3	33.3	0.0	45.5
CUCHOTHAID	First choice accepted college offered other financial aid for 1st academic yr	1	100.0	27.3	45.5
CUCHNOAID	First choice accepted college offered no financial aid for 1st academic yr	3	33.3	18.2	36.4
CUAIDANYCLG	Offered financial aid apart from offers from these schools	103	72.8	5.4	2.7
CUCOSTFALLCLG1	Total cost of fall 2012 college for 2012-2013 school year	86	76.7	12.6	6.7
CUFALLBORROW1	Amount will borrow to pay for fall 2012 college	70	90.0	21.4	13.5
CUFALLSCHOLAR1	Amount will receive in scholarships and grants for fall 2012 college	81	72.8	13.6	9.6
CUCOSTCHOICE1	Total cost of 1st choice accepted college for 2012-2013 school year	7	42.9	18.8	16.7
CUCHCBORROW1	Amount would have borrowed to pay for 1st choice accepted college	7	71.4	18.8	16.7
CUCHCSCHOLAR1	Amount would have received in scholarships and grants for 1st choice accepted college	7	57.1	12.5	16.7
CUREPUTATION	Importance of academic quality/reputation when choosing fall 2012 college/school	91	75.8	1.0	4.8
CUCOSTATTEND	Importance of cost of attendance when choosing fall 2012 college/school	72	75.0	1.9	1.9
CUCLOSEHOME	Importance of being close to home when choosing fall 2012 college/school	42	73.8	1.0	1.9
CUFARHOME	Importance of being far from home when choosing fall 2012 college/school	15	73.3	6.8	11.5
CUJOBPLC	Importance of job placement when choosing fall 2012 college/school	69	60.9	4.9	12.5
CUGRADSCHPLC	Importance of graduate school placement when choosing fall 2012 college/school	59	67.8	6.8	12.5
CU4YRBAPLC	Importance of placement in 4-yr Bachelor's program when choosing fall 2012 college/school	61	57.4	8.7	19.2
CUSPORTS	Importance of opportunity to play sports when choosing fall 2012 college/school	14	71.4	4.9	4.8
CURECOMMEND	Importance of family/friend recommendations when choosing fall 2012 college/school	45	64.4	5.8	6.7
CUOFFERSPGRM	Importance of program of study when choosing fall 2012 college/school	80	76.3	2.9	4.8
CUSOCIALLIFE	Importance of good social life when choosing fall 2012 college/school	59	64.4	4.9	6.7
CUWHERELIVE	Where student will live in fall 2012	101	95.1	1.0	0.0
CUDISLIKESCH	Not attending school in fall 2012 because does not like school	6	66.7	0.0	0.0
CUDIDPOORLY	Not attending school in fall 2012 because did not do well in school	6	83.3	0.0	12.5
CUCANTAFFORD	Not attending school in fall 2012 because can't afford it	6	66.7	0.0	12.5
CURATHERWORK	Not attending school in fall 2012 because needs to/would rather work	5	40.0	22.2	0.0
CUNOTACCEPTED	Not attending school in fall 2012 because not accepted where wanted	2	100.0	0.0	0.0
CUBADOPTIONS	Not attending school in fall 2012 because did not want to go where accepted	2	100.0	0.0	0.0
CUDEFER	Not attending school in fall 2012 because deferred enrollment	2	100.0	0.0	0.0
CUNOTENOUGH	Not attending school in fall 2012 because didn't receive enough financial aid	6	66.7	0.0	12.5
CUOTHRSN	Not attending school in fall 2012 for another reason	6	50.0	0.0	12.5
CUJOBNOW	Currently working for pay	30	73.3	0.0	0.0
CUJ1OCC2	Current job	18	72.2	3.0	5.7
CUJOBRELATE	Current job's relationship to job wants to have when education completed	19	79.0	3.0	0.0
CUAPPRENTSHP	Current job is a formal apprenticeship	20	65.0	0.0	2.9
CULICENSEHRS	Earning hours for license for occupational field on current job	15	100.0	9.1	11.4
CUHSJOB	Started current job while in high school	20	90.0	0.0	0.0
CUHSPRG	Got current job through high school-arranged program	20	100.0	0.0	0.0
CUHSASSIST	Got current job with other assistance from high school	20	95.0	0.0	0.0
CUJOBEARN	Job earnings (Recoded)	15	66.7	6.1	17.1
CUJOBPLAN	Plans to have current job on November 1, 2012	19	89.5	0.0	2.9
CUJ2OCC2	Job plans to have on November 1, 2012	2	0.0	29.6	41.2
CUCNSLCLG	How well counselor prepared teenager to gain admission to college	102	49.0	2.7	6.3
CUCNSLAID	How well counselor prepared teenager to apply for financial aid	98	38.8	5.4	7.1
CUCNSLJOB	How well counselor prepared teenager to find a job	88	40.9	4.5	17.9

¹ Continuous variable categorized for analysis.

# No data available