National Longitudinal Transition Study-2 Wave 4 non-response bias

April 2008

Analysis Of potential bias in the Wave 4 Respondents To the National Longitudinal Transition Study-2 (NLTS2)

Prepared for:

David Malouf

Project Officer

National Center for Special Education Research

Institute of Education Sciences

U.S. Department of Education

Prepared by:

Harold Javitz, Mary Wagner, and Lynn Newman

SRI International

SRI Project P11182

The National Longitudinal Transition Study-2 (NLTS2) has been funded by the U.S. Department of Education, Institute of Education Sciences, under contract number ED-01-CO-0003. The content of this publication does not necessarily reflect the views or policies of the U.S. Department of Education nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government.

CONTENTS

Wave 4 NLTS2 Instrument Response Rate 1

Implications for Statistical Precision 1

Implications for Potential Bias 2

Cumulative Response Rate 6

TABLES

Table 1. Response Rates for the Parent/Youth NLTS2 Instruments 1

Table 2. Expected and Actual Number of Parent and Youth Interview/Surveys to Date 2

Table 3. Distribution of Students to the Wave 1 Parent Interview/Survey and Wave 4
Parent and Youth Interview/Survey 3

Table 4. Comparison of Wave 4 Parent and Youth Interview/Survey Eligible Population
and W4 Parent and Youth Interview/Survey Respondents on their Wave 1
Parent Interview/Survey Responses 5

Analysis Of potential bias in the wave 4 Respondents to the National Longitudinal Transition Study-2 (NLTS2)

In evaluating the quality of a survey sample, there are two primary considerations: statistical precision and the potential for bias. The survey response rate is pertinent to both in that an unexpectedly low response rate can leave a study with insufficient statistical precision and it might produce, although does not necessarily produce, a biased sample—i.e., one that does not accurately represent the universe from which the sample was selected. Below we present the number of respondents for the fourth wave of the National Longitudinal Transition Study-2 (NLTS2) and a response rate that is calculated using the maximum eligible sample. We then discuss the implications for statistical precision and for the potential for sample bias.

Wave 4 NLTS2 Instrument Response Rate

Table 1 specifies the number of respondents for the Wave 4 Parent/Youth interview/survey, and the associated response rates, calculated using the maximum appropriate eligible population within responding LEAs, as indicated in the table notes. Response rates for Waves 1, 2, and 3 of NLTS2 are also provided for comparison purposes. In particular, these calculations include youth as eligible whether or not they could possibly be reached for an interview or survey because no location information is available. Note that the sample obtained for each instrument will be weighted so that it accurately represents the universe of students, defined by age and disability category, from which the NLTS2 sample was selected, regardless of response rate.

Table 1. Response Rates for the Parent/Youth NLTS2 Instruments

	Eligible Students	Number with Completed Instrument	Response Ratea
Wave 1 Parent interviews/mail survey	11,244a	9,230	82.09%
Wave 2 Parent interview/youth interview/survey	11,226b	6,859	61.10%
Wave 3 Parent interview/youth interview/survey	11,225c	5,657	50.40%
Wave 4 Parent/youth interview/survey	11,128d	5,570	50.06%

^a 26 deceased youth were eliminated from the pool of eligible sample members, reducing that pool from 11,270 originally selected members to 11,244.

^b 44 deceased youth were eliminated from the pool of eligible sample members in Wave 2, reducing that pool from 11,270 originally selected members to 11,226.

^c 45 deceased youth were eliminated from the pool of eligible sample members in Wave 3, reducing that pool from 11,270 originally selected members to 11,225.

^d 142 deceased youth were eliminated from the pool of eligible sample members in Wave 4, reducing that pool from 11,270 originally selected members to 11,128.

Implications for Statistical Precision

The NLTS2 sampling plan (available at http://www.nlts2.org/pdfs/final_sampling_plan.pdf) estimated the needed student sample using the following assumptions:

• Estimates in Year 9 (the fifth and final wave of data collection for the parent/youth interviews) should have standard errors of no more than 3.6% for the largest categories of disability (learning disabilities, speech impairments, emotional disturbances, mental retardation, hearing impairments, and other health impairments. Other categories are expected to range from 3.8% (visual impairments) to 8.2% and 10.1% for the very small categories of traumatic brain injuries and deaf-blindness.

• Ten percent of the initial sample would not have good contact information and, thus, would have no data from any instrument.

• Attrition would be 8% per year (i.e., sample members lost due to out-of-date contact information) of those with initial contact information.

• The parent/youth interview response rate would be 70% of the available sample (i.e., sample remaining after attrition) in a given wave.¹

With a starting sample of 11,270, these assumptions would produce the available sample indicated in column A of Table 2 for each year of the study and the number of completed parent/youth interviews indicated in Column B. The data indicate that 3,643 parent/youth interviews would be needed in year 9 (wave 5) to achieve the precision levels desired. Column C indicates the actual number of parent interviews completed in Waves 1 through 3.

Table 2. Expected and Actual Number of Parent and Youth Interview/Surveys to Date

	A	B	C
Study Year/Wave	Expected “Live” Sample	Expected Number Completed Parent and Youth Interview/Surveys	Actual Number Completed Parent and Youth Interview/Surveys
1 (Wave 1)	10,143	7,100	9,230
2	9,332
3 (Wave 2)	8,585	6,010	6,859
4	7,898
5 (Wave 3)	7,266	5,086	5,657
6	6,685
7 (Wave 4)	6,150	4,305	5,570
8	5,658
9 (Wave 5)	5,205	3,643

The number of Wave 1 parent interviews exceeded the expected number by 30%, the actual number of completes in Wave 2 exceeded expectations by 14%, the actual number of completes in Wave 3 exceeded expectations by 11.2%, and the actual number of completes in Wave 4 exceeded expectations by 29.4%. Thus, the study is going into Wave 5 with a higher number of sample members with completed interviews than was expected in order to reach the desired precision level in Wave 5. Because having had a previous interview increases the chances of completing a subsequent interview (because information on location and on third-party contacts through whom a youth’s location could be traced), there is a high likelihood that subsequent waves of interviewing will continue to reap more than the expected number of completed interviews. This likelihood is further increased by the fact that the incentive plan approved by OMB (which was not available in Waves 1 and 2) permits payments of $20 for each completed parent and youth interview. This should help achieve or exceed the response rates required to reach the required number of completed interviews in Wave 5, suggesting the statistical precision requirements of the study will be met.

Implications for Potential Bias

Although, as noted above, response rate and response bias are conceptually independent (i.e., it is possible to generate an unbiased, representative sample even with a relatively low response rate), the risk of bias increases as response rate decreases. To reduce the likelihood of bias, the NLTS2 sample for each instrument in each wave is weighted to represent the distribution on the key factors of disability category, age, and race/ethnicity of students with disabilities in the universe, as reported by states to OSEP for their entire special education population. No other items in the limited dataset on the universe of students receiving special education are common to NLTS2, so there are not additional factors from a known universe that could be compared to test for bias or to develop or adjust weights.

Other than the variables in the OSEP report to the states, the closest approximation to the universe that can be used to assess potential bias in the Wave 4 Parent/Youth interview/survey are the responses to the NLTS2 Wave 1 parent interview. We previously used Wave 1 Parent data to examine Wave 3 Parent/Youth responses for possible biases. On the basis of those results we modified our weighting algorithm to include consideration of household income, parental volunteering, and being held back a grade. That is, our weighting algorithm for Wave 4 attempts to replicate the national distribution of disability category, age, and race/ethnicity and the distribution of household income, parental volunteering and being held back a grade that we found in our Wave 1 data.

The objective of this analysis is to compare our Wave 4 respondents with our Wave 4 eligibles on both a weighed and unweighted basis with respect to their Wave 1 responses. The preliminary step in performing this analysis is to identify key variables from the NLTS2 Wave 1 parent interview that reflect or help shape students’ school experiences and outcomes. Those variables include disability category; age; gender; household income; race/ethnicity; school type; school experiences; parental involvement, satisfaction and expectations; students school work quality; family support score; and social skills score.

The second step is to categorize the NLTS2 participant population according to whether or not a student (1) had a Wave 1 parent interview, (2) was eligible for the Wave 4 Parent/Youth interview/survey, and (3) was a respondent to that survey. Ineligibility is narrowly defined as deceased. Table 3 shows the six mutually exclusive categories into which a student could be classified (excluding 26 students who were ineligible for the Wave 1 interview). Cells are labeled G1 (for Group 1) to G6. The table shows the number of students in each cell.

Each student in Table 3 represents a set of students in the universe; if both instruments had been administered to every student in the universe, the universe also would be divided into the categories in Table 3. The original weights for the Wave 4 Parent/Youth Survey projected all students in groups G3 and G4 to represent all students in the universe in groups G3 through G6.

Table 3. Distribution of Students to the Wave 1 Parent Interview/Survey and Wave 4 Parent and Youth Interview/Survey^a

	Respondents to Wave 1 Parent Interview	Nonrespondents to Wave 1 Parent Interview
Ineligible for Wave 4 Parent and Youth Interview/Survey but alive at Wave 1	G1 = 142	G2 = 0
Respondents to Wave 4 Parent and Youth Interview/Survey	G3 = 5,359	G4 = 211
Nonrespondents to Wave 4 Parent and Youth Interview/Survey	G5 = 3,740	G6 = 1,820

^a Excludes students who were ineligible for Wave 1.

For purposes of this nonresponse weighting analysis, two alternative weights were developed for participants in the Wave 4 Parent/Youth interview/survey. One set of weights (denoted as the G3 weights) project students in G3 to the portion of the universe represented by G3 through G6. The second set of weights (denoted as the G35 weights) project the students in G3 and G5 to the portion of the universe represented by G3 through G6.

Responses to the key questions from the Wave 1 Parent survey have been tabulated in four ways: using (1) the G3 group without weights, (2) the combination of the G3 and G5 group without weights, (3) the G3 group and the G3 weights, and (4) the combination of the G3 and G5 group and the G35 weights. The comparison of tabulations 1 and 2 (i.e., unweighted comparisons) can be used to assess the extent to which there is nonresponse bias before any weighting adjustments are made. The comparison of tabulations 3 and 4 (i.e., weighted comparisons) can be used to assess the extent to which there is nonresponse bias after weighting adjustments are made. For example, if Hispanic parents are disproportionately nonrespondents to the Wave 1 Parent Survey, this would be reflected in differences between tabulations 1 and 2, but not in differences between tabulations 3 and 4 because race/ethnicity is one of the variables considered in the weighting process.

The amount of bias caused by nonresponse in G5 can be estimated using the formula:
Bias = MG35 – MG3 where MG35 is the mean value for the key variable using the G35 weights and MG3 is the mean value for the key variable using the G3 weights.

When we examined the weighted results using our Wave 3 weighting algorithm we found that after weighting there were two remaining biases (differences in weighted percentages between all eligibles for Wave 4 and all respondents to Wave 4 of 3% or greater). Comparing Wave 4 and Wave 1 on a weighted basis, we found that in Wave 4 we had: (a) fewer students who had been suspended or expelled at Wave 1 (a difference of 3.0%), and (b) more students whose parents strongly agreed that their child was getting supports from the school that he/she needed at Wave 1 (a difference of 3.3%).

To eliminate these two residual differences, we adjusted the weights to equalize the distribution of parental opinions concerning whether their child was getting supports from the school that he/she needed at Wave 1. We considered "did not provide an answer" to be a legitimate category, in the sense that if these questions had been asked of all students with disabilities in the universe, a certain percentage would not provide an answer. We then recalculated weights using Deming's algorithm so that the marginal totals for the weighted data approximated the estimated totals for the universe. Table 4 shows both the unweighted results and the results using the extended weights.

Table 4. Comparison of Wave 4 Parent and Youth Interview/Survey Eligible Population and W4 Parent and Youth Interview/Survey Respondents on their Wave 1 Parent Interview/Survey Responses

	Unweighted		Weighted*
	W4 Respondents	W4 Eligible	W4 Respondents	W4 Eligible
Disability category
Learning disability	8.3	9.6	62.1	62.1
Speech/language impairment	8.8	9.5	4.0	4.0
Mental retardation	8.8	9.4	12.2	12.2
Emotional disturbance	7.3	9.1	11.3	11.3
Hearing impairment	9.3	9.4	1.3	1.3
Visual impairment	8.0	7.4	0.5	0.5
Orthopedic impairment	10.6	9.8	1.2	1.2
Other health impairment	10.4	9.9	4.6	4.6
Autism	12.2	10.1	0.7	0.7
Traumatic brain injury	4.1	4.1	0.3	0.3
Multiple disabilities	10.2	9.9	1.8	1.8
Deaf-blindness	2.1	1.8	0.2	0.2
Gender = male	64.8	64.8	65.3	66.9
Age on 7/15/2001
13 or 14	33.8	34.2	31.6	30.5
15	25.5	24.8	21.7	23.9
16	25.0	25.1	27.4	26.5
17	15.7	15.9	19.3	19.0
Household income
$25,000 or less	27.8	31.6	33.2	33.3
$25,001 to $50,000	27.2	27.3	27.3	26.8
More than $50,000	37.5	31.3	30.3	30.3
No value provided by survey respondent	7.6	9.8	9.3	9.6
Race/ethnicity
White	66.4	62.5	62.1	61.8
African-American	18.2	20.7	20.5	21.0
Hispanic	12.5	13.5	14.4	14.3
School type
Attends regular school for general population	81.6	81.3	91.6	91.6
Attends neighborhood school	61.0	61.8	71.4	71.7
School experiences
Has ever been held back a grade	32.1	32.8	35.8	36.2
Has ever been suspended or expelled	23.8	27.2	30.3	32.7
Parent has been through mediation over special education services	13.0	12.7	8.5	10.6
In Wave 1 school year, parent belongs to a group for parents of students with disabilities	19.1	16.5	9.3	9.1
Parent’s agreement that student is getting supports from school he/she needs
Strongly agree	30.1	29.5	27.1	27.1
Disagree/strongly disagree	19.5	19.9	21.0	20.9
Parent’s satisfaction with child’s school
Very satisfied	45.4	43.6	39.0	37.0
Somewhat/very dissatisfied	18.5	19.4	19.7	20.5

See notes at end of table.

Table 4. Comparison of Wave 4 Parent and Youth Interview/Survey Eligible Population and W4 Parent and Youth Interview/Survey Respondents on their Wave 1 Parent Interview/Survey Responses (concluded)

	Unweighted		Weighted*
	W4 Respondents	W4 Eligible	W4 Respondents	W4 Eligible
In Wave 1 school year, parent:
Attended general school meeting	78.3	76.6	77.3	77.0
Volunteered at school	28.5	25.6	24.1	23.8
Went to IEP meeting	91.8	90.6	86.4	87.8
Parent wanted to be more involved in decisionmaking at IEP meeting	30.5	32.9	33.8	33.7
Expectations of student's postsecondary education
Definitely will	25.9	25.5	24.1	25.5
Probably/definitely won’t	43.1	41.9	36.6	37.5
Youth's overall academic achievement
Excellent or above average	40.0	37.9	28.1	28.0
Average	38.0	38.3	42.4	43.2
Below average or failing	22.0	23.8	29.6	28.8
Family support score [scale of 2-8]
2-5 (Low)	26.0	27.2	28.6	30.9
6.0	27.8	27.0	32.9	30.2
7.0	21.5	21.3	20.0	19.0
8 (high)	24.7	24.5	18.6	29.9
Social skills score [scale of 0-22]
0-10 (low)	22.0	21.6	17.0	17.4
11-16 (medium)	54.8	55.4	61.3	59.9
17-22 (high)	23.2	23.0	21.7	22.7

Shaded comparisons in bold are differences of 3.0 or more percentage points.

* Weights include adjustment for level of parental agreement to statement that their child is receiving the services and support from the school that he or she needs.

As seen in Table 4, prior to weighting, there were modest differences between respondents to the Wave 4 and Wave 1 interviews. Compared to Wave 1, Wave 4 respondents were: (1) less often from low ($25,000 or less) income households (27.8% vs. 31.6%), (2) more often from high (more than $50,000) income households (37.5% vs. 31.3%), (3) more often White (66.4% vs. 62.5%), and (4) less often from families of students who had been suspended or expelled by Wave 1 (23.8% vs. 27.2%).

Weighting on disability category, age, and race/ethnicity of students with disabilities, household income, parental volunteering, being held back a grade, and adequacy of school services reduced all differences to 2.7% or less.

Overall, the bias analysis is encouraging. Prior to weighting the differences between Wave 4 and Wave 1 respondents, although large enough to require attention, were not so large as to invalidate survey results. Weighting on characteristics known for the universe and some critical characteristics of the Wave 1 respondents reduces all differences to less than 3%.

Cumulative Response Rate

A power analysis indicated that a total of 497 local education agencies (LEAs), stratified by region, district size (student enrollment), and community wealth (Orshansky percentile), was the appropriate sample for NLTS2. A total of 501 LEAs provided rosters from which to select students for the second stage sample, meeting both the requirements of LEA sample size and distribution across the sampling grid. A total of 3,634 LEAs were selected from the universe of those serving students with disabilities in the NLTS2 grade range and invited to participate to generate the needed sample of 501 LEAs, or 13.8% of the number invited. Using this as the first-stage response rate, the cumulative response rates for the Wave 4 Parent/Youth instrument is 13.8% x 50.0% = 6.9%.

As was discussed earlier, it is possible to generate an unbiased, representative sample even with a relatively low response. Analyses comparing the universe of LEAs and the LEA sample, both weighted and unweighted, on variables used in stratification revealed that the weighted LEA sample closely resembled the LEA universe with respect to those variables. To further confirm the representativeness of the NLTS2 LEA sample, OMB directed the Office of Special Education Programs to complete a nonresponse bias study; it was conducted in two stages. The first stage involved analyses of extant databases to determine whether variations in LEA characteristics contribute meaningfully to explaining variations in student-level experiences and outcomes. The second stage involved selecting a nationally representative sample of LEAs and conducting a telephone survey of those LEAs and LEAs participating in NLTS2 to compare various aspects of their special education policies and procedures. The results of both stages support the conclusion that bias in the NLTS2 LEA sample is not a significant issue. It appears to be a nationally representative sample of LEAs from which a nationally representative sample of students was selected, meeting the goals and technical requirements of the NLTS2 sampling plan.

1This rate assumes either the parent or youth interview is completed.

SRI International 333 Ravenswood Avenue Menlo Park, CA 94025