Justification rev

Volume I

Fast Response Survey System (FRSS) 107: Programs and Services for High School English Learners 2015

OMB# 1850-0733 v.31

June 30, 2015

Revised July 14, 2015

National Center for Education Statistics

U.S. Department of Education

Justification

The National Center for Education Statistics (NCES), within the U.S. Department of Education (ED), requests OMB approval under the NCES system clearance for the Quick Response Information System (QRIS) (OMB# 1850-0733) to conduct data collection for the Fast Response Survey System (FRSS) survey #107 on programs and services for high school English learners (ELs). The survey will provide nationally representative data, with a First Look report on the results to be released in the fall of 2016. The Office of English Language Acquisition (OELA) in the U.S. Department of Education requested that NCES conduct this FRSS survey.

According to the Center for Applied Linguistics, English Learners (ELs) are the fastest growing segment of the K-12 student population and they often struggle to be successful in school. A 2005 report by the Center for Adult English Language Acquisition notes that as high school exit criteria increase in rigor, many ELs are often unable to graduate on time and are turning to adult education programs to earn high school diplomas, acquire job skills, and improve their English language proficiency. However, little is known about the ways in which public school districts serve older adolescent and young adult ELs. The purpose of this FRSS survey is to collect the first nationally representative data from school districts on programs and services designed to serve high school ELs. Topics include instructional programs/approaches provided for high school ELs, the presence and characteristics of newcomer programs, use of online or computer-based programs to address the needs of English learners, participation of high school ELs in various district programs and services (e.g., summer school, tutoring, career and technical training), presence of programs or services designed specifically for ELs in high school, materials and services that the district has available in native languages for high school ELs and their parents/guardians, use of native language for content instruction and for instructional support, information about types of educational programs or services that the district provides to ELs ages 18 to 21 seeking to newly enroll in the district, and the extent to which the district considers various factors (e.g., English proficiency level, literacy in native language) when providing information about educational programs or services available to ELs ages 18 to 21 who are seeking to newly enroll in the district.

NCES is authorized to conduct the FRSS survey by the Education Science Reform Act of 2002 (ESRA 2002, 20 U.S.C. § 9543). NCES has contracted Westat to collect data for all stages of this survey.

Design

Overview of Survey Development

FRSS has established procedures for developing short surveys on a wide variety of topics. The techniques used to shape the survey design for FRSS 107 included literature reviews on EL programs, input from and review by the NCES Quality Review Board (QRB), three rounds of feasibility calls, and a pretest.

The current survey reflects lessons learned from topics and issues identified through literature review, with modifications based on three rounds of feasibility calls and a pretest with public school district personnel most knowledgeable about high school EL programs. The first round of feasibility calls was conducted with 12 districts in August and September 2014 (OMB# 1850-0803 v.109). Because this is a new survey topic, the first round of calls used an open-ended interview guide to learn more about the EL populations ages 14 to 21 that school districts serve, the terminology districts use regarding these populations, and the characteristics of the EL programs and services they provide at the high school level. The second round of feasibility calls was conducted with 13 districts in October and November 2014 and the third round of feasibility calls was conducted with 15 districts in January and February 2015. During the second and third rounds of feasibility calls, respondents were asked to review but not complete draft survey questions, instructions, and definitions based on the initial round of feasibility calls. Respondents then participated in a short telephone interview with Westat to provide feedback on the questionnaire. The resulting draft of the questionnaire was then reviewed by the NCES QRB and revised accordingly to prepare it for the pretest.

Pretest calls with 12 districts were conducted in May and June 2015 (OMB# 1850-0803 v.132). For the pretest, respondents were asked to complete the questionnaire and participate in a telephone debriefing with Westat to provide feedback on the questionnaire. Completed questionnaires were collected by fax prior to the debriefing with each respondent. The purpose of the pretest was to verify that all questions and corresponding instructions were clear and unambiguous, to determine if the information would be readily accessible to respondents, and to determine whether the burden on respondents could be reduced further. Changes to the questionnaires were made based on the feedback received from the pretest, and documented in a memorandum summarizing the pretest results (Attachment 6). The revised questionnaire (Attachment 1) is being submitted with this request for OMB clearance.

NCES Review and Consultations Outside of Agency

The NCES QRB members reviewed a draft list of questionnaire topics prior to the submission of the OMB package for the feasibility calls. Revisions were made to the list of topics based on input from the reviewers, and the list was used to develop an interview guide for the feasibility calls. During the second round of feasibility calls, a draft questionnaire was developed with input from OELA and OCTAE. During the later rounds of the feasibility calls, revisions were made to the draft questionnaire with input from OELA. Following the last round of feasibility calls, the QRB reviewed the draft questionnaire, and revisions were made based on their input. The revised version was used for the pretest.

In addition to staff from each of the three Divisions at NCES, the QRB also included staff from OELA; the Office of Career, Technical, and Adult Education (OCTAE); and the Office of Postsecondary Education (OPE). The QRB members for this survey are listed below:

Melissa Escalante, OELA

Carlos Martinez, OELA

Debra Suarez, OCTAE

Ricardo Hernandez, OCTAE

Lenore Garcia, OPE

Chris Chapman, NCES

Eugene Owen, NCES (PIAAC)

Arnold Goldstein, NCES (NAEP)

Grady Wilburn, NCES (NAEP)

Richard Reeves, NCES (IPEDS)

Sharon Boivin, NCES (ATES)

Kashka Kubzdela, NCES

Assurance of Confidentiality

Data to be collected will not be released to the public with institutional or personal identifiers attached. Data will be presented in aggregate statistical form only. In addition, each data file undergoes extensive disclosure risk analysis and is reviewed by the NCES/IES Disclosure Review Board before use in generating report analyses and before release as a public use data file. Respondents will be assured that their participation in the survey is voluntary and that their answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573).

Description of Sample and Burden

The proposed sample design is a nationally representative sample of approximately 1,700 regular public school districts with high schools from the 2012–13 (or most recent) NCES Common Core of Data (CCD) Local Education Agency Universe File. The questionnaire is limited to three pages of items readily available to respondents and can be completed by most respondents in about 30 minutes.

Any special requirements that districts have for approval of surveys will be met before those districts are contacted. Each of the approximately 15 districts that require special approval for district surveys has unique requirements for obtaining approval. The materials sent to special districts will be tailored to meet the specific requirements of each district, consistent with the materials included in this OMB package. For example, most districts request information on survey justification, confidentiality, sample size, and survey collection procedures, which will be copied from the appropriate sections of the OMB package after its approval.

Questionnaire packages, including information needed to access the Web survey, will be mailed to the superintendent of each sampled district in September 2015. The cover letter and questionnaire will include a description of the most appropriate respondent. Follow-up for nonresponse will be conducted both by mail and telephone and will begin about 3 weeks after the questionnaires have been mailed to the districts. Experienced telephone interviewers will be trained to conduct the nonresponse follow-up and will be monitored by Westat supervisory personnel. Telephone nonresponse follow-up is used to prompt respondents to complete the survey by web, mail, or fax and is expected to take about 5 minutes.

Based on previous FRSS studies, it is anticipated that approximately 15 districts with special clearance procedures will be contacted (table 1). The respondent burden is estimated to be on average 2 hours per special district. The estimated burden time for 1,700 districts to review the introductory letter requesting their participation (initial contact) is 5 minutes per district. Assuming a response rate of 90 percent, the initial sample of 1,700 districts will yield about 1,530 completed questionnaires, with a response burden of approximately 30 minutes per completed questionnaire¹. It is also anticipated that about 75 percent of the districts will receive a nonresponse follow-up call that will take about 5 minutes.

Table 1. Estimated burden for data collection and nonresponse follow-up for 1,700 districts: FRSS 107

Type of collection	Sample size	Estimated response rate (percent)	Estimated number of respondents	Estimated number of responses	Burden hours per respondent	Total respondent burden hours
Special clearance district review	15	100%	15	15	2.00	30
Initial district contact	1,700	100%	1,700	1,700	.083	141
Questionnaire	1,700	90%	1,530	1,530	.50	765
Nonresponse follow-up call to school	1,700	75%	1,275	1,275	.083	106
Total burden	-	-	1,700	4,520	-	1,042

Procedures and Data Collection Instrument

A questionnaire, cover letter (Attachment 2), and web information sheet (Attachment 3) will be mailed to each sampled district. The cover letter requests the participation of the district and introduces the purpose and content of the survey. It also notes that the survey should be completed by the person(s) in the district most knowledgeable about programs and services for English learners at the high school level. The cover letter includes instructions on how to complete and return the survey, as well as contact information in case of questions. The web information sheet is included in the mailing to provide information about the option to complete a web version of the survey. On the cover of the survey and in the cover letter, respondents are assured that their participation is voluntary and their answers may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573).

If a completed survey is not received for a sampled district within 3 weeks after the initial mailing, the district will receive a nonresponse follow-up letter (Attachment 4), another copy of the district’s web information sheet, and a brief, scripted telephone call (Attachment 5) prompting the respondent to return a completed survey via the web, fax, or mail.

Questionnaire

The questionnaire is designed to collect information from public school districts on programs and services for high school English learners (ELs) during the 2015-16 school year.

Question 1 asks whether the district currently enrolls any English learners at the high school level.

Question 2 asks for the current total number of high school English learners enrolled in the district.

Question 3 asks about the English learner instructional programs/approaches that the district currently provides for English learners in high school. Programs/approaches include bilingual instruction for ELs in content classes, two-way bilingual/dual language program in content classes, English as a second language (ESL) instruction in scheduled class periods, ESL push-in or pull-out instruction, instructional support by a paraprofessional who does not speak the student’s native language, instructional support by a paraprofessional who speaks the student’s native language, and sheltered English/content instruction.

Question 4 asks whether the district has a newcomer program for English learners in high school. Questions 5 through 8 ask about the characteristics of the newcomer program, including whether it is designed to serve a specific group of newly arrived students (such as those with limited or interrupted formal education), the structure of the newcomer program, and the typical length of time a high school student spends in the newcomer program.

Question 9 asks whether high school English learners in the district work with online or computer-based programs in various areas to address any of their needs as English learners. Areas include English language acquisition, English language and literacy instruction, content area instruction, native language support in content area instruction, and organizational and study skills.

Question 10 asks districts approximately how many high school English learners participate in various programs or services, with the response options of none, few, some, most, and don’t know. The programs and services are summer school, remediation classes, credit recovery course/program, flexible scheduling, alternative school/program for at-risk students, career and technical training, distance education course/program, district-administered GED courses, tutoring, and mentoring program.

Question 11 asks whether the district has various programs or services designed specifically for English learners in high school. The programs and services are tutoring, summer school, credit recovery course/program, mentoring program, and distance education course/program.

Question 12 asks about materials and services that the district has available in native languages for high school English learners and their parents/guardians. Districts are asked to report separately for materials and services in the most common native language of ELs in the district, and for materials and services available in other native languages of ELs in the district. Materials and services include written information about high school academic programs in the district, written information about high school career and technical education programs in the district, translation service upon request for printed materials, and interpreters upon request for school meetings or calls.

Question 13 asks districts for the approximate number of high school English learners with their native language used for content instruction, and the approximate number with their native language used for instructional support only. The response options are no students, few students, some students, and most or all students. Districts are asked to report separately for high school ELs whose native language is the most common native language of ELs in the district, and those who native language is another non-English language in the district.

Question 14 asks districts about how often in the last 12 months English learners ages 18-21 have newly enrolled in the district as a high school student. The response options are never, rarely, sometimes, often, and don’t know.

Question 15 asks whether the district provides English learners ages 18 to 21 who are seeking to newly enroll in the district with information about various educational programs and services. Programs and services include academic programs at the regular high school, alternative school or program for at-risk students, district-administered newcomer program, career and technical training offered by the public school district, career and technical training offered by other entities, GED or adult education programs offered by the public school district, GED or adult education programs offered by other entities, and free or low-cost English classes.

Question 16 asks about the extent to which the district considers various factors when providing information about educational programs or services available to English learners ages 18 to 21 who are seeking to newly enroll in the district. The response options are not at all, minor extent, moderate extent, and major extent. The factors include English proficiency level, literacy in their native language, limited or interrupted formal education, length of time needed to accrue sufficient credits to graduate, whether the student will be able to meet high school graduation requirements in content area classes, whether the student will be able to pass state tests required for graduation, and the age of the student at time of enrollment.

Survey Cost and Time Schedule

The survey is estimated to cost the federal government about $730,000, including about $680,000 for contractual costs and $50,000 for salaries and expenses. Contractual costs include the costs for survey preparation, data collection, data analysis, and report preparation.

Mailing of the survey will begin in September 2015, and about 3 weeks later, telephone follow-up for nonresponse will begin. Data collection is scheduled to end about 20 weeks after initial mailout.

Plan for Tabulation and Publication

The First Look report will be released on the NCES website in the fall of 2016 and include explanatory text and tables. Participating districts will be notified when NCES releases the report. A public use data file will also be released on the NCES website. Survey responses will be weighted to produce national estimates. Tabulations will be produced for each data item. Cross-tabulations of data items will be made with selected classification variables, such as district enrollment size, community type (locale), geographic region, poverty level, and concentration of ELL students in the district.

Statistical Methodology

Reviewing Statisticians

Chris Chapman, of NCES, is the Project Officer for this survey. Adam Chu, Senior Statistician, Westat, was consulted about the statistical aspects of the design.

Respondent Universe and Sampling Frame

The proposed survey will collect data on programs and services for English learners provided to high school students from a nationally-representative sample of public school districts. The sampling frame (i.e., universe list) from which the sample will be drawn will be constructed from the most recent NCES Common Core of Data (CCD) Local Education Agency (LEA) Universe File.² The CCD file contains a record of all known public school districts along with selected characteristics such as enrollment size by grades offered, urbanicity (type of locale), region of country, and poverty level. The file also contains a district-level variable (ELL) on the number of students served in ELL programs. Although the ELL counts in CCD are not available for individual grades, they can nonetheless provide a useful overall measure of the expected concentration of ELL students in each school district.

Table 2 shows that there are 11,405 regular school districts with grades 11-12 in the 2012–13 CCD universe file, of which almost 12 percent enroll more that 10 percent of their students in ELL programs. In general, districts with relatively high concentrations of ELL students tend to be more prevalent in large and urban school districts, in the western region, and in districts with relatively high proportions of children in poverty. Note that the counts of districts in Table 2 pertain only to the “regular” school districts in the CCD file (i.e., the type 1 and type 2 local school districts). Excluded from the frame are all other types of districts such as supervisory unions, regional educational services agencies, and state or federal agencies serving special needs populations. Also excluded are school districts operating in the outlying U.S. territories.

Table 2. Distribution of districts with grades 11–12 in the 2012–13 CCD LEA Universe File by size class and percentage of students in ELL programs

Enrollment size class	Percent of district enrollment in ELL programs
	Missing*	Under 10%	10% or more	Total
<1,000
Number	35	4,439	367	4,841
Row percent	0.7%	91.7%	7.6%	100.0%
1,000–2,499
Number	53	2,675	232	2,960
Row percent	1.8%	90.4%	7.8%	100.0%
2,500–9,999
Number	74	2,293	392	2,759
Row percent	2.7%	83.1%	14.2%	100.0%
10,000+
Number	12	488	345	845
Row percent	1.4%	57.8%	40.8%	100.0%
TOTAL
Number	174	9,895	1,336	11,405
Row percent	1.5%	86.8%	11.7%	100.0%

* Data are missing in CCD. Districts with missing data will be assigned to an appropriate category for sampling purposes.

Sample Design and Stratification

Traditionally, surveys conducted under the FRSS have employed stratified samples ranging in size from 1,200 to 1,800 districts depending on analytic goals and available resources. Since FRSS is designed to provide estimates for broadly-defined subgroups of interest as well as overall national estimates, a stratified sample design with primary strata defined by size class and other relevant characteristics has been found to be generally effective in meeting study objectives. Specification of explicit strata for sampling purposes allows for the selection of districts at varying rates to (a) ensure that key subgroups are adequately represented in the sample and (b) improve sampling precision for selected subgroup estimates. Moreover, use of enrollment size as the primary stratifier also helps to ensure that sample-based estimates that are correlated with the size of the district (e.g., the estimated numbers of students in ELL programs of specified types) can better achieve reasonable levels of precision.

In view of the above considerations, we will select a stratified sample of 1,700 districts for the FRSS survey with strata defined by cross-classifying districts in the sampling frame by (a) enrollment size class (i.e., the following six size classes: [1] under 1,000 students; [2] 1,000 to 2,499; [3] 2,500 to 9,999; [4] 10,000 to 24,999; [5] 25,000 to 99,999; and [6] 100,000+) and (b) ELL status (e.g., [1] under 10% of students enrolled in ELL and [2] 10% or more students in ELL). The total sample size will be allocated to the strata in proportion to the aggregate square-root of the enrollment of the districts in the size class. Since most estimates to be derived from the survey will be categorical (e.g., the estimated proportion of districts with a specified characteristic), use of the square root of enrollment rather than enrollment as the measure of size for sample allocation will limit the design effects (and associated increased variances) arising from the use of varying sampling rates. Other variables such as type-of-locale and region will be used to sort districts in the sampling frame prior to sample selection. The sorting is a form of “implicit” stratification that helps ensure that districts with the given characteristics are appropriately represented in the sample. Within each sampling stratum, districts will be selected systematically and with equal probability from the sorted list of districts. Note that under the proposed stratification, all districts with 100,000 or more students will be included in the sample with certainty. Table 3 summarizes the proposed allocation of the sample of 1,700 districts to strata.

Table 3. Distribution of district sample by size class and percentage of students in ELL programs

Enrollment size class	Percent of district enrollment in ELL programs
	Missing*	Under 10%	10% or more	Total
<1,000
Number	2	296	24	323
Row percent	0.7%	91.8%	7.6%	100.0%
1,000–2,499
Number	7	337	30	374
Row percent	1.8%	90.2%	8.0%	100.0%
2,500–9,999
Number	16	489	89	594
Row percent	2.8%	82.2%	15.0%	100.0%
10,000–24,999
Number	2	129	84	215
Row percent	1.1%	60.0%	38.9%	100.0%
24,999–99,999
Number	4	86	78	167
Row percent	2.1%	51.3%	46.6%	100.0%
100,000+
Number	0	11	17	28
Row percent	0.0%	6.6%	10.2%	16.8%
TOTAL
Number	31	1,347	322	1,700
Row percent	1.8%	79.2%	18.9%	100.0%

Expected Level of Precision

Table 4 summarizes the expected sample sizes and levels of precision for selected subgroup estimates derived from the proposed sample design. The numbers of “responding districts” shown in the table are calculated assuming an overall response rate of 90 percent. Also shown are 95% confidence bounds around an estimated percentage derived from the respondent samples. The confidence bounds given in the table are for reported respondent characteristics ranging from a 20% characteristic to a 50% characteristic. As can be seen in the table, for subgroups with at least 300 respondents, estimates are expected to be relatively precise with 95% confidence bounds ranging from ± 2.5% to 6.6% depending on the magnitude of the proportion being estimated. Moreover, under the proposed sample design, the minimum detectable difference (MDD) in estimated percentages between subgroups of approximately 300 respondents each would be about 10% (e.g., using a t test to test for significance).

Table 4. Expected sample sizes (number of completed interviews) and 95% confidence bounds around an estimated proportion by selected subgroups under proposed design

Subgroup	No. selected	No. resp. districts	95% Conf. Bounds Around Estimated Percentage Equal To:
			P = 20%	P= 33%	P = 50%
Size class
<1,000	323	290	±4.61%	±5.42%	±5.76%
1,000 to 2,499	374	336	±4.28%	±5.03%	±5.34%
2,500 to 9,999	594	535	±3.40%	±3.99%	±4.25%
10,000 +	410	369	±4.23%	±4.98%	±5.29%
ELL status
Low (<10%)	1,347	1,212	±2.64%	±3.11%	±3.30%
High (10%+)	322	290	±6.11%	±7.19%	±7.64%
Locale
City	248	223	±5.91%	±6.95%	±7.39%
Suburb	571	514	±3.92%	±4.61%	±4.90%
Town	326	293	±4.84%	±5.68%	±6.04%
Rural	556	500	±3.82%	±4.50%	±4.78%
Region
Northeast	351	316	±5.08%	±5.97%	±6.35%
Southeast	332	299	±5.30%	±6.23%	±6.63%
Central	525	473	±4.09%	±4.81%	±5.11%
West	492	443	±4.80%	±5.64%	±6.00%
Poverty level
< 10%	309	278	±5.53%	±6.50%	±6.92%
10 to 19.9%	627	564	±3.91%	±4.60%	±4.89%
20 to 29.9%	485	437	±4.59%	±5.39%	±5.73%
30% +	268	241	±6.11%	±7.18%	±7.63%
All districts	1,700	1,530	±2.41%	±2.84%	±3.02%

Estimation and Calculation of Sampling Errors

For estimation purposes, sampling weights reflecting the overall probabilities of selection and adjustments for nonresponse will be attached to each data record. To properly reflect the complex features of the sample design, standard errors of the survey-based estimates will be calculated using jackknife replication. Under the jackknife replication approach, 50-100 subsamples or "replicates" will be formed in a way that preserves the basic features of the full sample design. A set of estimation weights (referred to as "replicate weights") will then be constructed for each jackknife replicate. Using the full sample weights and the replicate weights, estimates of any survey statistic can be calculated for the full sample and each of the jackknife replicates. The variability of the replicate estimates is used to obtain a measure of the variance (standard error) of the survey statistic. Previous surveys, using similar sample designs, have yielded relative standard errors (i.e., coefficients of variation) in the range of 2 to 10 percent for most national estimates. Similar results are expected for this survey.

1 This estimate is the average amount of time district staff respondents reported the questionnaire took to complete during the pretest.

2 For example, see Keaton, P. (2014). Documentation to the NCES Common Core of Data Local Education Agency Universe Survey: School Year 2012-13 Provisional Version 1a (NCES 2015-008). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved from http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2015008.