Part B FRSS 109 Teacher Use of IT

Fast Response Survey System (FRSS) 109: Teachers’ Use of Technology for School and Homework Assignments

Supporting Statement Part B

OMB # 1850-0857 v.5

National Center for Education Statistics (NCES)

U.S. Department of Education

Institute of Education Sciences

Washington, DC

April 2018

Section B. Description of Statistical Methodology

B.1. Respondent Universe and Sample Design

The respondents for the proposed survey on teachers’ use of technology will include public school classroom teachers of grades 3 through 12 who teach one or more core academic subjects (e.g., English/language arts, math, science, social studies/social science). Teachers for the survey will be selected randomly from teacher lists to be compiled from a stratified sample of regular public elementary and secondary schools in the United States. The school sample will be selected from the most current Common Core of Data (CCD) Public School Universe File maintained by the National Center for Education Statistics (NCES). Vocational education, special education, alternative/other non-regular schools, and schools operated by the Department of Defense or Bureau of Indian Affairs are ineligible for the survey, as are schools with a high grade lower than 3,¹ ungraded schools, and schools in the outlying U.S. territories.

Using the definitions given in the 2014–15 CCD Public School Universe File, the following four categories of schools based on the low and high grades taught in the school will be defined for sample stratification purposes:

Primary – low grade of PK through 3 and a high grade of PK through 8

Middle – low grade of 4 through 7 and high grade of 4 through 9

High – low grade of 7 through 12 and high grade of 12 only

Other – all other schools not falling in the above three categories

Table 1 summarizes the distribution of regular schools, students, and full-time equivalent (FTE) teachers in the
2014–15 CCD Public School Universe File by instructional-level category and enrollment size class, indicating that 49 percent of all FTE teachers were in primary schools, 20 percent in middle schools, 27 percent in high schools, and 4 percent in schools with instructional level categorized as “other.”

Table 1. Distribution of schools, students, and teachers in the 2014–15 CCD Public School Universe File by instructional level and enrollment size class

Instructional level		Enrollment size class
	Population count	<300	300–499	500–999	1000–1499	1500+	Total
Primary	Schools	11,597	18,461	19,882	996	69	51,005
	Enrollment	2,181,513	7,430,869	13,070,501	1,128,980	122,261	23,934,124
	Teachers (FTE)	144,008	434,744	726,950	60,958	6,243	1,372,904
Middle	Schools	3,750	3,559	7,176	1,670	163	16,318
	Enrollment	649,895	1,423,175	5,158,438	1,947,752	280,587	9,459,847
	Teachers (FTE)	42,552	87,222	300,683	107,030	13,608	551,096
High	Schools	4,653	2,719	3,448	2,284	3,071	16,175
	Enrollment	719,494	1,071,816	2,468,613	2,824,071	6,452,532	13,536,526
	Teachers (FTE)	53,478	65,477	145,436	158,923	328,235	751,549
Other	Schools	1,771	739	926	254	182	3,872
	Enrollment	242,938	289,374	641,865	305,637	481,913	1,961,727
	Teachers (FTE)	18,340	17,717	35,317	16,314	19,876	107,565
TOTAL	Schools	21,771	25,478	31,432	5,204	3,485	87,370
	Enrollment	3,793,840	10,215,234	21,339,417	6,206,440	7,337,293	48,892,224
	Teachers (FTE)	258,378	605,160	1,208,386	343,226	367,963	2,783,113

B.2 Statistical Methodology

The sample design for the proposed teacher survey will be a stratified two-stage sample consisting of the selection of schools at the first stage of sampling and the selection of teachers within the sampled schools at the second stage. The primary strata for selecting schools will be defined by crossing the four instructional levels and five enrollment size classes given in Table 1. Within each primary sampling stratum, schools in the sampling frame will be sorted by type of locale (central city, suburban, town, rural) within size class, and categories of poverty level based on the percent of students eligible for free/reduced price lunch within type of locale. When used in conjunction with systematic sampling, the sorting will induce additional implicit substratification within the primary strata to ensure that relevant subcategories of schools are appropriately represented in the sample.

Selection of Schools

A total of 2,000 schools will be selected for the study. Roughly equal sample sizes will be allocated to the major instructional levels (primary, middle, and high) and a minimum of 150 schools will be allocated to the remaining “other” category. Such an allocation is approximately optimal for separate analyses of the three major instructional levels and comparisons between them, while ensuring that the other instructional levels are adequately represented in the sample for overall national estimates. Within each category of instructional level, the specified number of sample schools will be distributed to the five enrollment size classes indicated in Table 1 in proportion to the number of FTE teachers in the size class. After sorting the schools in the sampling frame by type of locale and poverty status category within each primary sampling stratum, the sample of schools will be selected with probabilities proportionate to the number of FTE teachers in the school. In addition to analyses by instructional level, the proposed sample sizes are expected to be large enough to permit limited analysis by type-of-locale and poverty status categories. Table 2 summarizes the number of schools to be selected for FRSS 109 by various subgroups and the corresponding numbers of participating schools (i.e., schools providing teacher lists for subsequent sampling purposes) assuming an overall school-level response rate of 85 percent.

Selection of Teachers

Participating schools will be requested to provide lists of their classroom teachers for subsequent sampling purposes. Eligible teachers are those with self-contained classrooms in grades 3 or higher (generally teaching in primary schools) and those with departmentalized classes in one or more of the core academic subjects (generally teaching in middle and high schools). On average 2.4 teachers will be randomly selected from each participating school for a total teacher sample size of 4,000. Based on experience in previous FRSS teacher surveys, we expect to obtain completed questionnaires from 85 percent of the sampled teachers, or about 3,400 completed questionnaires. The last two columns of Table 2 summarize the numbers of teachers to be selected and the corresponding expected numbers of respondents, respectively, for various subgroups of schools. Note that within each instructional level stratum, the teacher samples will be self-weighting (i.e., equal probability samples) if the number of eligible teachers reported by the schools is proportional to the FTE teacher count used to select the sample of schools. To the extent that the two sets of teacher counts depart appreciably from proportionality, the use of a fixed sample size per school will increase design effects and, hence, lower sampling precision. To minimize the impact of such design effects, it may be desirable on occasion to increase the sample size in some schools.

Table 2. Number of schools and teachers to be sampled and expected number of teachers completing questionnaires, by subgroup

Subgroup		Number of schools to be sampled	Expected number of participating schools*	Number of teachers to be sampled	Expected number of responding teachers*
Total		2,000	1,700	4,000	3,400
Instructional level
	Primary	616	524	1,232	1,047
	Middle	617	524	1,234	1,049
	High	617	524	1,234	1,049
	Other	150	128	300	255
Enrollment size class
	Under 500	553	470	1,106	940
	500 to 999	831	707	1,663	1,413
	1,000 or more	616	523	1,231	1,047
Type of locale
	City	576	490	1,152	979
	Suburban	764	649	1,528	1,299
	Town	242	205	483	411
	Rural	418	356	837	711
Percent of students eligible for free/reduced price lunch
	Under 35 percent	616	523	1,231	1,047
	35 to 49 percent	367	312	734	624
	50 to 75 percent	562	477	1,123	955
	75 percent or more	456	388	912	775
* Both school-level and teacher-level response rates are assumed to be 85 percent.

Expected Levels of Precision

Under the proposed sample design, the sampling precision of an estimated proportion derived from the survey (e.g., the estimated proportion of teachers who are very knowledgeable about their students’ access to computers outside of school) will be affected by two types of design effects. The total design effect, D_T, is the product of two design effects, D_c and D_w. The first is a design effect due to clustering. Such a design effect arises from the selection of approximately two teachers per school, which can lead to similar (correlated) responses from the sampled teachers for some survey items. This design effect can be expressed as D_c = (1 + rho), where rho = the intraclass correlation between teachers within schools. For purposes of estimating sampling precision under the proposed sample design, we speculate that rho = 0.05 for many types of items to be collected in the survey. The second type of design effect arises from variable sampling weights that could occur from (a) the use of a fixed sample size per school and (b) weighting adjustments to account for differential levels of nonresponse. As indicated previously, the use of a fixed sample size per school will not result in appreciable variable weighting effects within the major instructional levels unless there are large disparities between the measure of size (FTE teachers) used to select schools and the corresponding number of eligible teachers provided on the teacher lists. Together with the variation in weights introduced from the nonresponse adjustment process, we estimate that the weighting deign effect, denoted by D_w will be no greater than D_w = 1.10 for estimates by instructional level strata, and D_w = 1.20 for estimates that cover all instructional levels. Thus, depending on the subgroup being analyzed, the total design effect for an estimated proportion can be expected to range from D_T = 1.16 to D_T = 1.26. Based on these design effects, the standard errors of an estimated proportion to be expected under the proposed design are shown in Table 3. The standard errors can be converted to approximate 95 percent confidence bounds by multiplying the entries by 2. For example, from Table 3, an estimated proportion of the order of 20 percent (P = 0.20) based on a sample of 1,047 responding primary school teachers would be subject to a margin of error of ±2.6 percent at the 95 percent confidence level. An estimated proportion of the order of 50 percent (P = 0.50) based on a sample of 775 teachers in high poverty schools (schools where the percent of students eligible for free/reduced price lunch is 75% or greater) would be subject to a margin of error of ±4.0 percent at the 95 percent confidence level.

Table 3. Expected sample sizes for the teachers’ survey and standard errors of estimated proportions for selected subgroups

Subgroup		Responding teachers	Standard error for estimated proportion (P)
			P = 0.20	P = 0.33	P = 0.50
Total		3,400	0.008	0.009	0.010
Instructional level
	Primary	1,047	0.013	0.016	0.017
	Middle	1,049	0.013	0.016	0.017
	High	1,049	0.013	0.016	0.017
	Other	255	0.027	0.032	0.034
Enrollment size class
	Under 500	940	0.015	0.017	0.018
	500 to 999	1,413	0.012	0.014	0.015
	1,000 or more	1,047	0.014	0.016	0.017
Type of locale
	City	979	0.014	0.017	0.018
	Suburban	1,299	0.012	0.015	0.016
	Town	411	0.022	0.026	0.028
	Rural	711	0.017	0.020	0.021
Percent of students eligible for free/reduced price lunch
	Under 35 percent	1,047	0.014	0.016	0.017
	35 to 49 percent	624	0.018	0.021	0.022
	50 to 75 percent	955	0.015	0.017	0.018
	75 percent or more	775	0.016	0.019	0.020

Estimation and Calculation of Sampling Errors

For estimation purposes, sampling weights reflecting the overall probabilities of selection and adjustments for nonresponse will be attached to each data record. To properly reflect the complex features of the sample design, standard errors of the survey-based estimates will be calculated using jackknife replication. Under the jackknife replication approach, 50–100 subsamples or “replicates” will be formed in a way that preserves the basic features of the full sample design. A set of estimation weights (referred to as “replicate weights”) will then be constructed for each jackknife replicate. Using the full sample weights and the replicate weights, estimates of any survey statistic can be calculated for the full sample and each of the jackknife replicates. The variability of the replicate estimates is used to obtain a measure of the variance (standard error) of the survey statistic. Previous surveys, using similar sample designs, have yielded relative standard errors (i.e., coefficients of variation) in the range of 2 to 10 percent for most national estimates. Similar results are expected for this survey.

B.3. Methods for Maximizing the Response Rate

A number of public school districts have structured procedures that researchers must follow to obtain permission to contact their schools or teachers for a study. These districts’ cooperation is important in obtaining data that represents all public schools and teachers. Westat maintains a set of required materials for each special contact district, which will be updated as needed for this survey, and has a group of staff members that specializes in obtaining research approvals from these districts. Once the school sample is selected for FRSS 109, project staff will identify the relevant special contact districts and begin preparing research application materials to send to them (see Appendix A-1). FRSS 109 will follow a standardized approach to submit the applications as quickly as possible, and FRSS 109 staff will communicate with each district on a regular basis to check that the application is being processed and to respond to any additional requirements. The Westat survey manager and project director will monitor the progress of this activity weekly and add more staff to this task if needed.

In addition, the superintendent of each district with sampled schools will be sent a letter informing them about the survey (see Appendix A-1). This letter will be sent to all districts with sampled schools except for the special contact districts, because those are already being notified through the research application process per their specific requirements.

For collection of teacher lists, we will mail the request and materials to each sampled school (see Appendix B). We will begin telephone followup for nonresponse about 3 weeks after the request has been mailed to the schools. Experienced telephone interviewers will be trained in prompting the nonrespondents and will be monitored by Westat supervisory personnel during all interviewing hours. Recent response rates for FRSS school-level surveys average about 85 percent and a similar response rate is expected for the teacher lists. Ratio-weighting within adjustment cells will be used to partially compensate for the expected approximately 15 percent nonresponse for teacher list collection.

For the teacher questionnaire, respondents will have the option of completing either a paper or online version of the questionnaire. The questionnaire is limited to 3 pages of information readily available to respondents and can be completed by most teachers in 20 minutes (see Appendix C). These procedures are typical for FRSS surveys and result in minimal burden to respondents. Telephone follow up for nonresponse (see Appendix A-2) will begin about 3 weeks after the questionnaires have been mailed to the teachers. Experienced telephone interviewers will be trained to conduct nonresponse follow up and will be monitored by Westat’s supervisory personnel during all interviewing hours.

B.4. Tests of Procedures and Methods

Feasibility calls (OMB# 1850-0803 v.202) and a pretest (OMB# 1850-0803 v.226) are being conducted for this survey. The purpose of the feasibility calls and pretest is to determine what problems respondents might have in providing the desired information and to finalize the content, wording, and layout of the FRSS 109 questionnaire. Responses and comments on the questionnaire will be collected by fax and telephone during the pretest, and the results will be summarized as part of the final documentation for the survey. See Appendix D for the summaries of the feasibility calls and pretesting results.

B.5. Reviewing Statisticians

FRSS surveys are conducted by NCES. NCES contracted Westat to administer FRSS 109 developmental studies and the national data collection. Adam Chu, Senior Statistician at Westat, was consulted about the statistical aspects of the FRSS samples. Chris Chapman, the Associate Commissioner of the Sample Surveys Division within NCES, is the Contracting Officer’s Representative for QRIS. He supervises staff working on FRSS surveys and approves all aspects of the survey, including all aspects of statistical design.

1 Although schools with a high grade lower than 3 will not be eligible, we plan to keep these schools in the frame at the time of sampling. This is because the school grade span listed in CCD might be outdated and not reflect the current grade span of the school. The likelihood of selecting these schools is low because they only represent about 2 percent of schools (and probably a much smaller percent of teachers), but this approach will ensure that any of these schools that have teachers eligible for FRSS 109 will have a chance of selection.