Coronavirus Pandemic_OMB Part B_11Dec20

The coronavirus pandemic significantly disrupted K-12 educational operations and learning in spring 2020 and is likely to do so again during the 2020–21 school year. Federal education policies and funding are intended to support state and local agencies as they respond to the crisis. But the crisis may also shape the way federal programs are carried out. The U.S. Department of Education (the Department), through its Institute for Education Sciences (IES), is requesting clearance for a new data collection to examine how the pandemic may be influencing: (1) implementation of, and waivers from, key provisions of the Elementary and Secondary Education Act, reauthorized as the Every Student Succeeds Act of 2015 (ESSA), (2) state and district use of federal funds, including those provided specifically to help in the pandemic recovery, and (3) supports for English learners (ELs) in districts with high EL enrollments. The surveys of all state education agencies and a nationally representative sample of school districts are being conducted as part of an ongoing evaluation of Titles I and II-A of ESSA. Interviews of district and school personnel in high-EL districts are being conducted as part of an ongoing evaluation of Title III of ESSA.

B.1. Respondent Universe and Sampling Methods

The study will survey the universe of states, the District of Columbia, and Puerto Rico, and a nationally representative sample of school districts. The study will conduct interviews with administrators in 20 districts from the district survey sample as well as administrators of 60 schools within those districts.

B.1.1. State Sample

We will survey all 50 states, the District of Columbia, and Puerto Rico.

B.1.2. School District Sample

The spring 2021 data collection will involve a nationally representative sample of school districts. The study team received permission from the National Center for Education Statistics (NCES) to use the 2021 district frame for the National Assessment of Educational Progress (NAEP) as the starting frame for this data collection. The 2021 NAEP frame is based on the official 2018-19 NCES Common Core of Data (CCD) district universe file. The NAEP frame-building process filters out entities on the CCD that are not really districts such as intermediate units. The study team processed this frame to subset out entities not of interest to the study such as schools with only pre-kindergarten or kindergarten grades or enrollment. Districts with no resulting schools or enrollment were filtered out of the frame. The processed NAEP frame was supplemented with data from the U.S. Bureau of the Census’s district-level SAIPE (Small Area Income and Poverty Estimates) program for school-district percentages of children in families in poverty.

We will draw a district sample of 550 districts out of 17,501 school districts. See Table B-1 in Section B.2 for the universe counts by stratification classifications for the sample. (See section B.2.2 for additional information on the design for this sample.)

B.1.2. Interview Sample

The study team will select 20 districts for the interview sample from the broader sample of 550 districts chosen for the district survey sample. All selected districts will have greater than 10 percent EL enrollment, hence above the national average in terms of EL enrollment. In addition, eight of the 20 districts will be in the 100 largest districts in the country in terms of the number of ELs enrolled. The districts will be selected purposively to reflect a variety of other contexts, including districts in cities and rural settings, districts with varied levels of financial resources, and districts that are majority Hispanic (and Spanish-speaking) as well as districts that enroll ELs from different language backgrounds. To limit potentially confounding variation in state policy contexts, we will nest the 20 districts in six states. The six states will be selected to reflect geographic diversity and to ensure a sufficient number of districts that meet certain district level selection criteria. These district-level criteria include:

At least one elementary, middle, and high school
EL enrollment above the national average;
Urbanicity (urban, rural/town, suburban);District size;
Median household income; and
Implementation of social-emotional learning initiatives.

Once the 20 districts are selected, the study team will identify the elementary, middle, and high schools with the largest EL enrollment in each district for the 60 school interviews. If there are multiple schools with the same level of EL enrollment in a district, the study team will choose schools that will enable us to reflect language diversity across the full sample.

B.2. Information Collection Procedures

B.2.1. Notification of the Sample, Recruitment, and Data Collection

B.2.1.1 State Survey

The study team will send the chief state school officer and the state Title I administrator a notification letter (see Appendix C) explaining the study, thanking the state for its previous participation in the study, and emphasizing the importance of the state’s involvement in this data collection. States receiving education funds through the CARES Act have an obligation to participate in Department evaluations (Education Department General Administrative Regulations (EDGAR) (34 C.F.R. § 76.591)). The state letter will note the mandatory nature of the state’s response.

The study team will follow up with a phone call to the state Title I administrator to answer questions about the study and identify who should be the state-level respondent point of contact. The notification letter and invitation email will be sent to the chief state school officer in each of the 50 states, the District of Columbia, and Puerto Rico beginning in February 2021 and will include a URL to the web-based survey and reference an invitation email they will receive shortly after the letter is mailed. The state survey URL will include embedded login information to: (1) reduce the number of communications from the study team to the state to securely provide login information separate from the survey URL; and (2) reduce the burden of sharing access to the survey within the state if a different respondent is identified as the best person to complete the survey.

Project staff will monitor completion rates, review the instruments for completeness throughout the field period, and follow up by email and telephone as needed to answer questions and encourage completion. During these calls, respondents will be given the option of completing the survey by telephone with the researcher.

B.2.1.2 District Survey

In February 2021, the study team will send a notification letter (see Appendix C) by email and mail to the superintendents of the sampled districts. The notification letter will introduce the study and underscore the study’s importance and benefits. Sending the notification letters both by email and mail will increase the likelihood that addressees will receive our communications in a timely manner. Like the state letters, the district letter will note that districts receiving education funds through the CARES Act are expected to participate per the EDGAR requirement.

The district surveys will likely require input from several key individuals (e.g., federal programs director, finance officer). The district superintendent will serve as the primary contact for the survey (unless the superintendent offers a designee) and will coordinate input from the multiple respondents as needed. For the 20 districts in which EL interviews will be conducted, the notification letter will include mention of the interviews, but the interview team will follow up with more details on the interview procedures. We will follow all required procedures, and as necessary, obtain approval of the district for its participation in the study through submissions of the required research application.

Letters will include the district survey URL with embedded login information, which will be personalized for each district. All communications will include study contact information (i.e., toll-free study number and a study email address) for respondents’ questions and technical support. Based on Westat’s experience on large-scale data collections, we will assign several trained research staff to answer the study hotline and reply to emails in the study mailbox. We will train staff on the purpose of the study, the obligations of district respondents to participate in the evaluation, and the details for completing the web-based survey. Content questions will be referred to the study leadership. An internal FAQ document will be developed and updated as needed throughout the course of data collection to ensure that all research staff have the most current information on the study.

The study team will give respondents the option of emailing them an electronic version of the survey (e.g., PDF or Word document) to complete and return by email or completing a paper-and-pencil instrument. However, we have found that the vast majority of respondents prefer the web-based approach. A phone survey option will be offered to all respondents as part of the nonresponse follow-up effort. Since the web-based surveys will include data checks, we will use the web-based surveys to enter any surveys received on hard copy or by phone. A web-based data monitoring system (DMS) will record the status of district research application approvals, generate materials for mailings, and monitor survey response rates.

B.2.1.2 District and School Interviews

The study team will begin the interview recruitment and data collection process by holding an in-depth training session for all interviewers to ensure a clear and consistent understanding of the interviews’ purpose, data collection and management procedures, and protocol questions across the study team. Upon receiving OMB approval and in follow-up to the notification letter sent by the survey administration team, we will send the EL/Title III director in each sampled district a notification email explaining the purpose of the interviews and highlighting the importance and benefits of their participation. This email will note that districts receiving funds through the CARES Act are expected to participate per the EDGAR requirement. The study team will then follow up with each respondent via email and/or telephone until the interview is scheduled. Throughout the scheduling process, the study team will employ a case manager approach that assigns a specific team member to each district to serve as a single point of contact for all communication. Once the interview is scheduled, the district’s case manager will send a meeting invitation with details about accessing the video/teleconference line as well as an informed consent document outlining the terms of the respondent’s participation, including the interview’s purpose, anticipated benefits and risks, and confidentiality procedures.

Prior to conducting the district interviews, interviewers will review any publicly available information about the district’s approach, such as a written COVID-19 plan and relevant updates on the district website, and use this information to tailor the interview protocol questions for that district. The interviews will be conducted via a secure video/teleconference platform (e.g., GoToMeeting) and will be audio-recorded, with the respondent’s permission, to ensure data are captured accurately and completely. The study team will then use the audio recordings to generate verbatim transcripts of the interview. Interviewers will review each transcript and if needed, follow up with the respondent via email to clarify any information that is missing or unclear.

Recruitment and data collection procedures for the school-level interviews will closely mirror the district-level interview procedures. However, school-level respondents will be informed that their participation is voluntary. In addition, to help boost response rates, the study team may ask the district EL/Title III director to endorse the study to encourage school-level respondents’ participation.

B.2.2. Statistical Methodology for Stratification and Sample Selection

B.2.2.1. States

The study will include all 50 states, the District of Columbia, and Puerto Rico. Based on experience, the study design assumes that all states will respond. Therefore, the study team does not plan to construct weights to account for state-level nonresponse.

B.2.2.2. District Sample

The study team will select a nationally representative sample of 550 districts. District poverty and district size are important considerations for the district sample design. The $13.5 billion in Elementary and Secondary School Education Relief Funds will be allocated based on the Title I funding, which in turn is based on an estimate of the number of children in families in poverty in the school district. At equal poverty levels, we expect larger districts to receive a larger share of these funds, and at equal enrollment levels, we expect higher-poverty districts to receive a larger share of funds. As a result, the district sample design will oversample larger districts and higher-poverty districts.

The team will select the district sample using a stratified simple random sample approach. The frame will be stratified by district poverty level. The high-poverty stratum is defined as all districts above the 75^th percentile in poverty level. Poverty level for geographically based districts is based on the percentage of children in families below the poverty level, as estimated for geographically based school districts from the Small Area Income and Poverty Estimates (SAIPE) program. For non-geographically based districts not in the SAIPE file, we estimate this percentage by using the percentage of students eligible for free or reduced-price lunch within the schools in the district to impute the percentage of children in families below the poverty level.

Each of the two major poverty strata will be stratified by district-size strata, with stratum sampling rates proportional to the square root of mean enrollment for each stratum.¹ We will oversample high-poverty districts by a factor of two. The two-times high-poverty oversampling will provide a larger set of high-poverty districts that will likely have considerable relief funding than would occur without oversampling. Table B-1 presents a breakdown of district-size strata and the two poverty strata based on the working district frame developed from the processed 2021 NAEP district frame.²

Within these highest-level strata (high-poverty vs. low/medium-poverty), we will implicitly stratify districts to improve the representativeness of the sample. This implicit stratification determines the sort order for systematic sampling using the probabilities of selection. The highest levels in the implicit stratification hierarchy will have the greatest control in sample sizes. We will implicitly stratify by the following district characteristics:³

District Comprehensive Support and Improvement school status (district does not have a CSI school or district has at least one CSI school);
District charter status (charter district (i.e., only has charter schools), regular district with charters, or regular district with no charters);
Urbanicity (four strata: urban, suburban, town, rural).
District poverty strata (three poverty strata based on two quartiles for poverty (high-poverty greater than 75th percentile; medium-poverty between 25th and 75th percentile; low-poverty below 25th percentile);
Census Region (four geographic strata Northeast, South, Central, West); and
Student enrollment.

Table B-1. District sample by primary poverty strata and district-size strata

Poverty Stratum	District Size Strata	Number of districts	Mean enroll-ment	Relative sampling rate	Percent of frame districts	Percent of sampled districts	Expected sample size
High	E1--Enrll 1 to 500	2,092	240	1.00	44.00%	19.16%	42.2
High	E2--Enrll 501 to 1,500	1,384	855	1.89	29.11%	23.90%	52.6
High	E3--Enrll 1,501 to 5,000	854	2,673	3.34	17.96%	26.08%	57.4
High	E4--Enrll 5,001 to 15,000	296	8,215	5.85	6.23%	15.85%	34.9
High	E5--Enrll 15,001 to 50,001	107	27,344	10.67	2.25%	10.45%	23.0
High	E6--Enrll 50,001 to 900,000	21	108,796	21.29	0.44%	4.09%	9.0
High	E7--Enrll 900,000+	1	947,836	62.84	0.02%	0.45%	1.0
High	Total	4,755	2,641		100.00%	100.00%	220
Low/Med	E1--Enrll 1 to 500	4,603	240	1.00	36.11%	13.91%	45.9
Low/Med	E2--Enrll 501 to 1,500	3,716	900	1.94	29.15%	21.76%	71.8
Low/Med	E3--Enrll 1,501 to 5,000	2,942	2,761	3.39	23.08%	30.18%	99.6
Low/Med	E4--Enrll 5,001 to 15,000	1,066	8,273	5.87	8.36%	18.93%	62.5
Low/Med	E5--Enrll 15,001 to 50,000	350	25,270	10.26	2.75%	10.86%	35.8
Low/Med	E6--Enrll 50,000+	69	105,228	20.94	0.54%	4.37%	14.4
Low/Med	Total	12,746	2,941		100.00%	100.00%	330
Total	Total	17,501	2,859				550

B.2.3. Estimation Procedures

The primary goal of this data collection is to describe the effects of the coronavirus pandemic on key federal education policies and the use of CARES Act funds for K-12 education. Responses to survey questions will be tabulated into descriptive statistics (such as percentages) and simple statistical tests (such as tests for differences between percentages).

While simple descriptive statistics will provide answers to many of the study’s research questions, cross-tabulations will be important to answering questions about variation across state and district characteristics. The primary district characteristics of interest for the cross-tabulations are:

District poverty level. Poverty is included because the allocation method for the bulk of the CARES Act relief funds for K-12 education follows the Title I formula. Title I funds are specifically intended to ameliorate the effects of poverty on local funding constraints and educational opportunity.
District urbanicity. Urbanicity is included because of the relationship between educational opportunity and rural isolation and the concentration of poverty in urban schools. In addition, urbanicity may serve as a proxy for local effects of the coronavirus pandemic.

Because of the use of a statistical sample, survey data presented for districts will be weighted to generalize to the experiences of school districts nationally. In addition, the descriptive tables will indicate where differences between subgroups are statistically significant. We will use Chi-Square tests to test for significant differences among distributions and t-tests for differences in means.

B.2.4. Degree of Accuracy Needed

The sample design will provide unbiased, nationally representative estimates for all public school districts receiving funds under this program. Table B-2 presents a summary of the design and final predicted precision at the national level. The assigned sample sizes are 220 and 330 for each of the two primary poverty strata respectively, and accounting for the square-root design allocation to district-size strata, the effective sample sizes are 149.5 and 216.6 for high-poverty and low/medium-poverty strata, respectively, corresponding to design effects of 1.472 and 1.524 respectively. There is a very limited 1.077 design effect from the oversampling of high-poverty districts. Given the sample design, a 95 percent confidence interval around a sample percentage of 50 percent would be (44.7%, 55.3%), and the coefficient of variation would be 5.4 percent

Table B-2. Summarized precision for national estimates

Poverty Stratum	Number of districts	Percent of frame districts	Aggregate enroll-ment (in 1000s)	Percent aggregate enroll-ment	Assigned sample size	Design effect	Effective sample size
High	4,755	27.20%	12,558	25.10%	220	1.472	149.5
Low/Med	12,746	72.80%	37,482	74.90%	330	1.524	216.6
Total	17,501	100.00%	50,039	100.00%	550	1.618	339.9
			Coefficient of Variation Sample Pct 50%				5.42%
			Standard Error Sample Pct 50%				2.71%
			Design Effect Poverty Strat Oversampling				1.077
			Lower Bound 95% CI Sample Pct 50%				44.68%
			Upper Bound 95% CI Sample Pct 50%				55.32%

Table B-3 presents power calculations for comparing the two poverty strata. In comparing the two poverty strata, the design allows for a minimum detectable difference (MDD) of 15 percentage points.⁴ Table B-4 then presents power calculations for urbanicity comparisons. The sample design allows for an MDD of 15 percentage points for this subgroup comparison of interest.⁵ These MDD values are in line with those for sample design for the original Title I/II study district sample.

Table B-3. Power calculations for comparing poverty strata

Poverty Stratum	Nominal sample size	Effective sample size	Pop Pct Null	Std error	Cutoff Null Critical Region	Pop Pct Alternative	Std error	Power
High	220	149.5	50%	4.09%		50.0%	4.09%
Low/Med	330	216.6	50%	3.40%		64.8%	3.25%
Difference	550			5.32%	10.4%	14.8%	5.22%	79.92%

Table B-4. Power calculations for comparing urbanicity subgroups

Poverty Stratum	Nominal sample size	Effective sample size	Pop Pct Null	Std error	Cutoff Null Critical Region	Pop Pct Alternative	Std error	Power
Urb/Sub	293.4	173.8	50%	3.79%		50.0%	3.79%
Tw/Rur	256.6	181.9	50%	3.71%		64.8%	3.54%
Difference	550.0			5.30%	10.4%	14.8%	5.19%	80.21%

B.2.5. Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems requiring specialized sampling procedures.

B.2.6. Use of Periodic (less than annual) Data Collection to Reduce Burden

The coronavirus-pandemic focused surveys and interviews will be conducted during the 2020–21 school year.

B.3. Methods to Maximize Response Rates

The study team achieved very high response rates for the state survey (100 percent) and district survey (99 percent) during the study’s 2018 data collections. As a result, we expect to achieve very high response rates again for the 2021 surveys. We plan to work with states and school districts to explain the importance of this data collection effort and to make it as easy as possible to comply. For all respondents, a clear description of the study design, the nature and importance of the study, and the OMB clearance information will be provided.

For the states, we will be courteous but persistent in follow-up with participants who do not respond in a timely manner to our attempts. We also will be very flexible gathering our data, allowing different people to respond to the different content areas and in whichever mode is easiest -- electronic, hard copy or telephone format. Project staff will monitor completion rates, review the instruments for completeness throughout the field period, and follow up by email and telephone as needed to answer questions and encourage completion. During these calls, respondents will be given the option of completing the survey by telephone with the researcher.

For the district survey, we will initiate several forms of follow-up contacts with respondents who have not responded to our communication. We will use a combination of reminder postcards, emails, and follow-up letters to encourage respondents to complete the surveys. Westat’s project management system developed for this study will be the primary tool for monitoring whether surveys have been initiated. After 10 days, we will send an email message (or postcard for those without email) to all non-respondents indicating that we have not received a completed survey and encouraging them to submit one soon. Within seven business days of this first follow-up, we will mail non-respondents a hard copy package including all materials in the initial mailing. Ten days after the second follow-up, we will telephone the remaining non-respondents to ask that they complete the survey and offer them the option to answer the survey by phone, either at that time or at a time to be scheduled during the call.

To maximize response rates, we also will (1) provide clear instructions and user-friendly materials, (2) offer technical assistance for survey respondents using a toll-free telephone number or email, and (3) monitor progress regularly. In recognition of the fact that district administrators have many demands on their time, and typically, these administrators receive numerous requests to participate in studies and complete surveys for federal and state governments, district offices, and independent researchers, we plan to identify a district liaison. For most districts, completion of the district survey will require input from multiple respondents, and the district liaison’s role will be pivotal in positively impacting participation, collecting high quality data, and achieving a minimum 85 percent response rate.

For the district and school interviews, the study team will employ similar strategies to promote high response rates, including providing study information and instructions in a clear, user-friendly format; using multiple modes of communication (i.e., email and telephone); being flexible and accommodating to respondent needs (e.g., scheduling interviews at times that are convenient for the respondent), engaging in consistent but courteous follow up efforts, and systematically tracking all communication attempts to guide future outreach. Study notification letters and follow-up communication will emphasize the social incentive to respondents by stressing the importance of the data collection to inform future technical assistance and policy making at the federal, state, and local levels. If needed and appropriate, the study team will ask district EL/Title III directors to endorse the study to encourage school-level respondents to participate.

In addition, the study team has developed interview protocols that are appropriately tailored to the respondent group and are designed to place as little burden on respondents as possible. The team will use cognitive interviews with district and school administrators to pilot data collection instruments to ensure that they are user-friendly and easily understandable, all of which increases participants’ willingness to participate in the data collection activities and thus increases response rates.

B.3.1. Weighting the District Sample

After completing the field data collection, we plan to weight the district data to provide a nationally representative estimator at the district level. The district weighting process will involve developing unit-base sampling and replicate weights, then adjusting these weights as necessary to account for survey nonresponse.

Replicate weights will be generated to provide consistent jackknife replicate variance estimators (statistical packages such as STATA and SAS Version 9.2+ allow for easy computation of replicate variance estimates). The development of replicate weights will facilitate the computation of standard errors for the complex analyses necessary for this survey. For districts selected with certainty into the sample, the replicate weights will equal the base sampling weights. For the noncertainty districts, the replicate weights will be generated using the jackknife replication method.

We anticipate limited nonresponse at the district level, which we will adjust for by using information about the non-responding districts from the frame. This information will be used to generate nonresponse cells with differential response propensities. The nonresponse adjustments will be equal to the ratio of the frame-weighted count to the sum of weights for respondents. This will adjust for bias from nonresponse, and adjust for differences from the frame (accomplishing poststratification). If the cell structure is too rich, we may use raking (multi-dimensional adjustment). The cell structure for districts will include district poverty status, Census region, urbanicity, and district size.

B.4. Test of Procedures

The study team pretested the state and district surveys and the district and school interview protocols with nine or fewer respondents to ensure that questions are clear and that the average completion time is within expectations.

B.5. Individuals Consulted on Statistical Aspects of Design

The individuals consulted on the statistical aspects of the school district sample design include:

Patty Troppe, Westat, Vice President and Project Director

Lou Rizzo, Westat, Senior Statistician

Keith Rust, Westat, Senior Vice President Statistics and Evaluation Sciences Practice

References

U.S. Department of Education. (2016a). Retrieved from http://eddataexpress.ed.gov/

U.S. Department of Education. (2016b). Findings from the 2015–2016 Survey on the Use of Funds Under Title II, Part A. Retrieved from: http://www2.ed.gov/programs/teacherqual/leasurveyfundsrpt82016.pdf

1This approach is similar to the district sample design for the sample drawn for the 2014 Title I/II district survey. For that sample, the team used a ‘minimax’ approach in which we used a stratified random sample of districts (equal probabilities within strata), with higher sampling rates proportional to a root of mean district enrollment (with the stratum). The root was 0.535 and was carefully chosen to balance the precision of unit-based and enrollment-based estimates. For this data collection, there is less interest in enrollment-based estimates. As such, the team recommends the slightly smaller but easier-to-comprehend square root.

2Note that this frame includes Puerto Rico, but excludes other U.S. territories, BIE schools, and DoD schools.

3The Title I/II study will include a spring 2022 data collection of states, districts, principals, and teachers. We are expecting to maximize the overlap in the district samples for the 2021 and 2022 data collections. As a result, the implicit stratification includes district’s CSI school status and district’s charter status since these will be important for the school sample selection for the 2022 data collection. Urbanicity is included as an implicit stratification variable since it is likely to be important for differential effects of the coronavirus pandemic.

4This assumes a two-sided test of the null hypothesis of no difference, with both population percentages at 50%, with the given effective sample sizes, and 80% power.

5This assumes a two-sided test of the null hypothesis of no difference, with both population percentages at 50%, with the given effective sample sizes, and 80% power.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Lee, Erica
File Modified	0000-00-00
File Created	2021-01-13

Coronavirus Pandemic_OMB Part B_11Dec20