ISFFMS OMB Part B 4.24.18

part b

Impact Study of Federally-Funded Magnet Schools: OMB Data Collection Package

January 10, 2018

Submitted to:

Institute of Education Sciences
550 12th Street, SW

Room 4104
Washington, DC 20004

Project Officer: Lauren Angelo
Contract Number: ED-IES-17-C-0066

Submitted by:

Mathematica Policy Research

P.O. Box 2393
Princeton, NJ 08543-2393
Telephone: (609) 799-3535
Facsimile: (609) 799-0005

Project Director: Christina Tuttle
Reference Number: 50526.01.026.220.000

This page has been left blank for double-sided copying.

CONTENTS

PART B. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION 1

A. Collection of information employing statistical methods 1

1. Respondent universe and sampling methods 1

2. Procedures for the collection of information 4

3. Methods to maximize response rates and deal with nonresponse 7

4. Tests of procedures or methods to be undertaken 8

5. Individuals consulted on statistical aspects of the design 8

References 9

TABLE

B.1. Research questions and data sources 2

B.2. Individuals consulted on study design 8

This page has been left blank for double-sided copying.

PART B. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION

This Office of Management and Budget (OMB) package requests clearance for data collection activities to support a rigorous Impact Study of Federally-Funded Magnet Schools. The Institute of Education Sciences (IES) at the U.S. Department of Education (ED) has contracted with Mathematica Policy Research and its subcontractor, Social Policy Research Associates (SPR), to conduct this evaluation (ED-IES-17-C-0066). The evaluation includes an initial feasibility assessment, the focus of this clearance package, to determine whether t an impact study can be conducted appropriately. First, the study team will gather information from fiscal year (FY) 2016 and 2017 Magnet Schools Assistance Program (MSAP) grantee districts and schools on student recruitment and admissions policies and practices, paying particular attention to the use of randomized lotteries for student admissions. The feasibility phase will result in a brief describing how MSAP-funded schools recruit and select students for admission, a topic of interest to the program office. Second, if a sufficient number of students are being admitted to these schools through lotteries, a revised clearance package will be submitted and the impact study will collect survey data from principals and district administrative records on admissions lotteries and student progress. The study would use these data to estimate the impacts of magnet schools on student achievement and diversity and to describe whether particular features of magnet schools are associated with greater success.

B. Collection of information employing statistical methods

The most recent FY 2016 and FY 2017 MSAP grant competitions provide a potential opportunity to conduct a rigorous study of the MSAP program, drawing primarily on administrative records. The grant notices in those years included a competitive preference priority for grant applicants that proposed to select students to attend the magnet schools by methods such as lotteries. This incentive introduced by the program office at ED builds directly on findings from an earlier, descriptive study of magnets schools conducted by the Institute of Education Sciences, which raised concerns about MSAP schools’ success in recruiting and admitting students from outside the schools’ neighborhood to encourage educational and demographic diversity.

The potential impact evaluation would measure the effects of a wide range of MSAP schools, on student’s achievement and school diversity, using a rigorous lottery-based random assignment design. The impact study data collection would make it possible to estimate the impact of admission to a magnet school on: (1) student achievement and other available academic outcomes; and (2) school environment, including exposure to a more diverse set of peers. The study would also examine whether particular features of MSAP schools are associated with greater success which could inform program improvement efforts in the future.

This impact evaluation of MSAP schools is authorized by Part B section 8601 of the Elementary and Secondary Education Act of 1965 (ESEA) as amended by the Every Student Succeeds Act (ESSA), which allows the Department to pool resources across ESEA programs in order to fund rigorous evaluations of individual Federal education programs that currently lack sufficient evaluation dollars. The Magnet Schools Assistance Program (MSAP), itself, is authorized under Title IV, Part D of the ESEA, as amended by ESSA, and provides grants to local educational agencies (LEAs) and consortia of LEAs to support magnet schools under an approved, required or voluntary, desegregation plan.

To increase the policy relevance of the study, the analysis would focus on the districts currently receiving federal support through multi-year MSAP grants awarded in FY2016 and FY2017. We anticipate recruiting approximately 30 MSAP schools.¹ To reach that target, the study team will first collect information on schools’ admissions and lottery procedures, administrative data availability, and willingness to participate in an impact evaluation. This information will be particularly informative for policymakers interested in understanding magnet schools’ student recruitment and admissions practices.

Then, if enough schools meet the study’s criteria, a revised OMB package would be submitted and the impact evaluation would address specific questions about the impact of being admitted to a MSAP school and explore other questions that may help ED and the magnet school community in improving the program. The study design takes advantage of the lotteries that districts or schools conduct when there are more students interested in being admitted to a MSAP school than there is available space for them. Specifically, the analysis would estimate the impact of each magnet school in the study sample by comparing the outcomes of students who receive an admission offer through the lottery to the outcomes of students who do not receive an offer through the lottery. Our analytic approach takes into account that substantial variation will exist in schools’ lottery procedures, grade levels of applicants, and the ratio of applicants to available seats; including as many different magnet school lotteries as possible in the study will maximize the sample’s size and policy relevance.

Table B.1 describes these research questions, as well as data sources and analysis methods for addressing those questions.

Table B.1. Research questions and data sources

Research questions	Data sources and analysis methods
Feasibility study
How do the districts and schools funded through the 2016 and 2017 MSAP grants recruit and select students for admission?	Screener interviews with 2016 and 2017 MSAP districts and school surveys on student recruitment and admissions Comparison of reported school/district practices
How many schools hold eligible lotteries and are willing to participate in the study?	Screener interviews with 2016 and 2017 MSAP districts
Is the number of lottery participants and eligible schools sufficient to conduct the study?	Screener interviews on lottery procedures and information available in district administrative data Projections of the number of students participating in usable lotteries in spring 2018 and 2019 Analysis of statistical power to detect meaningful impacts
Impact evaluation
What is the impact of admission to the magnet program on student academic outcomes (achievement and/or other relevant measures of student success, such as persistence in school or graduation)?	Data on student outcomes from district student-level records Experimental impact analysis comparing outcomes of treatment group students (who win lotteries and are admitted to magnet schools) and control group students (who lose lotteries)
What is the impact of admission to the magnet program on the type of school that students attend, including their school’s educational programs and the students’ exposure to a diverse range of peers?	Student- and school-level administrative records, supplemented by data on school characteristics from principal survey Experimental impact analysis comparing characteristics of schools attended by treatment group and control group students
To what extent is there a relationship between school characteristics, including measures of diversity, and school impacts on student outcomes?	Subgroup analysis, in which impacts are calculated for subgroups of schools with particular characteristics Correlational analysis, examining relationship between school characteristics and measures of diversity (from school survey and extant data) and impacts

1. Respondent universe and sampling methods

For the district interviews, the respondent universe is all school districts receiving Magnet Schools Assistance Program (MSAP) grants in 2016 and 2017. Forty different school districts received these grants in FY 2016 and FY 2017. In addition, school survey supplements will include one administrator in all 162 2016 and 2017 MSAP schools. Those interviews/surveys will determine which schools are eligible for data collection associated with the impact study, which includes a principal survey and student records collection. The respondent universe for the student sample includes all students who apply to the eligible schools in 2018 or 2019 and participate in the schools’ admissions lotteries (if the full impact study is feasible). These lotteries would determine students in the treatment group (“lottery winners,” who are offered admission to the study’s magnet schools) and students in the control group (“lottery losers,” who are not offered admission). We would collect survey data from principals in the schools the treatment group and control group students attend, and would collect administrative data on these same schools from districts. We would collect administrative data on all sample students in the study schools. Next, we describe each stage in greater detail.

a. Selection of districts and schools

The study will conduct screening interviews with one MSAP coordinator in each of the FY2016 and FY2017 MSAP grantee districts. In addition, surveys will include one administrator in all 162 2016 and 2017 MSAP schools. From those interviews/surveys, we will identify potential districts and magnet schools to include in the impact study. If the impact study is pursued, the sample will comprise 30 MSAP schools that hold 2018 and/or 2019 admissions lotteries (power calculations indicate that a sample of 30 schools is needed to conduct a study with adequate statistical precision; if more than 30 MSAP schools are eligible for the impact study, we will select the schools that enroll larger numbers of students through a lottery-based admissions system). We expect a broad set of schools to participate in the study, including new, preexisting, and conversion magnets, as well as magnets with different themes.

b. School survey sample (impact study)

The school survey sample will include all principals in the 30 treatment group schools and those in up to 220 schools attended by students in the control group, as well as any replacement principals who join the study schools during the years of the study. If students in the control group attend more than 220 schools, to contain costs the study will select the sample by prioritizing schools that enroll the largest number of control group students. Given expected annual principal turnover rates of 27 percent (Goldring and Taie 2014), we expect approximately 100 new principals to join the study sample, resulting in a total sample of 350 principals. We expect that approximately 85 percent of principals in the sample will complete the survey.

c. Student sample (impact study)

The respondent universe for student data collection will include all students who participate in the 2018 or 2019 lotteries for admission to study MSAP schools. To conduct a study with adequate statistical precision, our power calculations indicate that we will need to recruit a sample of 30 MSAP schools and include two cohorts of students per school, representing a universe of approximately 4,000 students who applied to and participated in the lotteries for these schools. The lotteries will determine students in the treatment group (“lottery winners,” who are offered admission to the study’s magnet schools) and students in the control group (“lottery losers,” who are not offered admission). To examine effects of magnet schools on student outcomes, we will collect district administrative records data on student demographics (age, sex, race/ethnicity, and eligibility for free or reduced-price lunch), school enrollment, test scores, attendance, persistence, and graduation.

2. Procedures for the collection of information

a. Statistical methodology for stratification and sample selection

We will select a sample of 30 MSAP grantee schools that are willing and able to participate in the impact evaluation, and for each MSAP school the study sample will include two cohorts of students who applied for admission via a lottery. Under a set of conservative assumptions, we project that this sample of schools will yield a student-level sample of more than 4,000 treatment group and control group students. The proposed design and sample size is sufficient to detect policy-relevant impacts both for the full sample and for key subgroups of students and schools. The design supports a minimum detectable effect (MDE) of 0.08 on achievement outcomes for the overall sample, or an MDE of 0.11 for a subgroup such as the 2018 lottery cohort (for which we will have as many as four years of follow-up data). Other key student subgroups may include those with outcome data in different grades. This design will also give us reasonably high levels of statistical power for estimates of impacts for subsets of magnet schools. For example, the MDE for impact estimates for STEM-themed schools alone—which appear to make up roughly half of the MSAP schools supported by FY 2016 and FY 2017 grants, according to publicly available information—is 0.13. This level of power for subsets of schools also implies that the design will be able to conduct a useful correlational analysis of how magnet school characteristics relate to impacts.

Next, we describe our selection of the sample of principals and students for each data collection activity, should the impact evaluation go forward.

Selection of principal sample. We will not randomly sample principals for any data collection activities. Instead, we will attempt to collect survey data from all principals of all treatment group schools and up to 220 schools attended by control group students. Treatment group schools are defined as 2016 and 2017 MSAP grant-supported schools with eligible admission lotteries. Control group schools are defined as schools not supported by MSAP grants where students chose to enroll after participating in an admission lottery for a treatment school and not receiving an admission offer from that treatment group school. If control group students attend more than 220 schools, we will survey principals in the schools that enrolled the largest number of control group students.

Selection of student sample. We will not randomly sample students for the collection of district administrative records. Instead, we will attempt to collect administrative records data on all students who entered the sample by participating in a MSAP school lottery seeking admission in fall 2018 or fall 2019, regardless of which schools the students attended in the grantee district during the study’s follow-up period. The study team will not modify the lottery procedures used by MSAP-supported districts and schools—for example, any systems used by districts to give some types of students a higher chance of receiving an admissions offer than other types of students (such as lottery stratification) will remain in place and will not be altered by the study.

b. Data collection

District interview (feasibility study).

We will conduct a screening interview with each MSAP district to describe the study, collect school eligibility information, and seek support to encourage school participation in study activities, including screening interviews and school organization and instruction surveys.

We developed the protocol with the goal of completing a discussion with one staff member in 60 minutes. We will provide the protocol to respondents in advance of the discussion and assume that they may spend up to an hour locating and compiling information for the interview. To reduce burden on district staff, we will review grantees’ district websites and other publicly available information to identify information relevant to our items of interest and ask respondents to verify or update that information rather than submit new answers. Discussions with staff members will begin in fall 2018.

School survey on recruitment and admissions (feasibility study). Short school surveys will collect only essential information on student recruitment activities and student selection. This information will also help determine whether a school is eligible for the impact study and will provide information of interest to the program office.

School survey on organization and instruction (impact study). The data collection team will administer a 30-minute survey electronically to treatment group and control group principals. We also will provide an option for principals to complete the survey by telephone, to achieve an 85 percent response rate.

We will administer principal surveys in spring 2019, 2020, 2021, and 2022. The survey will gather information on school instruction and organization. Principals will respond to questions on curricular focus; admissions processes; student programs; instructional organization, approach, and resources; faculty and staff; safety and behavior policies; and community and parent engagement. Data from the principal surveys will allow us to assess whether schools change over time in their organization and instructional approaches, and also will allow us to analyze which characteristics may be related to the effectiveness of magnet schools in improving student outcomes and diversity.

We will draw valid and reliable items from instruments we have developed to survey principals at choice and traditional public schools, including the principal surveys from the IES charter school study and the National Evaluation of Charter Management Organization (CMO) Effectiveness, both used to identify promising practices in choice schools. We will draw from other relevant surveys, including the School Survey on Crime and Safety, Schools and Staffing Survey, and National Center on School Choice Surveys (Berends et. al 2009).

Lottery records memo (impact study). For the impact study, the study team will share a memo with districts that defines the data that districts and schools will need to provide for each cohort of lottery applicants to account for variation in lottery implementation. The lottery memo will describe detailed information for each cohort of lottery applicants to account for variation in lottery implementation and all school options. Details about how the lottery was conducted are crucial for compiling the appropriate data to accurately define students’ treatment group or control group status. The extraction memo is structured to describe lottery data in a manner that accounts for the following factors:

• Single-school versus districtwide common lotteries

• Lottery processes for making admissions offers

• Accounting for wait list offers for eventual admissions

• Identifying treatment group assignment compliers: matching lottery records to enrollment records

In eligible schools and districts, collecting and processing data will involve close coordination with staff before, during, and after lotteries. Multiple points of coordination will allow us to document and verify nuances in the lottery procedures and data that we should account for in the research design. We will establish a liaison at the school or district with whom we will communicate about lottery procedures and outcomes. We will communicate before the 2019 lotteries to confirm that the lottery process is consistent with the information gathered during recruitment. We will use updated information that we gather from liaisons at this stage to develop a plan for completing the lottery results extraction form retrospectively for 2018 lotteries and prospectively for 2019 lotteries. We will collect lottery data starting in spring 2019 through the beginning of the 2019–2020 school year. Districts will provide the associated records in the format used by the district.

District student-level records memo (impact study). For the impact study, the data collection team will collect data on students in treatment group and control group schools. We developed a standardized memo to share with districts that describes the data districts will provide so they can be collected in a consistent manner. This includes (1) student-level records, including demographic characteristics, school enrollment, and test scores, as well as attendance, persistence, and graduation where applicable and available; and (2) school-level data, such as teacher experience, principal experience, and average student characteristics. The data request form clearly and concisely summarizes (1) the samples of schools and students for whom we are requesting data (including identifying information where possible), (2) the data elements, and (3) the school years for which we are requesting each data element.

We also will use publicly available school-level data from the Common Core of Data and district websites to measure additional school and staff characteristics. These characteristics will include school size, racial/ethnic and socioeconomic student composition, and teacher/pupil ratio.

3. Methods to maximize response rates and deal with nonresponse

Next, we describe our methods for maximizing response rates and minimizing nonresponse in our collection of extant data from districts and our collection of primary data from principals (should the impact evaluation be feasible). All MSAP grantees are expected to participate in data collection activities, including completing the screener, as a condition of their grant funding.

Collection of administrative data. To reduce districts’ burden in the submission of administrative records (including student-level lottery documentation and longitudinal school records data) and maximize response rates, we will allow districts to submit data in the most convenient format. Federal rules permit ED and its designated agents to collect school records data from schools and districts without prior parental or student consent (Family Educational and Rights and Privacy Act [20 U.S.C. 1232g; 34 CFR Part 99]). To further maximize the response rate and minimize burden on schools, we will follow these federal rules.

School survey on organization and instruction (impact study). We expect to achieve a response rate of 85 percent for the principal survey. ED has indicated that grantees are expected to complete the survey as a condition of their funding, which we expect to assist with response rates in the study’s treatment group. However, this will not apply to principals of schools attended by students in the control group. We will use the following approach, which is designed to maximize efficiency and minimize costs.

We will send an initial welcome email to all sample principals. The email will contain information on the study and a link to access the survey. Principals will be given 12 weeks to complete the survey, and we will send an email every 2 weeks reminding them to do so. We will also forward a list of study principals to a coordinator in each district, and ask districts to encourage the principals to complete the survey. To validate data and ensure quality control, we will conduct (1) regular, real-time checks of survey responses to ensure completeness and face validity and to detect issues such as instructions that need clarification; (2) an interim review of aggregate data to validate instrument skip patterns and review preliminary statistics, early enough to make critical fixes for most of the sample; and (3) a full review, cleaning, and editing of the complete data files.

For treatment group principals, we will leverage the expectation that they will cooperate as a condition of their grant funding. For control group principals, we will implement an incentive ($30) and leverage district participation in the study by requesting help from the district liaison identified in the district memorandum of understanding. The district liaison will provide an advance letter and other communication and follow-up, as appropriate, to participating treatment group and control group principals to encourage their cooperation.

4. Tests of procedures or methods to be undertaken

Pre-testing the district/school screener was vital to the integrity of data collection. We reviewed previously used questions and developed new questions for the evaluation according to the following guidelines:

• Questions are worded simply, clearly, and briefly, as well as in an unbiased manner.

• Respondents can readily understand key terms and concepts.

• Question response categories are appropriate, mutually exclusive, and reasonably exhaustive, given the intent of the questions.

• Questions are accompanied by clear, concise instructions and probes so that respondents know exactly what is expected of them.

The goal of the pre-test was to assess how respondents understand the terms and questions presented in the survey, assess the accuracy and relevancy of our questions, and determine whether we are missing important elements in our questions. The pre-test allowed us to determine how long the survey took to complete.

We conducted pilot testing of the district/school screener during the 60-day comment period. Due to this testing, we learned that it was necessary to target items specifically to the district role and other items specific to the school administration. Therefore, the interview was broken-up into a district interview and short school survey on recruitment.

5. Individuals consulted on statistical aspects of the design

Table B.2 lists members of the study team who were consulted on aspects of the sampling plan.

Table B.2. Individuals consulted on study design

Name	Title and affiliation
Phil Gleason	Associate director, Human Services Research and senior fellow, Mathematica
Ira Nichols-Barrer	Senior researcher, Mathematica

References

Berends, Mark, Marisa Cannata, Ellen Goldring, and Roberto Penaloza. “Innovations in Schools of Choice.” Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA, April 13–18, 2009. Available at https://www.researchgate.net/publication/267563706_Innovation_in_Schools_of_Choice. Accessed July 13, 2017.

Goldring, R., and S. Taie. “Principal Attrition and Mobility: Results from the 2012–13 Principal Follow-up Survey.” (NCES 2014-064). Washington, DC: U.S. Department of Education, National Center for Education Statistics, 2014. Available at http://files.eric.ed.gov/fulltext/ED545366.pdf. Accessed October 23, 2017.

www.mathematica-mpr.com

Improving public well-being by conducting high quality,
objective research and data collection

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ SEATTLE, WA ■ TUCSON, AZ ■ Washington, DC ■ Woodlawn, MD

1 Though the focus is on schools funded with the new grants, we may consider including magnet schools that received MSAP grant funds between 2010 and 2014 in the same 2016–2017 grantee districts. These earlier-funded schools are likely to have more mature magnet programs and stronger demand for admission, making them potentially better candidates for a lottery-based impact study.