Att_1850-0832 v.4 rev4-29 MSAP Part A 4.25.11

Magnet programs were designed to help address racial equity issues in American public education and have become an important component of public school choice as well as a possible mechanism to improve the achievement of all students, particularly students who are disadvantaged. The Conversion Magnet School Evaluation is being conducted to determine if efforts to turn around low-performing schools through converting to a Magnet Schools Assistance Program (MSAP) supported magnet school are associated with improved achievement and a reduction in minority group isolation using a Comparative Interrupted Time Series (CITS) design.

An OMB clearance request that (1) described the study design and full data collection activities and (2) requested approval for the burden associated with the first three years of data collection was approved in 2007 (OMB Number 1850-0832 approval 7/13/07; expiration 7/31/10). As described in the original clearance request, the study included a feasibility phase to determine if an adequate sample permitted the full evaluation as well as an evaluation phase. In 2008, we completed the feasibility phase and updated OMB on the results including the viability of moving forward with the CITS study and a description of the actual study sample (please see Appendix Q; Approved on 3/21/08). We also adjusted the burden estimates for years one through three to reflect the actual rather than the expected sample. In 2010, we requested clearance for the burden associated with the fourth and fifth year of data collection necessary for the rigorous Comparative Interrupted Time Series (CITS) design including student records data collection for the 2009-2010 school year. (OMB Number 1850-0832 approval 6/14/10; expiration 6/30/13). The burden associated with the fourth and fifth year of data collection will be completed by the time this most recent package will be approved (anticipated 6/11). We are now requesting clearance for the burden associated with one additional round of student records data collection (student records data from the 2010-2011 school year) from participating districts due to the later than expected implementation of the magnet programs in the 2007 grantee cohort. As prior data collection (187 hours) will be complete when this 2-year extension package is approved, the burden request in the current package is only for the one year of student records collection (280). Therefore, the burden change is +93 hours (this is the difference between the to-be-completed currently approved package burden of 187 and the currently requested burden of 280 hours).

The 2007 MSAP grants were awarded later than anticipated and many grantees implemented a ‘planning year’ in 2007-2008 prior to converting the grantee schools to magnet schools and did not convert to magnet schools (or implement a magnet program) until 2008-2009, one year later than expected. The study design, interrupted time series, requires outcome data for three years after the school converts to a magnet school. Therefore, instead of collecting data through 2009-2010 as planned (which would now only be two years after conversion due to the planning year), both the program office and IES found it necessary to extend the data collection so that the 2010-2011 school year data are collected to permit the use of the third year of outcome data for the 2007 cohort grantees. The burden and cost of the additional year of data collection is modest compared to other collections because the study team has established efficient procedures in the prior years’ collections and the one study report will incorporate these data.

Student records data were collected (the collection will be completed by the approval of this current request; anticipated in 6/11) for the three years (07-08, 08-09, 09-10) as planned. The 2010-2011 school year data will be collected once after this 2-year package is approved. In all, we will have one additional year of pre-conversion data (2007-2008) and three years of post-conversion outcome data (2008-2009, 2009-2010, and 2010-2011) for the 2007 cohort grantees.

The content in this final OMB request is identical to the latest approved request, with one exception: the addition of student records data collection for the 2010-2011 school year. In addition to generally updating the package (e.g. removing previously accepted revisions; removing instruments for which the data collection is complete), we made these revisions:

Section Two, Purposes and Uses of Data, the prose in the student data collection plan now includes the collection of student records data from the 2010-2011 school year.
Section Twelve, Estimates of Response Burden, includes a new exhibit (3c) that includes that time burden for respondents as part of the additional round of data collection (student records data from the 2010-2011 school year). This table specifies 11 respondents, 20 responses, and 280 total burden hours.
Section Fourteen, Estimates of Costs to the Federal Government, was revised and now describes a 6.5 year study costing $3,620,128 overall, averaging $556,942.80 per year, with a feasibility phase cost of $495,279 and an evaluation phase cost of $3,124,849.
Section Fifteen, Changes in Burden, was revised and now describes an increase of 93 burden hours. This is the difference between the to-be-completed currently approved package burden of 187 and the currently requested burden of 280 hours.
Section Sixteen, Plans and Schedule for Analysis, Tabulation, and Publication of Results, includes the collection of student records data for the 2010-2011 school year which will occur December 1, 2011-March 31, 2012 and a report release in spring 2013.

These revisions are highlighted in yellow. This document contains two sections. This first introductory section provides background for the data collection instrument for which clearance is sought by describing the policy context within which magnet schools operate, the characteristics of the federally funded magnet schools that are the focus of the Conversion Magnet Schools Evaluation, and the major features of the evaluation study. The second section contains Part A (Justification) of the supporting statement for the Paperwork Reduction Act Submission. A set of appendices contains the instrument for which we are requesting clearance, and a companion document contains Part B (Statisical Methods) of the supporting statement for the Paperwork Reduction Act Submission.

Study Background

Since the mid-1970s, magnet schools have been critical to school districts’ efforts to implement voluntary desegregation plans and, in some cases, court desegregation orders. More recently, they have become an important component of public school choice options available to parents, and admissions policies of such programs have evolved to include considerations other than ethnic balance—e.g., promoting socio-economic diversity, and providing options to families who want to move their children out of underperforming schools. It is estimated that, in 1999-2000, the U.S. had about 3,000 magnet schools enrolling 2.5 million students.¹

The federal government has supported the development of magnet schools through the Magnet Schools Assistance Program (MSAP) since the mid-1980s. Legislative authority for MSAP is found in the Elementary and Secondary Education Act of 1965, as amended, Title V, Part C; 20 U.S.C. 7231-7231j.² The program was most recently amended by sections 5301-5311 of the No Child Left Behind Act of 2001 (NCLB). The program awards 3-year grants to school districts for use in implementing new or revised magnet programs in elementary and secondary schools. In a given year, about 10 percent of the nation’s public magnet schools receive support from MSAP. During the 2007 cohort funding cycle (grants awarded in 2007), MSAP supported programs in over 175 schools located in 41 districts. The annual appropriation for MSAP between 2004 and 2010 averaged $106 million.³

Section 5310 of the NCLB statute authorizes the secretary of education to use MSAP monies to evaluate the program. Participation in the evaluation by MSAP grantees is required to obtain or retain benefits. Under the Education Department General Administrative Regulations (EDGAR), section 75.591, all eligible MSAP grantees are required to participate in a national evaluation as EDGAR specifies a grantee shall cooperate in any evaluation of the program by the Secretary. The MSAP is subject to EDGAR. The National Center for Education Evaluation (NCEE) of the Institute of Education Sciences (IES), in collaboration with the Office of Innovation and Improvement (OII), has awarded a five-year contract to the American Institutes for Research (AIR) and its subcontractor, Berkeley Policy Associates (BPA), investigate the relationship between the introduction of elementary magnet programs using funds from MSAP grants awarded in 2004 or 2007 and student outcomes that include student achievement and minority group isolation.

Conversion Magnet Schools and Their Students

Federal statute defines a magnet school as a public school or education center that offers a special curriculum capable of attracting substantial numbers of students of different racial backgrounds.⁴ Most elementary magnet schools begin as “regular” schools that serve the population that lives within the boundaries of a residentially defined service area—the school’s attendance area (also often referred to as an attendance zone). Students in this area who attend the school are its “resident” students. Typically, these schools serve attendance areas with high concentrations of low-income, minority group students who historically have had low levels of academic achievement.

Schools become conversion magnet schools when they introduce a special curriculum or instructional approach (the magnet program) and seek to attract and enroll non-resident students (i.e., students living outside the school’s regular attendance area). The expectation is that by increasing the diversity of students attending the school and involving students in engaging and rigorous academic programs, the conversion will improve the resident students’ academic performance and reduce minority group isolation of students in schools with substantial proportions of minority students.⁵ Non-resident students are expected also to benefit academically from the quality of the magnet program and the diversity of their classmates. In addition, non-resident students may benefit because the magnet school that their family has chosen for them may be better able to provide them with opportunities that match their particular needs and/or interests.

The Conversion Magnet Schools Evaluation

Despite the popularity and persistence of magnet programs, there have been only a few quantitative studies of their relationship to important student outcomes. Results have been mixed, and no definitive conclusions can yet be drawn. Drawing broad conclusions is particularly challenging because the structures and target populations of magnet school programs are so varied, but the studies conducted have treated magnet schools as a single type of intervention. For instance, elementary and secondary programs differ in the degree to which they can capitalize on students’ career and college plans and the pattern of standardized testing that provides evidence of student achievement over time. Some schools operate “programs-within-a-school” (PWSs) in which the magnet program serves only a fraction of their enrollment, while others operate whole-school programs that are designed to affect all of their students. While most magnet schools (particularly at the elementary level) serve both the residents of an attendance area and non-residents who must apply for admission, some magnets have no attendance area and require all students to apply for admission. Magnet programs also vary in maturity—they may be new, well-established, or “stale” and in need of revision to strengthen the curriculum and generate new interest among potential applicants.

The Conversion Magnet Schools Evaluation to which this request pertains is an investigation of the relationships between some MSAP-funded magnet schools and the academic achievement and minority group isolation of the students who attend them. The study avoids some of the limitations of earlier studies by focusing on a single, relatively large group of these schools that have several characteristics in common: they are elementary schools that converted into whole-school magnets during the 2004 or 2007 MSAP grant cycle, and serve a mixed enrollment of resident and non-resident students.

The differing circumstances of the two groups of students necessitate the use of different research designs to study the relationship between conversion magnet schools and student outcomes. Because resident students are not randomly selected for admission to the magnet school, resident student outcomes cannot be examined using an experimental design. Rather, we are conducting a rigorous quasi-experimental, comparative interrupted time series design in which outcomes for resident students in MSAP-funded conversion elementary magnet schools are compared with those for resident students in matched non-magnet comparison schools in the same district. For non-resident students a fixed effects model is being conducted in which the achievement gains of individual students switching between traditional public schools and conversion magnet schools are examined. This approach removes the influence of unobserved characteristics such as students’ level of motivation, to the extent that these characteristics are fixed during the period under study⁶.

Supporting Statement for Paperwork Reduction Act Submission

A. Justification

1. Circumstances Making Collection of Information Necessary

Through the provision of the Magnet Schools Assistance Program (MSAP), the federal government aims to make a significant contribution to supporting magnet schools by promoting an intervention strategy that:

offers distinctive educational curriculum or teaching methods in a manner that increases parental choice options by making programs available to students both inside and outside of neighborhood attendance areas; and
prevents, reduces, or eliminates minority group isolation by attracting students of diverse backgrounds.

The research literature identifies each of these mechanisms—curricular focus or teaching method, parental choice, and diversity in student compositions—as possible avenues for enhancing the academic achievement of students. Moreover, the explicit objective of reducing racial and ethnic minority group isolation makes MSAP unique among the federally funded programs of the Elementary and Secondary Education Act.

MSAP priorities include supporting schools and students most in need of educational improvement, the primary target of ESEA and new programs under the American Recovery and Reinvestment Act of 2009 (Pub.L. 111-5). Specifically, MSAP prioritizes support to districts that (a) establish magnet programs in schools identified for school improvement, corrective action, or restructuring under Title I and to improve the quality of teaching and instruction in the schools or (b) maximize the opportunity for students in low-performing schools to attend higher-performing magnet schools, thereby reducing minority group isolation in the low-performing sending schools.⁷ Participation in the evaluation by MSAP grantees is required to obtain or retain benefits. Under the Education Department General Administrative Regulations (EDGAR), section 75.591, all eligible MSAP grantees are required to participate in a national evaluation as EDGAR specifies a grantee shall cooperate in any evaluation of the program by the Secretary. The MSAP is subject to EDGAR.

As mentioned above, magnet programs are a popular and longstanding intervention with an estimated 2.5 million students enrolled as of 1999-2000. MSAP has supported magnet schools since the mid-1980s and with an annual appropriation that recently averaged $106.6 million (2004 through 2009). Most recently, MSAP is supporting 175 schools in 41 school districts (grants awarded in 2007).

Research on Magnet School Programs

Despite the popularity and durability of the magnet school concept, scientifically rigorous research on the effectiveness of magnet school programs is inconclusive. A review of the research literature on the effects of magnet programs on student achievement identifies 3 random assignment studies using lotteries and 12 quasi-experimental studies using non-randomized comparison groups with pre and post-tests controls.

Collectively, the lottery based studies of the effects of magnet schools on student achievement are inconclusive. A study of programs-within-a-school (PWSs) and whole school “career” magnet programs in New York City high schools in the late 1980s and early 1990s determined that the programs not only failed to have an effect on reading scores, absenteeism, or on the likelihood of students taking advanced graduation/college admissions tests, they also appeared to have a negative effect on high school graduation rates and mathematics test scores.⁸ In a recent study of elementary and secondary schools in San Diego, the authors generally found no differences in mathematics or reading test scores between lottery winners and losers in any of the district’s school choice programs, including magnet programs. As an important exception, the authors did report that winners of lotteries to attend magnet high schools performed better on mathematics achievement tests 2 and 3 years later.⁹ In the third study of lottery winners and losers to middle school magnet programs in a mid-sized Southern school district, the positive effect of magnet programs on mathematics achievement tests disappeared when the authors controlled for student demographics and prior achievement. The authors suggest that the most likely explanation for this is a differential pattern of attrition among lottery winners and losers.¹⁰^,¹¹

The 12 quasi-experimental studies of the relationship between magnet programs and student achievement date largely from student cohorts of the 1980s and early 1990s and include analysis of the relationship of magnet programs to test scores in reading, mathematics, and other subjects.¹² Seven studies were conducted on elementary school magnet programs, three on middle school programs, and two on high school magnets. Some of the studies examine whole school programs, while others focus on program-within-a-school (PWS) magnets, and a few consider both. The studies of PWS magnets tend to be more consistent in showing positive outcomes than studies of whole school programs. The PWS magnets, however, are often very selective of students, and the studies of those programs may be particularly subject to selection bias. Although whole school magnets provide programs that are generally more available to all students, the results from studies of those magnet programs tend to be mixed.

Research on the relationship of magnet schools to desegregation is even more limited than research on the relationship of magnet schools to student achievement. Two descriptive studies reported that over half of desegregation-targeted schools in MSAP funded districts succeeded in preventing, eliminating or reducing minority group isolation.¹³^,¹⁴ The San Diego study cited previously, uses the outcomes of lotteries to examine the effect of school choice options, including magnet schools, on racial, socioeconomic and other forms of integration district-wide.¹⁵ The results indicated that magnet schools increased the exposure of White to non-White students, and vice versa. The effect of magnet schools on socioeconomic integration was inconclusive.¹⁶ While the San Diego study makes an important contribution to examining the relationship of magnet schools to integration, the study is restricted to a single district. The earlier descriptive studies of the relationship of MSAP funded magnet schools to minority group isolation do not provide for a controlled comparison between magnet and non-magnet schools.

Each of the prior studies has limitations, and the mixed findings indicate that no definitive conclusions can yet be drawn about the effects of magnet schools and programs on important student outcomes. Drawing broad conclusions is particularly challenging because the structure and target population of magnet school programs are so varied.¹⁷^,¹⁸ Another important limitation of existing studies is that most focus on students who have actively applied for a program, thereby overlooking the effect of magnet programs on resident students who may have been admitted to the program because they live in the school’s attendance area.

Conversion Magnet Schools Evaluation

IES, in collaboration with OII, initiated the Conversion Magnet Schools Evaluation due to the popularity and persistence of magnet programs and the inconclusive research on the relationship of these programs to important student outcomes. Section 5310 of ESEA authorizes the Secretary of Education to carry out evaluations that address, among other things, how and to what extent magnet school programs lead to educational quality and improvement, and the extent to which they lead to elimination, reduction, or prevention of minority group isolation. The legislation also directs the Secretary to collect and disseminate information on successful magnet schools to the general public (see Appendix A).

This study is addressing limitations to previous work identified above by (1) focusing on schools that are similar in that they represent the most common category of school receiving funding through MSAP (elementary whole-school magnets) and are new (i.e. conversion) magnets (2) studying the relationship between participation in magnet programs and important outcomes for both resident and non-resident students, and (3) examining the relationship of magnet school conversion to minority group isolation through a controlled comparison of magnet and non-magnet schools.

2. Purposes and Uses of the Data

The data collected as part of the Conversion Magnet School Evaluation are needed to address the following main research questions:

How does the conversion of a neighborhood school to a magnet school relate to the educational achievement of resident (neighborhood) students?
To what extent does the conversion of a neighborhood school to a magnet school reduce minority group isolation in the school?
How does magnet school attendance relate to the educational achievement of non-resident students?
To what extent do the new magnet schools funded through the 2004 and 2007 MSAP evolve over time in terms of their program structure and content?

A grantee screening protocol was used to address the feasibility study research questions (now complete). To address the evaluation research questions, we are using student records data, a principal survey, and an interview with MSAP project directors/school choice coordinators. The descriptions of the grantee screening protocol, principal survey, and interview with MSAP project directors/school choice coordinators are provided for background information. However, as the use of these instruments will be completed by time the current package is approved, the instruments in the appendix of this package have been deleted.

Grantee Screening Protocol

The purpose of the district screening protocol was to gather information about the 2004 and 2007 MSAP grantee districts and schools needed to determine whether a rigorous analysis of the relationship between attending a magnet school and student achievement was feasible, and whether the districts had the capacity to provide the necessary data for such a study. The protocol is organized by topic into four modules directed at a district official who is knowledgeable about the subject. These include:

Module A: MSAP Project Director Interview, which covers the number and characteristics of both elementary magnet schools funded by MSAP and potential comparison schools.
Module B: District Assessment Representative Interview, which covers the assessments used by the district since 2001-2002.
Module C: District School Choice Coordinator Interview, which covers the operation of the district’s magnet assignment procedures, including the numbers of students winning and losing in lotteries to attend the magnet elementary schools, and record-keeping of these data.
Module D: District Data Management System Representative Interview, which covers the content, format, and linkability of record-data in the district’s student data management system(s).

A notification letter was sent to the 2004 and 2007 MSAP grantees that were potentially eligible to be in the study. A brochure describing the overall study accompanied the notification letter and was distributed to all interviewees in the screening process.

The screening of grantee districts was, of necessity, a semi-structured process as districts varied considerably in size, data management capacity and sophistication, and the details of their choice systems. While the protocol indicates specific information needed by the study, the interviewers asked for clarification when initial responses were ambiguous, incomplete, or unanticipated. While the information needed to assess the feasibility of including each district was fairly extensive, the screening process was designed to reduce burden in three ways. First, it was organized by topic into four modules (each of which required approximately 30 minutes to complete) for use with officials who were particularly knowledgeable about specific subjects. The two modules that were administered first (those pertaining to the district’s magnet/comparison schools and student achievement tests) allowed researchers to eliminate some districts from consideration without administering the two other modules (pertaining to the school choice system/lotteries and the data management system). Thus, while the estimated average time to complete all four modules was 2 hours (distributed among three or four different individuals), the burden for some districts was an hour or less. We estimated that all four modules would be administered to about two-thirds of the districts. Second, to the extent possible, existing data about schools and assessments (e.g., enrollment data from the CCD and assessment information from federal, state, and district websites) were collected and pre-coded into each district’s protocol prior to the interviews.

Pre-coding enabled district staff to verify some information rather than search for it themselves. In addition, it helped to highlight areas in which a district may not meet study criteria so that interviewers can focus their initial questions on the factors most likely to eliminate the district from consideration. Finally, advance copies of the pre-coded protocol were sent to the district so that officials would know what information would be requested during the interview.

Student Data Collection Plan

Student test scores (the outcome measure) and data on student background characteristics (covariate measures) are the core data needed by the Conversion Magnet Schools Evaluation to answer the research questions pertaining to magnet schools’ relationship to student achievement and minority group isolation. Districts participating in the study from both the 2004 and 2007 cohorts have been asked to provide individual student record data for each of the 3 years prior to their receiving an MSAP grant and for at least 3 years after the grant award.¹⁹ The 2004 grantees have been asked to provide data for the 6-years from the award of their grants, which is to say from 2004-2005 through 2009-2010. The 2007 grantees have been asked to provide data for the 3 years from the award of their grants, which is from 2007-2008 to 2009-2010. Data for all years prior to 2008-2009 were requested in early 2009, and data for 2008-09 were requested in late 2009 and early 2010. These data collections were approved as part of the first OMB review. The data for 2009-2010 will be completed by the time this most recent package will be approved (anticipated 6/11).

For the reasons provided in the introduction, it is now necessary to collect data for the 4^th year after award from the 2007 grantees (the 2010-2011 school year). The data for the 2010-2011 school years will be collected in late 2011 and early 2012 as scores for spring 2011 state tests become available (after the approval of the current OMB request).

Data required by the evaluation include individual student test scores in English language arts and mathematics, information allowing each student to be identified as a resident or non-resident of the school he or she attended; and demographic variables that will be used as covariates to reduce variance in the analysis. Data for each student stored in different data systems must either be linked together in the files provided by the district or transmitted in a form that will allow the study team to make the linkages themselves.

In all of the participating districts, student record data are being collected for all elementary school students in the district rather than for selected schools. This is done for three reasons. First, it enables the study to examine outcomes for resident students in the magnet and comparison schools, and students who transfer to the magnet school. Second, having data for all elementary students in the district will permit supplementary analyses that might either strengthen the interrupted time series or student fixed effects analysis, or allow alternative analyses should those approaches not be feasible for a particular district. Third, extracting data for all students is easier than extracting data for selected schools and/or individual students, and thus reduces the burden of data collection for the districts’ data managers.

Most of the data required to investigate the reduction of minority group isolation in conversion magnet schools is being ascertained from extant data sources (e.g. the National Center for Education Statistics’ (NCES) Common Core of Data (CCD) or ED’s EDFacts, a centralized performance database). In addition to examining the composition of entire schools, the minority group isolation study is also investigating, to the extent feasible, the composition of classrooms within schools. The study is requesting information on individual students’ classroom assignments as part of the student data request. Where student-level data are not available, the study is determining whether summary data are maintained and can be readily provided on classroom composition by such factors as gender, race-ethnic group, income, and English language status.

The student data collection plan describes the longitudinal student records data and the classroom- and school-level enrollment summaries that are being requested from each district. The Student Data Collection Plan appears in Appendix I. It is accompanied by a brochure describing the evaluation (Appendix J) that is given to data managers for districts in the study. The variables being collected with the student record data and their purposes are detailed in Appendix K.

The study is taking steps to minimize the burden on districts of providing this data. First, AIR has systems programmers with experience in dealing with complex data formats and a variety of data platforms. The experience of these programmers is being used to provide assistance, for example, in linking data files. Additionally, an individual AIR or BPA staff member has been assigned to work with each district to compile detailed information on the data being provided. For some districts, the compilation of disparate sets of documentation may, in fact, be a side benefit that districts receive by participating in this evaluation.

Principal Survey

The principal survey was administered to the principals in the 2004 MSAP grant cohort of districts in 2008, and is being administered in fall 2010/winter 2011 to principals in both the 2004 and 2007 MSAP grant cohorts (This data collection will be completed by the anticipated approval of the current request – 6/11). The survey provides key information needed to (1) interpret the results of the achievement and desegregation analyses; (2) document the differences between the magnet and comparison schools; (3) place these schools in the larger context of the nation’s schools (through comparisons on key variables between the study schools and the schools surveyed in NCES’s Schools and Staffing Survey (SASS) and Early Childhood Longitudinal Study (ECLS); and (4) describe the nature and evolution of the magnet schools in the years following the grant award. Questions relating to the first three of these purposes are answered by both magnet and comparison school principals; questions relating to the evolution of the school’s magnet program are answered only by the magnet school principals. The fall 2010 principal survey (approved by OMB in June 2010) is a revised version of the survey that was approved by OMB in July 2007 and administered in May 2008 to magnet and comparison school principals in the 2004 MSAP grant cohort.

MSAP Project Director/School Choice Coordinator Interview Protocol

This semi-structured interview is designed to obtain information from the MSAP Project Directors or School Choice Coordinator in each of the 14 grantee districts that will inform the analyses of student achievement and minority group isolation and help describe the evolution of the magnet school programs in those districts (This data collection will be completed by the anticipated approval of the current request – 6/11). On the basis of the screening protocols used at the beginning of the study, it was estimated that the average time to complete the revised interview would be 30 minutes. To facilitate an efficient interview process, portions of the protocol are pre-coded with information gathered during the earlier feasibility or screening phase of the study so that our informants can simply confirm that the information is still accurate or indicate changes that may have occurred. Additionally, interviewers are directed to complete the interviews within 30 minutes.

3. Use of Information Technology to Reduce Burden

During the feasibility phase, the initial screening of the grantees involved semi-structured telephone interviews with district officials. The participating officials were sent copies of the interview protocols via email in advance of the interviews so they could prepare themselves for the discussion, but they were not required to produce written responses. Where possible, the study team also reduced burden on district officials by obtaining enrollment and accountability data from the CCD, EDFacts, and other electronic sources (e.g., from district websites and state testing agencies).

During the evaluation phase, the major data collections involve extraction of student demographic and achievement data from pre-existing electronic data files maintained by districts. All data and documentation will be submitted to the study team via secure electronic transmissions.

Only one paper-and-pencil survey was administered during this study. During the evaluation phase, a survey of magnet and selected comparison school principals was administered at the end of each grant period. This survey did not involve information technology, as developing web-based methods would not have been cost-effective given the relatively modest sample size of the study. In some circumstances, information technology was used to reduce burden. For instance, telephone calls were used to remind respondents to complete the surveys, and project staff offered to help non-respondents complete the survey over the phone. The survey is designed to reduce burden on respondents: most of its questions were answered by checking a response option, and the form is short enough to be answered in 35 minutes or less. In addition, principals were given the option of completing a paper-and-pencil or fillable electronic version of the survey.

4. Efforts to Identify Duplication

The Conversion Magnet Schools Evaluation is the only large-scale study currently underway that will apply rigorous experimental and quasi-experimental methods to examining the relationship between magnet school programs and the achievement and minority group isolation of the students who attend them. Magnet schools are identified in NCES’s CCD and SASS surveys, and this information is also included in the National Longitudinal School-Level Standardized Score Database. However, data available through these surveys do not allow analysts to differentiate between resident and non-resident students and thus would not permit rigorous analyses based on disaggregated data.

States and districts nationwide do collect student assessment and background data. The Conversion Magnet Evaluation will assemble these existing data rather than administering its own achievement tests, thereby reducing burden on the participating districts and schools and avoiding duplication of data collections.

5. Methods to Minimize Burden on Small Entities

No small entities will be impacted by this project.

6. Consequences of Not Collecting the Information

As required by ESEA, school districts must provide families whose children attend underperforming schools additional school choices in which to enroll their children. The Conversion Magnet Schools Evaluation represents one of the first efforts by the Department of Education to conduct a rigorous study of therelationship between magnet school programs and student outcomes. Failure to collect the information proposed in this request will prevent ED from evaluating this form of special educational programming that is supported by federal Magnet Schools Assistance Program grants. More generally, without this study, policymakers will have a limited basis on which to judge the value of investing in magnet schools, and parents will lack information for deciding whether to send their children to magnet schools.

7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.5

No special circumstances apply to this study with respect to any requirements of respondents, pledges of confidentiality, or use of statistical data classification.

8. Consultation Outside the Agency

The study employs a technical work group (TWG) to advise AIR/BPA on data collection instruments, study feasibility, research designs, and methodological issues including combining results across sites that use diverse types of student assessments. The consultants bring expertise in magnet schools, school choice, studies of racial segregation and achievement gaps, experimental and interrupted time series designs, and the analysis of complex student assessment data. The consultants and their affiliations are listed in Exhibit 2.

Exhibit 2. Technical Working Group Members for the Evaluation of Conversion Magnet Schools Study

Name	Position and Institution	Expertise
Adam Gamoran	Professor of Sociology and Educational Policy Studies (University of Wisconsin—Madison)	Empirical studies of magnet schools and school choice
Dale Ballou	Associate Professor of Public Policy and Education (Vanderbilt University)	Empirical studies of magnet schools and school choice
Ellen Goldring	Professor of Education Policy and Leadership (Vanderbilt University)	Studies of magnet program characteristics
Ronald Ferguson	Lecturer in Public Policy and Senior Research Associate (Kennedy School of Government, Harvard University)	Studies of racial segregation and achievement gaps
Steven Rivkin	Associate Professor of Economics (Amherst College)	Studies of racial segregation and achievement gaps
Thomas Dee	Associate Professor of Economics (Swarthmore College—on leave at Stanford University School of Education, 2006-7)	Studies of racial segregation and achievement gaps
Jason Snipes	Vice President, Center for Research Evaluation and Technology (CERET), Academy for Educational Development	Experimental and interrupted time series designs
Larry Hedges	Professor of Sociology and Psychology (University of Chicago)	Combining measurements from different state tests

During the feasibility phase of the project, the TWG members were available for consultation on an as-needed basis to comment on data collection instruments and to review drafts of the memorandum on evaluation feasibility and study options. TWG members meet twice in the evaluation phase. During the first meeting in October 2008, the TWG reviewed and commented on the analysis plan for the evaluation. The second meeting will be to review and comment on preliminary results from the analyses. TWG members may also review drafts of the final report.

The 60 day FR notice was published VOL 76, page 10573, February 25, 2011. One comment of support was received.

9. Payments or Gifts to Respondents

Principals were compensated $25 to complete the principal survey, which requires about 35 minutes of their time. Principals have many demands on their time during the school day and thus have limited time to respond to surveys. Typically, they complete surveys outside normal work hours. Therefore, we are paying to offset the time principals spend completing the survey, with the payment proportional to their estimated hourly wage. Since the survey was expected to take 35 minutes to complete, about 60% of an hour, the principals are compensated $25 or 60% of the hourly rate for elementary principals based on the median elementary school principal’s hourly rate. We believe that this is a medium burden activity.²⁰

10. Assurances of Confidentiality

None of the information collected will be reported or published in a manner that would identify individual respondents.

To ensure that the data collected are not available to anyone other than authorized project staff of the contractor and subcontractor, a set of standard confidentiality procedures is being followed during the data collection process:

All project staff agree in writing to an assurance of confidentiality.
All project staff keep completely confidential the names of all respondents, all information or opinions collected during the course of the study, and any information about respondents learned incidentally.
Reasonable caution is being exercised in limiting access to data during transfer, storage, and analysis to persons working on the project who have been instructed in the applicable confidentiality requirements for the project. In particular, electronic data files are encrypted for transfer between the districts and the contractor, and stored on secure servers at AIR and BPA. Respondents send paper surveys directly to AIR/BPA via FedEx (which provides rapid transmission and tracking if problems arise). Paper surveys are stored in locked files. After the project is completed, the contractors will destroy all identifying information.
To allow the linking of student data across files (e.g., connecting demographic information and test scores, or test scores from successive years) without using student names or the identification codes used by their districts, all of the participating districts are creating a randomly assigned pseudo-ID number for each student. Pseudo-ID numbers rather than the actual ID codes are included in all data files provided to the study in order to protect the identities of individual students. The cross-walk between actual identifiers and the pseudo-IDs is being maintained by the districts and not provided to the study team.
The Project Director is responsible for ensuring that all contractor personnel involved in handling data on the project are instructed in these procedures and comply with these procedures throughout the study.
The Project Director ensures that the data collection process adheres to provisions of the U.S. Privacy Act of 1974 with regard to surveys of individuals for the Federal government.

Finally, the following explicit statement regarding confidentiality is be included in notification letters, study descriptions, and instructions to survey respondents: “Responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district or individual. We will not provide information that identifies you, your school, or your district to anyone outside the study team, except as required by law.”

11. Questions of a Sensitive Nature

The data collection instruments do not request any data of a sensitive nature to individual persons.

12. Estimates of Response Burden

Exhibit 3a presents the original estimates of total respondent burden for the data collections during the 3-year period for which OMB approval of instruments was sought in 2007. These estimates were based on the assumptions that 40 MSAP grantee districts would be screened, that 50 conversion magnet schools and 100 comparison schools from 20 districts would participate in the study, and 10 of the participating districts would come from each of the two grantee cohorts. The total estimated hour burden for these 3 years was 3,186 hours: 64 hours for district officials; 3,085 hours for district data managers; and 37 hours for principals. Across these 3 years, the average annual number of responses was 94 and the average response hours were 1,062.²¹

By spring 2008, it was apparent that the sample of districts and schools recruited for the study was smaller than the numbers upon which the numbers in Exhibit 3a were based. The revised estimates were based on the actual sample of 25 magnet and 50 comparison schools rather than the 50 magnet and 100 comparison schools projected in the original OMB request. Accordingly, in March 2008, NCEE submitted a revised burden estimate in which the estimated total numbers of respondents and hours for the first three years of the study were halved. Specifically, the number of respondents was reduced from 282 to 141 (which reduced the annual average respondents from 94 to 47), the number of respondent hours was reduced from 3,186 to 1,593 (which reduced the annual average hours from 1,062 to 531).

Exhibit 3b summarizes the estimated respondent burden for the data collections planned for the final two years of the study, for which clearance was attained in June, 2010 and for which data collection will be completed by the time this current package is approved (anticipated 6/11) . The exhibit below is a revised version of the exhibit that appeared in the April 2007 clearance request. Revisions are based on current information about the numbers of districts and schools now in the study sample. The total estimated hour burden for these 2 years is 373 hours: 6 hours for district officials; 329 hours for district data managers; and 38 hours for principals. Across these 2 years, the average annual number of responses is 50 and the average annual response hours are 187. This burden estimate includes the following data collection activities:

Local MSAP project directors or school choice coordinators will participate in semi-structured interviews focused on changes that have occurred over the years of their MSAP grants. The interviews will take approximately 30 minutes to complete. (Assumes 14 grantee districts and an 85% response rate.)
District data management staff from 13 data management departments representing the 14 participating districts (two of which share a common data management system), will respond to one request for student data for the 2009-2010 school year. They will (1) extract, format, and submit student records data along with a data dictionary; and (2) create and maintain a file of pseudo-ID numbers cross-walked with students’ actual identifiers.²²,²³
District data management personnel from 10 school districts will, as necessary, extract and submit classroom composition data for the 2009-2010 school year. It is expected that 10 MSAP grantee districts will be able to provide classroom composition data.²⁴
Principals of 64 magnet and comparison schools in the 14 MSAP grantee districts will spend about 35 minutes completing a principal survey in 2010 (assumes 75 schools and an 85 percent response rate).

Exhibit 3c summarizes the estimated respondent burden for current request. This collection includes the additional round of student records data for the 2010-2011 school year.

Exhibit 3a. Time Burden for Respondents During the First Three Years of the Evaluation (April 2007 Estimates; Already Approved by OMB)

Task	Total Sample Size	Estimated Response Rate	Projected Number of Respondents	Time Estimate (in person-hours)	Number of Administrations or Collections	Total Hours
District Officials
Grantee Screening Interview (Module A)¹	40	100%	40	0.5	1	20
Grantee Screening Interview (Module B)	36	100%	36	0.5	1	18
Grantee Screening Interview (Module C)	26	100%	26	0.5	1	13
Grantee Screening Interview(Module D)	26	100%	26	0.5	1	13
District Data Managers
First Student Data Request for 2004 Grantee Cohort (2001-2002 through 2006-2007)²	10	100%	10	121	1	1,210
First Student Data Request for 2007 Grantee Cohort (2004-2005 through 2006-2007)²	10	100%	10	76	1	760
Second Student Data Requests for 2004 and 2007 Grantee Cohorts (2007-2008)²	20	100%	20	23	1	460
Third Student Data Requests for 2004 and 2007 Grantee Cohorts (2008-2009)²	20	100%	20	23	1	460
First Classroom Data Request for 2004 Grantee Cohort (2001-2002 through 2006-2007)³	5	100%	5	18	1	90
First Classroom Data Request for 2007 Grantee Cohort (2004-2005 through 2006-2007)³	5	100%	5	9	1	45
Second Classroom Data Requests for 2004 and 2007 Grantee Cohorts (2007-2008 )³	10	100%	10	3	1	30
Third Classroom Data Requests for 2004 and 2007 Grantee Cohorts (2008-2009)³	10	100%	10	3	1	30

(continued on next page)

Exhibit 3a. Continued

Task	Total Sample Size	Estimated Response Rate	Projected Number of Respondents	Time Estimate (in person-hours)	Number of Administrations or Collections	Total Hours
Principals
2007 Principal Survey (2004 Grantee Cohort)⁴	75	85%	64	0.6	1	37
Totals			282			3,186

¹ The grantee screening protocol consists of four modules, each of which will require an average of 30 minutes to complete. Information gathered in Modules A and B will eliminate some districts from the simple, making administration of other modules unnecessary.

²Hours estimated for providing student records data include time in consultation with AIR/BPA staff about content and formatting needs, identifying lottery applicants, retrieving and extracting data, linking data across files, and documenting and transmitting files. Estimates assume that providing the three earliest years of data (2001-2002 through 2003-2004) will require 20 hours per year of data while providing more recent years of data will require 15 hours per year of data. Estimates also include 16 hours to create a file of student pseudo-IDs during the first wave of student data collection and 8 hours for each of the three subsequent waves to update the file.

³ Assumes that half of the districts in each grantee cohort supply classroom data, and that file extractions require 3 hours each per year of data.

⁴ The principal survey is estimated to take 35 minutes (0.58 hours) to complete.

Exhibit 3b. Time Burden for Respondents During the Final Two Years of the Evaluation (2010 Estimates; Already Approved by OMB)

Task	Total Sample Size	Estimated Response Rate	Projected Number of Respondents	Time Estimate (in person-hours)	Number of Administrations or Collections	Total Hours
District Officials
Coordinator Interview in 2010	14	85%	12	0.50	1	6
District Data Managers
Fourth Student Data Request for 2004 and 2007 Grantee Cohorts (2009-2010)^1,2	13	100%	13	23	1	299
Fourth Classroom Data Request for 2004 and 2007 Grantee Cohorts (2009-2010)³	13	75%	10	3	1	30
Principals
2010 Principal Survey (2004 and 2007 Grantee Cohorts)⁴	75	85%	64	0.6	1	38.4
Totals			99			373.4

¹Hours estimated for providing student records data include time in consultation with AIR/BPA staff about content and formatting needs, identifying lottery applicants, retrieving and extracting data, linking data across files, and documenting and transmitting files.

²The study includes 14 grantees of which 2 share a common data management system. Consequently, while there are 14 magnet program coordinators, student data will be requested from 13 data managers.

³Assumes that 75 percent of the districts in each grantee cohort supply classroom data, and that file extraction requires 3 hours per year of data.

⁴The principal survey is estimated to take 35 minutes (0.58 hours) to complete.

Exhibit 3c. Time Burden for Respondents During the Additional Round of Data Collection (Current Request)

Task	Total Sample Size	Estimated Response Rate	Projected Number of Respondents	Time Estimate (in person-hours)	Number of Administrations or Collections	Total Hours
District Data Managers
Fifth Student Data Request for 2007 Grantee Cohorts (2010-2011)^1,2	11	100%	11	23	1	253
Fifth Classroom Data Request for 2007 Grantee Cohorts (2010-2011)³	11	75%	9	3	1	27
Total Responses			20			280

²The study includes 12 school districts that received MSAP awards in 2007, and 2 of them share a common data management system. Consequently, student data for 2010-2011 will be requested from 11 school district data managers. Therefore, there are 11 total respondents, 9 of whom will respondent twice, bringing the total number of responses to 20.

³Assumes that 75 percent of the districts in each grantee cohort supply classroom data, and that file extraction requires 3 hours per year of data.

13. Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record-Keepers

There are no capital or start up costs.

14. Estimates of Costs to the Federal Government

Cost of the evaluation contract: The estimated cost for the 6.5-year study contracted to AIR/BPA—including conducting a feasibility investigation; developing a detailed study design, data collection instruments, and justification package, and completing data collection, data analysis, and preparing reports—is $3,620,128 overall, averaging $556,942.80 per year. The total cost of the feasibility phase is $495,279, and total cost of evaluation phase is $3,124,849.

15. Changes in Burden

As prior data collection (187 hours) will be complete when this 2-year extension package is approved, the burden request in the current package is only for the one year of student records collection (280). Therefore, the burden change is +93 hours (this is the difference between the to-be-completed currently approved package burden of 187 and the currently requested burden of 280 hours).

16. Plans and Schedule for Analysis, Tabulation and Publication of Results

The schedule for the publication of results for the two phases of the Conversion Magnet Schools Evaluation is shown in Exhibit 4.

A. Description of Study Reports

The schedules for data collections and the publication of results for the two phases of the Conversion Magnet Schools Evaluation are shown at the end of this section in Exhibits 4 and 5, respectively.

Feasibility Phase Reports

Two documents were submitted to ED at the end of the feasibility phase. The first was a memorandum documenting the results of the feasibility investigation and presenting recommendations about whether and how to proceed with a full evaluation of conversion magnet schools. The second was a memorandum that focuses on school characteristics during the years leading up to the conversion of the magnet schools funded through either the 2004 or 2007 MSAP grants. Tabulations show levels and trends in characteristics such as total enrollment, proportions of students from minority groups, and proportions of students eligible for free and reduced price meals of the conversion magnet schools compared with other schools in their districts.

Evaluation Phase Report

A final report will be submitted at the end of the evaluation phase. It will present the results of analyses estimating relationships between conversion magnet schools and key desired outcomes of the Magnet Schools Assistance Program—improvement of students’ academic achievement and reduction of minority group isolation. These analyses will be based on data for the school years 2001-2002 through 2010-2011. The report will include the following tabulations and analyses:

Descriptive Information

Description of the characteristics of conversion magnet schools in the 2004 and 2007 MSAP grantee cohorts (e.g., location, size, and themes)
Comparison of study schools (conversion magnets and non-magnet comparison schools) and schools nationwide to provide context for understanding how the study schools differ (if they do) from other magnet and non-magnet schools
Description of dependent variables (student achievement scores, minority group isolation in the schools)
Description of changes in enrollment composition of the districts, conversion magnets, and comparison schools between 2001-2002 and 2010-2011 (from 3 years prior to the MSAP grant to 6 years after the 2004 MSAP grants and 4 years after the 2007 MSAP grants)
Description of changes in the content and structure of the magnet programs during the 4 years following the award of their MSAP grants

Estimation of the relationship between conversion magnets and key outcomes: student achievement and minority group isolation

Results of analyses of the achievement of resident students in conversion magnet schools in the 2004 and 2007 MSAP grantee cohorts before and after the schools converted.
Results of analyses of the achievement of non-resident students who switch between a traditional public school and a conversion magnet school in the 2004 and 2007 MSAP grantee cohorts after the schools converted²⁵
Results of analyses of trends in minority group isolation in the conversion magnet schools in the 2004 and 2007 MSAP cohorts

B. Complex Techniques for Analyses of the Relationship Between Conversion Magnet Schools, Student Achievement, and Minority Group Isolation

As has been explained earlier, differences in the manner in which resident and non-resident students are assigned to conversion magnet schools require different methods for analyzing the relationship between conversion magnet schools and student achievement. This section provides additional detail on the estimation methods that will be used for the core and supplemental achievement analyses that will be conducted for each group of students. The section also discusses models for analyzing the relationship between magnet conversions and minority group isolation.

Core Analysis for Resident Students: Comparative Interrupted Time Series

The primary estimation method for analysis of the relationship between conversion magnet schools and the achievement of resident students will be an interrupted time series analysis that uses a three-level hierarchical linear model (HLM), with students at the first level, year at the second level, and school at the highest level. The analysis will be conducted separately by grade level (2nd, 3rd, 4th, or 5th grade) and by subject area (English language arts/reading or mathematics). Each grade level of resident students will be compared before and after the conversion of the school. A grade level sample is comprised of the students in a particular grade level, 3 years before and at least 3 years after their school converts; for example, the third-grade class in years 2001 to 2006 comprise a sample for analysis.²⁶^,²⁷

The HLM is specified as follows:

Level 1 – Individual Student Level

(1)

where

the outcome for student i, in grade level j, from school pair k and school s.

mean-centered background characteristics q for student i in grade level j from school pair k and school s.

a random error term for student i in grade level j from school pair k and school s (assumed independently and identically distributed (iid) across students in a grade level).

Level 2 – Year/Grade Level (e.g., third-grade students over 6 years or more)

(2)

where

= the mean outcome for grade level j from school pair k and school s for students with mean background characteristics for grade level j and school s.

= dummy variable equal to 1 for first year after conversion. 0 otherwise.

= dummy variable equal to 1 for second year after conversion. 0 otherwise.

= dummy variable equal to 1 for third year after conversion. 0 otherwise.

= dummy variable equal to 1 if school s is a magnet school, pre- or post-conversion.
0 otherwise.

= random error term for grade level j, school pair k, iid across school pairs.

= random error term for grade level j, school pair k, school s, iid across schools

Level 3 – School Pair Level

(3)

where are random error terms, iid across school pairs.²⁸

The first level is a simple regression of outcomes for individual students in a given grade j as a function of their background characteristics. The equation is included in the model to control statistically for any compositional shifts that might occur over time in the measured background characteristics of students at a given school.

The second level of the model is the comparative interrupted time-series analysis of regression-adjusted mean outcomes for a given grade-level group from a single school within a single magnet/pooled-comparison schools pair. π_0k is the regression-adjusted baseline mean student outcome for the 2 comparison schools combined. π_1k, π_2k, and π_3k are the deviations from the baseline mean student outcome for the comparison schools in the three follow-up years. π_4k is the difference between the regression adjusted baseline mean outcome for the magnet school (in years before and after conversion) and that of its comparison schools. π_5k,π_6k,and π_7k are the differences between the deviations from the baseline mean for the magnet school and its comparison school counterpart—the estimated effects of transforming the magnet school in the first, second, and third years of magnet status, respectively. For the 2004 cohort, it may be possible to include as many as six years after conversion in the analysis.

The third level represents the distribution of parameters across magnet/comparison school pairs, providing a way to summarize these findings.π_5k, π_6k,and π_7k are the best available estimates of the typical effects of a magnet school. The standard deviations of these estimates provide measures of the consistency of these effects.

Core Analysis for Non-Resident Students: Fixed-Effects Analysis

Non-resident students who switch between traditional public schools and the conversion magnet schools will be studied separately from resident students if there are enough students switching schools. The comparative interrupted time series approach we will use for resident students is not useful in the analysis of non-resident students, in part because there are no obvious comparison schools in this case. Moreover, students who self-select to apply to enroll in a magnet school are likely to differ in both observable and unobservable ways from students in their local schools who do not apply to the magnet school, as well as from resident students in the magnet schools.

Students who switch into magnet schools are likely to differ in important ways from students who decide not to do so. This heterogeneity makes it problematic to compare switchers with non-switchers as a method of inferring the relationship of switching to magnet schools and student achievement. As a remedy to this problem we will use student-level fixed-effect models in which we do not compare one student with another, but we instead compare the student’s achievement growth before he or she switches to a magnet with the growth he or she experiences after switching.

If a sufficient number of students switch between non-magnet and magnet schools, a student-fixed effects analysis will be conducted to assess the relationship of attending magnet schools to student achievement for non-resident students. The analysis will make use of data on all students in the district attending a non-magnet school and those non-resident students attending a magnet school (i.e., those students attending a school that adopted a magnet program who live outside the magnet school’s attendance zone). That is, the only students in the district excluded from the analysis are the resident magnet school students who live in the attendance zone of the magnet school that they attend. The achievement gains of individual students will be compared before and after students switch between the conversion magnet schools and traditional public schools. This approach avoids making inter-student comparisons, and removes the influence of unobserved characteristics such as student motivation, to the extent that these characteristics are fixed during the period under study.

The main requirement of this fixed-effects model is there are students who have at least one test-score gain when attending a non-magnet school, and one test-score gain when the student is in the magnet school. For example, in a district that test students starting at grade three, a student that switches from a non-magnet to a converted, MSAP magnet schools at the beginning of the fifth grade would have a gain score between third and fourth grade from the traditional public school and a gain score between fourth and fifth grade attributed to the switch to the magnet school. In districts in which elementary schools include sixth grade, the student would also have a gain score between fifth and sixth grade attributable to the switch to a magnet school if he was still in that school.²⁹

The planned fixed-effects model to estimate the effect of magnet schools on non-resident student achievement is specified as follows:

Where the observation subscript refer to student i in grade g attending school s and district d in school year t. The variables in the model are defined as follows:

= a set of dummy variables for all students where there are I in total and where is a dummy variable equal to 1 for observations from student i.

= set of dummy variables for all schools, where there are S schools in total, and where is a dummy variable equal to 1 for observations from school s, district d at time t and 0, otherwise.

= a set of grade dummy effects, where λ_igt is a dummy variable equal to 1 for observations from student i in grade g at time t, and 0 otherwise.

= a set of time-varying student background characteristics, where there are Q in total, for student i enrolled in grade g at school s, district d at time t.

= dummy variable equal to 1 if observation for student i and year t corresponding to year j, and 0 otherwise, defined separately for j = 2,…,6.

= dummy variable equal to 1 if school s attended by student i in year t is a magnet, and 0 otherwise, where year t corresponds to year j defined separately for j = 4, 5 and 6.

= random error term for student i in grade g, school s, district d and year t.

The dependent variable is a year-to-year change in achievement (we will be modelling both ELA and math outcomes separately) from the previous year to the current year (i.e., from years t – 1 to t). The coefficients represent the regression-adjusted mean outcome for each of the students across all schools attended and grades they were enrolled in. The estimated parameters show deviations from the student mean outcome for students in school s, while the coefficients show deviations from the individual mean outcome attributable to grade level g. The coefficients represent deviations from individual mean outcomes due to background characteristics that vary over time. The estimated parameters , , and show year-specific effects of being observed in the study years 3, 4, 5 or 6. Of particular interest are the final three coefficients, , and , which denote the effect on indiviudal student outcomes from switching to a magnet school for study years 4, 5 or 6.

The main challenge to this analysis is that we cannot know until the data are examined exactly how many non-resident students switch into magnet schools during the first three years in which those schools operate their magnet programs. The statistical power of this analysis will increase with the number of students who switch.

Supplementary Estimation Technique for Resident Student CITS Analysis

The CITS analysis discussed above will provide a quasi-experimental measure of the effect of attending a conversion magnet school for resident students. This section briefly addresses a supplementary student fixed-effects model to estimate the effects of staying in a school after conversion to magnet. This model can improve the internal validity of the CITS analysis by controlling for all fixed characteristics of students, both observed and unobserved, thereby reducing the chance that unobserved differences between students are being confounded with the magnet status of the school.

One approach to this fixed-effect model that is highly analogous to the comparative interrupted time series HLM model is to estimate a difference-in-difference model that compares individual students’ gains in achievement before and after a school converts to magnet status, while using gains of individual students at comparison schools to control for district-wide trends.

Core Analysis of the Estimated Effect of Magnet Conversion on Minority Group Isolation

The study will also evaluate the relationship between converting elementary schools into magnets and various measures of minority group isolation, such as the percentage of students at the school who are from minority groups. Let Rjks = one of the measures of minority group isolation for grade level j from school pair k and school s. Then this variable can be modeled using equations (2) and (3) from the HLM model proposed earlier for test scores. (A student-level model (1) is dispensed with because measures of minority group isolation are typically measured at the school or a higher level.) In this version of the HLM model, the coefficients on π5k, π6k, and π7k indicate whether the conversion magnet school changed its trend in the given measure of minority group isolation relative to the comparison school(s).

It is also possible to test whether district-wide measures of minority group isolation have changed from their pre-existing trends in the years after one or more elementary schools are converted to magnet status. However, such an approach does not allow for a comparison set of schools to be used, and so this corresponds to an interrupted time series research design rather than a comparative interrupted time series design.

C. Schedules of Data Collections and Reports

Exhibit 4 presents the schedule for the data collection activities during the feasibility and evaluation phases of the Conversion Magnet Schools Evaluation, and Exhibit 5 presents the schedule of deliverables for the two phases.

Exhibit 4. Schedule for Data Collections

Feasibility Phase	Beginning and End Dates
Screen 2004 MSAP Cohort Screen 2007 MSAP Cohort	May 1, 2007-August 1, 2007 July 2, 2007-September 28, 2007
Evaluation Phase
Survey principals in 2004 cohort	December 3, 2007-March 31, 2008
Student records data collection for 2001-2002 through 2006-2007	December 3, 2007-March 31, 2008
Student records data collection for 2007-2008	December 1, 2008-March 31, 2009
Student records data collection for 2008-2009	December 1, 2009-March 31, 2010
Student records data collection for 2009-2010	December 1, 2010-March 31, 2011
Student records data collection for 2010-2011	December 1, 2011-March 31, 2012
Final interview of MSAP project directors or district choice representatives	April 15, 2010-December 1, 2010

Exhibit 5. Schedule for Dissemination of Study Results

Feasibility Phase	Deliverable Dates
Feasibility Memorandum	February 25, 2008
Descriptive Report	March 31, 2008
Evaluation Phase Deliverables
Evaluation Report	Spring 2013

17. Approval to Not Display Expiration Date

All data collection instruments will include the OMB expiration date.

18. Exceptions to Item 19 of OMB Form 83-1

No exceptions are requested.

1 See Christenson, B., Eaton, M., Garet, M., & Doolittle, F. (2004). Review of literature on magnet schools (Report submitted to U.S. Department of Education). Washington, DC: American Institutes for Research and MDRC, p. 3. See also U.S. Department of Education. (2005). Magnet schools assistance. Retrieved January 8, 2007, from http://www.ed.gov/legislation/ESEA02/pg65.html; and Rossell, C. H. (2005, Spring). What ever happened to magnet schools? Education Next, 2. Retrieved January 8, 2007, from http://www.educationnext.org/20052/44.html

2 Program regulations are in 34 CFR 280.

3 Information downloaded from www.ed.gov/programs/magnet/funding.html

4 The statute authorizes funding of grants to districts to carry out magnet programs that are part of an approved desegregation plan and are designed to bring students from different social, economic, ethnic, and racial backgrounds together. (Sec. 5303 of NCLB)

5 The Code of Federal Regulations definition of terms for the federal government’s magnet assistance program identifies minority group to include American Indian or Alaskan Natives, Asian or Pacific Islanders, Hispanics, and Blacks (not of Hispanic origin). The code defines minority group isolation in reference to a school as, “the condition in which minority group children constitute more than 50 percent of the enrollment of the school.” (34 CFR 280.4(b) )

6As mentioned in the introduction, we are conducting a student-fixed effects analysis to assess the relationship of attending magnet schools to student achievement for non-resident students instead of a lottery study due to the existence only a few magnet schools that were oversubscribed and relied on lotteries for admittance. Most magnet schools accepted all applicants.

7 See Federal Register, February March 9, 2007 (Volume 72 Number 46), page 10723.

8 Crain, R., Allen, A., Thaler, R., Sullivan, D., Zellman, G., Little, J., & Quigley, D. (1999). The effects of academic career magnet education on high schools and their graduates. Berkeley, CA: University of California Berkeley, National Center for Research in Vocational Education.

9 Betts, J., Rice, L., Zau, A., Tang ,Y., & Koedel, C. (2006). Does school choice work? Effects on student integration and achievement. San Francisco: Public Policy Institute of California.

10 Ballou, D., Goldring, E., & Liu, K. (2006). Magnet schools and student achievement. New York: Columbia University, Teachers College, National Center for the Study of Privatization in Education.

11The lottery applicants in the study by Ballou are non-resident students applying to schools outside of the assigned school zone. Although not specifically discussed, the magnet lottery applicants in the San Diego study by Betts et al. are also likely to be non-resident students, particularly at the elementary school level. In Cain et al.’s New York City study, however, students had to apply to participate in the career magnet academies and were selected through a lottery process without apparent regard to residents.

12 See Christenson, B., Eaton, M., Garet, M., & Doolittle, F. (2004). Review of literature on magnet schools (report submitted to U.S. Department of Education). Washington, DC: American Institutes for Research and MDRC.

13 A descriptive evaluation of the 1998 MSAP funded school districts reports that 57 percent of schools targeted for desegregation with MSAP funds succeeded in preventing, eliminating, or reducing minority group isolation. (U. S. Department of Education, Office of the Under Secretary. (2003). Evaluation of the Magnet Schools Assistance Program, 1998 grantees. Washington, DC: Author. See page xii. [Retrieved January 8, 2007, from http://www.ed.gov/rschstat/eval/choice/magneteval/finalreport.pdf]). An earlier descriptive study of the 1989 and 1991 MSAP funded school districts reported that 64 percent of desegregation targeted schools met or made progress in meeting their desegregation objectives, with 57 percent moving closer to the district-wide average in terms of minority enrollment. (U.S. Department of Education, Office of the Under Secretary. (1996). Reducing, eliminating, and preventing minority isolation in American schools: The impact of the Magnet Schools Assistance Program. Washington, DC: Author.)

14 It should be noted that not all of the magnet schools that MSAP supports are the target for desegregation. In a small proportion of cases, the magnet program in one school is intended to affect the enrollment composition of one or more other schools by drawing students from those schools into the magnet school.. In such cases, these other schools are the targets for desegregation.

15 Betts, J., Rice, L., Zau, A., Tang ,Y., & Koedel, C. (2006). Does school choice work? Effects on student integration and achievement. San Francisco: Public Policy Institute of California.

16While the exposure of students of parents with a low education to students whose parents have a high education increased, so did the exposure of students whose parents education is unknown to students of parents with a high education. Since it is unclear what the unknown category represents, it is difficult to draw a conclusion.

17 The 2004 and 2007 MSAP grants provided funding for 385 schools. Nearly two-thirds (253 schools) were new magnet schools, and of these over half (137 schools) were elementary schools.]

18 Nationally in 1999-2000, over three-fifths of all magnet schools were located in elementary schools (Christenson et al., 2004, p. 3).

19 All of the districts are providing data in which each student’s records are linkable across years. Having longitudinally linked data for individual students makes possible stronger analyses than are possible with individual records that cannot be longitudinally linked.

20 According to the Department of Labor’s Occupational Outlook Handbook, 2006-2007 edition, the median annual salary for elementary school principals in 2004-2005 was $74,062. Assuming 220 workdays (or 1760 work hours) per year, the hourly rate for elementary principals without fringe benefits is $42.

21 Each row in Exhibits 3a and 3b represent a distinct data collection activity. The total projected number of respondents for the first 3 years of the project is obtained by summing the projected number of respondents shown in column 4 of Exhibit 3a. As shown in the bottom row of that exhibit, we projected a total of 282 respondents during the first 3 years of the evaluation, or an average of 94 respondents per year. The total number of hours per data collection activity (column 8) is obtained by multiplying the projected number of respondents (column 4) by the estimated time it will take each respondent to complete that activity (column 5). As shown in the bottom row of Exhibit 3a, we projected a total of 3,186 hours of respondent time during the first 3 years of the evaluation, or an average of 1,062 hours per year.

22 Actual student identifiers will be replaced by pseudo-ID codes in the files provided to AIR/BPA to ensure confidentiality of individual students’ records.

23 The estimate assumes that the response rate for this data collection will be 100 percent because the participating districts have all agreed to provide data.

24 The estimate assumes that the response rate for this data collection will be 75 percent because some of the participating districts may be able to provide other types of student record data but not classroom assignment data to the study.

25 As indicated in the introduction,-a lottery-based analysis of the achievement of non-resident students in conversion magnet schools has been dropped from the evaluation. However, a quasi-experimental analysis based on fixed effect models described below may provide an alternative for studying achievement of non-resident students in conversion magnet schools at no additional burden to districts.

26 For this analysis, students in the two comparison schools for each magnet will be pooled together as one pseudo-comparison school before being compared with the magnet school.

27 Schools using the same achievement tests will be grouped together for analysis, so it is likely that the analyses will be grouped by state.

28 The model as shown treats variation in estimated effects across magnet schools as random effects; it would also be possible to treat this variation as fixed effects.

29 In practice, the analysis will make use of the total number of switchers into or out of magnet schools to identify the relationship of magnet schools to achievement of nonresident students.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Authorised User
File Modified	0000-00-00
File Created	2021-02-01

Att_1850-0832 v.4 rev4-29 MSAP Part A 4.25.11