Att_OMB package Impact Evaluation of the DC Opportunity Scholarship Program 03-29-05

Att_OMB package Impact Evaluation of the DC Opportunity Scholarship Program 03-29-05.doc

Impact Evaluation of the DC Opportunity Scholarship Program

OMB: 1850-0800

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1850-0800 can be found here:

Document [doc]

Download: doc | pdf

U.S. Department of Education

Impact Evaluation of the DC Opportunity Scholarship Program

Office of Management and Budget

Clearance Package Supporting Statement

And Data Collection Instruments

March 30, 2005

TABLE OF CONTENTS

INTRODUCTION 1

A. JUSTIFICATION

A.1 Circumstances Making The Collection of Information Necessary 4

A.2 Purposes and Uses of the Data 4

A.3 Use Of Technology to Reduce Burden 7

A.4 Efforts To Identify Duplication 7

A.5 Methods to Minimize Burden on Small Entities 7

A.6 Consequences of Not Collecting Data 8

A.7 Special Circumstances 8

A.8 Federal Register Comments and Persons Consulted Outside The Agency 8

A.9 Payments or Gifts 9

A.10 Assurances of Confidentiality 9

A.11 Justification of Sensitive Questions 11

A.12 Estimates of Hour Burden 11

A.13 Estimates of Cost Burden to Respondents 13

A.14 Estimate of Annual Cost to the Federal Government 13

A.15 Program Changes or Adjustments 13

A.16 Plans For Tabulation And Publication of Results 13

A.17 Approval To Not Display The OMB Expiration Date 18

A.18 Explanation of Exceptions 18

COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Potential Respondent Universe and Sampling or Respondent Selection

Methods to be Used 18

B.2 Procedures for the Collection of Information 18

B.3 Methods to Maximize Response Rates and to Deal with Issues of

Non-Response 19

B.4 Tests of Procedures or Methods to be Undertaken 19

B.5 Contact Information 19

APPENDIX A. Title III--DC School Choice Incentive Act of 2003

APPENDIX B. Application and Consent Form

APPENDIX C. Confidentiality Agreement

APPENDIX D. Proposed Data Collection Instruments

IMPACT EVALUTAION OF THE DC OPPORTUNITY SCHOLARSHIP PROGRAM

SUPPORTING STATEMENT

FOR PAPERWORK REDUCTION ACT SUBMISSION

INTRODUCTION

In early 2004, the U.S. Congress passed the DC School Choice Incentive Act, Title III of the District of Columbia Appropriations Act of 2004, Division C of HR 2673 (PL 108-199). The legislation established a new, five-year school choice program for low-income residents of Washington, DC, and provided for a program operator to design and oversee parent outreach efforts, school recruitment, the student application process, and the distribution of scholarships.^¹ The program provides scholarships of up to $7,500 per student per year to enable low-income elementary and secondary students to attend private schools in lieu of the public schools already available to them. It is anticipated that, given annual appropriations of $13 million, up to 2000 students could be supported by scholarships each year, since most private schools in DC charge less than the ceiling amount for tuition and fees. The law requires that students be assigned scholarships by lottery if there are more eligible applicants than can be accommodated by the appropriation or the availability of seats in participating private schools.

The law also requires an evaluation of the program “using the strongest possible research design for determining the effectiveness” of the program (Section 309, see Appendix A). The U.S. Department of Education (ED) awarded contracts to Westat, and its research partners, Georgetown University and Chesapeake Research Associates, to (1) provide technical assistance to the program operator, particularly with respect to the design and conduct of the random assignment of participants during the baseline year of 2004, and (2) perform a 5-year impact evaluation of the program.

This document represents the Supporting Statement for the data collection and analysis to be conducted under the Impact Evaluation of the DC Opportunity Scholarship Program. In particular, we are requesting approval for: (1) parent, student, and principal surveys, (2) ongoing testing of student applicants, and (3) records abstraction from DC Public School (DCPS) administrative files.

Study Design

The foundation of the DC Opportunity Scholarship Program evaluation will be a Randomized Control Trial (RCT) comparing outcomes of eligible applicants (students and their parents) assigned by lottery to receive or not receive a scholarship. This design is consistent with the requirement for a rigorous evaluation as well as the need to fairly allocate the scholarships if the program is oversubscribed. At the same time, the law specified other kinds of comparisons and analyses, resulting in a planned evaluation study that includes both quantitative and qualitative components, and both performance (progress) reporting and measures of impact.

Research Questions

The study is designed to address the following key questions:

What is the impact of the program on student academic achievement? The law places high priority on examining whether the program—the availability and offer of scholarships—improves the academic achievement of eligible students. This question can be addressed most rigorously by comparing the academic achievement of student applicants randomly assigned to receive and not receive scholarships. However, the law also asks for a comparison of the academic achievement of students who participate in the program with their grade-level counterparts in DCPS.

What is the impact of attending private versus public schools? Because it is likely that some students offered scholarships will choose not to use them, the evaluation will also use accepted econometric methods to examine the effects for students who take the scholarship offer and enroll in a private school.

What is the impact of the program on other student measures? The law calls for examining other indicators of student school success, including persistence, retention, graduation and, if possible, college enrollment. In addition, Congress required the evaluation to assess the school safety of students who receive the scholarships relative to those who did not receive scholarships.

What effect does the program have on student and parent satisfaction with the educational options available in DC and with children’s actual school experiences? A key desired outcome of scholarship programs is an increase in both the school choices possible and parents’ and students’ satisfaction with the choices they have made. These issues will be examined by comparing the satisfaction and reasons for applying to the DC Opportunity Scholarship Program among applicants assigned by lottery to receive scholarships and those assigned to not receive scholarships.

To what extent is the program having an impact on schools in Washington, DC? Scholarship programs have been hypothesized to affect not only the students who receive the scholarships but also the broader population of public schools and students. Theory suggests that these broader outcomes could occur when public school systems respond to a fear of losing students, and therefore revenues, to private schools. These competitive effects might include changing curricula, adopting new themes or missions, and other modifications to existing policies and practices to make the public schools more attractive. Choice programs might also affect the larger population of private schools, beyond those in which the programs’ participants are currently enrolled; if choice programs are successful, additional private schools may choose to participate or new schools may be established to meet enrollment demand. However, exploring these potential systemic effects of the DC Opportunity Scholarship Program will be challenging, given the existing design of the program and the limited resources available to address this question.

Data Collection

Evaluation data will be collected for two cohorts of program applicants and include a variety of data collection methodologies. To achieve the sample sizes necessary for statistical power, the evaluation will track the progress and experiences of applicants in spring 2004 and in spring 2005. The evaluation team is collecting pre-program (“baseline”) measures of family background and student achievement and is planning to collect annual “in program” measures in order to conduct a rigorous evaluation of program impacts. These measures will be collected from the data sources described in Table 1.

Table 1. Data Measures for the Evaluation of the DC Opportunity Scholarship Program
Data Source	Description
Student assessments	Baseline test scores (SAT-9) will be abstracted from DCPS records for public school applicants; applicants who did not participate in DCPS spring assessments (primarily children below grade 3) will be tested the following fall by the evaluators using the same assessment at Saturday sessions. After the baseline year, the evaluators will administer the SAT-9 to all treatment and control group members in the spring of each year, at annual events to re-establish eligibility for the program The study will also obtain DCPS test score data for all public school students in those years, in order to draw a random sample of similarly low-income DCPS nonapplicant students, stratified by grade level, to compare with the applicants to the DC Opportunity Scholarship Program, as required by law.
School records	Administrative records will be collected from DCPS and charter school authorizers to obtain data on attendance, persistence, disciplinary actions, and grades for members of the treatment and control groups at baseline.^² In addition, the study will seek to obtain these data for all public school students, including those in charter schools, so that the program applicants can be compared to nonapplicant DCPS students in the relevant grade levels, as required by the DC Choice Act
Parent surveys	The study will conduct surveys of parents (of students in the treatment and control groups) in all four years of data collection for the evaluation. These surveys will examine such issues as reasons for applying, satisfaction with school choices, and perceptions of school safety, educational climate, and offerings. It is likely that these surveys will be administered during the annual program renewal events, with telephone follow up as necessary.
Student surveys	Each year, the study will conduct surveys of treatment and control group students who are in grades four and above, to collect information about students’ satisfaction with their schools, perceptions of safety, and other characteristics of their school program and environment. The surveys will be administered each year of the program and are likely to occur at the same time (and place) as the student assessments.
Principal surveys	The study design calls for a survey of principals in the spring of each year from (1) principals of all 109 private schools, and (2) principals of all of the 160 regular public and charter schools in DCPS. The surveys will be administered each year of the program, and will collect information about school conditions and the school environment that might affect student achievement, and awareness of and response to the DC Opportunity Scholarship Program.
DC Opportunity Scholarship Program Operator Records	As the administrator of the DC Opportunity Scholarship Program, the operator is responsible for confirming ongoing eligibility for the program and continuing participation for scholarship recipients. Although surveys of parents and students will also be conducted, Westat will collect annual data from the program operator about individual student program participation.

A. JUSTIFICATION

Circumstances Making the Collection of Information Necessary

As described in the introduction, Section 309 of the DC School Choice Incentive Act, Title III of the District of Columbia Appropriations Act of 2004 mandates the conduct of an independent evaluation of the program “using the strongest possible research design for determining the effectiveness” of the program. The legislation also lays out a series of topics and issues that the evaluation must address (see PS 108 199 Section 309, attached in Appendix A). The information collected through this study will be used as the basis for this mandated evaluation.

Purposes and Uses of the Data

Information on the DC Opportunity Scholarship Program and the outcomes of program applicants will be collected primarily by Westat, with data analyzed by Westat and its research partners, Georgetown University and Chesapeake Research Associates. This work will be conducted under Contract Number ED-04-CO-0126. The data to be collected will be obtained from student assessments, school records, and surveys of parents, students, and principals and used to address the research questions and topics identified by in the authorizing legislation. The legislation also specifies that the evaluation report annually on the performance of the program and the students; thus, annual data collection is necessary and cannot be reduced to a lesser frequency.

The student, parent, and principal surveys will all include the universe of respondents. In no case do we anticipate any unusual problems requiring specialized sampling procedures.

Table 2, on the next page, shows how each of the sources of data relates to the study questions followed by detailed descriptions of the data sources.

Table 2. Relationship Between the Study Questions and Proposed Sources of Data
Study Question	Student assessments	School records	Parent surveys	Student surveys	Principal surveys	DC Opportunity Scholarship Program Operator Records
What is the impact of the program on student academic achievement?						
What is the impact of attending private versus public schools?					
What is the impact of the program on other student measures?						
What effect does the program have on student and parent satisfaction with the educational options available in DC and with children’s actual school experiences?						
To what extent is the program having an impact on schools in Washington, DC?						

Student Assessments

Based on the legislated language, the key outcome measure for judging the effectiveness of the program is student achievement. Moreover, the law requires the independent evaluator to measure student achievement each year. For the purposes of the evaluation, we have interpreted “student achievement” as students’ skills in reading and mathematics (not science or history).

There are several key considerations that must be taken into account in order to ensure that the measurement of student achievement is a valid indicator of program impacts. Most importantly, to the extent possible, the same administration and testing environments must be maintained for both scholarship recipients (treatment group) and those who applied for but did not receive scholarships (control group). This is easy in the case of the “baseline” measurement of achievement. DCPS annually administers the SAT-9 in April in all of its schools, following a consistent test administration guide for each grade level; we plan to abstract these data for all public school applicants to the DC Opportunity Scholarship Program.

However, going forward beyond the baseline year offers some challenges. Only the control group and members of the treatment group who have declined to use their scholarships or who attrited from the program will be attending DCPS schools and participate in DCPS testing. Most treatment group members will be dispersed throughout a set of participating private schools. Private schools are unlikely to allow us to pull members of the treatment group out of their school day in order to administer the DCPS test to them. Moreover, comparing test results in those circumstances to results for students in the public schools who took the DCPS test along with all students in at their schools introduces a substantial bias. For the DCPS students, the DCPS test is likely to be more consequential, with teachers planning and preparing for it for at least several weeks. In contrast, students in the private schools will have little warning or preparation, placing them at a serious disadvantage in the comparison of achievement with public school students (the control group). This option, although requiring less burden on the control group, would lay the evaluation open to serious criticism in estimating and interpreting the key program impacts.

Instead, at the current time, we plan to administer the SAT-9 math and reading assessments when the treatment and control group families come in to renew their eligibility for the Program, so that the test administration will be similar across all types of evaluation members. The scholarship users will clearly be the most motivated to attend and we will be conscious of the need to take steps to encourage the scholarship non-users (decliners) and control group members to fulfill the requirements to participate in the evaluation’s data collection. These assessments will be administered in early April of each year, for the four years of the evaluation’s data collection.

School Records

Administrative records will be collected from DCPS and charter school authorizers to obtain data on attendance, persistence, disciplinary actions, and grades for members of the treatment and control groups at baseline. In addition, Westat will seek to obtain these data for all public school students, including those in charter schools, so that the program applicants can be compared to other students in the relevant grade levels, as required by the DC Choice Act.

Parent Surveys

The legislation requires the evaluation to examine the impact of the program on parents. The study will conduct surveys of parents (of students in the treatment and control groups) in all four years of data collection for the evaluation. These surveys will examine such issues as reasons for applying to and remaining with the program, satisfaction with school choices, and perceptions of school safety, educational climate, and offerings. These surveys will be administered to the parents when they come in to renew their child’s program eligibility, with telephone follow up as necessary.

Student Surveys

Each year, the study will conduct surveys of treatment and control group students who are in grades four and above, to collect information about students’ satisfaction with their schools, perceptions of safety, reports of behavior both within and outside of school, and other characteristics of their school program and environment. The surveys will be administered each year of the program and are likely to occur at the same time (and place) as the student assessments – the family events where the parents come in to renew program eligibility.

Principal Surveys

The study design calls for two separate principal surveys: (1) principals of all 109 private schools in DC, administered toward the end of each of the four years and (2) principals of all of the 160 regular public and charter schools in DCPS, administered toward the end of each of the four years.

The private school principal survey will focus on knowledge of the DC Opportunity Scholarship Program and ask specific questions about perceptions of the program, why the school does (or does not) participate, and how the program is integrated within their school. The public school principal survey will collect information about school characteristics, climate, how much they know about the DC Opportunity Scholarship Program, and whether they are changing anything in response to the program.

Use of Technology to Reduce Burden

The data collection plan has been designed to maximize efficiency and accuracy, and to minimize respondent burden. A key consideration in the decision to abstract baseline student achievement data from DCPS records (rather than administer our own evaluation assessment) was to minimize evaluation costs and reduce respondent burden. We will ask parents to complete a paper survey form at the time they come in to renew their eligibility, and we will follow up with telephone interviewing to offer parents the opportunity to provide the information in the format most convenient to them.

Efforts to Identify Duplication

As an examination of a new program, serving students at least half of whom will be outside public school district records, the evaluation must collect much of its own data. We are using existing data to the extent possible—for example, relying on the DCPS assessment for the baseline measures of student achievement. However, other information collected as part of the evaluation — the ongoing student assessments, the surveys of parent, students, and principals — is not available elsewhere.

Methods to Minimize Burden on Small Entities

There is no anticipated impact on small business or other small entities (as stated on Item 5 of OMB Form 83-I).

The primary entities for this study are students and parents, although some data will be collected from principals in public and private schools. Burden is reduced for all respondents by requesting only the minimum information required to meet the study objectives. The burden on schools has also been minimized through the careful specification of information needs, restricting questions to generally available information where possible, and designing the data collection strategy—particularly the survey methods—to minimize burden on respondents. For example, we will obtain some descriptive information on public and private schools from the Common Core Data (CCD) available from the National Center on Education Statistics. We will also administer the surveys to students and parents when they are attending events to re-establish their eligibility for the program.

Consequences of Not Collecting the Data

This data collection is necessary in order to evaluate the DC Opportunity Scholarship Program and comply with the evaluation mandate in the DC School Choice Incentive Act. Virtually all of the data collection activities—respondents, topics, and the need for annual collection—stem directly from the legislative requirements.

Special Circumstances

None of the special circumstances listed apply to this data collection.

Federal Register Comments and Persons Consulted Outside the Agency

Consultations on the research design, sample design, data sources and needs, and study reports have occurred during the study’s design phase and will continue to take place throughout the study. The purpose of such consultations is to ensure the technical soundness of the study and the relevance of its findings, and to verify the importance, relevance, and accessibility of the information sought in the study.

Westat and its subcontractors, Georgetown University and CRA, have provided substantial input to ED for the study. Senior technical staff from these organizations who are conducting the study are listed below:

Westat Ms. Babette Gutmann, Project Director (301) 738-3626

Alex Ratnofsky, Vice President (301) 251-8249

Juanita Lucas-McLean, Senior Analyst (301) 294-2866

Georgetown Professor Patrick Wolf, Principal Investigator (202) 687-9152

Professor Nada Eissa, Senior Analyst (202) 687-0626

CRA Dr. Michael Puma, Senior Analyst (410) 897-4968

The Department has also consulted with an Expert Advisory Panel, a group that includes both eminent school choice experts and evaluation methodologists. This advisory panel includes:

Professor Julian Betts, University of California, San Diego

Professor Thomas Cook, Northwestern University

Professor Jeff Henig, Columbia University

Assistant Professor William Howell, Harvard University

Professor Guido Imbens, University of California, Berkeley

Dr. Larry Orr, Abt Associates

Professor Rebecca Maynard, University of Pennsylvania.

The notice for this data collection was published in the Federal register on January 7, 2005.

Payments or Gifts

We realize that participation in the evaluation of the DC Opportunity Scholarship Program will place demands on each of the respondents. Specifically, it is critical to the study design that parents, students, and principals participate in the assessments and complete the survey forms each year, as we will be following each cohort and their parents for up to four years. We propose that the study provide the following modest incentives to the parents and principals in order to (1) maximize the annual response rates and (2) encourage ongoing participation from one year to the next.

Parents. The study not only needs the parents to complete a parent survey but also relies upon the parents to bring students in to participate in assessments and complete the student survey. For the parent surveys, we propose offering them $25 cash upon receipt of their completed survey form.

Principals. For the principal survey, we propose to send $10 cash out with each survey. Research has shown that this type of immediate pay off–cash in hand–can significantly increase response rates in household surveys^³. We anticipate that it will have a similar effect in schools. As we follow up by phone and re-send surveys to principals we will not re-send the $10 incentive.

Table 3, below, shows the anticipated costs associated with this incentive program. For the purpose of this estimate, we have assumed that using the incentive we will achieve a response rate of 80 percent among the parents of students in the DC Opportunity Scholarship Program.

Table 3. Costs Associated with Incentive Program for the Evaluation
Survey Form	Number of Respondents Receiving Incentive	Cost per Respondent	Total Cost
Parent	2,240 (80% of 2,800)	$25.00	$56,000
Principal Private School Public School	109 160	$10.00	$2,690
Total	2,509		$58,690

Assurances of Confidentiality

All data collection activities will be conducted in full compliance with Department of Education regulations to maintain the confidentiality of data obtained on private persons and to protect the rights and welfare of human research subjects as contained in Department of Education regulations. These activities will also be conducted in compliance with other applicable federal regulations. Research participants will be informed about the nature of the information that will be requested and confidentiality protection, and they will be assured that information will be reported only in aggregate, statistical form in reports and public use data files. Respondents will also be informed that their names will not be associated with their answers and that no one will have access to this information except as may be required by law, regulation, or subpoena or unless permission is given by both the parent and participating child.

In particular, it is very important that parents or legal guardians of sample members understand that information is being collected regarding their children, and that this information is being held confidential. When parents apply to the DC Opportunity Scholarship Program on behalf of their child(ren), they receive an oral presentation on evaluation activities and requirements and a written statement of the same; they are asked to sign the consent form and only those who sign are part of the program and the evaluation (see the consent form attached in Exhibit B). All parent, and principal surveys will also contain a statement regarding the confidentiality of their responses (see survey instruments in the appendices).

Specific Procedures to Maintain Confidentiality

The Evaluation of the DC Opportunity Scholarship Program will be conducted in accordance with all relevant regulations and requirements, including the Privacy Act of 1974 (5 usc 552 a), the Family Educational Rights And Privacy Act Of 1974 (20 usc 1232 g), the Freedom Of Information Act (5 usc 522), The Protection Of Pupil Rights Act (20 usc 1232 h), the Confidentiality Provisions Of The Education Sciences Reform Act (20 usc 9573), related regulations (41 cfr part 1-1 and 45 cfr part 5b), and, as appropriate, other federal or ED regulations on the protection of human subjects.

In addition, Section 309 of the DC Choice Act includes a particular specification that no personally identifiable information can be disclosed as part of the evaluation. As a result of this provision, in publishing the Privacy Act Notice for the System of Records for this evaluation, ED has eliminated all possible routine disclosures to which any data collected or obtained for the evaluation might be subjected. Under the notice, personal information (names, addresses, student ID numbers) may only be disclosed to Westat and in the unlikely case of a terrorist threat.

Westat, as ED’s “authorized representative” for the collection and maintenance of data for the Evaluation, will take the confidentiality requirements very seriously. Employees of Westat are required to sign Westat’s “employee or contractor’s assurance of confidentiality of data” (see Appendix C). This document outlines the general requirements and responsibilities of employees and contractors with regard to maintaining the confidentiality and privacy of data. In addition, each project at Westat is required, upon inception, to develop a customized confidentiality. The Westat project director develops the confidentiality plan for the evaluation that takes into account assurances made to respondents, what project information is confidential, who is authorized to have access to it, and how access can be controlled. This plan will be shared with all project staff, who will then be expected to implement it. Some of the components of the plan include:

Keeping hard-copy confidential information under lock and key.
Storing confidential electronic information in a secure location.
Communicating about cases via email without violating confidentiality and privacy.
Clearly labeling documents containing confidential information “confidential.”
Limiting to the number of copies of confidential documents.
Arranging for security when sending confidential jobs to a network printer.
Ensuring that only authorized personnel see faxes containing confidential information.
Adhering to the telephone research center’s (TRC) protocols for transporting confidential data to and from the TRC.
Adhering to data entry’s protocols for transporting confidential data to and from data entry.
Using mail and delivery services appropriate for the sensitivity level of the confidential data.
Not bringing confidential data home.
Disposing of confidential information properly when it is no longer needed.

Institutional Review Board (IRB)

Westat has sought clearance from its Institutional Review Board (IRB) for the DC Opportunity Scholarship Program application and consent form, and for all other protocols associated with student’s participation in the study. In the case of the DC Choice Act, the Congress specified a requirement that all applicants, even those who ultimately do not receive a scholarship through the lottery process, participate in the evaluation’s data collection in order to be eligible for a scholarship in succeeding years. The Congress considered such support for data collection critical to ensure that comprehensive and comparable data was collected from both the treatment and control group members. The IRB provided guidance on how to clarify these requirements on the application and consent forms that all applicants must sign.

Justification of Sensitive Questions

The parent and student surveys each have 3 questions regarding religion that are designed to determine the motivation for applying for (and using) a DC Opportunity Scholarship. We feel it is critical to include these questions in the survey instruments because the question of whether or not scholarship programs affect religious behavior is central to legal and policy debates about such programs.

Many opponents of scholarship programs argue that they establish religion or at least bring about an excessive entanglement of government with religion, in violation of the First Amendment to the U.S. Constitution, by promoting religious activities and instruction. Some sociologists who have studied private religious schooling in the U.S. agree that such schools increase the religiosity of students, and that the change has positive effects for student achievement and personal well-being. Thus, to bring evidence to bear on this important debate, it is necessary to ask participants in the D.C. Opportunity Scholarship Program if their participation in religious activities or receipt of religious instruction has changed as a result of the program.

As a side note, the survey forms only ask questions about religion at a very general level. They do NOT ask participants if they belong to a certain religion, the type or denomination of any religious activities in which they participate, or anything about their private religious beliefs.

Estimates of Hour Burden

The study calls for surveys of students, parents, and principals, as well as records abstraction and test administration. The instruments, found in Appendix D, were developed to maximize respondent completion of the surveys and to minimize respondent burden. All survey instruments are brief and focus on collecting only information essential to the study.

The research team will administer the student surveys as part of the student assessment that will be administered to students at the family renewal events. The parent survey will be administered at the family renewal events, with telephone follow up. These surveys are designed to be completed in paper and pencil format and will collect information on the respondents’ perception of the school program and environment.

The two principal surveys will be administered as a mail survey with telephone follow up. The surveys will be mailed to principals with instructions to complete the survey and mail or fax it back to the research team. Principals who do not respond by the stated deadline will be contacted by telephone in an attempt to obtain a completed response.

The research team will administer the assessment to the treatment and control groups each spring. In addition, they will collect administrative records from DCPS and charter schools authorizers to obtain data on attendance, persistence, disciplinary actions, and grades for members of the treatment and control groups at baseline.

Table 4 below shows the estimated burden for each of the data sources.

Table 4. Annual Burden Estimates, by Data Source
Data Source	Respondents	Estimated Number of Responses	Estimated Annual Burden per Response (in Hours)	Total Estimated Annual Burden (in Hours)
Student Assessments	Eligible applicants in grades K-12	2,800	2.5	7,000.00
School Records	DCPS staff and charter school authorizers	2	40	80.00
Student Survey	Eligible applicants in grades 4-12	2,800	0.25	700.00
Parent Survey	Parents of eligible applicants	2,800	0.25	700.00
Private School Principal Survey	Private school principals of participating and non-participating schools	109	0.17	18.53
Public School Principal Survey	Principals of DC public schools	150	0.17	25.50
DC Opportunity Scholarship Program Operator Records		1	40	40.00
Total		8,662		8,564.03

Note: The information in this table describes the surveys and burden for one (annual) cycle of data collection. This cycle will be repeated for a total of 4 data collection years.

Estimate of Cost Burden to Respondents

There are no additional respondent costs associated with this data collection other than the hour burden estimated in item A12.

Estimate of Annual Cost to the Federal Government

The estimated cost to the federal government of conducting the Impact Evaluation of the DC Opportunity Scholarship Program is based on the government's contracted cost of the data collection and related study activities along with personnel cost of government employees involved in oversight and/or analysis. For the data collection activities for which OMB approval is currently being requested, the overall cost to the government is $2,356,073. This includes:

$491,869 for the first year of data collection, including instrument development
$603,838 for the second year of data collection
$621,177 for the third year of data collection

$639,189 for the fourth year of data collection

The overall costs to the government of the full range of evaluation activities over the entire study period will be $5,489,394 over a five-year period. When annualized, this cost amounts to $1,097,879 per year. This estimate is based on the evaluation contractor's previous experience managing other research and data collection activities of this type.

Program Changes or Adjustments

This is a new data collection and therefore does not require any changes or adjustments.

Plans for Tabulation and Publication of Results

All data will be analyzed according to rigorous technical standards, and woven together to provide a complete assessment of whether the DC Opportunity Scholarship Program achieved its goals. The focus of the analysis and report will be evidence regarding: (1) who applies for and uses a scholarship; (2) what impacts does the offer and use of a scholarship have on student test scores, parental satisfaction, school safety, and other participant outcomes; and (3) do the principals at DC public schools and private schools plan to manage their educational institutions differently in response to the establishment of the DC Opportunity Scholarship Program.

General Analytic Strategy

It is well known that the independent effects of school choice on student outcomes are difficult to estimate. Perhaps the most significant difficulty faced by researchers is selection bias -- the self-selection of families to even seek out a new school choice for their child, and the mutual student/school decision process that selects students into different types of schools. Because this bias is generally a result of unmeasurable factors, most researchers have preferred the use of a randomized experiment to a dependence on non-experimental statistical methods. Since the DC Opportunity Scholarship Program provides for the random distribution of scholarships using a lottery, under certain conditions and within certain parameters, we will therefore use experimental methods to the extent possible to estimate most programmatic impacts.

To motivate the discussion of how we identify the effect of the scholarship program on test-scores, it is useful to begin with a simple representation of the selection problem as a missing data problem, using the potential outcomes approach. This approach defines causal effects in terms of potential outcomes or counterfactuals. Conceptually, the causal effect of treatment is defined as the difference between the “outcome for individuals assigned to the treatment group” and “outcome for the treatment group if it had not received the treatment,“ or:

(E.1) “E(Y_i|X_i, T_i =1)” - “E(Y_i |X_i, T_i =0)”

In the case of scholarships, the treatment effect–the effect of the scholarships on academic achievement–would be defined as the difference between “test scores for program students” and “test scores for program students if they had not received a scholarship.” The fundamental problem is that a student is never observed simultaneously in both states of the world. What is observed is a student in the treatment group (T_i =1) or in the control group (T_i =0). The outcome in the absence of treatment, E(Y_i |X_i, T_i =0), is then the counterfactual--what would have occurred to those students receiving the scholarships if they had not received them.

If students receiving scholarships were identical to other students in both observable and unobservable characteristics, the counterfactual could be generated directly from an appropriately selected comparison group. Valid comparison groups are rarely found in practice, however. The random assignment of students into the program generates the counterfactual from the control group – eligible applicants who did not receive a scholarship.^⁴ If correctly implemented, random assignment yields statistically equivalent groups, and allows estimation of the program impact through differences in mean outcomes between the two groups.

Consistent with this approach is the following basic analytic model of the effects of school choice scholarships on outcomes. Consider first the outcome equation for the test score of student i in year t. It is reasonable to assume that test scores (Y_it ) are determined as follows:

(E.2) Y_it =α+ τ T_it + X_iγ+ ε_it if t>k (period after program takes effect)

In equation (E.2), T_it is equal to one if the student has the opportunity to participate in the voucher program (i.e., the award rather than the accrual use of the voucher) and equal to zero otherwise. X_iis a vector of student characteristics (measured at baseline) known to influence future academic achievement, such as prior test scores, mother’s level of education, family income, etc. In this model, τ represents the effect of vouchers on test scores for students in the program, conditional on X_i. With a properly designed experiment, using a concise and judiciously chosen set of statistical controls for characteristics that predict future achievement should improve the precision of the estimated impact. That is, the estimated treatment effect, τ, should be identical to the difference in mean outcomes between the treatment and the control groups.

Customization of General Analytic Strategy

Since the initial applicants were randomized within certain relevant subgroups, we propose a randomized block design for analyzing scholarship program impacts. The randomized block design divides the program group into relatively homogenous groups (called blocks). The program group is then randomly assigned vouchers within each block. We are interested in how academic achievement (Y) is affected by the assignment into a voucher program. Suppose we could identify b blocks -- based on grade and scholarship priority status -- that are of size n. Consider then the following statistical model for this Randomized Block Design:

(E.3) Y_ikt = μ+ τT_ikt +∑^b_j=2ρ_j B_ik+ X_ik γ+ ε_ik,t

where

i = 1,…..,n observations and k=1,….,b blocks(defined by grade and priority status);

Y_ji is the outcome for student i in block j, at time t;

μ is the overall mean outcome (e.g. test score);

τ is the treatment (scholarship program) effect;

ρ_jis the j^th block effect;

T_itis assignment into the voucher program

B_ji is the block assignment

X_jirepresents observable characteristics, measured at baseline

ε_ij is the random error; independent, Ν(0,σ_ε²).

This analytical framework follows naturally from the randomization scheme and is easily implemented and interpreted. Y can be measured in several different dimensions, including test scores, school satisfaction, parental satisfaction, grade completion, including where appropriate, high school graduation, etc. μ is average outcome for all program members; ρ_jis the average block effect. τ is the effect of vouchers on academic achievement. The remainder of this discussion discusses econometric concerns and associated empirical methods.

Take-Up of Scholarships

Even with a properly implemented experiment, we may expect slippage between the random assignment into the experiment and use of the scholarship at a private school. This occurrence has been observed in very different experimental settings, including medical trials, job training and health insurance experiments. More relevant to our exercise is the slippage that has been observed in previous school voucher experiments, such as the Milwaukee Parental Choice Program. Such slippage has important implications for the estimators of the effect of the scholarship program. Generally we define two broad estimators of interest. The first, commonly referred to as the "Intent to Treat" (ITT), is the effect of the offer of a scholarship on student outcomes. All students randomized into the sample make up the experimental sample, regardless of whether they use the scholarship to attend a private school.

Policymakers are typically also interested in the effect of scholarship use on student achievement. This estimator, commonly referred to as the "Impact of the Treated" (IOT), is based on the sample of scholarship users. Instrumental variable analysis provides us with a well-established method to generate an estimate of the scholarship impact on the treated from the ITT estimator.^⁵

Using only the sample of scholarship users in this case could introduce a form of selection bias, in that the sample of students using the voucher to attend private schools is selected (from the randomized-in sample). Self-selection bias results in the case where family (observable and unobservable) characteristics that affect student outcomes also affect the decision to use the voucher. For example, families who care more about education and are more able to gather and analyze relevant information about the schools are also the families whose children are more likely to make use of the voucher, all else equal. Students in such families are also more likely to do better once in a private school setting than their randomized-in counterparts who do not use the voucher. To see the point, consider the following models of actual use of the voucher, and student test scores.^⁶

(E.4) V_it = σ₀+ σ₁T_it + X_iσ₂+ ε_it

where

i represents student, t time

V represents use of the voucher

Trepresents treatment status (=1 if selected in the lottery)

X represents observable characteristics

Note that when schools randomly select from applicants when they are over-subscribed, L is random, conditional on the school and grade of the applicant. Such effects would be controlled for in the randomized block design proposed in equation E.3.

We also recognize that a model of student outcomes would be based on actual voucher use/attendance at a private institution.

(E.5) Y_it = π₀+ π₁V_it + X_i π₂+ ν_it

Combining equations (E.4) and (E.5), we get

(E.6) Y_it = ψ₀+ ψ₁T_it + X_i ψ₂+ ξ_it

What these equation show is that the estimated treatment effect ψ₁is equal to a combination of the effects of selection into the program on voucher use and of school attendance on student outcomes (ψ₁= π₁ *σ₁). Note that ψ is the treatment effect in the empirical models E.2 and E.3. What we estimate in the ITT model is therefore the reduced form effect of both margins of response-student learning in private schools and family take-up of scholarship dollars. It is important to note that ψ is in some respects the policy parameter of interest since families cannot be compelled to use available scholarships. Its decomposition is of course incredibly useful for learning about the effectiveness of different types of schools on educational attainment; and of the success of, in this case, publicly funded scholarships. Our empirical analysis will examine, among other margins, family choices regarding take-up of the scholarships as well as types of schools selected.

These types of analyses will be performed for the various outcome measures called for in the law, including academic achievement, safety, satisfaction, and other student outcomes.

Non-experimental Comparisons

In addition to an impact analysis, the law calls for a comparison of students participating in the scholarship program with students in the same grades in the DC Public Schools. DCPS students who did not apply to the scholarship program are likely to be quite different from those who applied and are participating—in ways we can observe and ways we cannot. Comparing outcomes between participants and nonapplicants is therefore not a reliable measure of program effects. Instead, this type of performance reporting will be combined with other data collection and analysis that examines the context in which the program is operating

The performance reporting comparisons will focus on student achievement, both as specified in the legislation and because that is the only measure that will be similar for DCPS students and those participating in private schools through the DC Opportunity Scholarship program.^⁷ In order to ensure comparability in student assessment, the evaluation will make every effort to administer the same test to program participants that is used by DCPS. DCPS will provide the evaluation team with test-score and background data on public school students.

The analysis will be conducted by comparing the mean test scores of program participants and DCPS non-applicants, testing for the statistical significance of the difference. To create the most relevant group of DCPS students for comparison, we will draw from the DCPS database the group of non-applicant students who qualify for the program (i.e. eligible for free/reduced lunch), stratified by grade level to match our scholarship performance reporting sample. We will present these comparative results as descriptive findings, since the absence of random-assignment to the scholarship or public school conditions would render any causal claims highly speculative.

Reports

The first report, due to Congress on December 1, 2004, will describe who applied to the DC Opportunity Scholarship Program, largely by comparing the demographic characteristics and achievement of program applicants with those of other DCPS students. Subsequent reports will focus on the impact of the program, using the experimental and multivariate regression techniques described above to estimate differences in outcomes between the treatment and control group members. Based on the guidance in the legislation, the reports will focus primarily on conditions and outcomes involving student academic performance, parental satisfaction, school safety, and the process by which parents select schools. Within 6 months after delivering each report to Congress, Westat will prepare and disseminate a public use file with all of the data collected and analyzed so far. A schedule for the reports and data files is provided in Table 5.

Table 5. Deliverable Schedule

Deliverable

Schedule

Interim Reports for December 2004, 2005, 2006, 2007, 2008

Draft #1

Draft #2

Final

September 1 of year

October 1 of year

November 1 of year

Final Report

Draft #1

Draft #2

Final

March 1, 2009

April 1, 2009

May 1, 2009

Data Files with Documentation.

For Interim Reports

For Final Report

6 months after submission of each Interim Report

March 1, 2009

Approval to Not Display OMB Expiration Date

All data collection instruments will include the OMB expiration date.

Explanation of Exceptions

No exceptions are requested.

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Procedures

The Impact Evaluation of the DC Opportunity Scholarship Program will be based on administering assessments and surveys to the universe of program applicants and their parents as well as school principals. The surveys that will provide information on schools’ competitive response to the program will also include the universe of DCPS principals and of private school principals. Therefore, there is no sampling proposed for this study.

B.2. Statistical Methods for Sample Selection

No sampling stratification is planned for any of the data collection activities. The student, parent, and principal surveys will all include the universe of respondents. In no case do we anticipate any unusual problems requiring specialized sampling procedures.

B.3. Methods to Maximize Response Rates

We will maximize the response rate for this portion of the study both by distributing survey instruments that are fairly easy for respondents to complete and by following up with non-responders by mail, fax, and telephone. Although the primary respondents are quite disadvantaged (eligibility requirements include family income less than or equal to 185 percent of poverty), this study is striving for a response rate of 80 percent.

Obtaining high response rates in the Impact Evaluation of the DC Opportunity Scholarship Program will be critical to the success of the study. It will be particularly important to obtain response rates that are not only high overall, but that are approximately equal in the treatment and control groups. This will be challenging due to the fact that while most of the treatment group will presumably be in a relative small set of participating private schools, control group students will likely attend a large number of different DCPS schools, and the identities of these schools will not be known in advance.

We have several strategies for ensuring a high rate of response. First, we have planned to conduct most of the data collection—student assessments, student and parent surveys—at events the program operator will hold for the treatment and control groups to re-establish eligibility for the program. Second, because of a key provision in the law, in our communications with parents we can stress that participation in the evaluation’s data collection is required for students to keep their scholarship or remain eligible to receive a scholarship in the future. We believe these requirements will be a formidable incentive to respond to the surveys and assessments. Finally, we will employ a sophisticated tracking system to ensure that we follow up with non-response in a timely and comprehensive way.

B.4. Pilot Testing

We will pretest each of the surveys with nine or fewer people who are similar demographically to respondents in the study. We plan to ask the pretest respondents to first complete the relevant survey and then participate in a focus group about it. In the focus group discussions we will test for completion times and feelings of burden, salience of language, concept recognition, and understanding of terms. After the pretest, we will revise the surveys as needed based on the pretest results.

B.5. Individuals and Organizations Involved in this Project

The statistical aspects of the design have been reviewed thoroughly by staff at the Institute of Education Sciences, as well as by members of the study’s expert panel (listed in Section A8). Table 6 shows the individuals most closely involved in developing the statistical procedures and who will be responsible for data collection and analysis.

Table 6. Individuals Involved in this Project
Name	Affiliation	Role	Phone Number
Babette Gutmann	Westat	Project Director	(301) 738-3626
Patrick Wolf	Georgetown University	Principal Investigator	(202) 687-9152
Mike Puma	CRA	Senior Analyst	(410) 897-4968
Juanita Lucas-McLean	Westat	Director of Data Collection	(301) 294-2866
Marsha Silverberg	ED/IES	Economist, COR	(202) 208-7178

Appendix A

Authorizing Legislation

TITLE III--DC SCHOOL CHOICE INCENTIVE ACT OF 2003

SEC. 301. SHORT TITLE

This title may be cited as the "DC School Choice Incentive Act of 2003".

SEC. 302. FINDINGS

The Congress finds the following:

(1) Parents are best equipped to make decisions for their children, including the educational setting that will best serve the interests and educational needs of their child.

(2) For many parents in the District of Columbia, public school choice provided for under the No Child Left Behind Act of 2001 as well as under other public school choice programs, is inadequate due to capacity constraints. Available educational alternatives to the public schools are insufficient and more educational options are needed. In particular, funds are needed to assist low-income parents to exercise choice among enhanced public opportunities and private educational environments, whether religious or nonreligious. Therefore, in keeping with the spirit of the No Child Left Behind Act of 2001, school choice options, in addition to those already available to parents in the District of Columbia (such as magnet and charter schools and open enrollment schools) should be made available to those parents.

(3) In the most recent mathematics assessment on the National Assessment of Educational Progress (NAEP), administered in 2000, a lower percentage of 4th-grade students in the District of Columbia demonstrated proficiency than was the case for any State. Seventy-six percent of the District of Columbia fourth-graders scored at the "below basic" level and of the 8th-grade students in the District of Columbia, only 6 percent of the students tested at the proficient or advanced levels, and 77 percent were below basic. In the most recent NAEP reading assessment, in 1998, only 10 percent of the District of Columbia fourth-graders could read proficiently, while 72 percent were below basic. At the 8th-grade level, 12 percent were proficient or advanced and 56 percent were below basic.

(4) A program enacted for the valid secular purpose of providing educational assistance to low-income children in a demonstrably failing public school system is constitutional under Zelman v. Simmons-Harris, 536 U.S. 639 (2002), if it is neutral with respect to religion and provides assistance to a broad class of citizens who direct government aid to religious and secular schools solely as a result of their genuine and independent private choices.

(5) The Mayor of the District of Columbia, the Chairman of the Education Committee of the City Council of the District of Columbia, and the President of the District of Columbia Board of Education support this title.

(6) This title provides additional money for the District of Columbia public schools and therefore money for scholarships is not being taken out of money that would otherwise go to the District of Columbia public schools.

(7) This title creates a 5-year program tailored to the current needs and particular circumstances of low-income children in District of Columbia schools. This title does not establish parameters or requirements for other school choice programs.

SEC. 303. PURPOSE

The purpose of this title is to provide low-income parents residing in the District of Columbia, particularly parents of students who attend elementary schools or secondary schools identified for improvement, corrective action, or restructuring under section 1116 of the Elementary and Secondary Education Act of 1965 (20 U.S.C. 6316), with expanded opportunities for enrolling their children in higher-performing schools in the District of Columbia.

SEC. 304. GENERAL AUTHORITY

(a) AUTHORITY- From funds appropriated to carry out this title, the Secretary shall award grants on a competitive basis to eligible entities with approved applications under section 305 to carry out activities to provide eligible students with expanded school choice opportunities. The Secretary may award a single grant or multiple grants, depending on the quality of applications submitted and the priorities of this title.

(b) DURATION OF GRANTS- The Secretary may make grants under this section for a period of not more than 5 years.

(c) MEMORANDUM OF UNDERSTANDING- The Secretary and the Mayor of the District of Columbia shall enter into a memorandum of understanding, as described in the statement of the managers, regarding the design of, selection of eligible entities to receive grants under, and implementation of, a program assisted under this title.

SEC. 305. APPLICATIONS

(a) IN GENERAL- In order to receive a grant under this title, an eligible entity shall submit an application to the Secretary at such time, in such manner, and accompanied by such information as the Secretary may require.

(b) CONTENTS- The Secretary may not approve the request of an eligible entity for a grant under this title unless the entity's application includes--

(1) a detailed description of--

(A) how the entity will address the priorities described in section 306;

(B) how the entity will ensure that if more eligible students seek admission in the program than the program can accommodate, eligible students are selected for admission through a random selection process which gives weight to the priorities described in section 306;

(C) how the entity will ensure that if more participating eligible students seek admission to a participating school than the school can accommodate, participating eligible students are selected for admission through a random selection process;

(D) how the entity will notify parents of eligible students of the expanded choice opportunities and how the entity will ensure that parents receive sufficient information about their options to allow the parents to make informed decisions;

(E) the activities that the entity will carry out to provide parents of eligible students with expanded choice opportunities through the awarding of scholarships under section 307(a);

(F) how the entity will determine the amount that will be provided to parents for the tuition, fees, and transportation expenses, if any;

(G) how the entity will seek out private elementary schools and secondary schools in the District of Columbia to participate in the program, and will ensure that participating schools will meet the applicable requirements of this title and provide the information needed for the entity to meet the reporting requirements of this title;

(H) how the entity will ensure that participating schools are financially responsible and will use the funds received under this title effectively;

(I) how the entity will address the renewal of scholarships to participating eligible students, including continued eligibility; and

(J) how the entity will ensure that a majority of its voting board members or governing organization are residents of the District of Columbia; and

(2) an assurance that the entity will comply with all requests regarding any evaluation carried out under section 309.

SEC. 306. PRIORITIES

In awarding grants under this title, the Secretary shall give priority to applications from eligible entities who will most effectively--

(1) give priority to eligible students who, in the school year preceding the school year for which the eligible student is seeking a scholarship, attended an elementary school or secondary school identified for improvement, corrective action, or restructuring under section 1116 of the Elementary and Secondary Education Act of 1965 (20 U.S.C. 6316);

(2) target resources to students and families that lack the financial resources to take advantage of available educational options; and

(3) provide students and families with the widest range of educational options.

SEC. 307. USE OF FUNDS

(a) SCHOLARSHIPS-

(1) IN GENERAL- Subject to paragraphs (2) and (3), a grantee shall use the grant funds to provide eligible students with scholarships to pay the tuition, fees, and transportation expenses, if any, to enable them to attend the District of Columbia private elementary school or secondary school of their choice. Each grantee shall ensure that the amount of any tuition or fees charged by a school participating in the grantee's program under this title to an eligible student participating in the program does not exceed the amount of tuition or fees that the school customarily charges to students who do not participate in the program.

(2) PAYMENTS TO PARENTS- A grantee shall make scholarship payments under the program under this title to the parent of the eligible student participating in the program, in a manner which ensures that such payments will be used for the payment of tuition, fees, and transportation expenses (if any), in accordance with this title.

(3) AMOUNT OF ASSISTANCE-

(A) VARYING AMOUNTS PERMITTED- Subject to the other requirements of this section, a grantee may award scholarships in larger amounts to those eligible students with the greatest need.

(B) ANNUAL LIMIT ON AMOUNT- The amount of assistance provided to any eligible student by a grantee under a program under this title may not exceed $7,500 for any academic year.

(4) CONTINUATION OF SCHOLARSHIPS- Notwithstanding section 312(3)(B), an eligible entity receiving a grant under this title may award a scholarship, for the second or any succeeding year of an eligible student's participation in a program under this title, to a student who comes from a household whose income does not exceed 200 percent of the poverty line.

(b) ADMINISTRATIVE EXPENSES- A grantee may use not more than 3 percent of the amount provided under the grant each year for the administrative expenses of carrying out its program under this title during the year, including--

(1) determining the eligibility of students to participate;

(2) providing information about the program and the schools involved to parents of eligible students;

(3) selecting students to receive scholarships;

(4) determining the amount of scholarships and issuing the scholarships to eligible students;

(5) compiling and maintaining financial and programmatic records; and

(6) providing funds to assist parents in meeting expenses that might otherwise preclude the participation of their child in the program.

SEC. 308. NONDISCRIMINATION

(a) IN GENERAL- An eligible entity or a school participating in any program under this title shall not discriminate against program participants or applicants on the basis of race, color, national origin, religion, or sex.

(b) APPLICABILITY AND SINGLE SEX SCHOOLS, CLASSES, OR ACTIVITIES-

(1) IN GENERAL- Notwithstanding any other provision of law, the prohibition of sex discrimination in subsection (a) shall not apply to a participating school that is operated by, supervised by, controlled by, or connected to a religious organization to the extent that the application of subsection (a) is inconsistent with the religious tenets or beliefs of the school.

(2) SINGLE SEX SCHOOLS, CLASSES, OR ACTIVITIES- Notwithstanding subsection (a) or any other provision of law, a parent may choose and a school may offer a single sex school, class, or activity.

(3) APPLICABILITY- For purposes of this title, the provisions of section 909 of the Education Amendments of 1972 (20 U.S.C. 1688) shall apply to this title as if section 909 of the Education Amendments of 1972 (20 U.S.C. 1688) were part of this title.

(c) CHILDREN WITH DISABILITIES- Nothing in this title may be construed to alter or modify the provisions of the Individuals with Disabilities Education Act.

(d) RELIGIOUSLY AFFILIATED SCHOOLS-

(1) IN GENERAL- Notwithstanding any other provision of law, a school participating in any program under this title that is operated by, supervised by, controlled by, or connected to, a religious organization may exercise its right in matters of employment consistent with title VII of the Civil Rights Act of 1964 (42 U.S.C. 2000e-1 et seq.), including the exemptions in such title.

(2) MAINTENANCE OF PURPOSE- Notwithstanding any other provision of law, funds made available under this title to eligible students that are received by a participating school, as a result of their parents' choice, shall not, consistent with the first amendment of the United States Constitution, necessitate any change in the participating school's teaching mission, require any participating school to remove religious art, icons, scriptures, or other symbols, or preclude any participating school from retaining religious terms in its name, selecting its board members on a religious basis, or including religious references in its mission statements and other chartering or governing documents.

(e) RULE OF CONSTRUCTION- A scholarship (or any other form of support provided to parents of eligible students) under this title shall be considered assistance to the student and shall not be considered assistance to the school that enrolls the eligible student. The amount of any scholarship (or other form of support provided to parents of an eligible student) under this title shall not be treated as income of the parents for purposes of Federal tax laws or for determining eligibility for any other Federal program.

SEC. 309. EVALUATIONS

(a) IN GENERAL-

(1) DUTIES OF THE SECRETARY AND THE MAYOR- The Secretary and the Mayor of the District of Columbia shall jointly select an independent entity to evaluate annually the performance of students who received scholarships under the 5-year program under this title, and shall make the evaluations public in accordance with subsection (c).

(2) DUTIES OF THE SECRETARY- The Secretary, through a grant, contract, or cooperative agreement, shall--

(A) ensure that the evaluation is conducted using the strongest possible research design for determining the effectiveness of the programs funded under this title that addresses the issues described in paragraph (4); and

(B) disseminate information on the impact of the programs in increasing the student academic achievement of participating students, and on the impact of the programs on students and schools in the District of Columbia.

(3) DUTIES OF THE INDEPENDENT ENTITY- The independent entity shall--

(A) measure the academic achievement of all participating eligible students;

(B) use the same grade appropriate measurement every school year to assess participating eligible students as the measurement used by the District of Columbia Public Schools to assess District of Columbia Public School students in the first year of the program; and

(C) work with the eligible entities to ensure that the parents of each student who applies for a scholarship under this title (regardless of whether the student receives the scholarship) and the parents of each student participating in the scholarship program under this title, agree that the student will participate in the measurements given annually by the independent entity for the period for which the student applied for or received the scholarship, respectively.

(4) ISSUES TO BE EVALUATED- The issues to be evaluated include the following:

(A) A comparison of the academic achievement of participating eligible students in the measurements described in this section to the achievement of--

(i) students in the same grades in the District of Columbia public schools; and

(ii) the eligible students in the same grades in the District of Columbia public schools who sought to participate in the scholarship program but were not selected.

(B) The success of the programs in expanding choice options for parents.

(D) A comparison of the retention rates, dropout rates, and (if appropriate) graduation and college admission rates, of students who participate in the programs funded under this title with the retention rates, dropout rates, and (if appropriate) graduation and college admission rates of students of similar backgrounds who do not participate in such programs.

(E) The impact of the program on students, and public elementary schools and secondary schools, in the District of Columbia.

(F) A comparison of the safety of the schools attended by students who participate in the programs and the schools attended by students who do not participate in the programs.

(G) Such other issues as the Secretary considers appropriate for inclusion in the evaluation.

(5) PROHIBITION- Personally identifiable information regarding the results of the measurements used for the evaluations may not be disclosed, except to the parents of the student to whom the information relates.

(b) REPORTS- The Secretary shall submit to the Committees on Appropriations, Education and the Workforce, and Government Reform of the House of Representatives and the Committees on Appropriations, Health, Education, Labor, and Pensions, and Governmental Affairs of the Senate--

(1) annual interim reports, not later than December 1 of each year for which a grant is made under this title, on the progress and preliminary results of the evaluation of the programs funded under this title; and

(2) a final report, not later than 1 year after the final year for which a grant is made under this title, on the results of the evaluation of the programs funded under this title.

(c) PUBLIC AVAILABILITY- All reports and underlying data gathered pursuant to this section shall be made available to the public upon request, in a timely manner following submission of the applicable report under subsection (b), except that personally identifiable information shall not be disclosed or made available to the public.

(d) LIMIT ON AMOUNT EXPENDED- The amount expended by the Secretary to carry out this section for any fiscal year may not exceed 3 percent of the total amount appropriated to carry out this title for the fiscal year.

SEC. 310. REPORTING REQUIREMENTS

(a) ACTIVITIES REPORTS- Each grantee receiving funds under this title during a year shall submit a report to the Secretary not later than July 30 of the following year regarding the activities carried out with the funds during the preceding year.

(b) ACHIEVEMENT REPORTS.-

(1) IN GENERAL.-In addition to the reports required under subsection (a), each grantee shall, not later than September 1 of the year during which the second academic year of the grantee's program is completed and each of the next 2 years thereafter, submit a report to the Secretary regarding the data collected in the previous 2 academic years concerning-

(A) the academic achievement of students participating in the program;

(B) the graduation and college admission rates of students who participate in the program, where appropriate; and

(2) PROHIBITING DISCLOSURE OF PERSONAL INFORMATION.- No report under this subsection may contain any personally identifiable information.

No report under this subsection may contain any personally identifiable information.

(1) IN GENERAL.-Each grantee shall ensure that each school participating in the grantee's program under this title during a year reports at least once during the year to the parents of each of the school's students who are participating in the program on-

(A) the student's academic achievement, as measured by a comparison with the aggregate academic achievement of other participating students at the student's school in the same grade or level, as appropriate, and the aggregate academic achievement of the student's peers at the student's school in the same grade or level, as appropriate; and

(B) the safety of the school, including the incidence of school violence, student suspensions, and student expulsions.

(2) PROHIBITING DISCLOSURE OF PERSONAL INFORMATION.- No report under this subsection may contain any personally identifiable information, except as to the student who is the subject of the report to that student's parent.

(d) REPORT TO CONGRESS.-The Secretary shall submit to the Committees on Appropriations, Education and the Workforce, and Government Reform of the House of Representatives and the Committees on Appropriations, Health, Education, Labor, and Pensions, and Governmental Affairs of the Senate an annual report on the findings of the reports submitted under subsections (a) and (b).

SEC. 311. OTHER REQUIREMENTS FOR PARTICIPATING SCHOOLS

(a) REQUESTS FOR DATA AND INFORMATION.-Each school participating in a program funded under this title shall comply with all requests for data and information regarding evaluations conducted under section 309(a).

(b) RULES OF CONDUCT AND OTHER SCHOOL POLICIES.-A participating school, including those described in section 308(d), may require eligible students to abide by any rules of conduct H. R. 2673-132 and other requirements applicable to all other students at the school.

SEC. 312. DEFINITIONS

As used in this title:

(1) ELEMENTARY SCHOOL.-The term "elementary school" means an institutional day or residential school, including a public elementary charter school, that provides elementary education, as determined under District of Columbia law.

(2) ELIGIBLE ENTITY.-The term ''eligible entity'' means any of the following:

(A) An educational entity of the District of Columbia Government.

(B) A nonprofit organization.

(3) ELIGIBLE STUDENT.-The term "eligible student" means a student who-

(A) is a resident of the District of Columbia; and

(B) comes from a household whose income does not exceed 185 percent of the poverty line.

(4) PARENT.-The term "parent" has the meaning given that term in section 9101 of the Elementary and Secondary Education Act of 1965 (20 U.S.C. 7801).

(5) POVERTY LINE.-The term "poverty line" has the meaning given that term in section 9101 of the Elementary and Secondary Education Act of 1965 (20 U.S.C. 7801).

(6) SECONDARY SCHOOL.-The term "secondary school" means an institutional day or residential school, including a public secondary charter school, as determined under District of Columbia law, except that the term does not include any education beyond grade 12.

(7) SECRETARY.-The term "Secretary" means the Secretary of Education.

SEC. 313. AUTHORIZATION OF APPROPRIATIONS

There are authorized to be appropriated to carry out this title $14,000,000 for fiscal year 2004 and such sums as may be necessary for each of the 4 succeeding fiscal years.

Appendix B

DC Opportunity Scholarship Program

Application and Consent Form

O MB: 1855-0015

Expiration Date: 11/30/04

D.C. Opportunity Scholarship Program Application

and Evaluation Questionnaire
Your Child’s Future, Your Peace of Mind

Thank you for your interest in the D.C. Opportunity Scholarship Program. This form should be filled out by the parent or guardian who lives with the child(ren) applying for a scholarship. This form is in two sections.

Section 1: Scholarship Application and Contact Information

Part A: A description of the Program and your signed agreement to participate.

Part B: Scholarship Eligibility Form. Household composition and other information

needed to determine eligibility for the D.C. Opportunity Scholarship Program.

Part C: Additional contact information so that we can find you in case you move or

change your phone number.

Section 2: Evaluation Questionnaire

These questions will not affect your chances of getting a scholarship, and individual family answers will not be seen by anyone outside the evaluation team. The answers will be combined so that Congress and policy makers can use the information to see how well the children in the program are doing. These questions need to be answered by all applicants.

Part D: General questions about the adults in your household.

Part E: General questions about the children in your household.

Applying Parent/Guardian (please put this name on the top of each page of the application):

Parent/Guardian Name _________________________________________________________

Street Address _______________________________________________________________

City _____________________________ State ________ Zip Code ______________

Home Phone Number (_______) ___________________ Work Phone: (_____) _________

Mobile Phone (_____) ____________________ E-Mail: _______________________

NOTICE: According to the Paperwork Reduction Action of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB Control Number. The valid OMB Control Number for this information collection is 1855-0015 (expires). The time required to complete this information collection is estimated at 25 minutes per respondent, including time to review instructions, and complete and review the information collection.

Part A. Description of the Program and Agreement to Participate

1. Description of the D.C. Opportunity Scholarship Program and Evaluation

In early 2004, the U.S. Congress passed the DC School Choice Incentive Act. This law established a new, five-year school choice program for low-income residents of Washington, DC. The Program will provide scholarships to enable low-income elementary and secondary education students to attend private schools in addition to the public schools already available to them.

Scholarships are for up to $7,500 per year for tuition fees (for example books, uniforms, and transportation expenses). Under the current law, Congress has authorized the Program to run until the end of the school year 2009-2010.
Receiving a scholarship will not interfere in any way with other public assistance your family may receive.

Scholarships may only be used at a school participating in the D.C. Opportunity Scholarship Program.
Once a child receives a scholarship, they will receive one every year the program is funded, as long as they remain eligible and maintain good academic standing in a participating school.
If there are more applicants than slots in schools or available funds, the law requires that scholarships will be given out through a lottery. Scholarships will be considered conditional until the student is actually placed in a school. Children who do not receive a scholarship this year and are eligible may receive a scholarship next year.
Only families with completed applications will be included in the lottery. If your application is determined to be incomplete (meaning that we do not have enough documentation proving eligibility) we will contact you and give you a limited time in which to complete your application. If you do not complete your application in time, we will not be able to include you in the lottery. The determination of whether or not your application is complete is not made when you submit it. The determination of eligibility and if your application is complete is made by one of Washington Scholarship Fund’s (WSF) partners once they have had time to review your submission in detail.
Scholarships are awarded to individual students, not to families. Scholarships can not be transferred.
Once a student has a conditional scholarship, you are responsible for applying to the schools that you are interested in. Each school has its own application process, and you must contact participating schools directly to apply.
WSF encourages conditional scholarship recipients to apply to as many schools as possible. In case your child does not get his/her first-choice school, it is important to have other choices
Schools that you apply to will inform you whether or not each applicant is admitted.
While application to the Program is voluntary, all applicants must participate in the Evaluation, whether or not they receive a scholarship. The Evaluation is important because it lets Congress know how successful the Program has been. As part of the Evaluation, applicants must agree to:

Annual testing of your child
Parent surveys and voluntary focus groups where you will be asked your opinions
Surveys of children in grades 4 and above
Collection of files and records from your child’s school
- If you have any questions about the application or your eligibility for the Program, please call the Washington Scholarship Fund at 202-293-5560 or at 1-888-DC-YOUTH
- If you have any questions about the evaluation requirements, please call Juanita Lucas-McLean at

Westat at 301-294-2866.

No school charged any families tuition, even when tuition was well over the $7,500 scholarship amount.

2. Agreement to Participate

When the U.S. Congress created the D.C. Opportunity Scholarship Program, it established rules for who is eligible to apply and how those applications should be handled. Congress also required that an evaluation be conducted to study the Program and students’ experiences before, during, and after being part of the Program. This form is your agreement that you understand these important requirements for the Program.

In submitting this application, I agree to the following for each child named below:

To be eligible for participation in the D.C. Opportunity Scholarship Program, I must be a resident of the District of Columbia and my annual household income must be below certain specified amounts. (Questions in Part B on the next page will be used to determine your initial eligibility.)

I understand that, if eligible, my child’s name will be placed in a lottery for a scholarship. My child may or may not receive a scholarship under this Program.

If my child is not selected to receive a scholarship this year, he or she may be included in a lottery drawing for the following year, as long as the family remains eligible under the Program’s rules.

I understand that my child and I are required to participate in all aspects of the evaluation, including the annual testing of my child, filling out annual surveys, and allowing records to be collected from my child’s school. If my child and I do not participate in these evaluation activities, my child will not be eligible for a scholarship in any year.

I consent to the disclosure of information about my child(ren) and me contained in this application to the U.S. Department of Education and its contractor(s) for the purposes of evaluating this program. I understand that the Department and its contractors will not release to anyone or any organization personally identifiable information in this application and evaluation questionnaire, except as required by law.

Parent/Guardian Name….. ________________________________________