Att_TTI OMB package Part A revised 07-14-09

Contract No.: ED-04-0112/0007

MPR Reference No.: 6398-113

An Impact Evaluation of Moving High-Performing Teachers to Low-Performing Schools

Part A: Supporting Statement for Paperwork Reduction Act Submission

July 2009

Submitted to: U.S. Department of Education Institute of Education Sciences 555 New Jersey Avenue, NW Washington, DC 20208 Telephone: (202) 208-7169 Project Officer: Elizabeth Warner

Submitted by: Mathematica Policy Research, Inc. 600 Maryland Ave. SW, Suite 550 Washington, DC 20024-2512 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 Project Director: Steve Glazerman

CONTENTS

Chapter Page

Part A: Supporting Statement for Paperwork Reduction Act Submission 1

A. JUSTIFICATION 1

1. Circumstances Necessitating the Collection of Information 2

a. Overview of the Study 2

b. Research Questions 3

c. Data Collection Plan and Study Timeline 3

2. How, by Whom, and for What Purpose the Information Is to Be Used 6

3. Use of Automated, Electronic, Mechanical, or Other Collection Technology 6

4. Efforts to Avoid Duplication of Effort 7

5. Sensitivity to Burden on Small Entities 7

6. Consequences of Not Collecting Data 7

7. Special Circumstances 7

8. Federal Register Announcement and Consultation 8

a. Federal Register Announcement 8

b. Consultations Outside of the Agency 8

9. Payments or Gifts 8

10. Assurances of Confidentiality 9

11. Additional Justification for Sensitive Questions 10

12. Estimates of Hours Burden 10

13. Estimates of Total Annual Cost Burden to Respondents or Record Keepers 11

14. Estimates of Annual Costs to the Federal Government 12

15. Reasons for Program Changes or Adjustments 12

16. Plans for Tabulation and Publication of Results 12

a. Tabulation Plans 12

b. Publication Plans 14

17. Approval to Not Display the OMB Expiration Date 14

18. Explanation of Exceptions 14

REFERENCES 15

APPENDICES

APPENDIX A: candidate survey

APPENDIX B: Teacher background surveys

1. Transfer Teacher Career and Satisfaction Survey

2. Teacher Career and Satisfaction Survey

3. Teacher Career and Satisfaction Survey, Year 2

APPENDIX C: principal survey

1. Principal Survey

2. Principal Survey, Year 2

APPENDIX D: district administrative records collection

1. Request for Student Records Data Cover Letter

2. Student Records Data Collection Form

APPENDIX E: Teacher roster collection form

tables

Table Page

A.1 schedule of major study activities A-5

A.2 estimated response time A-11

Part A: Supporting Statement for Paperwork Reduction Act Submission

This submission is a request for approval of data collection activities that will be used to support An Impact Evaluation of Moving High-Performing Teachers to Low-Performing Schools. This evaluation is being funded by the Institute of Education Sciences (IES), U.S. Department of Education (ED); it is being implemented by Mathematica Policy Research, Inc. (MPR) and its subcontractors – The New Teacher Project (TNTP) and Optimal Solutions Group (OSG). The program being evaluated is called the Talent Transfer Initiative (TTI). This program uses value-added student learning gains to identify teachers with consistently high performance and offers them recruitment and retention bonuses to transfer to schools identified as low-performing based on average student test scores.

This is the second submission of a two-stage clearance request. The first submission (approved on November 5, 2008 under OMB number 1850-0861) requested approval to recruit school districts for the study, collect student records data from recruited districts, and administer a survey to 64 teachers participating in a pilot study. In this package, IES is requesting approval for all data collection activities that will support the full-scale study.

A. JUSTIFICATION

Research shows that high-quality teachers are critical to raising student achievement (Rivkin et al. 2005; Rockoff 2004; Rowan et al. 2002), yet schools most in need of effective strategies for improving achievement often experience difficulty in attracting and retaining these kinds of teachers (Carroll et al. 2000; Lankford et al. 2002; Roellke 2007). On average, this leads to less experienced and less effective teachers teaching the neediest and lowest achieving students. Increasingly, districts and schools are experimenting with teacher compensation reform as one mechanism to address this less than optimal distribution of teacher quality.

In recent years, multiple States and districts implemented, or considered implementing, various incentives to improve teacher quality in low-performing schools (Education Week 2008; Kowal et al. 2008). However, policymakers lack rigorous research evidence about the forms of teacher incentives that succeed in improving the quality of teachers assigned to these low-performing students. One of the allowable uses of Title II funds includes developing merit-based pay plans and using financial incentives to recruit and retain teachers in high-need schools (Title II, Part A of Elementary and Secondary Education Act (ESEA)-the Improving Teacher Quality State Grants program). This study will provide information about a compensation strategy designed to redistribute teacher quality within a district. Information from this report will also potentially inform performance based pay systems such as those supported by Teacher Incentive Fund grants.

There is some empirical support for the claim that teacher mobility decisions are related to salary differences between districts (Hanushek et al. 2001; Imazeki 2005), and researchers have tried to estimate the additional salary amount needed to attract teachers into high-need schools (Hanushek et al. 2001; Goldhaber et al. 2008). Nevertheless, there is little information about the impact of incentives on attracting teachers to such schools. The few recruitment and retention bonus programs studied previously offered a relatively low bonus amount, did not use an experimental design, or faced difficulties in implementing the incentive (Clotfelter et al. 2008; Rasberry et al. 2006; Stoops 2008). Additionally, these studies did not measure the effect recruited and retained teachers on student achievement in targeted schools. The growing number of States and districts using teacher incentives to address teacher quality warrants a rigorous evaluation of the impact of a teacher incentive program for attracting high-quality teachers into low-performing schools.

1. Circumstances Necessitating the Collection of Information

ESEA emphasizes the importance of teacher quality in improving student achievement, and Section 9601 of ESEA permits ESEA program funds to be used to evaluate activities authorized under the Act. In fiscal year 2009, Title II, Part A, of ESEA is providing over 2.9 billion to States to help recruit and retain highly-qualified teachers. One allowable use of Title II, Part A funds is for developing merit-based performance systems and strategies that provide differential and bonus pay for teachers in high-need academic subjects, such as mathematics, as well as teachers in high-poverty schools and districts.

a. Overview of the Study

In October 2007, IES began working with MPR and its subcontractors to design and implement an evaluation of a selective program offering monetary incentives to high-performing teachers who transfer into low-performing schools. This program, labeled the Talent Transfer Initiative (TTI), identifies teachers with consistently high performance based on two to three years of value-added student learning gains. It then offers these teachers a series of bonuses totaling $20,000 over two years for transferring to schools identified as low-performing based on average test scores on district-administered tests in the most recent year for which data are available. The objective of the evaluation is to estimate the impact that the high-performing teachers who transfer to low-performing schools have in their new settings. To maintain fairness and not undermine the overall goal of helping low-performing schools, the program provides retention bonuses totaling $10,000 over two years for the identified high-performing teachers who are already in low-performing schools (in both treatment and control schools). As part of the evaluation, we will measure the retention of such teachers in their schools.

The evaluation is an experiment in which the researchers will randomly assign schools with a teaching position vacancy in targeted grades and academic subjects to the intervention that can hire an identified high value-added teacher, or a control group that will fill vacancies as they normally would if they were not part of a study. The study’s key outcome will be assessed through comparing student achievement in the intervention and control group schools.

Pilot Study and District Recruitment. In the 2008-2009 school year, the evaluation began with a pilot study in one school district using four control schools and four program schools, each of which received a transfer teacher. Concurrently, the study team endeavored to recruit schools for the full-scale study. By spring, 2009 seven districts had agreed to participate in the full-scale study starting in the 2009-2010 school year. Three additional districts will be recruited in 2010 if the desired sample size is not achieved within the current set of districts to bring the total to ten districts in the full-scale study. A detailed study timeline for the project is provided in Table A.1 on page A-6.

b. Research Questions

The study’s central research question is the following:

• When high-performing teachers are placed in low-performing schools, what is the impact on student achievement?

Several additional research questions also important for policymaking will be addressed in this study. Survey data and school records data – collected from treatment and control group schools – will be used to answer these questions:

• What is the distribution of teacher talent within the districts included in the study?

• How responsive to incentives are high-performing teachers? Will a two-year $20,000 financial incentive encourage high-performing teachers to transfer into and remain in selected low-performing schools in their districts?

• What factors, other than financial, influence the career decisions of high-performing teachers?

• In the absence of incentives, what are the characteristics and experiences of those who fill teaching vacancies in low-performing schools?

• What effect do financial incentives have on teacher retention for transfer teachers receiving the $20,000 incentive to transfer to low-performing schools?

• What is the retention rate of teachers receiving the $10,000 incentive to remain in their low-performing schools?

c. Data Collection Plan and Study Timeline

The study includes several complementary data collection efforts that together will provide answers to these research questions. A brief description of each data collection activity is provided below.

• Candidate Survey. IES is currently fielding this instrument with high-performing teachers identified for the pilot study based on OMB approval received in fall 2008. IES is seeking additional clearance to conduct the survey beginning in fall 2009 to be administered to all high-performing teachers identified in the full-scale study. The survey will help to characterize the background of teachers identified as high-performing and will provide information about the factors affecting teachers’ willingness to apply to the program, interview at low-performing schools, and, ultimately, to transfer, as well as their experiences during the hiring process (Appendix A).

• Teacher Background Surveys. This survey will be administered in fall 2009 to all teachers in the full study who filled one of the vacancies in intervention or control schools and to their colleagues in the same grade, an eligible population we estimate to include 600 teachers (see Part B). It will collect information on teachers’ experiences at the study schools, along with information on their educational and professional background and other factors that may affect their students’ achievement. There are two versions of this survey: the Transfer Teacher Career and Satisfaction Survey (Appendix B.1) omits questions that will already have been answered by teachers who completed a Candidate Survey and will be administered to identified high-performing teachers who elect to transfer to one of the treatment schools; the Teacher Career and Satisfaction Survey (Appendix B.2) will be administered to all other teachers in the study sample. MPR is pretesting both versions of this instrument with eight respondents in spring 2009 as part of the pilot study. In addition to the fall 2009 administration of the teacher survey, an abbreviated “refresher” Teacher Background Survey (Appendix B.3) will be administered in late fall 2010 to teachers who replace sample members who have left their schools. These three versions of the Talent Transfer Initiative Teacher Career and Satisfaction Survey will be referred to as the “teacher background survey” throughout this document.

• Principal Surveys. The principal survey will be administered in January 2010 and spring 2011 to obtain data from the principals in intervention and control schools. The survey will focus on teacher recruitment and hiring, principals’ assessments of the teachers hired in the study’s target grades, and any redistribution of resources across classrooms (including those related to the arrival of the new hire). The initial survey (January 2010) includes a series of questions about hiring during the period of the program transfers. The emphasis in the follow-up survey (spring 2011) is on teacher performance and school environment throughout the school year, so the hiring questions have been deleted and the survey is scheduled for later in the school year. As part of the pilot study, MPR is pretesting the initial instrument in spring 2009 with eight respondents (Appendix C contains both the initial and follow-up versions of the survey).

• District Administrative Records Collection. MPR will collect data from district records on students in targeted classrooms following each district’s preferred collection approach. (In general, districts are able to provide these records electronically.) These data will be collected at the end of each of the program’s two school years (2009-10 and 2010-11). The data will include standardized test scores and related information on student mobility, demographic characteristics, special education status, and other factors that help explain test scores (Appendix D).

• Teacher Roster Collection. To measure retention of teachers in their new schools, MPR will collect teacher rosters for all schools in the fall and spring of each of the program’s two school years, and once in the fall of 2011. (The latter collection will be used to determine whether teacher retention changes after the incentive period has expired.) Schools will be asked to submit a list of all regular classroom teachers and the grade(s) and subject(s) that they teach in whatever format is most efficient to those submitting the data. In addition to treatment and control schools, the data collection plan includes a collection of rosters from schools in which identified high-performing teachers were offered a “retention” incentive to continue in their current positions because they were already teaching in a low-performing school; this will allow measurement of whether teachers who received incentives stayed in their schools, or in teaching, at different rates than teachers who did not receive retention incentives. (Appendix E includes a form that indicates the kinds of information that we will be collecting from schools.)

This clearance request pertains to the administration of the candidate survey (Appendix A), teacher surveys (Appendix B), principal surveys (Appendix C), the district records form (Appendix D) and an example of a teacher roster form (Appendix E). Table 1.A provides a schedule of data collection activities and the overall timeframe of the study.

Table A.1
Schedule of Major Study Activities

Activity	Fall 2008	Spring 2009	Summer 2009	Fall 2009	Winter 2010	Spring 2010	Summer 2010	Fall 2010	Spring 2011	Summer 2011	Fall 2011
Pilot study with one district (8 schools)	X	X
Recruit districts and obtain data to identify high-performing teachersa	X	X
Identify low-performing schools with fall 2009 vacancies		X
Conduct school random assignment		X
Conduct Candidate Survey				X
Conduct Teacher Career and Satisfaction Survey (background survey)				X				X
Conduct Principal Survey					X				X
Collect student records data for impact analysis			X				X			X
Collect teacher rosters for retention analysis				X		X		X	X		X

^a Pilot candidate survey and district recruitment justified in 2008 submission to OMB.

2. How, by Whom, and for What Purpose the Information Is to Be Used

The primary purpose of the evaluation is to estimate the impacts on student achievement of moving high-performing teachers to low-performing schools. The information collected also will be useful in that it will:

• Provide information about the implementation of the TTI, including challenges to identifying and transferring prospective high-performing teachers

• Provide new evidence on the degree to which teacher quality is distributed equitably across schools, and so inform policy decisions about strategies for addressing the distribution of teacher quality and how those strategies affect student achievement

• Inform the debate about teachers’ responsiveness to incentives, including the types of teachers who change their behavior and the role that school characteristics and other factors play in teachers’ choice of schools and willingness to relocate

Findings from this study will help inform policy on supporting programs designed to recruit and retain effective teachers, and how these programs can affect student achievement. The data also will be useful for State and local policymakers and school districts. Findings will be presented in two reports, one for each program year studied.

3. Use of Automated, Electronic, Mechanical, or Other Collection Technology

The data collection plan reflects sensitivity to the issues of efficiency, accuracy, and respondent burden. Where feasible, information will be gathered from existing data sources, using the most efficient methods available. Existing data sources will include test scores for school-administered tests and student demographic information. This information will be obtained in the form of computer files provided by the school district. Some data, however, can be obtained only from teachers and school staff.

The teacher background survey will be mailed to respondents to complete and return, with telephone and email follow-up for no response or consistency checks. The study team considered other modes of survey administration for these instruments, such as a computer automated telephone interview (CATI) or a web-based survey. However, because the sample size is relatively small, the predicted cost of developing a computer-assisted survey outweighed the expected benefits. Some respondents may find a mail questionnaire to be less burdensome because computer-assisted interviews typically need to be conducted when the respondent has access to a telephone or computer; also, teachers’ access to telephones and private access to computers is uneven.

The Principal Survey will be administered as a web-based survey; principals also will have the option of completing a self-administered paper questionnaire or completing the survey by telephone. The online survey will enable principals to complete it at a location and time of their choice. In addition, its automated nature will allow principals to respond to the same set of questions about several teachers sequentially; with a paper questionnaire, information on each teacher will require a separate page or pages, significantly increasing the overall length of the instrument.

To collect the student demographics and test score data, MPR will work with district personnel to determine the most efficient and least burdensome procedures for their staff, and will capitalize on any electronic systems the districts have in place. Wherever possible, school districts will be able to simply upload or enter the necessary data into an electronic spreadsheet, or to an equivalent file, and transfer it to MPR through a secure FTP site.

Finally, teacher rosters will be collected in whatever form is most convenient and least burdensome for schools, whether as paper records sent by fax and scanned into electronic form by the study team, electronic records stored by the school, or electronic records stored centrally by the district. Wherever possible, the study team will gather such rosters from schools’ Internet websites, obviating the need to impose any burden on schools or gather the data centrally from the district department of human resources, if such information is deemed sufficiently accurate.

4. Efforts to Avoid Duplication of Effort

No similar evaluations are being conducted and there is no equivalent source for the information to be collected. Moreover, the data collection plan reflects careful attention to the potential sources of information for this study, particularly to the reliability of the information and the efficiency in gathering it. The data collection plan avoids unnecessary collection of information from multiple sources.

5. Sensitivity to Burden on Small Entities

The primary entities for the study are districts, principals, and teachers. Burden is minimized for all respondents by requesting only the minimum data required to meet the study’s objectives and by reducing the burden on school employees wherever possible. The sample sizes and data requirements were determined by careful consideration of the information needed to meet the study’s objective and have been reviewed by the study’s Technical Working Group (TWG), listed below in Section 8.b – Consultations Outside of the Agency.

6. Consequences of Not Collecting Data

The data collection plan described in this submission is necessary for conducting ED’s Evaluation of Moving High-performing Teachers to Low-Performing Schools and, consistent with the goals of the NCLB, to address the uneven distribution of teacher quality and the issue of raising student achievement by requiring that all students be taught core subjects by highly qualified teachers.

7. Special Circumstances

There are no special circumstances associated with this data collection.

8. Federal Register Announcement and Consultation

a. Federal Register Announcement

The 60-day Federal Register notice was published in Volume 74, page 22158, on May 12, 2009. No public comments have been received.

b. Consultations Outside of the Agency

During the preparation of the study’s design and associated data collection plan, IES sought input from the TWG, which includes a number of the nation’s leading experts in areas relevant to this study:

• Dale Ballou (Vanderbilt University)

• Brad Jupp (Denver Public Schools)

• Tom Kane (Harvard Graduate School of Education)

• Rob Meyer (University of Wisconsin Center for Education Research)

• Tony Milanowski (University of Wisconsin Center for Education Research)

• Jeff Smith (University of Michigan)

• Louise Sundin (Minneapolis Federation of Teachers, formerly)

• Jake Vigdor (Duke University)

Throughout the study, the study team will continue to consult with the TWG on issues that will benefit from its input. While providing input on the study design, Mr. Jupp was affiliated with the Denver Public Schools. He is currently affiliated with the U.S. Department of Education, Office of the Secretary.

9. Payments or Gifts

Incentives have been proposed for the Candidate and the teacher background surveys to partially offset respondents’ time and effort in completing the surveys. In early fall 2009 (after OMB approval is received),

MPR will administer the Candidate Survey to an estimated 600 out of 750 eligible teachers (80 percent) across multiple districts. We propose offering a $25 incentive for completing the survey so as to acknowledge the 30 minutes required to complete the instrument. This proposed amount is within the incentive guidelines outlined in the March 22, 2005 memo, “Guidelines for Incentives for NCEE Evaluation Studies,” prepared for OMB and is the same amount as was approved by OMB for the administration of this survey in the pilot phase. The incentives have been proposed for the candidate survey to offset anticipated reluctance from candidates—a majority of whom may not see any benefit in participating in the survey. In fact, the candidates whose survey response are of greatest interest include those who chose not to engage with the MTRP or those who were rejected from MTRP teaching positions for which they applied. These two groups are less likely to complete surveys, but their perspectives are key to learning all that we can from the study. 

We also propose an incentive of $25 for completing the teacher background survey in fall 2009 and fall 2010. Both of these surveys are estimated to be 30 minutes in length and are considered high burden teacher surveys. They are administered by mail, and high response rates are needed from both the treatment and control group. Teachers are reported to be the target of numerous requests to complete surveys on a wide variety of topics from State and district offices, independent researchers, and the Department of Education (PPSS and NCES). According to NCEE contractors, the collective bargaining agreements in many districts do not allow teachers to complete surveys during school time. Therefore, we propose this incentive of $25 as an efficient way to obtain response rates of at least 80 percent. This proposed incentive is consistent with incentives approved on similar NCEE impact evaluations and teacher instruments.

The research team does not plan to offer respondent payments or gifts to principals because typical practice suggests that such incentives are not cost-effective,

10. Assurances of Confidentiality

The data collection efforts that are the focus of this clearance package will be conducted in accordance with all relevant regulations and requirements. These include the Education Sciences Institute Reform Act of 2002, Title I, Part E, Section 183, which requires “All collection, maintenance, use, and wide dissemination of data by the Institute” to “conform with the requirements of Section 552 of Title 5, United States Code, the confidentiality standards of subsections (c) of this section, and sections 444 and 445 of the General Education Provisions Act (20 U.S.C. 1232 g, 1232h).” These citations refer to the Privacy Act, the Family Education Rights and Privacy Act, and the Protection of Pupil Rights Amendment. In addition, for student information, the director will ensure that all individually identifiable information about students, their academic achievements, and their families, as well as information with respect to individual schools, shall remain confidential in accordance with Section 552a of Title 5, United States Code, the confidentiality standards subsection (c) of this section, and sections 444 and 445 of the General Education Provisions Act. The study also will adhere to the requirements of subsection (d) of Section 183, which prohibits the disclosure of individually identifiable information and makes the publishing or inappropriate communication of individually identifiable information by employees or staff a felony.

MPR and its subcontractors, TNTP and Optimal Solutions Group, will use information from the study for research purposes only, and individually identifiable information will not be disclosed. No information that identifies any study participant will be released. Further, personally identifiable data will not be entered into the analysis file, and data records will contain a numeric identification code only. When reporting the results, data will be presented in aggregate form only, such that individuals and institutions will not be identified or identifiable. A statement to this effect will be included with all requests for data. When data are collected through telephone or in-person follow-up interviews, respondents will be reminded about the confidentiality protections, the voluntary nature of the survey, and their right to refuse to answer individual questions.

All members of the study team having access to the data will be trained and certified on the importance of confidentiality and data security. All data will be kept in secured locations, and identifiers will be destroyed as soon as they are no longer required.

The following safeguards will be employed routinely by MPR to carry out confidentiality assurances during the study:

• All employees at MPR will sign a confidentiality pledge (Appendix F) emphasizing its importance and describing their obligation.

• Access to sample selection will be limited to those who have direct responsibility for providing and maintaining sample locating information. At the conclusion of the research, these data will be destroyed.

• Identifying information will be maintained on separate forms and files, linked only by a sample identification number.

• Access to the file linking sample identification numbers with the respondents’ ID and contact information will be limited to a small number of individuals who have a need to know this information.

• Access to the hard copy documents will be strictly limited. Documents will be stored in locked files and cabinets. Discarded materials will be shredded.

• Computer data files will be protected with passwords, and access will be limited to specific users. Especially sensitive data will be maintained on removable storage devices kept physically secure when not in use.

The Privacy Act of 1974 applies to this collection. MPR will make certain that all surveys are held strictly confidential, as described above, and that in no instance will responses be made available, except in tabular form. Under no condition will information be made available to school personnel. District and school staff responsible for assisting MPR in the data collection will be fully informed of MPR’s policies and procedures regarding confidentiality of the data. A System of Records Notice was published in the Federal Register on March 24, 2009.

11. Additional Justification for Sensitive Questions

None of the data collection forms contain items considered to be of a sensitive nature.

12. Estimates of Hours Burden

Burden hours from the first phase (pilot and recruitment efforts) are nearing completion and will no longer be needed at the time of the anticipated clearance of this request. Therefore, the program office is deducting the 840 hours for the recruitment data collection activities completed in the 7 recruited districts. Additionally, because the initial collection was not put into Rocis on an annual basis, the adjustment to the annual figure will also be made at this point. The program office does intend to recruit the remaining 3 districts (included in the initial submission) in the fall of 2009, accounting for the total carryover of 397 hours (133 when this is adjusted for annual burden hours). The annual carryover of burden hours from the original submission will be 133 burden hours (1237-1104). Although there is an addition of 428 new annual burden hours being added to 133 annual burden hours carryover (with rounding this is 560 total annual burden hours), the resulting program change is calculated as being an overall decrease because the first phase was submitted in Rocis using total burden hours (rather than annual burden hours as it should have been).

Table A.2 presents burden hours, broken down by instrument and respondent. The number of respondents for the Candidate Survey assumes 750 eligible respondents and an 80 percent response rate, resulting in 600 completed surveys.

The teacher background survey assumes an 80 percent response rate for the 600 initially eligible respondents (who are different from the Candidate Survey population), resulting in 480 completed responses.

For the refresher survey, we assume 15 percent attrition of the original 600 teachers, resulting in 90 eligible respondents, which translates into 72 completes, assuming 80 percent response rate (600 * 15% * 80% = 72).

The numbers of respondents for the teacher rosters assume that 80 schools will have a teacher subject to a retention incentive in addition to the 120 schools in the transfer teacher study (treatment or control).

TABLE A.2

ESTIMATED RESPONSE TIME

Respondent/data request	Number of respondents	Unit response time (hours)	Total response time (hours)
Teacher Surveys
Candidate Survey (fall 2009)	600	0.5	300
Teacher Background Survey, initial (fall 2009)	480	0.5	240
Teacher Background Survey, refresher (fall 2010)	72	0.33	24
Principals/schools
Principal Survey (winter 2010; spring 2011)	120	2 x 0.5 = 1.0	120
Teacher Rosters (fall & spring 2009-10; fall & spring 2010-2011; fall 2011)	200	5 x 0.2 = 1.0	200
District Staff
Records File Preparation (spring 2010 and 2011)	10	2 x 20 = 40	400
Total			1,284

13. Estimates of Total Annual Cost Burden to Respondents or Record Keepers

None.

14. Estimates of Annual Costs to the Federal Government

The total estimated cost of the study is $11,692,524; the estimated annual cost for each of the five years of the study is $2,338,504.

15. Reasons for Program Changes or Adjustments

Although there is an addition of 428 new annual burden hours being added to 133 annual burden hours carryover (with rounding 560 total annual burden hours), the resulting program change of -677 is calculated as being an overall decrease because the first phase was submitted using total burden hours (rather than annual burden hours as it should have been.) A further explanation can be found in #12.

16. Plans for Tabulation and Publication of Results

Our tabulation plans for the study include two sets of analyses: (a) descriptive analysis based on information gathered on teachers, schools, and implementation of the TTI; and (b) estimation of the impacts of the TTI on student test scores and teacher retention.

a. Tabulation Plans

Tabulation plans cover both the impact and implementation evaluations. Each is discussed below.

Tabulating Descriptive Information. To identify challenges and develop strategies for refining the intervention, the data collected through the teacher and principal surveys will be used to describe implementation of the TTI. This description will include how teachers are recruited and hired in low-performing schools, their characteristics, as well as the factors that influence teachers’ decisions to apply for and pursue positions in those schools. These data also will provide important context for the impact evaluation. Data from the two-stage identification of high-performing teachers will be combined with publicly available data on schools to describe the distribution of teacher quality. Although the primary focus of the evaluation will be on estimating impacts on student achievement, we also will estimate the impact of the program on teacher retention of the transferring teachers. In addition, we will tabulate data to describe the broader effects of the TTI on schools and school districts by examining other outcome measures, such as the subjective ratings of newly hired teachers’ performance from the principal survey, as well as the retention rates of high-performing teachers who were already in the study schools.

Estimating Impacts of the Talent Transfer Initiative. For the impact evaluation, average student achievement for students in treatment and control schools, as measured by district-administered assessments, will be compared and analyzed to identify statistically significant differences. We hypothesize that placing a high-performing teacher in a low-performing school can result in three fundamental effects: direct effects of high-performing teachers on students in their own classrooms, indirect effects of transfer teachers on their colleagues and thus students in other classrooms, and distributional effects resulting from principals redirecting students or resources due to the presence of transfer teachers.

• Direct Effects on Student Achievement. Student test scores will be used to examine whether transfer teachers raise the achievement of the students in their own classroom relative to the achievement that would have been attained (the “counterfactual”) had the students been taught by (a) another teacher in the same school, or (b) whomever would have been hired by the school had the TTI not been in existence (represented by the control teacher group).

• Indirect Effects on Student Achievement. Student test scores and teacher and principal surveys will be used to examine whether the potential benefits of a transfer teacher may spill over to colleagues and affect students in other classrooms where teachers collaborate on lesson planning and curriculum design, or where transfer teachers provide information, mentoring, or other support of their colleagues.

• Distributional Effects on School Resources. Principals may assign transfer teachers to the hardest-to-teach students or redirect the mentoring or supervisory time they normally would devote to newly hired teachers toward other teachers in the school. The presence of distributional effects would make it difficult to distinguish between true direct and indirect effects. The principal survey poses specific questions about the allocation of school resources and assignment of students to teachers.

The sum of these three effects is the total impact of the TTI on student achievement in receiving schools. We can derive a pure estimate of the total effect of TTI on receiving schools by comparing average achievement growth for the whole treatment school (or grade level within the school) to that of the whole control school (or grade level).

Decomposing that total effect into a pure direct effect, an indirect effect, and a distributional effect as described is more challenging because we cannot easily quantify the distributional effect. We can report on an estimate of the direct effects by comparing average outcomes for the students of transfer teachers in treatment schools to average outcomes for the students of their counterparts (typically new hires) in control schools. This estimate of the direct effect combines the pure direct effect with some distributional effects. Similarly, we can report the on the corresponding estimate of the indirect effects that is confounded with distributional effects as well by comparing outcomes for the transfer teacher’s colleagues to outcomes of the newly hired control teacher’s colleagues.

While we cannot assume distributional effects are zero, we will use evidence from the teacher and principal surveys relating to student assignment and other resources to gauge the likely role that distributional effects play, including the sign if not the magnitude of these effects. For example, if we find that principal reports on the method by which students were assigned to teachers in both treatment and control schools is the same, particularly if this method is believed to produce equivalent groups of students, and if principals and teachers report equal sharing of mentoring and professional development supports across the grade within school, then we could assume the distributional effects are negligible and interpret the estimated direct and indirect effects being pure. On the other hand, if we find that principals and teachers report that transfer teachers are given more challenging assignments and they require less intervention from support staff such as literacy or math coaches than their colleagues, then we can interpret the estimated direct effect as a lower bound on the pure direct effect attributable to the transfer teacher.

Building upon the simple comparison of means, we will compute regression-adjusted estimates of the direct and total impacts of the TTI. Using regression procedures increases the statistical precision of the impact estimates by enabling us to account for student, teacher, and school characteristics other than TTI status that could affect the outcome.

b. Publication Plans

Two reports will be prepared under the full-scale study. The first report, with a projected release date in the fall of 2011, will describe the implementation, including challenges in identifying high-performing teachers willing to move to low-performing schools, and will examine the impacts of these teachers on student achievement during the intervention’s first year. The second report has a projected release date in the fall of 2012. It will address impacts on student achievement during the intervention’s first and second years and will report on the retention rates of transfer teachers relative to new hires in control schools.

17. Approval to Not Display the OMB Expiration Date

Approval not to display the expiration date for OMB approval is not requested.

18. Explanation of Exceptions

No exceptions to the certification statement are being sought.

REFERENCES

Carroll, S., R. Reichardt, and C. Guarino. “The Distribution of Teachers Among California’s School Districts and Schools.” Santa Monica, CA: RAND Corporation, 2000.

Clotfelter, C., E. Glennie, H. Ladd, and V. Jacob. “Teacher Bonuses and Teacher Retention in Low Performing Schools: Evidence from the North Carolina $1,800 Teacher Bonus Program." Public Finance Quarterly, vol. 36, no. 1, 2008, pp. 63-87.

Education Week. “Quality Counts 2008: Tapping into Teaching.” Washington, DC: Education Week, 2008.

Goldhaber, D., M. DeArmond, A. Liu, and D. Player. “Returns to Skill and Teacher Wage Premiums: What Can We Learn By Comparing the Teacher and Private Sector Labor Markets?” University of Washington, School Finance Redesign Project, August 2008.

Hanushek, E., J. Kain, and S. Rivkin. “Why Public Schools Lose Teachers.” Working Paper 8599. Cambridge, MA: National Bureau of Economic Research, 2001.

Imazeki, J. “Teacher Salaries and Teacher Attrition.” Economics of Education Review, vol. 24, 2005, pp. 431-449.

Kowal, J., B. C. Hassel, and E. A. Hassel. “Financial Incentives for Hard-To-Staff Positions: Cross-Sector Lessons for Public Education.” Washington, DC: Center for American Progress, 2008.

Lankford, H., S. Loeb, and J. Wyckoff. “Teacher Sorting and the Plight of Urban Schools: A Descriptive Analysis.” Educational Evaluation and Policy Analysis,
vol. 24, no. 1, spring 2002, pp. 37-62.

Rasberry, M., E. Hirsch, D. Montgomery, H. Muhammad, and M. Raschko. “Transforming Teaching and Learning in Mobile: Understanding Reform in MCPSS Five Transformation Schools.” Hillsborough, NC: Center for Teaching Quality, April 2006.

Rivkin, S. G., E. A. Hanushek, and J. F. Kain. “Teachers, Schools, and Academic Achievement.” Econometrica, vol. 73, no. 2, 2005, pp. 417-458.

Rockoff, J. “The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data.” American Economic Review, vol. 94, no. 2, 2004, pp. 247-252.

Roellke, C. “Teachers.” Education Finance and Policy, vol. 2, no. 1, 2007, pp. 1-9.

Rowan, B., R. Correnti, and R. J. Miller. “What Large-Scale, Survey Research Tells Us About Teacher Effects on Student Achievement: Insights from the Prospects Study of Elementary Schools.” Teachers College Record, vol. 104, no. 8, 2002, pp. 1525-1567.

Stoops, T. Performance Pay for Teachers: Increasing Student Achievement in Schools with Critical Needs. Raleigh, NC: John Locke Foundation, September 2008.