Att_Recruitment Part A FINAL.1_27_11

A Study of Teacher Residency Programs

Part A

January 27, 2011

Contract Number: ED-IES-10-C (0001) Mathematica Reference Number: 06748-510 Submitted to: Institute of Education Sciences U.S. Department of Education 555 New Jersey Avenue, NW Washington, DC 20208 Project Officer: Melanie Ali Submitted by: Mathematica Policy Research 600 Maryland Avenue, SW Suite 550 Washington, DC 20024-2512 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 Project Director: Philip Gleason

A Study of Teacher Residency Programs Part A January 27, 2011

CONTENTS

SUPPORTING STATEMENT For Paperwork REDUCTION ACT 1

PART A: Justification 1

1. Circumstances Necessitating the Collection of Information 1

a. Statement of Need for a Study of TRPs 1

b. Research Questions 2

c. Study Design 2

d. Recruitment of TRPs and Districts 3

e. Data Collection Plan 5

f. Study Activities and Timeline 7

2. Purposes and Uses of the Data 7

3. Use of Technology to Reduce Burden 7

4. Efforts to Avoid Duplication 8

5. Methods to Minimize Burden on Small Entities 8

6. Consequences of Not Collecting Data 8

7. Special Circumstances 8

8. Federal Register Announcement and Consultation 9

a. Federal Register Announcement 9

b. Consultations Outside the Agency 9

c. Unresolved Issues 9

9. Payments or Gifts 9

10. Assurances of Confidentiality 9

11. Justification for Sensitive Questions 10

12. Estimates of Hours Burden 10

13. Estimates of Cost Burden to Respondents 11

14. Annualized Cost to the Federal Government 11

15. Reasons for Program Changes or Adjustments 11

16. Plan for Tabulation and Publication of Results 11

Publication Plans 13

17. Approval Not to Display the Expiration Date for OMB Approval 13

18. Exception to the Certification Statement 13

REFERENCES 14

APPENDIX A: NOTIFICATION LETTERS

APPENDIX B: NON-TECHNICAL BROCHURE

APPENDIX C: CONFIDENTIALITY PLEDGE

APPENDIX D: TALKING POINTS

TABLES

Table 1. Overview of TRP Involvement in the Study 3

Table 2. Schedule of Major Study Activities 7

Table 3. Research Questions and Data Sources 8

Table 4. Estimated Response Time for Recruitment Activities 11

SUPPORTING STATEMENT For Paperwork REDUCTION ACT

This OMB package requests clearance to recruit teacher residency programs (TRPs) and districts for a study of TRPs. The study will provide important implementation information on TRPs funded by the U.S. Department of Education (ED), as well as information on the achievement outcomes of students whose teachers participate in TRPs. The study will focus primarily on TRPs that received Teacher Quality Partnership (TQP) grants from ED in late 2009 and early 2010. TRPs that did not receive TQP grants may be included if there are not enough grantees to satisfy study needs. ED’s Institute of Education Sciences (IES) has contracted with Mathematica Policy Research and its partner, Decision Information Resources, to conduct the study.

The main objective of the study is to describe the characteristics of TRPs and their participants. The study will also summarize the academic outcomes of students taught by novice TRP teachers and examine the retention rate of novice TRP teachers. This request for clearance focuses on sample recruitment activities. A future request will seek clearance for data collection activities for the full-scale study. We are submitting the package in two stages because the study schedule requires that recruitment begin before the data collection instruments are finalized.

A. Justification

1. Circumstances Necessitating the Collection of Information

a. Statement of Need for a Study of TRPs

The TRP study is authorized in Title II, Part A of the Higher Education Act, as amended on August 14, 2008, by the Higher Education Opportunity Act (Publ. L. 110-315, Sections 201–204) (HEA). The law stipulates that federal funds are to be used to evaluate activities that are authorized under this act. The TQP grant program was funded in fiscal year 2009 with $43 million, and received an additional $100 million from the American Recovery and Investment Act (ARRA) of 2009.

The study is also authorized under Title II, Part A of the Office of Elementary and Secondary Education Act (ESEA) Title II, Part A, 20 U.S.C. 6601-6641. This legislation includes the Improving Teacher Quality State Grants program, aimed to increase the academic achievement of all students by helping schools and districts improve teacher and principal quality and ensuring that all teachers are highly qualified. Providing support for TRPs is an allowable use of program funds.

Research suggests that for many teachers the early years represent a difficult transition period—first-year teachers tend to be less effective than their experienced counterparts (Clotfelter et al. 2007; Hanushek et al. 2005) and newer teachers are more likely to leave the profession (Ingersoll and Smith 2003). The situation may be more pronounced at high-need schools where teachers leave at higher rates and positions are more difficult to fill compared with more advantaged schools (Hanushek et al. 2004; Lankford et al. 2002; Berry 2008). While reasons for these patterns are complex, some teachers attribute staffing difficulties at high-need schools to a lack of support and training on how to effectively teach students with social and behavioral challenges (Berry 2008).

Some districts have responded to staffing challenges by creating or adopting TRPs. The TRP model combines elements from different models of teacher preparation. As with alternative routes to teaching, TRPs give candidates a “fast track” to the classroom without having to complete an undergraduate major in education, enabling them to start teaching prior to receiving initial certification. TRPs involve a year-long “clinical” experience (the “residency”) shadowing and co-teaching with an experienced mentor, similar to but longer than the usual student teaching component of traditional routes. TRPs also provide continued support and mentoring after participants become teachers of record, similar to what is provided in teacher induction programs. Before and during their residencies, participants in TRPs take coursework usually resulting in a master’s degree.

TRPs represent an innovative training strategy, but there is little rigorous evidence connecting specific aspects of teacher training programs to teacher effectiveness (Constantine et al. 2009; Wilson et al. 2002). Given this lack of evidence and the recent infusion of federal grant money to create or expand TRPs, ED has requested a study of TRPs as a model for preparing new teachers. The study will summarize the outcomes of students with novice TRP teachers, examine the retention rates of TRP novice teachers, and describe the programs and their participants. The results of this study will provide critical information on the implementation of this model of teacher preparation to educators, policymakers, and researchers.

b. Research Questions

The study’s primary research questions are:

1. What are the characteristics of teacher residency programs?

2. What are the characteristics of participants in teacher residency programs?

3. What is the average performance of novice¹ TRP teachers as measured by value-added benchmarked against novice and all non-TRP teachers in the district?

4. What are the retention rates of novice TRP teachers and their novice colleagues who did not go through TRPs?

c. Study Design

To the extent possible, we will put the information about TRPs in context by also describing the traditional and other alternative-route programs. We will also examine differences in attributes across the TRPs included in the study (for example, differences in selection criteria or training strategies), and if we notice important variation in any attributes, we will attempt to explore how those differences are correlated with student outcomes.

The study was originally envisioned as focusing solely on the TQP grantees that are operating TRPs, but preliminary information suggests it may be useful to expand the pool to include non-grantee TRPs to boost the sample size for the outcomes study while maintaining the focus on programs that have at least one prior year of experience as of 2010-2011 (so that residents who become teachers of record in fall 2011 will not have gone through the TRP in the program’s first year of operation). We are targeting 8 TRPs and 8 of their partner districts.

As part of the TRP study, and to address the research questions, Mathematica will:

• Collect student records data to summarize the achievement of students in classrooms of TRP teachers and non-TRP teachers.

• Track teacher mobility through district records and teacher mobility surveys in order to examine retention among novice TRP and novice non-TRP teachers.

• Survey and interview TRP administrators to describe the programs and their implementation.

• Survey teachers as well as TRP residents and their mentors to describe these sample members and their teaching experiences.

Different sets of TRPs will be needed for the four major analytical components of the study, as shown in Table 1. For example, all TQP grantees operating TRPs will be surveyed about basic program characteristics, but only a subset of about 15 of their administrators will be interviewed for more program details.

Table 1. Overview of TRP Involvement in the Study

	Number	Student Achievement Outcomes Study	Teacher Retention	Descriptive Analysis of TRPs	Descriptive Analysis of TRP Participants
All TRPs that Received TQP grants in 2009-2010	28			√
Subset of Above Group for In-Depth Study	15a			√	√
Experiencedb Grantees Specifically Targeted For Outcomes Study	6	√	√	√	√
Other Experienced TRPs (Non-Grantees) Specifically Targeted For Outcomes Study	2	√	√	√	√

^a Estimate; at this time not enough is known about TRPs—especially the number and type of teachers they will have placed in fall 2011, and how those placements will be distributed across partner districts and schools—to cite specific numbers with certainty. These determinations will be made during the selection and recruitment process.

^b Experienced grantees are those that began operation in 2009 or earlier.

d. Recruitment of TRPs and Districts

This study will not statistically sample TRPs, districts, schools, or teachers. Instead, it will rely on a purposefully selected convenience sample of TRPs that have been in existence since at least 2009 and are best suited for the outcomes study (a determination based primarily on the availability of student-teacher linked data and the number and type of teachers they prepare). Within the study districts, we will collect data on all students in grades tested in math and reading. The study does not aim to make statements that generalize beyond the TRPs and teachers under study.

Collect and review information on TRPs. The universe of TRPs is quite small (the 28 TQP grantees operating TRPs plus a small number of other TRPs²) relative to the number of traditional (typically undergraduate level) training programs and alternative certification programs, which makes it possible for us to initially consider all of these TRPs for inclusion in the study. We will begin by obtaining and reviewing the grant applications submitted by TQP grantees; reviewing websites; and following up with brief, semi-structured telephone calls to gain an initial sense of the number and type of teachers they expect to place in full-time teaching positions in partner districts for the 2011-2012 school year.

Target a subset of feasible TRPs. Drawing on the information gathered in the aforementioned activities and subsequent conversations with program officials, we will identify a subset of TRPs that appear most feasible for inclusion in the in-depth study. As noted above, we will focus on programs that have been operating since at least 2009. Another important factor will be the availability of student-teacher linked data in math and reading in the partner districts, excluding, for example, programs that only place teachers in non-tested grades and subjects or that only place teachers in self-contained special education classrooms in secondary schools. Student-teacher linked data will allow us to employ value-added techniques to measure the average growth of students. Based on preliminary information, we anticipate we will target 15 grantees. These 15 grantees will (1) contribute additional information to the descriptive analysis of TRPs through a program director interview—information not easily obtained through the program survey described below—and (2) provide data for the descriptive analysis of TRPs through surveys of the program’s current residents and mentor teachers. We will use memoranda of understanding (MOUs) to document the programs’ agreement to participate in subsequent aspects of the study.

Recruit TRPs for the outcomes study. Eight of the TRPs that have been in operation for at least one year will be recruited for the outcomes study. We will seek to include only TQP grantees in the study. If we are unable to identify 8 TQP grantees with student-teacher linked data or are unable to secure the participation of a sufficient number of grantees, we will supplement the sample with about 2 non-grantees. During a site visit with each TRP, a member of the study team will discuss topics such as the study overview, what program participation entails, features of the program and partner districts, and each TRP’s willingness to participate in the study (Appendix D).

Recruit districts to cooperate with the study. Using information from the subset of TRPs described above, we will contact a prioritized list of districts in spring 2011 and seek their agreement to participate in the study. We will begin by sending a packet containing two documents: 1) a notification letter, on ED letterhead and signed by the project officer, highlighting the importance of the study, outlining the study objectives and design, and noting that a member of the study team will follow up soon by telephone to provide more details and discuss the district’s participation (see Appendix A); 2) a two-page brochure describing the study in non-technical terms, briefly explaining what participation would entail for district staff and for teachers, highlighting reasons for participation, identifying the study team, and providing contact information (see Appendix B).

Shortly after the packet’s delivery, a recruitment team member will call the district liaison to initiate a discussion about participating in the study. We will arrange in-person visits with the targeted districts as needed. Through these discussions, we will seek information about TRP teacher placements and assurances that districts maintain and can provide the student and teacher administrative records required to address various components of the study. Finally, we will document the districts’ agreement to participate in the study with an MOU.

e. Data Collection Plan

This package does not request OMB clearance for data collection at this time, only clearance to solidify the participation of grantees and districts. The study includes several complementary data collection efforts that will be used to address the research questions. A brief description of each data collection activity is provided below. The forms for these activities will be developed and submitted in a subsequent clearance package along with estimated burden time for each. Mathematica will develop all the instruments described below, drawing on existing models when possible.

• Student records data. Following the 2011-2012 and 2012-2013 school years, we will ask districts to provide data for all students in grades tested in math and reading. We will collect both demographic data (that is, age, race/ethnicity, English language proficiency, disability status, and eligibility for school lunch program) and data on students’ performance on state or district tests in the current year and two previous school years. We will send the district a letter that will specify the data items requested and a non-technical brochure providing additional study information.

• TRP survey. We will mail a self-administered 35-minute hardcopy survey to a director at each of the 28 TRPs that received a TQP grant in fall 2009 or spring 2010 and to any non-grantees included in the outcomes study. This survey, to be administered in spring 2011, will address TRP characteristics, admission requirements, and key program features, providing the foundation for answering the first research question. The mailing will contain a grantee or non-grantee cover letter, questionnaire and a non-technical brochure providing additional study information (Appendix B). The letter, which will be on ED’s stationery, will describe the study and its objectives and the need for TRP participation, address confidentiality, and provide a telephone number and email address for questions or concerns.

• TRP director interview. In spring 2011, we will conduct semi-structured interviews, by telephone or in person, with the directors of the 15 TRPs identified for the in-depth implementation study. The 45-minute interview will collect detailed information on the amount of instruction in different substantive and pedagogical subject areas provided to candidates at various points in the program (before, during and after the residency year; during and after the first year as a teacher of record). The open-ended questions in the TRP interviews will allow us to collect more in-depth information than that collected from the survey, and to probe for clarification if necessary. We will contact potential respondents in advance and provide them with a list of topics to be covered and any general information about the study as needed. (They will be familiar with the study from our previous contacts with them during the recruitment stage.)

• Resident teacher survey. A 25-minute mail survey of 300 TRP participants who serve their residency during the 2010-2011 school year will be conducted in spring 2011. It will collect descriptive information on resident teachers’ backgrounds as well as experiences during their residency year—for example, interactions with resident mentors, classroom responsibilities, and views on the program. This survey will be administered to all residents from the same set of TRPs included in the program director interviews. The mailing will contain a cover letter, questionnaire and a non-technical brochure providing additional study information (Appendix B). In the cover letter, we will describe the study and its objectives and the need for resident teacher participation, address confidentiality, and provide a toll-free telephone number and email address for questions or concerns.

• Mentor teacher survey. In spring 2011, we will mail a 20-minute self-administered hardcopy survey to the 300 mentors associated with each of the teachers targeted for the resident teacher survey. Respondents will be asked to provide descriptive information on their backgrounds, qualifications, and training for the mentor role, their residents’ responsibilities, and their interactions with residents. Parallel questions across the resident and mentor surveys will allow for corroboration during analysis. The mailing will contain a mentor teacher cover letter, questionnaire and a non-technical brochure providing additional study information (Appendix B). In the cover letter, we will describe the study and its objectives and the need for mentor teacher participation, address confidentiality, and provide a toll-free telephone number and email address for questions or concerns.

• Teacher of record survey. In fall 2011, we will contact districts to request a list of teachers hired within the last two years, their current school placement, and date of hire. The lists will provide the sample for the teacher of record survey. In spring 2012, all novice teachers in the 8 districts in the outcomes study will be asked to complete a 25-minute self-administered hardcopy mail survey on their background characteristics, experiences during the 2011-2012 school year, and views on teaching. Teachers of record who will have completed the resident survey the previous year will complete a shorter version of the teacher of record survey—one that excludes questions about background characteristics. The mailing will contain a teacher of record cover letter, a questionnaire, and a non-technical brochure providing additional study information (Appendix B). The cover letter will describe the study and its objectives and the need for participation from teachers of record, address confidentiality, and provide a toll-free telephone number and email address for questions or concerns. We expect to survey 800 teachers.

• Teacher employment data. In fall 2012 and fall 2013, districts will be asked to verify whether the novice teachers in the outcomes study are still employed by the district. We will send each district a data request form that will specify the data requested, the list of novice teachers, and a non-technical brochure providing additional study information (Appendix B).

• Teacher mobility survey. In fall 2012 and fall 2013, all teachers in the outcomes study will be asked to complete a 15-minute self-administered hardcopy mail survey on their current employment status and reasons for job changes, if applicable. The mailing will contain a cover letter that will describe the purpose of the survey and the need for participation from all teachers regardless of current employment status, address confidentiality, and provide a toll-free number and email address for questions or concerns. We expect to survey 800 teachers in each wave of the survey.

f. Study Activities and Timeline

The study will be completed in five years. Table 2 shows the overall estimated timeline, highlighting the major study activities.

Table 2. Schedule of Major Study Activities

Activity	Spring 2011	Fall 2011	Spring 2012	Fall 2012	Fall 2013	Fall 2014
Recruit TRPs and Districts	√
Conduct TRP Survey	√
Conduct TRP Director Interview	√
Conduct Resident Teacher Survey	√
Conduct Mentor Teacher Survey	√
Obtain List of Novice Teachers		√
Conduct Teacher of Record Survey			√
Collect Student Records Data				√	√
Collect Teacher Employment Data				√	√
Conduct Teacher Mobility Survey				√	√
Submit First Report					√
Submit Second Report and Restricted-Use Data File						√

2. Purposes and Uses of the Data

Teacher quality is a critical component of student achievement, and while TRPs represent an innovative training strategy—one the federal government has, through the TQP grants, invested substantial funds in—little is known about TRP participants or approaches to teacher training. This study will describe characteristics of TRPs and their participants, summarize the academic outcomes of students taught by novice TRP teachers, and examine the retention rate of novice TRP teachers. The purpose of recruitment is to find TRPs and districts that will support the study by providing the needed data. The data will provide policymakers and practitioners with important insight into practices for teacher recruitment, preparation, support, and retention in high-need schools.

Table 3 lists the study’s research questions and the data to be collected to address each question. Study findings will be presented in two reports, and the data collected by the study will be submitted to ED as restricted-use data files that will serve as a valuable resource for other researchers.

3. Use of Technology to Reduce Burden

The recruitment plan is designed to obtain reliable information in an efficient way that minimizes respondent burden. As much information as possible will be gathered from existing data sources, such as successful TQP TRP grant applications (provided by ED) and programs’ websites. If TRP officials have data to share, we will encourage them to submit it electronically.

Table 3. Research Questions and Data Sources

Research Question	Data Sources
1. What are the characteristics of TRPs?	TRP surveys TRP interviews Resident teacher surveys Mentor teacher surveys
2. What are the characteristics of participants in TRPs?	TRP surveys TRP interviews Resident teacher surveys Teacher of record surveys
3. What is the average performance of novice TRP teachers as measured by value-added benchmarked against novice and all other teachers in the district?	District administrative records on student achievement and background characteristics
4. What are the retention rates of novice TRP teachers and their novice colleagues who did not go through TRPs?	Teacher employment data Teacher mobility survey

4. Efforts to Avoid Duplication

No other national study has been conducted or is underway to address the same research questions as this study. This submission represents the sole, official study of TRPs being sponsored by ED. Prior to issuing the performance work statement and request for proposals, ED determined that a national study examining TRPs is needed, and that this study would not be duplicative.

5. Methods to Minimize Burden on Small Entities

The primary entities for the study are TRPs (most of which are operated by colleges and universities), school districts, and teachers. We will minimize burden for all respondents by requesting only the minimum data required to meet study objectives. The sample size and data requirements were determined by careful consideration of the information needed for the study and have been reviewed by the study’s technical working group (TWG).

6. Consequences of Not Collecting Data

The recruitment plan described in this submission is necessary for ED to conduct the study of TRPs and to understand the characteristics of TRPs and their participants. The study represents an important step in developing a systematic study agenda in the area of teacher training and retention. Without recruiting TRPs and school districts, ED will not have the pool of participants from whom to collect the study data. .

7. Special Circumstances

There are no special circumstances associated with this data collection.

8. Federal Register Announcement and Consultation

a. Federal Register Announcement

The 60-day notice to solicit public comments was published in Volume 75, Number 102, page 29731 of the Federal Register on May 27, 2010. No comments were received. The 30-day notice to solicit public comments was published in Volume 75, Number 150, page 47284 of the Federal Register on August 5, 2010. No comments were received.

b. Consultations Outside the Agency

In formulating the study design, the study team sought input from the TWG, which includes some of the nation’s experts in teacher preparation, study methodology, and education policy. We will continue to consult with the TWG throughout the study on other issues that would benefit from their input. The TWG members are:

• Jeffrey Smith, professor of economics, University of Michigan

• James Wyckoff, professor of education, University of Virginia

• Sandra Odell, professor of education, University of Nevada-Las Vegas

• Pamela Grossman, professor of education, Stanford University

• Diana Montgomery, independent research consultant

• Jon Fullerton, executive director, Project for Policy Innovation, Harvard University

• Jason Snipes, director of research, Council of Great City Schools

• Elizabeth Stuart, assistant professor of biostatistics, Johns Hopkins University

c. Unresolved Issues

There are no unresolved issues.

9. Payments or Gifts

The study will not give gifts to TRPs, districts, or any individuals affiliated with these entities for participating in the recruitment process.

10. Assurances of Confidentiality

The efforts that are the focus of this clearance package will be conducted in accordance with all relevant regulations and requirements, including:

• The Privacy Act of 1974, P.L. 93-579 (5 U.S.C. 552a);

• The Family Educational and Rights and Privacy Act (FERPA) (20 U.S.C. 1232g; 34 CFR Part 99);

• The Protection of Pupil Rights Amendment (PPRA) (20 U.S.C. 1232h; 34 CFR Part 98);

• The Education Sciences Reform Act of 2002 (ESRA), (20 U.S.C. 9573; Title I, Part A, Section 183).

Mathematica and its subcontractor DIR will protect the confidentiality of all study information and use it for research purposes only. The project director will ensure the confidentiality of all individually identifiable respondent information.Data will be kept in secured locations and identifiers will be destroyed when no longer required. Study team members with data access will be trained and certified on the importance of confidentiality and data security. In reporting results, data will be presented in aggregate form, such that individuals and institutions will not be identified. Included in all requests for data and communications about the study will be the following statement:

The contractor follows the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). The contractor will protect the confidentiality of all information collected for the study and will use it for research purposes only. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific program, district, school, or individual. We will not provide information that identifies any study participant to anyone outside the study team, except as required by law.

The following safeguards are routinely employed by Mathematica to carry out confidentiality assurances during the study:

1. All employees sign a confidentiality pledge (Appendix C) that emphasizes its importance and describes employees’ obligation.

2. Personally identifiable information (PII) is maintained on separate forms and files, which are linked only by sample identification number.

3. Access to hard copy documents is strictly limited. Documents are stored in locked files and cabinets and discarded materials are shredded.

4. Access to computer data files is protected by secure usernames and passwords, which are only available to specific users.

5. Especially sensitive data is encrypted and stored on removable storage devices that are kept physically secure when not in use.

11. Justification for Sensitive Questions

We do not anticipate that any of the data collected during recruitment will contain items considered to be of a sensitive nature.

12. Estimates of Hours Burden

Table 4 reports the estimated burden hours for TRP officials and district staff who will participate in our recruitment process. The 223 burden hours include 102 hours for TRP staff and 121 hours for district staff. These estimates are based on our experience recruiting grantees and districts for similar studies. Burden estimates for other data collection efforts described in section A.1 above that are not the subject of this request will be included in a future request.

Table 4. Estimated Response Time for Recruitment Activities

Respondent/Data Request	Number of Respondents	Unit Response Time (Hours)	Total Response Time (Hours)
TRP Staff
Initial phone conversation with most TRP directors	35	0.75	26
Site visits with 2 respondents at each of the 15 TRPs most promising for outcomes study	30	2.0	60
Follow-up conversations with 1 respondent at the 8 TRPs selected for outcomes study	8	2.0	16
District Staff
Initial phone conversation with 1 respondent at each of the 15 districts most promising for outcomes study	15	1.0	15
Site visits with 3 respondents at each of the 15 districts most promising for outcomes study	45	2.0	90
Follow-up phone conversations with 2 respondents at each of the 8 districts selected for outcomes study	16	1.0	16
Total			223

13. Estimates of Cost Burden to Respondents

There are no start-up costs for respondents.

14. Annualized Cost to the Federal Government

The estimated average annual cost of the study over the five years of the base contract is $802,618, or a total estimated cost of $4,013,092 for the base contract that includes recruiting TRP grantees as well as district staff, designing and administering all data collection instruments, processing and analyzing the data and preparing reports. Including the study option, the total estimated cost of the study is $5,322,690, an annual cost of $1,064,538.

15. Reasons for Program Changes or Adjustments

This is a new data collection.

16. Plan for Tabulation and Publication of Results

We discuss our plans below for tabulating data for all reports to address the research questions and for publishing results.

Describing the achievement outcomes of students with TRP teachers. The study will not seek to identify the causal effect of TRP teachers on students’ achievement levels. Instead, the study will describe the average growth in achievement of students of novice TRP teachers benchmarked against the average growth of students of all other teachers, as well as the subset of novice non-TRP teachers in the district. Rather than using the simple change in test scores, we will attempt to get the most precise measure of growth possible using a value-added model:

(1)

where Y_ijk is the test score of student i in a class taught by teacher j in year t, Yi_(-t) is a vector of the previous two years of test scores for student i, X_ijk is a vector of student baseline characteristics, the T_i’s are indicator variables for each teacher j, µ_j is a classroom-specific random error term, ε_ij is a student-level random error term, and β, , and γ represent parameters to be estimated. The model will be estimated with ordinary least squares (OLS), using standard errors that account for classroom-level clustering.

The estimates of γ represent the change in student achievement that each teacher produces in excess to what would have been expected based on the characteristics and prior achievement level of their students. We will take an average of the s for TRP teachers and provide the average value of for all teachers³ and the average for novice non-TRP teachers as benchmarks.

TRP and non-TRP teacher retention rates. The study will also summarize the retention rates of TRP and non-TRP teachers in the district. Measures of teacher retention may include whether the teacher remained at the same school, moved to another school in the district, moved to another district, or left the teaching profession.

The timeline for retention data collection is as follows. In fall 2012 and fall 2013, we will contact districts to request data on employment status among all novice teachers. In fall 2012, we will determine which of the novice teachers from the previous year are still teaching in the district. In fall 2013, we will again determine which of the teachers in our analysis sample are still teaching in the district. In addition to the district employment data, we will administer a teacher mobility survey in fall 2012 and fall 2013. We will use the responses from these surveys to supplement the information collected through the district employment data.

Describing the characteristics of TRPs. Drawing on data from the TRP survey and interviews with the directors of TRPs considered for the study, we will use descriptive statistics to paint a profile of these programs. Our analyses will provide information on the types of teachers that TRPs train; the amount, content, and timing of the coursework they provide; and their support and mentoring activities. To the extent possible, we will put the information about TRPs in context by comparing them with traditional and other alternative-route programs. We will also examine differences in attributes across the TRPs included in the study (for example, differences in selection criteria or training strategies), and if we notice important variation in any of these attributes, we will explore how it is correlated with student outcomes. All TRP surveys and director interviews will take place in spring 2011.

Describing the characteristics of participants in TRPs. We will conduct three sets of teacher-level analyses. First, drawing on the more detailed data from the resident teacher survey, we will use descriptive statistics to provide additional information on characteristics of program participants and their experiences and opinions regarding the program. Second, we will use descriptive statistics to profile characteristics of TRP teachers’ mentors during the residency year. We will compare the responses of residents and mentors to questions about their joint activities, which will provide a more balanced and comprehensive view of the residency experiences than would be possible from examining the responses of only one of these groups. Third, to provide context for the study, we will describe TRP and non-TRP teachers’ backgrounds and experiences. Potential characteristics to examine include prestige of undergraduate institution, college major, and prior work experience.

Conducting exploratory analyses assessing variation in outcomes. Our outcomes analysis may be enhanced through exploratory analyses that provide information on key program features and teacher characteristics associated with the outcomes of TRP teachers as well as the mechanisms through which teachers trained in TRPs may influence student achievement. These findings can inform policymakers and program directors seeking to improve the effectiveness of existing and future TRPs. Our exploratory analyses will include subgroup analyses as well as an analysis of potential mediators of TRP outcomes (that is, the characteristics or experiences of TRP teachers that are associated with outcomes that are more positive). In each case, we will note that observed relationships may not be causal, and may reflect the influence of additional, unobserved factors.

Publication Plans

We will prepare two reports presenting the results of the study. The first report, with a projected release date in fall 2013, will address all four research questions based on data from the 2010-2011 and 2011-2012 school years. The second report, with a projected release date of fall 2014, will update our findings on student outcomes and teacher retention using data from the 2012-2013 school year, and include exploratory analyses of factors related to outcomes.

17. Approval Not to Display the Expiration Date for OMB Approval

The study will display the OMB expiration date.

18. Exception to the Certification Statement

No exceptions are being sought.

REFERENCES

Berry, Barnett. “Staffing High-Needs Schools: Insights from the Nation’s Best Teachers.” Phi Delta Kappan, vol. 89, no. 10, 2008, pp. 766–771.

Clotfelter, Charles, Helen Ladd, and Jacob Vigdor. “How and Why Do Teacher Credentials Matter for Student Achievement?” CALDER Working Paper No. 2. 2007.

Constantine, Jill, Daniel Player, Tim Silva, Kristin Hallgren, Mary Grider, and John Deke. “An Evaluation of Teachers Trained Through Different Routes to Certification. Final Report.” Princeton, NJ: Mathematica Policy Research, 2009.

Hanushek, Eric, John Kain, and Steven Rivkin. “The Market for Teacher Quality.” NBER Working Paper 11154. Cambridge, MA: National Bureau of Economic Research, 2005.

Hanushek, Eric, John Kain, and Steven Rivkin. “Why Public Schools Lose Teachers.” Journal of Human Resources, vol. 39, no. 2, 2004, pp. 326–354.

Ingersoll, Richard, and Thomas Smith. “The Wrong Solution to the Teacher Shortage.” Educational Leadership, vol. 60, no. 8, 2003, pp. 30–33.

Lankford, Hamilton, Susanna Loeb, and James Wyckoff. “Teacher Sorting and the Plight of Urban Schools: A Descriptive Analysis.” Educational Evaluation and Policy Analysis, vol. 24, no. 1, 2002, pp. 37–62.

Wilson, Suzanne, Robert E. Floden, and Joan Ferrini-Mundy. “Teacher Preparation Research: An Insider’s View from the Outside.” Journal of Teacher Education, vol. 53, no. 3, 2002, pp. 190–204.

www.mathematica-mpr.com

Improving public well-being by conducting high-quality, objective research and surveys

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC

Mathematica® is a registered trademark of Mathematica Policy Research

1 Throughout this document, novice teachers are defined as those in their first or second year of teaching.

2 Our initial investigation of other TRPs revealed no more than 10 additional programs that were similar to the TRPs operated by TQP grantees in their basic structure—that is, participants took coursework that usually resulted in a master’s degree and had a yearlong residency prior to becoming a teacher of record.

3 In order to make the test scores comparable, we intend to normalize test scores to have a mean of zero and a standard deviation of one. Therefore, the average value of for all teachers will be equal to zero.