Att_Recruitment Part B FINAL.1_27_11

A Study of Teacher Residency Programs

Part B

January 27, 2011

Contract Number: ED-IES-10-C (0001) Mathematica Reference Number: 06748-510 Submitted to: Institute of Education Sciences U.S. Department of Education 555 New Jersey Avenue, NW Washington, DC 20208 Project Officer: Melanie Ali Submitted by: Mathematica Policy Research 600 Maryland Avenue, SW Suite 550 Washington, DC 20024-2512 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 Project Director: Philip Gleason

A Study of Teacher Residency Programs Part B January 27, 2011

CONTENTS

SUPPORTING STATEMENT for paperwork REDUCTION ACT 1

Collection of Information Employing Statistical Methods 1

1. Respondent Universe and Sampling Methods 1

2. Procedures for the Collection of Information 2

3. Methods to Maximize Response Rates and Deal with Nonresponse 4

4. Tests of Procedures or Methods to be Undertaken 5

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 5

APPENDIX A: NOTIFICATION LETTERS

APPENDIX B: NON-TECHNICAL BROCHURE

APPENDIX C: CONFIDENTIALITY PLEDGE

TABLES

Table 1. Overview of TRP Involvement in the Study 2

SUPPORTING STATEMENT for paperwork REDUCTION ACT

This OMB package requests clearance to recruit teacher residency programs (TRPs) and districts for a study of TRPs. The study will provide important implementation information on TRPs funded by the U.S. Department of Education (ED), as well as information on the achievement outcomes of students whose teachers participate in TRPs. The study will focus primarily on TRPs that received Teacher Quality Partnership (TQP) grants from ED in late 2009 and early 2010. TRPs that did not receive TQP grants may be included if there are not enough grantees to satisfy study needs. ED’s Institute of Education Sciences (IES) has contracted with Mathematica Policy Research and its partner, Decision Information Resources, to conduct the study.

The main objective of the study is to describe the characteristics of TRPs and their participants. The study will also summarize the academic outcomes of students taught by novice TRP teachers and examine the retention rate of novice TRP teachers. This request for clearance focuses on sample recruitment activities. A future request will seek clearance for data collection activities for the full-scale study. We are submitting the package in two stages because the study schedule requires that recruitment begin before the data collection instruments are finalized.

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The respondent universe for the outcomes and retention studies will consist of novice TRP teachers and non-TRP teachers. Different sets of TRPs will be needed for the four major analytical components of the study, as shown in Table 1. For example, all TQP grantees operating TRPs will be surveyed about basic program characteristics, but only a subset of about 15 TRP directors will be interviewed for more program details and asked to provide data on applicants.

At the conclusion of each school year, we will request student-level data for all teachers in the district who are teaching math or reading in grades in which students are tested in these subjects (grades 3 through 8 in all districts, and additional grades in some districts). For the student achievement outcomes analysis, we will include all of the students and all of the teachers in the targeted districts in these grades. For the retention analysis, we will focus on all teachers in these districts in their first or second year of teaching.

Table 1. Overview of TRP Involvement in the Study

	Number	Student Achievement Outcomes Study	Teacher Retention	Descriptive Analysis of TRPs	Descriptive Analysis of TRPs and Participants
All TRPs That Received TQP Grants in 2009-2010	28			√
Subset of Above Group For In-Depth Study	15a			√	√
Experiencedb Grantees Specifically Targeted For Outcomes Study	6	√	√	√	√
Other Experienced TRPs (Non-Grantees) Specifically Targeted For Outcomes Study	2	√	√	√	√

^a Estimate – at this time not enough is known about TRPs—especially the number and type of teachers they will have placed in residency in fall 2011, and how those placements will be distributed across partner districts and schools—to cite specific numbers with certainty. These determinations will be made during the selection and recruitment process.

^b Experienced grantees are those that began operations in 2009 or earlier.

2. Procedures for the Collection of Information

Statistical methods for sample selection. This study will not statistically sample TRPs, districts, schools, or teachers. Instead, it will rely on a purposefully selected convenience sample of TRPs that have been in existence since at least 2009 and are best suited for the outcomes study (a determination based primarily on the availability of student-teacher linked data and the number and type of teachers they prepare). Within the study districts, we will collect data on all students who were in grades tested in math and reading. The study does not aim to make statements that generalize beyond the TRPs and teachers under study.

Data collection plan. Our data collection plans will involve different sample members in different ways. We will collect some basic information from the universe of TRPs that received TQP grants and from any other TRPs considered best qualified for the outcomes study; additional information from a subset of the grantees and from any other TRPs considered best qualified for the in-depth implementation study; information from all teachers selected for the outcomes analysis; and student and teacher information from all districts in the outcomes study. We will also collect data on resident teachers and their mentors. Following is an overview of the data collection plans that will be described in greater detail in the follow-up package.

• TRP survey. All TRP grantees (regardless of their inclusion in the outcomes study) plus any non-grantees targeted for the outcomes study will be asked to complete a survey on program characteristics.

• TRP director interviews. The directors of the 15 TRPs targeted for the outcomes study will be interviewed to collect additional details on program operations.

• Resident and mentor surveys. Participants at the 15 TRPs targeted for the outcomes study who serve their residency during the 2010-2011 school year, and their classroom mentors, will be surveyed on their backgrounds and experiences in the residency component.

• Teacher of record survey. All 2011-2012 novice teachers in the outcomes study will be surveyed on their backgrounds and teaching experiences.

• Teacher mobility surveys. All 2011-2012 novice teachers in the outcomes study will be surveyed in two follow-up years on their current employment status and, if they have changed jobs, their current position and reasons for moving.

• Student administrative records. Districts will be asked to provide demographic and achievement data on all students in the outcomes study, including students in the classrooms of TRP teachers as well as those in other classrooms in those same districts.

• Teacher administrative records. Districts in the outcomes study will be asked to provide data on the teaching assignments of 2011-2012 novice TRP and non-TRP teachers and to verify their continued employment in the district in 2012-2013 and 2013-2014. .

Confidentiality. Mathematica’s internal confidentiality pledge, which will guide all staff who work on this study, is presented in Appendix C.

Estimation procedures. The study will not seek to identify the causal effect of TRP teachers. Instead, the study will describe the average growth in achievement of students of novice TRP teachers benchmarked against the average growth of students of all other teachers, as well as the subset of novice non-TRP teachers in the district. Rather than using the simple change in test scores, we will attempt to get the most precise measure of growth possible using a value-added model:

(1)

where Y_ijk is the test score of student i in a class taught by teacher j in year t, Y_i(-t) is a vector of the previous two years of test scores for student i, X_ijk is a vector of student baseline characteristics, the T_i’s are indicator variables for each teacher j, µ_j is a classroom-specific random error term, ε_ij is a student-level random error term, and β, , and γ represent parameters to be estimated. The model will be estimated with ordinary least squares (OLS), using standard errors that account for classroom-level clustering.

The estimates of γ represent the change in student achievement that each teacher produces in excess to what would have been expected based on the characteristics and prior achievement level of their students. We will take an average of all of the s for TRP teachers and present the benchmark of the average of the ’s for all teachers¹ and for other novice teachers.

TRP and non-TRP teacher retention rates. We will also summarize the retention rates of novice TRP and non-TRP teachers in the district. Measures of teacher retention may include whether the teacher remained at the same school, moved to another school in the district, moved to another district, or left the teaching profession.

The timeline for retention data collection is as follows. In fall 2012 and fall 2013, we will contact districts to request data on employment status among all novice teachers. In fall 2012 we will determine which of the novice teachers from the previous year are still teaching in the district. In fall 2013 we will again determine which of the teachers in our analysis sample are still teaching in the district. The teacher mobility survey will also be administered in fall 2012 and fall 2013 to collect information on novice teachers’ employment status.

Mediators analysis. In addition to our main outcomes analysis, a complementary approach will help shed light on the outcomes we observe. We will estimate a version of equation (1) that includes controls for additional covariates, or mediators, representing possible mechanisms through which TRPs may influence student achievement. In one case, for example, we will control for teacher background characteristics to determine whether they are correlated with outcomes. We will use a similar approach to estimate the relationship between outcomes and specific aspects of the training received in TRP programs. For example, we will examine relationship between coursework and training while teaching (reflected by relevant measures from the teacher survey) and student outcomes.

Degree of accuracy needed. The study will not attempt to measure impacts, so statistical power is not a concern for the student outcomes analysis and the teacher retention analysis. However, the study will seek to provide measures of the average outcomes that are as generalizable as possible. To this end, the study will include all TRP and non-TRP teachers who are teaching in the eight districts in grades in which students are tested in reading and math in the student achievement outcomes study. The retention analysis will include all TRP and non-TRP teachers in these districts in their first or second years of teaching.

3. Methods to Maximize Response Rates and Deal with Nonresponse

We anticipate virtually no nonresponse by TRPs we approach during the recruitment phase of this study, since TQP grantees have committed to cooperating with the national study and the small community of other TRPs is likely interested in hearing about the proposed study. The non-grantee TRPs may also be interested in learning more about their programs through the study. We will consider offering to share (with TRPs in both groups) aggregated data from any surveys completed by their residents and the residents’ mentors, along with student outcomes data, so long as doing so does not compromise the confidentiality of survey respondents. As districts are partners in sponsoring and supporting TRPs, we also expect any districts we approach to be highly responsive to our requests for discussions about the study. Notification letters on ED letterhead (see Appendix A) and a non-technical brochure (see Appendix B), both of which will be sent via Federal Express to grantees and districts prior to our telephoning them, will capture their attention and help increase the response rate. Recruitment discussions and meetings, whether by telephone or in person, will be scheduled at times that are convenient to the respondents. Finally, we will be courteous but persistent in following up with officials who do not respond in a timely manner to our attempts to reach them.

4. Tests of Procedures or Methods to be Undertaken

We do not plan to pretest any of the procedures associated with recruitment. Procedures and materials outlined above have worked well on several prior studies.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following individuals were consulted on the statistical aspects of the study:

Name	Title	Telephone Number
Philip Gleason	Senior Fellow, Mathematica	(315) 781-8495
Melissa Clark	Senior Researcher, Mathematica	(609) 750-3193
Dan Player	Researcher, Mathematica	(609) 945-3368
Allison McKie	Researcher, Mathematica	(202) 484-4681
Heinrich Hoch	Researcher, Mathematica	(202) 250-3557

The following individuals will be responsible for data collection and analysis for this study:

Name	Title	Telephone Number
Philip Gleason	Senior Fellow, Mathematica	(315) 781-8495
Timothy Silva	Senior Researcher, Mathematica	(202) 484-5267
Melissa Thomas	Senior Survey Researcher, Mathematica	(202) 484-3478
Mary Grider	Senior Systems Analyst, Mathematica	(202) 484-4820
Annette Luyegu	Survey Researcher, Mathematica	(202) 264-3463
Linda Mendenko	Survey Researcher, Mathematica	(609) 275-2329
Dan Player	Researcher, Mathematica	(609) 945-3368
Allison McKie	Researcher, Mathematica	(202) 484-4681
Christina Clark Tuttle	Researcher, Mathematica	(202) 554-7570

www.mathematica-mpr.com

Improving public well-being by conducting high-quality, objective research and surveys

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC

Mathematica® is a registered trademark of Mathematica Policy Research

1 In order to make the test scores comparable, we intend to normalize test scores to have a mean of zero and a standard deviation of one. Therefore, the average value of for all teachers will be equal to zero.