Data Collection Part B_FINAL

A Study of Teacher Residency Programs

Part B

April 28 2011

Contract Number: ED-IES-10-C (0001) Mathematica Reference Number: 06748-510 Submitted to: Institute of Education Sciences U.S. Department of Education 555 New Jersey Avenue, NW Washington, DC 20208 Project Officer: Melanie Ali Submitted by: Mathematica Policy Research 600 Maryland Avenue, SW Suite 550 Washington, DC 20024-2512 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 Project Director: Philip Gleason

An Study of Teacher Residency Programs Part B April 28, 2011

CONTENTS

SUPPORTING STATEMENT for paperwork REDUCTION ACT 1

PART B: COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS 1

1. Respondent Universe and Sampling Methods 1

2. Procedures for the Collection of Information 2

3. Methods to Maximize Response Rates and Deal with Nonresponse 6

4. Tests of Procedures or Methods to be Undertaken 7

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 7

REFERENCES 9

appendix a: trp, DISTRICT, SCHOOL, and teacher letters 10

appendix B: district data request documents 12

appendix c: teacher employment data form 14

appendix d: trp survey 16

appendix e: mentor teacher survey 18

appendix f: resident teacher survey 20

appendix g: teacher of record survey 22

appendix H: trp director interview protocol 24

appendix I: teacher mobility survey 26

appendix J: non-technical brochure 28

appendix k: confidentiality pledge 30

appendix L: pretest memo 32

TABLES

1 Overview of TRP Involvement in the Study 2

2 Minimum Detectable Effects on Student Test Scores and Teacher
Retention 6

SUPPORTING STATEMENT for paperwork REDUCTION ACT

This OMB package requests clearance for data collection activities to support a study of teacher residency programs (TRPs). This study will provide important implementation information on TRPs funded by the U.S. Department of Education (ED), as well as information on the achievement outcomes of the students whose teachers participate in TRPs. The study will focus primarily on TRPs that received Teacher Quality Partnership (TQP) grants from ED in late 2009 and early 2010. ED’s Institute of Education Sciences (IES) has contracted with Mathematica Policy Research and its partner, Decision Information Resources (DIR), to conduct the Study.

The main objective of the study is to describe the characteristics of TRPs and their participants. We will also summarize the academic outcomes of students taught by novice TRP teachers and examine the retention rate of novice TRP teachers. This is the second of two requests for the study, the first of which requested approval for recruitment of TRPs, school districts and schools. We are submitting the package in two stages because the study schedule required that participant recruitment begin before all the data collection instruments are developed and tested. In this package, we are requesting approval for data collection activities that will support the full-scale study.

PART B: COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

The respondent universe for the outcomes and retention studies will consist of novice TRP teachers and non-TRP teachers. Different sets of TRPs will be needed for the four major analytical components of the study, as shown in Table 1. For example, all TQP grantees operating TRPs will be surveyed about basic program characteristics, but only a subset of about 15 TRP directors will be interviewed for more program details and asked to provide data on applicants.

At the conclusion of each school year, we will request student-level data for all teachers in the district who are teaching math or reading in grades 3 through 8. All of the students and all of the teachers in the targeted districts will be included in the outcomes analysis.

Table 1. Overview of TRP Involvement in the Study

	Number	Student Achievement Outcomes Study	Teacher Retention	Descriptive Analysis of TRPs	Descriptive Analysis of TRPs and Participants
All TRPs That Received TQP Grants in 2009-2010	28			√
Subset of Above Group For In-Depth Study	15a			√	√
Experiencedb Grantees Specifically Targeted For Outcomes Study	6	√	√	√	√
Other Experienced TRPs (Non-Grantees) Specifically Targeted For Outcomes Study	2	√	√	√	√

^a Estimate – at this time not enough is known about TRPs—especially the number and type of teachers they will have placed in residency in fall 2011, and how those placements will be distributed across partner districts and schools—to cite specific numbers with certainty. These determinations will be made during the selection and recruitment process.

^b Experienced grantees are those that began operations in 2009 or earlier.

2. Procedures for the Collection of Information

Statistical methods for sample selection. This study will not statistically sample TRPs, districts, schools, or teachers. Instead, it will rely on a purposefully selected convenience sample of TRPs that have been in existence since at least 2009 and are best suited for the outcomes study (a determination based primarily on the availability of student-teacher linked data and the number and type of teachers they prepare). Within the study districts, we will collect data on all students who were in a tested grade and subject. The study does not aim to make statements that generalize beyond the TRPs and teachers under study.

Data collection plan. Our data collection plans will involve different sample members in different ways. We will collect some basic information from the universe of TRPs that received TQP grants and from any other TRPs considered best qualified for the outcomes study; additional information from a subset of the grantees and from any other TRPs considered best qualified for the in-depth implementation study; information from all teachers selected for the outcomes analysis; and student and teacher information from all districts in the outcomes study. We will also collect data on resident teachers and their mentors. Following is an overview of the data collection plans.

Student records data. Following the 2011-2012 and 2012-2013 school years, we will ask districts to provide data for all students in the district who are in tested grades and subjects. We will collect both demographic data (that is, age, race/ethnicity, English language proficiency, disability status, and eligibility for school lunch program) and data on students’ performance on state or district tests in the current year and two previous school years. We will send the district a letter that will specify the data items requested and a non-technical brochure providing additional study information (Appendix J).

Federal rules permit the U.S. Department of Education and its designated agents to collect student demographic and existing achievement data from schools and districts without prior parental or student consent (The Family Educational and Rights and Privacy Act (FERPA) (20 U.S.C. 1232g; 34 CFR Part 99). To maximize response rates and minimize burden on schools and parents, we will follow these federal rules. We do not anticipate district opposition to this plan. If a district interprets federal rules in a way that does not permit data collection without consent, that district will be excluded from the study.

TRP survey. We will mail a self-administered 35-minute hardcopy survey to a director at each of the 28 TRPs that received a TQP grant in fall 2009 or spring 2010 and to any non-grantees included in the outcomes study. This survey, to be administered in spring 2011, will address TRP characteristics, admission requirements, and key program features, providing the foundation for answering the first research question. The mailing will contain a grantee or non-grantee cover letter (Appendix A), questionnaire (Appendix D) and a non-technical brochure providing additional study information (Appendix J). The letter, which will be on ED’s stationery, will describe the study and its objectives and the need for TRP participation, address confidentiality, and provide a telephone number and email address for questions or concerns.

TRP director interview. In spring 2011, we will conduct semi-structured interviews, by telephone or in person, with the directors of the 15 TRPs identified for the in-depth implementation study. The 45-minute interview will collect detailed information on the amount of instruction in different substantive and pedagogical subject areas provided to candidates at various points in the program (before, during and after the residency year; during and after the first year as a teacher of record). The open-ended questions in the TRP interviews will allow us to collect more in-depth information than that collected from the survey, and to probe for clarification if necessary (Appendix H). We will contact potential respondents in advance and provide them with a list of topics to be covered and any general information about the study as needed. (They will be familiar with the study from our previous contacts with them during the recruitment stage.)

Resident teacher survey. A 25-minute mail survey of 300 TRP participants who serve their residency during the 2010-2011 school year will be conducted in spring 2011. It will collect descriptive information on resident teachers’ backgrounds as well as experiences during their residency year—for example, interactions with resident mentors, classroom responsibilities, and views on the program. This survey will be administered to all residents from the same set of TRPs included in the program director interviews. The mailing will contain a cover letter (Appendix A), questionnaire (Appendix F) and a non-technical brochure providing additional study information (Appendix J). In the cover letter, we will describe the study and its objectives and the need for resident teacher participation, address confidentiality, and provide a toll-free telephone number and email address for questions or concerns.

Mentor teacher survey. In spring 2011, we will mail a 20-minute self-administered hardcopy survey to the 300 mentors associated with each of the teachers targeted for the resident teacher survey. Respondents will be asked to provide descriptive information on their backgrounds, qualifications, and training for the mentor role, their residents’ responsibilities, and their interactions with residents. Parallel questions across the resident and mentor surveys will allow for corroboration during analysis. The mailing will contain a mentor teacher cover letter (Appendix A), questionnaire (Appendix E) and a non-technical brochure providing additional study information (Appendix J). In the cover letter, we will describe the study and its objectives and the need for mentor teacher participation, address confidentiality, and provide a toll-free telephone number and email address for questions or concerns.

Teacher of record survey. In fall 2011, we will contact districts to request a list of teachers hired within the last two years, their current school assignment, and date of hire (Appendix A). The lists will provide the sample for the teacher of record survey. In spring 2012, all novice teachers in the 8 districts in the outcomes study will be asked to complete a 25-minute self-administered hardcopy mail survey on their background characteristics, experiences during the 2011-2012 school year, and views on teaching. Teachers of record who will have completed the resident survey the previous year will complete a shorter version of the teacher of record survey—one that excludes questions about background characteristics. The mailing will contain a teacher of record cover letter (Appendix A) questionnaire (Appendix G) and a non-technical brochure providing additional study information (Appendix J). In the cover letter, we will describe the study and its objectives and the need for participation from teachers of record, address confidentiality, and provide a toll-free telephone number and email address for questions or concerns. We expect to survey a total of 800 teachers.

Teacher employment data. In fall 2012 and fall 2013, districts will be asked to verify whether the novice teachers in the outcomes study are still employed by the district. We will send each district a data request form that will specify the data requested (Appendix C), the list of novice teachers, and a non-technical brochure providing additional study information (Appendix J).

Teacher mobility surveys. In fall 2012 and fall 2013, all teachers in the outcomes study will be asked to complete a 15 minute self-administered hardcopy mail survey on their current employment status and reasons for job changes, if applicable (Appendix I). The mailing will contain a cover letter (Appendix A) that will describe the purpose of the survey and the need for participation from all teachers regardless of current employment status, address confidentiality, and provide a toll-free number and email address for questions or concerns. We expect to survey 800 teachers in each wave of the survey.

Confidentiality. Mathematica’s internal confidentiality pledge, which will guide all staff who work on this study, is presented in Appendix K.

Estimation procedures. The study will not seek to identify the causal effect of TRP teachers. Instead, we will describe the average growth in achievement in the students of novice TRP teachers benchmarked against the average growth of students of all other non-TRP teachers in the district and against other novice non-TRP teachers. Rather than using the simple change in test scores, we will attempt to get the most precise measure of growth possible using a value-added model:

(1)

where Y_ijk is the test score of student i in a class taught by teacher j in year t, Y_i(-t) is a vector of the previous two years of test scores for student i, X_ijk is a vector of student baseline characteristics, the T_i’s are indicator variables for each teacher j, µ_j is a classroom-specific random error term, ε_ij is a student-level random error term, and β, , and γ represent parameters to be estimated. The model will be estimated with ordinary least squares (OLS), using standard errors that account for classroom-level clustering.

The estimates of γ represent the change in student achievement that each teacher produces in excess to what would have been expected based on the characteristics and prior achievement level of their students. We will take an average of all of the s for TRP teachers and present the benchmark of the average of the ’s for all teachers¹ and for other novice teachers.

TRP and non-TRP teacher retention rates. We will also summarize the retention rates of TRP and non-TRP teachers in the district. Measures of teacher retention may include whether the teacher remained at the same school, moved to another school in the district, moved to another district, or left the teaching profession.

The timeline for retention data collection is as follows. In fall 2012 and fall 2013, we will contact districts to request data on employment status among all novice teachers. In fall 2012 we will determine which of the novice teachers from the previous year are still teaching. In fall 2013 we will again determine which of the teachers in our analysis sample are still teaching in the district.

Mediators analysis. In addition to our main outcomes analysis, a complementary approach will help shed light on the outcomes we observe. We will estimate a version of equation (1) that includes controls for additional covariates, or mediators, representing possible mechanisms through which TRPs may influence student achievement. In one case, for example, we will control for teacher background characteristics to determine whether they are correlated with outcomes. We will use a similar approach to estimate the relationship between outcomes and specific aspects of the training received in TRP programs. For example, we will examine relationship between coursework and training while teaching (reflected by relevant measures from the teacher survey) and student outcomes.

Degree of accuracy needed. The study will not attempt to measure impacts, so statistical power is not a concern for the student outcomes analysis and the teacher retention analysis. However, the study will seek to provide measures of the average outcomes that are as generalizable as possible. To this end, the study will include all TRP and non-TRP teachers who are teaching in the eight districts that participate in the outcomes study.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Mathematica has developed and refined a wide range of methods to minimize attrition from survey samples and to maximize response. These methods focus on containing burden on respondents and techniques for avoiding refusals. Mathematica has found that the following techniques are major contributions to a high completion rate: establishing positive relationships with respondents and school and program staff; sending advance letters; and establishing efficient and flexible scheduling. We will include a statement on confidentiality and data collection requirements (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183) in all letters (Appendix A), data collection instruments (Appendix C through Appendix I) and study brochure (Appendix J).

We anticipate 100 percent response from TRP interviews and an 80 percent response from the surveys since grantees have committed to cooperating with the national study and the small community of other TRPs is likely interested in contributing to the proposed study. School districts are partners in sponsoring and supporting TRPs, and we will have established a rapport with them during the recruitment phase of the study. We therefore expect that they will be highly responsive to our requests for most student and teacher data. To further solidify administrators’ cooperation, we will adhere to additional data collection requirements that districts may have such as preparing research applications and seeking IRB approvals. In our correspondence, we will also send notification letters on ED letterhead (Appendix A) to TRPs and districts to capture their attention and to help increase the response rate.

The high response rates on the teacher surveys (85 percent to 90 percent) are based on Mathematica’s experience in conducting other surveys with teachers. To ensure response, follow-up will be initiated through email and telephone calls to teachers who do not respond within two to three weeks of the initial mailing. Second, nonrespondents will be given the option of providing data during the telephone follow-up. Data collectors can read the questions aloud and enter the responses on the hard copy instrument. Third, experienced interviewers will be recruited and extensively trained. These interviewers will be thoroughly trained on data collection procedures, including methods for promoting cooperation among school staff. Interviewers especially skilled at encouraging cooperation will be available to persuade reluctant teachers to participate (except for hostile refusals).

Reducing districts’ burden in providing study data to submit for the study will facilitate attaining a response rate of at least 85 percent on student administrative data and teacher employment data.

Finally, we will be courteous but persistent in following up with participants who do not respond in a timely manner to our attempts to reach them.

4. Tests of Procedures or Methods to be Undertaken

Data collection instruments for the study drew heavily on surveys and protocols that have been used successfully on previous studies. The pretests assessed the content and wording of individual questions, the organization and format of the questionnaire, respondent burden time, and potential sources of response error. Most of the survey questions have had prior OMB approval and have been administered to many respondents already. We piloted instruments that were new, that were adaptations and extensions of existing ones, that had limited information on reliability and validity for the population in this study, and for which we wished to examine how measures perform when combined with others. Each of the pilot tests were conducted with no more than nine respondents, after which a member of the study team debriefed with each respondent over the telephone. Results of the pilot testing are included in the memo in Appendix L

.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following individuals were consulted on the statistical aspects of the study:

Name	Title	Telephone Number
Philip Gleason	Senior Fellow, Mathematica	(315) 781-8495
Melissa Clark	Senior Researcher, Mathematica	(609) 750-3193
Dan Player	Senior Researcher, Mathematica	(609) 945-3368
Allison McKie Seifullah	Senior Researcher, Mathematica	(202) 484-4681
Heinrich Hoch	Researcher, Mathematica	(202) 250-3557

The following individuals will be responsible for data collection and analysis for this study:

Name	Title	Telephone Number
Philip Gleason	Senior Fellow, Mathematica	(315) 781-8495
Timothy Silva	Senior Researcher, Mathematica	(202) 484-5267
Melissa Thomas	Senior Survey Researcher, Mathematica	(202) 484-3478
Mary Grider	Senior Systems Analyst, Mathematica	(202) 484-4820
Annette Luyegu	Survey Researcher, Mathematica	(202) 264-3463
Linda Mendenko	Survey Researcher, Mathematica	(609) 275-2329
Dan Player	Senior Researcher, Mathematica	(609) 945-3368
Allison McKie	Senior Researcher, Mathematica	(202) 484-4681
Christina Clark Tuttle	Researcher, Mathematica	(202) 554-7570

REFERENCES

Agodini, Roberto, and Dynarski, Mark. “Are Experiments the Only Option?” The Review of Economics and Statistics, vol. 86, no. 1, February 2004, pp. 180–194.

Angrist, J., G. Imbens, and D. Rubin. “Identification of Causal Effects Using Instrumental Variables.” Journal of the American Statistical Association, vol. 91, no. 434, 1996, pp. 444–445.

Campbell, M.J., S.A. Julious, and D.G. Altman. “Estimating Sample Sizes for Binary, Ordered Categorical, and Continuous Outcomes in Two Group Comparisons.” BMJ, vol. 311, no. 7013, 1995, pp. 1145–1148.

Hanushek, Eric, John Kain, and Steven Rivkin. “The Market for Teacher Quality.” NBER Working Paper 11154. Cambridge, MA: National Bureau of Economic Research, 2005.

Jackson, Russell, Ann McCoy, Carol Pistorino, Anna Wilkinson, John Burghardt, Melissa Clark, Christine Ross, Peter Schochet, and Paul Swank. “National Evaluation of Early Reading First: Final Report.” Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, May 2007.

Kane, T., J. Rockoff, and D. Staiger. “What Does Certification Tell Us About Teacher Effectiveness? Evidence from New York City.” Economics of Education Review, vol. 27, no. 6, 2008, 615–631.

Marvel, John, Deanna M. Lyter, Pia Peltola, Gregory A. Strizek, and Beth A. Morton. “Teacher Attrition and Mobility: Results from the 2004–05 Teacher Follow-up Survey, First Look.” 2007. Available at [http://nces.ed.gov/pubs2007/2007307.pdf]. Accessed November 10, 2009.

Peikes, Deborah N., Lorenzo Moreno, and Sean Michael Orzol. “Propensity Score Matching: A Note of Caution for Evaluators of Social Programs.” The American Statistician, vol. 62, no. 3, August 2008, pp. 222–231.

Rivkin, S., E. Hanushek, and J. Kain. “Teachers, Schools, and Academic Achievement.” Econometrica, vol. 73, no. 2, 2005, pp. 417–458.

Rockoff, J. “The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data.” American Economic Review (AEA Papers and Proceedings), vol. 94, no. 2, 2004, pp. 247–252.

Schochet, Peter Z. “Statistical Power for Random Assignment Evaluations of Education Programs.” Journal of Educational and Behavioral Statistics, vol. 33, no. 1, 2008, pp. 62–87.

Smith, Jeffrey A., and Petra E. Todd. “Reconciling Conflicting Evidence on the Performance of Propensity-Score Matching Methods.” American Economic Review, vol. 91, no. 2, May 2001, pp.112–118.

www.mathematica-mpr.com

Improving public well-being by conducting high-quality, objective research and surveys

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC

Mathematica® is a registered trademark of Mathematica Policy Research

1 In order to make the test scores comparable, we intend to normalize test scores to have a mean of zero and a standard deviation of one. Therefore, the average value of for all teachers will be equal to zero.