District Visit Protocol

APPENDIX C
DISTRICT VISIT PROTOCOL

Study of Secondary Math Teachers from
Alternative Routes to Certification
Visit to _______________________ [District]
Talking Points

Introductions and overview
Introduce self
Distribute business cards
Get attendees’ names, affiliations
Ask them to please fill in the Sign In Sheet, so we have record of participants
Thank everyone for attending (and whoever set it up, for doing so)
Distribute, discuss agenda
Please feel free to ask questions at any time.
Describe purpose of meeting, give study overview, mention research team
Here today to describe an important new research study sponsored and fully paid for by the U.S.
Department of Education and, we hope, help your district move toward a positive decision about
supporting the study.
Study concerns the effectiveness of secondary math teachers from two programs: Teach For
America and the set of Teaching Fellows programs (or similarly named programs) established by
a group called The New Teacher Project.
Main research question is what impact secondary math teachers from these programs have on
student math achievement. The study gets at this issue by comparing them to other teachers who
teach the same subjects to similar students. (More on this in a minute.)
In addition, the study will explore how various differences in teachers’ backgrounds (e.g.,
training, other experiences; demographics; math content knowledge) are related to any
differences in effectiveness.
The reason we are appealing to your district now is its past/current/future use of secondary math
teachers from Teach For America and/or the local Teaching Fellows program.
Organization selected to lead the study for ED is Mathematica Policy Research, with Chesapeake
Research Associates and Branch Associates as partners. All three have extensive experience
conducting studies for ED, helping produce answers to important education questions of the day
for national, state, and local officials.
If known, refer to the team’s past involvement with this particular district
Briefly review history of contact with district so far on this new study—who we’ve spoken to,
when, what steps (if any) have been taken so far
District visit talking points

1

Provide some background/context for why ED wants this study done
Let me answer a few key questions you might have, to put this proposed study in context.
First, why do a study of any teachers from these programs?
In recent years, the number and proportion of teachers entering the profession from alternative
routes to certification have increased dramatically. And while some people argue one way or
another about whether that’s a good thing, there’s little solid information on the relative
effectiveness of alternatively certified teachers.
TFA and Teaching Fellows programs are worth studying because they are distinct, interesting,
and have a large number of participants. Although they differ in some ways—e.g., (a) TFA
mainly recruits brand new college graduates, while Teaching Fellows programs focus on careerchangers; (b) TFA directs participants into existing alternative certification programs, while
Teaching Fellows programs are set-up within and in partnership with districts—they share one
important distinction: they target some of the best and brightest people out there; they are highly
selective.
TFA goes after high-achieving graduates from some of the country’s top
colleges/universities; TNTP goes after high-achieving people who have been working in a
wide variety of fields.
Both put their applicants through a fairly rigorous screening process—for example,
involving challenging interviews or sample teaching lessons. And this distinguishes them
from the large majority of alternative certification programs in existence today, which rarely
require an undergraduate GPA over 2.5 and rarely involve more than a simple written
application for admission.
Also, they are by far the biggest of the selective programs out there, with thousands of teachers
placed in numerous districts around the country. So, ED realized that if they wanted to study
teachers who take highly selective routes to alternative certification, they’d have to study TFA
and Teaching Fellows teachers.
Finally, worth noting that while it was Department of Education’s idea to sponsor this study,
both TFA and TNTP are very interested in research on their own program participants and both
strongly support this study.
Second, why focus on math teachers at the secondary level?
There are shortages of such teachers nationwide, so schools/districts may want to know where
they can find qualified candidates, what programs to look to. Education Department wants to
know whether certain types of programs or routes to certification are worth fostering or
promoting.
Math is a vital skill for national economic competitiveness, and secondary math scores of
American students lag behind those of peers in many other industrialized countries. (Much more
so than at elementary level.)
District visit talking points

2

Most studies of alternatively certified teachers have focused on the elementary level—including
Mathematica’s study of TFA teachers and another study of less selective alternative certification
programs. A study of NYC Teaching Fellows included middle school, but no good studies have
included high school teachers.
Any questions before I go into study details?

Describe key study details
Now I’d like to describe the study in a bit more detail, then discuss whether/how we might
include schools from your district in this study during the 2009-2010 school year.
At its core, study compares student test scores for matching teachers in a given school, with one
teacher from Teach For America / the ______________[TNTP] program and the other from a
different route to certification, but where they teach the same math course to similar students,
typically (but not necessarily) during same period.
For example, the two teachers might both teach basic 6th grade math during 3rd period, or
both might teach algebra during 5th period. (Same two teachers might even have 2-3 courses
in common, in which they could form 2-3 matches. Also one teacher might form a match
with two or more different teachers.)
(If we can find enough teacher matches like this, teaching enough courses, across the
country, then we can produce definitive findings on the effectiveness of teachers from these
different routes to certification. Goal is roughly 450 matches nationwide, about half in
middle schools, half in high schools.)
Here’s what participation in the study would involve for your district and schools. Describe the 7
major study activities chronologically:
Study Activity #1. Now through next spring we need to identify secondary schools that
expect to have one or more eligible teacher matches in place next school year, and are
willing to help us.
Here’s the process we [are using / propose to use] to do that:
First, send to principal a notification letter from Dept. of Ed. / study description, and
Teach For America / the ______________[TNTP] program support letter—similar
materials as sent to district.
Second, call to briefly explain the study and tentatively explore suitability—e.g., by
asking a few key questions about teacher backgrounds and potential teaching assignments
next year.
Third, if merited, schedule in-person meeting to go over details that are too complex to
cover on phone, seek voluntary agreement to cooperate.
District visit talking points

3

During the visit we would talk about which teachers would likely teach which
courses during which period of the day, and how students are assigned to courses
or large groups.
We would also ask the teachers who we and the principal think might make up a
match to answer about 3 questions on a 1-page background form. (Principals
don’t always know what kind of training program or route their teachers took
toward initial certification, or how many years they’ve been teaching.)
IF NOT ALREADY STARTED: Any questions/concerns about that? Any sense of how
schools will react to our contacting them? What’s the ideal timeframe for schools being able
to give us good information about which teachers will be teaching which courses/sections
next year?
Study Activity #2. Spring through beginning of fall 2009 we would work with each school
to ensure that students assigned to the matched teachers’ courses are similar to one
another – on average, allowing for a fair comparison. This is the key to the whole
study; if two 6th grade math teachers don’t have students with similar characteristics, the
results comparing their test scores will not be scientifically credible; skeptics would just
argue that student outcomes are a function of pre-existing student characteristics, and not the
teachers’ teaching skills.
No one would see a comparison between gifted students in an honors 8th grade math section
and students in a regular 8th grade math section as fair. Even if schools say their objective in
assigning students to specific classrooms or larger grade-level groups is to create balance,
they might not achieve it, because (1) they can only account for so many factors, and some
things that could affect student outcomes—like motivation—can’t necessarily be measured,
and (2) subjectivity could creep into it.
So for this study we have to use a process called “random assignment” to ensure that the
students assigned to matching teachers are comparable. This is similar to what is done to test
new drugs, where study volunteers are randomly assigned to either try the new drug or to
take a placebo, a treatment group and a control group. It’s like a lottery, and it’s the same
process we have used with great success in several other major education studies we have
conducted for the Department of Education, including the TFA study and the alternative
certification study.
You may wonder how this can work with secondary schools’ diverse and complex
scheduling processes, but we have a plan that will avoid our upsetting the process.
In cases where the teacher match teaches the same course at the same time, the school
will use its normal process to assign students to the designated classrooms. Then an
official sends the original course rosters to Mathematica, and we’ll use a computer
program to simply reshuffle or re-assign this group of students to the teachers in the
match, e.g., the TFA/Teaching Fellows teacher or to the non-TFA/Teaching Fellows
comparison teacher.

District visit talking points

4

In cases where students are assigned to one of a few similar groups or families within a
school, where the goal is balanced groups, and if the TFA/Teaching Fellow teacher
teaches a certain course to students in one family and an eligible comparison teacher
teaches the same course to students in a different family, we could use random
assignment to create the families.
In both cases, we can ensure that the study classes/families are balanced by up to two
factors such as gender, grade level (if appropriate), academic ability, or other student
characteristics. We can also allow a small percentage of exceptions to random
assignment, to accommodate special needs.
We would do this in two stages. First, in spring or summer 2009, whenever schools are at
the right stage in their scheduling process. Second, because we know that after course
schedules are initially set, student mobility can affect course enrollments—with some
students not returning to the school and other new ones showing up—just before and
during first 2 weeks of fall semester we would work with schools to deal with class
roster changes due to student mobility. It’s a straightforward process we call “rolling”
random assignment.
School staff would call a toll-free number, give us a new student’s name and any key
characteristics (e.g., sex, poverty), and the subject they need to take (e.g., algebra 1), and we
will use an instantaneous lottery process to determine which teacher the student should be
assigned to. We will ensure one class does not get substantially larger than another and all
remain balanced on other key factors used during initial random assignment.
We’re known from experience that if we can explain these procedures to principals and
work with schools’ scheduling staff, we can make this work.
Any questions/concerns?
Study Activity #3. Shortly after that, we would do what we call a “roster check.” This
involves our getting lists of the course enrollments for the study classrooms, to verify that
the students we assigned there are still enrolled, and there hasn’t been any crossover
between teachers who comprise a match. If students have left the school, we will want to
find out where they have transferred to; if at another school in this district, we will attempt
to collect outcome data from them. Roster checks are quick and easy for school staff.
Study Activity #4. Slightly later in fall 2009 semester we would seek parental consent to
administer a math test to participating high school students. Will say more about the test
in a minute, but our approach to parent notification would be as follows. We’ll send a letter
to each parent saying that their child’s class is part of a national study involving one math
test, and that if they do not want their child to take the test, all they need to do is return the
form or call a toll-free number and we won’t include the student in that aspect of the study.
This is known as “passive consent,” and it’s a process we have used in many other studies.
Any reaction to this proposal? Any firm district policies that might push us in another
direction?

District visit talking points

5

Study Activity #5. We need to collect some information from teachers participating in the
study. First, at some point in the school year, most likely in fall 2009 but possibly not
until spring 2010, we will very likely ask all participating teachers to take a test
designed to assess their own math content knowledge. Details of this study activity have
not yet been finalized by ED. It would likely take place outside of regular school hours (not
interfere with teaching responsibilities), and though they are free to say “no,” we will
compensate them financially for completing the assessment (amount to be determined).
Second, in spring 2010 we will ask teachers to complete a 30-minute online survey on
their background and training, and their teaching and training experiences. We’ll pay
them $30 as a thank-you for filling it out.
Study Activity #6. In spring 2010 we will administer a subject-specific math test to just
the high school students participating in study. There will be four different multiplechoice tests, one each for general math, algebra 1, geometry, and algebra 2. The tests have
been developed by the Northwest Evaluation Association, and used in many districts.
They are computer-based adaptive tests. The “adaptive” part means that an algorithm
considers how well a student is doing on the test and adjusts subsequent questions
accordingly. For example, if a student is doing poorly on an algebra 1 test, it will adapt to
start asking more general math questions. This adaptivity can make low-achieving students
feel better (because they don’t keep struggling with tough questions), and also can keep all
students more engaged. Because we know schools may not have the computer resources to
accommodate this testing, we will bring in and set up enough laptop computers for all
students to have one, loaded with the test software. We will also bring trained staff to
proctor the class and trained technicians to deal with any hardware or software issues.
Because we understand the sensitivity of additional testing versus regular instruction, the
goal is for students to be able to complete the test in just one regular class period. And it
would be scheduled in consultation with school staff, to avoid conflicting with regular
teaching or testing activities.
Any questions about this?
Study Activity #7. Now, what about an outcome measure for middle school students? In
summer/fall 2010 we will ask schools/districts to provide spring 2010 standardized
math test scores for middle school students participating in the study. Basic math
achievement tests administered to meet NCLB requirements will be a good measure for the
large majority of middle school students, and avoids our having to increase the testing
burden on students.
At the same time, we will also request prior year achievement test data for all
participating students (as a retrospective baseline measure). For most students in grades
6-9 this may be their test score from spring 2009. For students in upper high school grades,
we may have to go back more than one year to get a prior score, if testing is not done
annually. Although we trust random assignment to produce the best possible comparison
groups, including on students’ average prior achievement levels, we can use data on actual

District visit talking points

6

prior achievement levels in our statistical analyses to improve the accuracy of our impact
estimates.
Finally, our data request will include basic background / demographic data found on
student records.
That’s the final study activity that would involve school or district staff, or students. The primary
activities that we are asking school staff to do are (1) provide the rosters we need in order to do
initial random assignment, and deal with the rosters we send back after re-shuffling them, (2)
deal with us to implement rolling random assignment, (3) help with roster checks to verify
student enrollment at a couple points, help with scheduling testing for high school students, and
help provide student records.
We hope this proposal eliminates anyone’s potential concern about burden associated with
the study.
I would also like to point out some features that may be part of other studies you’ve
considered, but are not part of our plan:
No new/additional testing for middle school students; we collect scores from the
standardized tests that districts already use
No teacher/classroom observations
No curriculum changes
No surveys or interviews of students or principals or district officials
No requirement that teachers teach particular courses
No new training or professional development for teachers
In summary, real key is finding schools with eligible matching teachers. Once that match is
identified and the classrooms are included in the study, we pretty much stay out of the way;
teachers are not asked to change their teaching approach in any way and we collect test scores
and aggregate up across schools and districts to compare their relative effectiveness.
Get initial assessment of interest or possible issues.
How does this study sound to you?
Level of interest?
What questions or concerns do you have? Are they major or minor?
What would you need to hear to make it seem (more) palatable?
Always seek explanation for reluctance / lack of interest. Seek to address them, or at least
express possible flexibility. Points for possible emphasis:
Study will produce new, much-needed info on relationship between teacher preparation and
background, and student math achievement
Without good research, ED will not know whether/how to support different training
programs, and district officials/principals won’t have solid basis for decisions, either
Teach for America / ____________[TNTP] program reps strongly support this study; they
really want to know how effective their teachers are
Study takes just one year of involvement with schools/district
Absolute minimal amount of additional testing

District visit talking points

7

If it turns out district is unalterably opposed to any new testing at high school level, we
could consider using only middle schools there
Minimal other data collection
No interference in classrooms/instruction
No cost to district
No distractions for principals
Our team’s experience in doing studies like these; we are sensitive to school issues

Questions we would like answered about your district
Math achievement testing and record keeping
We would want standardized math test scores from before the year of the study, to control for
prior achievement levels before students were assigned to one of the study classrooms/teachers.
What is the top grade level at which your district administers an annual standardized math test?
IF GRADE 8 OR HIGHER (HIGH SCHOOL STUDENTS) ARE TESTED: Since what year has
testing been done at that level?
IF LOWER THAN GRADE 9 (LOWER THAN FIRST YEAR OF HIGH SCHOOL) OR IF HIGH
SCHOOL TESTING IS SO NEW THAT PRIOR-YEAR SCORES WOULD NOT BE AVAILABLE
FOR FRESHMEN: When students move from middle school to high school, from 8th to 9th
grade, are their final 8th grade test scores automatically included on their transcript/permanent
record and forwarded to their new school? (If we wanted to know spring 2008 test score for a
student currently in 9th grade, or spring 2007 test score for a student currently in 10th grade,
would that score be available from the high school? Or would we have to go back to their middle
schools)
What is the timeframe for spring testing? (Approximately which week of which month does it
take place?)

Propose possible next steps, for outcome of this meeting.
OK, as for next steps, we are not necessarily seeking final decision right now, on the spot. We
know districts have different processes for deciding on research requests. Eventually, if the
district decides to support this study, we would want to have both the district and a study team
representative sign a “workplan”—essentially, a memorandum of understanding—that lays out
their respective responsibilities for the various study activities. And we’d also want each
participating school to sign something acknowledging their role in the study.
For now, there are some options for how we could move forward.

District visit talking points

8

IF NOT ALREADY STARTED CALLING SCHOOLS:
ONE OPTION is—we could start calling schools soon, to explore which of them might be
eligible for consideration.
Although we know from Teach For America / the ______________[TNTP] program
which of your secondary schools they’ve placed math teachers into, there’s no way for
us to know whether those schools might have eligible teacher matches unless we talk
directly to school officials about potential course schedules and verify teachers’
backgrounds. (Some of the Teach For America / the ______________[TNTP] program
teachers may not even be there any more.)
This would be most efficient for you, because we would not take up any more of district
officials’ time until we gained a sense of how well district schools might work for the
study. It’s also easier on schools to start thinking about the study sooner, rather than
deal with it in a condensed time period toward the end of the year, when things are
busy.
We could get a simple e-mail or a verbal “OK” from the district, just for contacting
schools, in case schools question our legitimacy or approval to be contacting them.
We would not say district has approved the whole study, but would say it has approved
calls to explore school suitability. We would later come back to you and tell you what
we learned and—if schools look promising—then start any formal research approval
process that is necessary.
Would you have any concerns/objections to this approach?
If they approve this approach, go to Closing section
SECOND OPTION – if you prefer that we submit a formal research application before
district approves exploratory calls to schools, we can certainly do that.
Some districts we’ve worked with on similar evaluations have used their discretion to
waive a formal research application because (1) we’re working on behalf of the U.S.
Department of Education (it’s not like a local education graduate student project), and
(2) all aspects of the study are being reviewed by both the U.S. Office of Management
and Budget (OMB) and an independent Institutional Review Board (IRB). (And we’re
happy to provide documentation of each of their approvals, once they’re available.)
But we are certainly willing to submit an application before going further.
If that’s the preference: inquire about the process (required documents/ presentations/
timeline/ etc.), then go to Closing section.
THIRD OPTION – if research application is not required, what steps/process would be
involved in the district deciding whether or not to support?

District visit talking points

9

IF ALREADY STARTED CALLING SCHOOLS:
Summarize progress, what we’ve learned so far, etc.
Make any necessary requests for district-level information/assistance.
If appropriate, raise issue of formal research request process, as under Second Option, above.
Definitely ask about steps/process for district reaching decision on study and signing workplan,
as in Third Option, above.

Closing
Thank you all very much for your time and consideration.
Whichever is applicable:
We’ll wait and hear back from you with a decision.
We’ll let you know what we learn from calls to schools and get back to you.
We’ll get started on the research application right away.
We look forward to continuing the discussion of your district’s possible participation.
If you have any more questions, you know how to reach me.

District visit talking points

10