Pre-test Memo-FINAL

APPENDIX L
PRETEST MEMO

MEMORANDUM

P.O. Box 2393
Princeton, NJ 08543-2393
Telephone (609) 799-3535
Fax (609) 799-0005
www.mathematica-mpr.com

DATE:
TO:

Melanie Ali

FROM:

Annette Luyegu, Linda Mendenko, Melissa Thomas

SUBJECT:

Teacher Residency Program (TRP) Pretest Findings

1/4/2011

This memo describes the procedures and results of pretesting activities conducted by
Mathematica staff for the Teacher Residency Program (TRP) study from October through
December 2010. Pretesting was conducted on four of the six data collection instruments: the
Resident Teacher Survey, the Mentor Survey, the Teacher of Record Survey and the Program
Survey.
The purpose of pretesting the survey instruments was to ensure that they are functioning as
planned. The pretests addressed such issues as questionnaire length, readability, skip pattern
logic, sentence structure, instructions, and subject matter.
RECRUITING THE PRETEST SAMPLE
The pretest sample for the teacher surveys was recruited from a New Jersey public school
through a Mathematica employee’s relationship with the school staff. The middle school serves
grades six through eight, and the pretest sample comprised of seven teachers who teach art,
health and physical education, special education, and math. Three teachers with two to three
years of teaching experience were recruited for the Teacher of Record Survey, and four mentors
of novice teachers agreed to participate in the Mentor Survey pretest.
Emails explaining the study and requesting participation in the pretest were sent to six
universities that received TRP grants. A study brochure was included in the email. Follow-up
telephone calls were made to each university to obtain contact information for the people most
suited to respond to the surveys. Three universities agreed to participate in the pretest and names
and contact information for mentors, program directors, and novice teachers were requested or
confirmed. Two mentors, two program directors, and five resident teachers agreed to pretest the
appropriate instruments.
PRETESTING PROCESS
Participants recruited through the public school were administered pretests at the school at
the end of the school day. Two members of the Mathematica research team met with all seven
participants in a classroom, providing the participants with an overview of the project, the
purpose of the pretest, instructions for completing the instruments, and the debriefing session..
Participants were then divided into two groups, one for the Mentor Survey and one for the
An Affirmative Action/Equal Opportunity Employer

MEMO TO: Melanie Ali
FROM:
Annette Luyegu, Linda Mendenko, Melissa Thomas
DATE:
1/4/2011
PAGE:
3
Teacher of Record Survey. In each group, a survey packet was given to each participant, with
instructions to read through the packet materials (cover letter, study brochure, and questionnaire)
and to answer the questionnaire items. The pretest administrators stayed in the room for the
duration of the pretests.
Once all participants completed the survey, a group discussion was held for each survey
type. Mathematica staff used a debriefing protocol (see below) to guide the discussion about
overall understanding and formatting of the survey, ease of completing, and specific questions or
responses.
Survey materials were sent to participants at the universities by FedEx. Each packet
included the cover letter, a copy of the study brochure, the hard copy questionnaire, and a FedEx
envelope and air bill to send the questionnaire back to Mathematica. Within a week of the
mailing, Mathematica staff conducted the debriefing with each participant over the telephone
using the debriefing protocol below to guide the discussion.
Debriefing Protocol
• Did the accompanying letter clearly convey the reason for the survey? If no, why?
• Was it difficult or time-consuming to recall any of the data you needed to complete
the survey? For which items was data most difficult to recall?
• What did you think of the general format of the survey? Was it clear on how to
respond to the questions?
• Were any questions confusing or difficult to understand? If yes, which one(s)?
• Were there any questions that seemed redundant? If yes, which one(s)?
• Were there questions for which the response options seemed inadequate? Please
specify (include item number(s) and the response categories you would recommend).
• Were there any additional topics you think should have been included in this survey?
Please describe.

MEMO TO: Melanie Ali
FROM:
Annette Luyegu, Linda Mendenko, Melissa Thomas
DATE:
1/4/2011
PAGE:
4
PRETEST RESULTS
Overall, the completion times for the five surveys were on target, with the exception of the
Teacher of Record Survey, which took the less time than what was initially estimated (Table 1).
The estimated completion time for the Teacher of Record was not adjusted to reflect the pretest
findings because (1) questions have been added to the final version, and (2) our experience with
questionnaires of similar length indicates that the typical respondent would take at least 25
minutes.
TABLE 1. Completion Time for Each Survey

No. of Pretests

Estimated
Completion Time

Average Completion
Time

Resident Teacher
Survey

3

25

26

Mentor Survey

6

20

18

Teacher of Record
Survey

3

25

9

Program Survey

1

30

30

Instrument

In general, participants had no problems with the survey format and clearly understood the
instructions for answering the questions. Participants reported that the overall question flow was
easy to follow, the order of topics was logical, and no questions seemed redundant. Several
participants commented that the study brochure was a helpful resource to enhance their
understanding of the project. Specific comments on each of the instruments are detailed below.
Resident Teacher Survey
•

Sections C, D, and E. Two teachers had difficulty answering questions in these sections
for the reasons below.
o Comment: Their school operates on quarters and not semesters.
o Response: Changed question wording from ‘first semester’ to ‘first half of your
residency year’ and from ‘second semester’ to ‘second half of your residency
year.’
o Comment: They were not as far along in the program as originally thought; they
were in the “observation mode,” and not actively participating in the classroom.
They were observing more than one teacher at the time of completing the survey.
o Response: The respondents are not the typical resident we expect to find in the
actual study, and we did not adjust the survey.

MEMO TO: Melanie Ali
FROM:
Annette Luyegu, Linda Mendenko, Melissa Thomas
DATE:
1/4/2011
PAGE:
5

Mentor Survey
•

A1: Please describe your ongoing or completed post secondary degrees is the chart below
(table listing degrees, names and locations of colleges attended, year of degree
completion, major field of study, and minor field of study).
o Comment: More room needed to write response for name and location of the
college. Associate’s degree not listed as an option.
o Response: Revised table to include associate’s degree as an option. Added a
second line for respondents to write college information.

•

A2: For how many school years have you been a teacher?
o Comment: Specify that respondents should count both private and public schools.
o Response: Added instruction that reads, ‘Include full-time teaching positions in
public and/or private schools.’

•

B6: Are there other teacher residency program (TRP) mentors in your school?
o Comment: Suggested wording change from ‘your school’ to ‘your school district’
to get a broader picture of interactions with other mentors.
o Response: Added question B7: Are there TRP mentors at other schools in your
school district?

•

B12: How much do you agree or disagree with the following statements?
o Comment: Specify reference to current or past experience.
o Response: Added instruction that reads, ‘Please think about your current or
most recent mentoring experience when responding to these statements.’

Teacher of Record Survey
•

A1: Table listing degrees, names and locations of colleges attended, year of degree
completion, major field of study, and minor field of study.
o Comment: All teachers indicated that the table was not very clear. It was difficult
to figure out “what to put where,” and it took them a while to figure out where to
write information on their bachelor’s degree. They would have preferred a
specific row for each type of degree. One teacher did not write his associate’s
degree, because it was not listed in the table. He could have used an ‘other’
category, but this did not occur to him.
Response: Revised table to include associate’s degree as an option. Added a
second line for respondents to write college information.
o Comment: One teacher said he majored in both health education and physical
education, but since the question asked for one major and one minor, he used
codes 182 and 181 in columns D and E, respectively.
o Response: No adjustments made; the teacher used the codes correctly.

MEMO TO: Melanie Ali
FROM:
Annette Luyegu, Linda Mendenko, Melissa Thomas
DATE:
1/4/2011
PAGE:
6
•

B1: Since graduating from college, how many years have you worked a full-time nonteaching job?
o Comment: Two respondents misread the question. Despite the underlined ‘nonteaching jobs,’ they gave the answer for all jobs. They suggested rewording to
‘worked a full-time job besides teaching.’
o Response: Implemented suggestion.

•

B3: Did you work as a …
o Comment: One teacher wrote the years and months without checking the ‘yes/no’
column; it appeared he did not read the question. Even though three teachers had
been student teachers, they did not consider it as work; rather, they perceived it to
be part of unpaid college curriculum.
o Response: Removed ‘student teacher’ from the answer choices.

•

B4: During this school year, do you or will you…
o Comment: All teachers suggested emphasizing the word ‘this.’
o Response: We underlined the word ‘this.’

•

D7: Please indicate if you teach any of the subjects listed below…
o Comment: One teacher only checked the one ‘yes’ box that applied to him, and
rather than check the ‘no’ boxes that did not apply to him, he left them blank. He
suggested better instructions.
o Response: We believe this was a rare instance of a respondent not reading the
question before answering. No changes made. In the unlikely event that this
happens with actual respondents, we will call the respondent for clarification.

•

F2: How satisfied are you with the following aspects of your job this school year, in
terms of the following?
o Comment: All respondents found the response scale confusing; previous
questions listed the satisfaction scale from negative to positive but F2 listed them
from positive to negative.
o Response: Reordered response scale to place ‘completely dissatisfied’ to the left
and ‘completely satisfied’ to the right.

Program Survey
Despite repeated requests, one program director did not return the questionnaire. The results
presented in this memo reflect answers from one program director.
•

C8: How is the typical second residency assignment similar to or different from the first
one?
o Comment: The director wrote, “Don’t have a second residency assignment.” The
four follow-up questions (C8a through C8d) therefore did not apply to her
program.

MEMO TO: Melanie Ali
FROM:
Annette Luyegu, Linda Mendenko, Melissa Thomas
DATE:
1/4/2011
PAGE:
7
o Response: Added a third answer option to C7 – residents work with one mentor
throughout the year. If respondents select this new option, they will be directed to
C9, skipping C8.
•

A5: Please complete the table below about the number of residency assignments for
participants who started their (first) residency during 2010.
o Comment: The respondent reported that it might be easier for the person doing the
survey if the question is separated into two parts, a) A5a and b) A5 b-k.
o Response: Left question as is. We do not believe this will be a problem.

•

A7: Please complete the table below regarding other types of teacher training programs at
the postsecondary institution that is part of your teacher residency program partnership.
o Comment: Respondent reported that the question was time consuming since she
did not have all the information. She had to contact the director of teacher
education services who keeps track of the data, and it took several days to get a
response.
o Response: Left question as is. The beginning of the questionnaire informs
respondents that they may need to consult multiple staff members at the program.

•

A8: Please provide the requested information on the number and background experiences
of the individuals who have taught courses or workshops leading to a teaching credential
via the teacher residency program during the 2010-2011 academic year. The table asks
for: (1) Total number of instructors, (2) Current affiliation – full-time faculty; adjunct or
part-time college faculty, but not school district employees; district level employee;
school-level employee, and (3) Prior teaching experience – have experience as an
elementary teacher; have experience as a secondary teacher.
o Comment: The respondent reported that it might be difficult to know the
experience of some of the school and district employees’ teaching experience.
o Response: We understand that this might be a challenge for some programs. For
such programs, we will use any estimates they are able to provide.

•

C11: Please complete the table below regarding first and second semester residency
assignments. The table asks for the following information for the typical first residency
semester (Column A) and typical second residency semester (Column B): (1) Length of
assignment; (2) Minimum number of full-length school days resident fully in charge of
classroom; (3) Number of days in a typical 5-day school week that the resident spends –
full time in the mentor’s classroom, part time elsewhere, and no time in the mentor’s
classroom and full time elsewhere.
o Comment: The respondent reported that it was difficult to answer the question
because the data can vary by resident.
o Response: We will seek clarification and get more details during the director
interview.

cc: Phil Gleason; Tim Silva