Responses to OMB comments

Memorandum

TO: Sandra Garcia, REL COR

FROM: Robert Marzano and Trudy Cherasaro, REL Central

SUBJECT: Response to OIRA and RMO comments on ED-2013-ICCD-0089-0004 A Study of Feedback in Teacher Evaluation Systems

DATE: 2/4/14

Comment 1: Our understanding is that teachers who will be sampled for the study will have received at least one round of performance evaluation.  Does your sampling plan factor in these performance ratings?  We think it would be interesting to see if there are variations in all three types of measures between teachers with different ratings (i.e. is a teacher with a high rating more responsive than a teacher with a low rating)?  Thus, we think it is important to include an adequate representation of teachers with various performance ratings.

Response 1: We are not considering prior performance ratings because the states were all piloting their evaluation models over the last two to three years (see Table 1 Supporting Statement A). Because the models were being piloted, ongoing adjustments were made to the criteria of effectiveness and the type of evidence included in the evaluation ratings. Therefore, the ratings from these years are not precise and should not be used for statistical purposes.

Comment 2: Will you factor prior performance evaluations in to your analysis?

Response 2: We are not considering prior performance ratings because the states were all piloting their evaluation models over the last two to three years (see Table 1 Supporting Statement A). Because the models were being piloted, ongoing adjustments were made to the criteria of effectiveness and the type of evidence included in the evaluation ratings. Therefore, the ratings from these years are not precise and should not be used for statistical purposes. We added a sentence to Supporting Statement A to indicate that prior performance is not included: “Prior teacher performance ratings are not included in the analysis because they are not precise due to changes in criteria and evidence of effectiveness made while the states were piloting their evaluation models over the last few years.”

Comment 3: Proportional quota sampling is generally considered a non-probability sampling method and its use would seem to weaken the study’s rigor.  Please clarify whether the study intends to use probability sampling in the smaller jurisdiction, and if not, clarify specifically why this is not possible, and how the results will need to be characterized if a nonprobability method is used instead.

Response 3: We agree that the use of the proposed non-probability sampling technique weakens the study’s rigor, so we adjusted our sampling plan to include only probability sampling techniques using a simple random sampling method that will allow for generalization to the population. We revised the sampling section in Supporting Statement B and removed the information related to non-probability sampling techniques.

Comment 4: For IES in particular--  what efforts it has undertaken to ensure that Survey Monkey meets the rigorous FISMA and other security requirements, such as staff training and affidavits of non-disclosure, that would permit it to offer such strong confidentiality assurances?

Response 4: We have opted to use SurveyGizmo survey software rather than SurveyMonkey because SurveyGizmo has a more sophisticated interface for sending invitations and reminders to survey participants. Survey Gizmo has Safe Harbor Certification and is HIPAA-compliant. SurveyGizmo uses an encrypted (SSL/HTTPS) connection to transfer and collect survey data. According to SurveyGizmo’s webpage, SurveyGizmo uses Advanced Encryption Standard (AES), the same method used by the US Government for classified top secret information. Per a report examining SurveyGizmo’s security features, third parties will not access nor be granted access to data, and the survey data can be scrubbed on request (http://wac.osu.edu/workshops/survey_of_surveys/). We added additional description of the security of SurveyGizmo to Section A.10 in Supporting Statement A. As stated in Supporting Statement A, Section A.10, the research team for this project is trained to follow strict guidelines for soliciting consent, administering data collection instruments, and preserving respondent confidentiality. All members of the research team have successfully completed the Collaborative Institutional Training Initiative (CITI) course in the Protection of Human Research Subjects through Liberty IRB. All members of the research team have signed an affidavit of non-disclosure; copies are provided in Attachment F.

Comment 5: Please clarify.  It looks like ED is requesting clearance for both their pilot and their full study, even though the sampling plan and questionnaire are likely to change after the pilot.  Given the discussion that public comment is needed on the full study, could Education clarify its expectations for this review

Response 5: We are requesting clearance for both the pilot and the full study. The sampling plan will not be affected by the results of the pilot of the survey instrument. We do not expect substantial revisions to the survey based on the pilot because the survey was carefully developed with input from educators in our research alliance, and content and methods experts serving as our Technical Working Group members. Although revisions to the survey, such as changes to the wording of questions, will be required based on the pilot results, the number of items and the level of public burden will not change significantly. We will submit the full study survey with changes based on the pilot results.

Response to OIRA and RMO Comments on ED-2013-ICCD-0089-0004 REL Central 2/4/14 Page 1