Technical Working Group

APPENDIX D – Technical Working Group Suggestions

In January 2014, Southwest REL researchers obtained comments from two TWG reviewers: Dan Goldhaber and Geoff Borman. This Appendix summarizes in bullet form the conceptual suggestions that Southwest REL researchers received. Line edits are omitted from this summary. The researchers response are noted under each bullet prefaced by “RESPONSE.”

• The power of the intervention to produce changes in practice may be compromised if there are limited quality PD opportunities for principals to recommend in order for teachers to improve their instruction. This could lead the investigators to incorrectly conclude that the guide was not useful for changing practice and, eventually, achievement.
  
  

  • RESPONSE: While ED’s contractor agrees that the quality of the PD modules could be an important moderator of the impact of the Guide, the decentralized acquisition or creation of PD by districts or by schools within districts makes it impractical to know ahead of time what the PD will be on offer during the school year 2015-2016. However, ED’s contractor believes that this lack of a standardized menu of pre-approved/pre-screened PD sessions is representative of the norm and thus a good test of the efficacy of a feedback guide in other states besides New Mexico.

• Unless it is vital for treatment group teachers to receive the treatment guide, it seems inadvisable because the threat of cross-over seems quite strong.

  • RESPONSE: ED’s contractor decided to distribute the guide to teachers as well as to principals for the following two reasons: (a) To minimize Hawthorne effects, ED’s contractor intends to trump principals’ survey responses regarding usage of the guide with teachers’ responses about having seen or ever used the feedback checklist. (b) ED’s contractor intends to disseminate the checklist to teachers because the point of the treatment guide is to structure a two-way, mutual conversation instead of principals talking at teachers (as is currently the common practice). ED’s contractor agrees this increases the risk that the treatment guide will be shared with control group school staff. ED’s contractor think the best idea to discourage contamination and reduce John Henry effects is to disseminate a “Guide” to all principals, blinding them to their status in treatment or control group, which is addressed in the next point.

• The provision of [only] the treatment guide and the reliance on self-reported survey data (and principal-reported classroom observation scores) could cause Hawthorne effects due to both principals and teachers expecting improvement solely because they have been offered a new treatment. Providing a "guide" to all participating principals could, potentially, minimize the risk of cross-over (because everyone gets something), and would certainly address the issue of a Hawthorne effect.

  • RESPONSE: ED’s contractor agrees and has adopted this suggestion.

• None of the outcome measures are well-validated outcomes, and it is not clear whether the authors intend to construct composite measures from the surveys or whether outcomes will be based on single items.

  • RESPONSE: ED’s contractor agreed and has included student achievement as an outcome (a well-validated outcome) and developed composite indexes derived from survey questions.

• The estimated recruitment rates of 65% for principals and 60% of teachers seem relatively high.

  • RESPONSE: ED’s contractor has powered the study for a range of participation rates, between 50 and 65 percent for principals and between 50 and 60 percent for teachers, and demonstrated that for the research design the drop-off is very small from the upper to the lower ends of these ranges (approximately 0.02 to 0.03 in MDES). ED’s contractor based the upper end of our anticipated participation rates on prior experience with a similar efforts in New York City Public Schools, where there was also a direct connection to regular school responsibilities (although somewhat weaker than in the case of the current study) and participation was well over 60% (McCombs, Kirby, and Mariano, 2009).

• The teacher inclusion criteria for the study could bias the results of the study. 

  • RESPONSE: ED’s contractor agreed, and has dropped eligibility screeners for either principals or teachers. The only remaining eligibility criteria is that the study participant work in NM public schools at the time of the survey.

• The study would be considerably better if it had value added components.

  • RESPONSE: ED’s contractor agreed that value-added would be a highly useful, validated outcome. Therefore, ED’s contractor added student achievement as an outcome measure.

• It would be beneficial to include charter schools.

  • RESPONSE: This suggestion has been enacted.