Memo 8-21-07

February 5, 2021

IC, 1850-NEW, EDICS #33003, A Study of the Differential Effects of ELL Training and Materials

RE: Response to OMB (questions in bold).

Prepared by REL Central for Sandra Garcia for submission to Kate Mullan

Follow up on #3. Has Harcourt been contacted to determine if they have evaluated (or plan to evaluate) the impact on student achievement of RISE or RISE and OWE in combination?

Per the suggestion of OMB, we contacted Jennifer Watts, Product Research Manager of Harcourt this morning (August 20, 2007). Ms. Watts confirmed that no studies have been conducted on the combination of RISE and OWE, although Harcourt believes this is a more effective way to prepare teachers to teach ELL students than with OWE alone. As indicated in the response to the first set of OMB questions, two very limited studies (commissioned by Harcourt) investigating the benefits of the OWE curriculum as a stand alone have been conducted (Educational Research Institute of America, 2004 and Marketing Works, 2005). The first of these, completed in early 2004, was a quasi-experimental study. Design limitations reduced the utility of mixed findings. The second, the 2005 study, was fatally flawed when the study school reassigned students mid-year and contaminated the groups. A third qualitative study, also on OWE as a stand alone, has just been completed. Harcourt has no plans, at this time, for further study of OWE by itself, nor of RISE, nor the combination of the two programs.

Follow up on #13. Can we get additional specifics on how negative findings (on teachers’ use of the OWE curriculum and the knowledge and skills emphasized in the RISE program) will be addressed?

Negative findings, either in OWE curriculum usage (fidelity of implementation) and/or teacher knowledge and skills, will be addressed in the results and discussion sections of the final report with the same rigor and attention as that given positive findings. We do not anticipate large numbers of teachers not implementing the intervention, since it is widely in use indicating teachers’ acceptance of it, and teachers involved in the study will have expressed interest in participating. Nonetheless, we will have data on teacher implementation. We will be including all teachers in the analyses – the intent to treat sample – regardless of their level of implementation. If there is no discernable positive relationship between treatment and student achievement, we will be able to determine if level of implementation is a factor.

Follow up on #14. OMB believes these correlations should be imbedded into the proposed evaluation design. [Note: We believe that this follow up is addressing item #15 in the previous OMB feedback and comments.]

Pursuant to the OMB request we will include these correlations in the design of the analysis.

OMB Responses:
A Study of the Differential Effects of ELL Training and Materials

Updated: 8/20/2007