10-12-07 memo

OMB Question	Response	Location in Document
Supporting Statement A:
What would be the cost to a school of implementing Success in Sight?	The cost per school to implement Success in Sight would be approximately $18,000 for schools that participate in the consortium model, as is being tested in this study. In the consortium model, groups of schools are brought together for the large group professional development sessions while working individually with the mentor assigned to their school.
The intervention is described as intended to be implemented over the course of two 9 month school years.  Please provide OMB a timeline of when the data collections will occur during this two year period.  Is any data collection going to take place after implementation is complete?	The timelines for data collection have been added follows: “Baseline data collection: March 1– April 30, 2008 First follow-up data collection: March 1– April 30, 2009 Second follow-up data collection: November 15 – December 15, 2009 Because the intervention, Success in Sight, was designed to be fully implemented over a period of two years, the second and final follow-up data collection is scheduled to occur at the end this implementation period in November 2009.”	p. 4
Could the second set of student achievement data be drawn from the regularly occurring State assessments in the spring rather than through a separate administration of the NALT?  How well aligned are the NALT and the State assessments?	We interpret the “second set of student achievement data” to mean the second “follow-up data collection” after the initial baseline data collection (i.e., the final data collection). To accommodate the timelines for preparing the final report for the study, including the time required for review and revision within IES, we were advised by IES to conclude data collection in the fall of 2009. Thus, the study must conclude prior to the next regular administration of the state assessment in spring 2010, making it necessary to administer the NALT as the final data point for student achievement. Linking studies indicate a high degree of alignment between the NALT and the state assessment. The Predictive Index ranges from 92-96%.
Please provide a description of the intervention Success in Sight.	Further information on the two-year intervention has been added, including: “Success in Sight is designed to build the capacity of the entire school to engage in four school reform practices: (1) data-based decision-making, (2) purposeful community, (3) shared leadership, and (4) other practices known to be related to improved student achievement.” and “Specific elements of the intervention include: three, two-day professional development sessions; monthly onsite mentoring for school leadership teams between sessions; and online support for principals and school leadership teams.”	p. 3
How will the evaluation determine if changes in student achievement are related to the intervention or to other things going on in the school?  Is the intervention sufficiently well specified to distinguish between the two?	As noted under the response to question 4, the intervention is designed to effect change in the capacity of schools to engage in four specific school reform practices: data-based decision-making, purposeful community, shared leadership, and other school improvement practices known to improve student achievement. In this randomized control trial study, schools are randomly assigned to treatment and control groups. As such, the treatment and control groups will be equal on all variables other than the intervention. Therefore, any impacts detected will be the result of the intervention and not other variables. Still, we will use covariates in our models to increase precision.
Incentives: please confirm that IES is not asking for an exception to your proposed incentives policy and revise the supporting statements accordingly.	IES is not asking for an exception. No payments will be made to respondents.	p. 13
Attrition: the study implies that tangible and intangible incentives may help with student and teacher attrition (specifically mentioning challenge of treatment teachers to control schools). This implication should be substantiated or removed.	This statement was removed.	p. 23
In item 16, page 27, there is an interrupted sentence that reads "School improvement practices are measured using the state..."  Please explain how school improvement practices will be measured.	This statement regarding student achievement was not relevant to the specification of the HLM model on page 27. It has been removed.	p. 27
Please clarify whether McREL intends to put out microdata files.  If so, please note that NCEE must ensure clearance by NCES's Disclosure Review Board.	McREL subcontracted with ASPEN Associates, an external research firm, to conduct the study under the REL Central lab contract in recognition of the importance of having a third party conduct research on an intervention that was developed by McREL. To maintain the integrity of the study, clearly defined roles and responsibilities were outlined to establish and maintain a firewall between the McREL staff who implement the intervention and the study team at ASPEN Associates. ASPEN Associates has been responsible for developing the final research design and budget for the study. Microdata files will be produced. The text has been clarified as follows: “Upon completion of the study, McREL will submit the final data file, along with a complete and accurate codebook prepared by ASPEN Associates to the IES program officer who will forward the files to the Chair of the Disclosure Review Board. This board will review and approve the files so that potential data users may apply to IES for restricted use license.”	p. 1 p. 28
Supporting Statement B:
Part B, page 2 indicates that preliminary recruiting is already underway. Please clarify. This is not acceptable prior to OMB approval.	No recruitment has occurred, only informational meetings. The text has been clarified as follows: To date, the study team has focused on identifying districts actively seeking comprehensive school improvement interventions and providing information about the study to these districts. A state department representative provided those districts in the state that are failing or in danger of failing to meet AYP. This information is a matter of public record. When an opportunity to meet with the leadership of these districts presented itself, staff members of Aspen Associates, a partner in this study, attended these informational meetings to inform district personnel about the upcoming study opportunity. The study team provided an overview of the intervention, the intended outcomes, key timelines, and school eligibility and selection criteria. No baseline or other study data has been collected from any of these district representatives, and will not be collected until OMB approval to proceed is received.	p. 2
Part B indicates that Minnesota was chosen in part because..."The intervention, Success in Sight, was developed and tested within the Central Region states, making it difficult to find a set of schools within the region that had not already been exposed to the intervention."  Is this saying that Minnesota is not in the Central Region or that MN has below average exposure or something else?	This is correct. The text has been clarified as follows: “The intervention, Success in Sight, was developed and tested within the Central Region states, making it difficult to find a set of schools within the region that had not already been exposed to the intervention. Minnesota is not in the Central Region and thus has not had prior exposure to Success in Sight. Minnesota also was selected because it is demographically similar to the states in the Central Region. Minnesota is adjacent to North Dakota and South Dakota, two of the Central Region states. Minnesota is similar to other states in the region in the percentage of the population that is under age 17, over age 65, possessing a high school diploma, possessing a college degree, per capita income, unemployment rate, and per pupil spending.	p. 2-3

Page 4 of 4