OMB-ED Q&A July 2010

Evaluation of Response to Intervention Practices in Elementary School Reading:

Support Statement for OMB Site Recruitment Clearance Request

Responses to Questions from OMB

General Questions

1.      Please clarify the sequence of contacts with schools and districts.  Some of the materials seem to suggest recruiting individual schools, then circling back and talking to the district.  Our understanding is that gaining district approval first is often a required or at least, a productive, first step.

ED agrees and the study team will contact districts before approaching schools that could be included in the evaluation.

2.      Also, could ED confirm that its intent is to recruit districts and schools that would be included in both aspects of the study if the data are available, or whether different schools will intentionally be recruited for the RDD and CITS analyses?

ED’s plan is to recruit sites that can be included in at least one of the two quasi-experimental impact designs, RDD and CITS. The programmatic and data requirements for the two designs are somewhat different. RDD requires a quantitative score (or combination of scores) for making decisions to provide more intensive reading support. A CITS design would require consistent data on student reading achievement and other key outcomes, such as identification for special education, both before and after a school’s adoption of RtI practices. ED anticipates that some schools and districts may fit the requirements for both designs, and the study team would then include them in both analyses rather than only one.

3.      Is there some analytic goal of this phase of data collection apart from the later phases?  In other words, is ED planning to produce some results from this phase only?  If so, what are they?  If not, is it the case that the purpose is to identify schools that can be in the next phase?

The purpose of this phase is to identify and recruit appropriate districts and schools for the planned in-depth study of the implementation and impact of RtI practices for elementary school reading. No reports from this study would be released prior to the data collection that would be described in a follow-on forms clearance package to be submitted for OMB review later in 2010.

4.      What is the primary purpose of the referenced site visits?  Are these intended to be only of schools that are intended to be included in the RDD and/or CITS analyses or are other schools candidates for the site visits?  If the latter, please explain why.

Site visits are under consideration as part of data collection from a subset of schools included in the RDD or CITS analyses. Such visits would be a means of gathering additional data on RtI implementation and may be combined with independent measurement of student reading performance. To inform the data collection package to be submitted to OMB later in 2010, the study team is planning to visit up to 9 schools similar to those proposed for the study.

5.      Can ED confirm that it will provide the results of this site selection and recruiting activity prior to asking OMB to approve the next set of materials?

ED will provide OMB with updates on the status of site selection and recruitment during the submission of the next set of materials for this study. However, ED does not want to delay the beginning of data collection from study schools by requiring that site selection be complete at the time of submission of the next forms clearance package.

6.      We understand that there is an exploratory aspect to this phase of work.  However, we would like to have explicit reassurance that NCEE intends to use the most rigorous methods and standards that are feasible, reserving those less rigorous approaches mentioned as “back ups” to invoke only if necessary.  Said another way, much of the information collected in this package is necessary to determine if an impact evaluation (either an RDD or a CITS) will be possible.  We would like a report back from IES on whether or not they think, based on these recruitment activities, an impact evaluation will be possible and if so using which design before any additional data collection (beyond what is necessary for recruitment) takes place.

ED is committed to addressing the research questions for the study using the most rigorous designs and analytic methods feasible. This evaluation will use a RDD and/or a CITS design. While, all else equal, the RDD is a more rigorous means of causal inference than the CITS design, the RDD can only address a narrow set of impact questions, focused on the effects on reading achievement of identifying at-risk students for more intensive instruction. The added value of the CITS design is that it would permit the study team to estimate effects of the adoption of RtI practices on the achievement of students as a whole, and on patterns of grade promotion and identification for special education. ED would be happy to brief OMB at any time on the status of recruitment of sites for the study and the feasibility of the different impact study designs, and will include the most up-to-date information in the future request for clearance of new data collection.

7.      The quasi-experimental impact analyses seem to assume that schools consistently administer Response to Intervention (RTI) in a rigid way-- that is they use a set protocol for determining the different tiers of intervention, have cut-scores, and clear beginnings and ends to the whole process.  However, our understanding is that there are two approaches to RTI – one that does have a standard treatment protocol and the other that’s more fluid (the problem solving approach).  Will this study look at both of these approaches and determine how prevalent/effective each is?

The study team will describe the varying practices used by the schools to implement RtI to address the needs of struggling readers. ED expects the study team to find schools across the spectrum of practice from standard treatment protocols to team-based problem solving. For the current impact evaluation, the study team will be not be able to make a strong causal statement about the relative effectiveness of RtI practices since these practices are not assigned randomly. Within this sample of schools, the study team will describe how prevalent these practices are.

8.      Also, we understand that the tiers of intervention can differ.  Some schools consider special education as the top (and general education as the bottom two tiers) and others only administer RTI within special education or actually consider extended days and summer schools as the top tier.  How will the study capture these differences?

RtI implementation includes a set of multiple practices, one of which is likely to be the use of multiple tiers for increasing the intensity of instruction. ED anticipates schools/districts will vary in how they choose to define and use multiple tiers of intervention. ED plans to begin to understand the differences through the questions asked to principals or RtI coordinators on the screening protocol (see questions 2 and 6), and in greater depth through the data collection plans that ED will describe in the next forms clearance package.

9.      In addition, our understanding is that most of the dollars that support RTI (through the coordinated early intervening services provision of IDEA Part B) go to professional development.  Will IES be looking at the professional development teachers get who implement RTI and determine the effectiveness of that training?  If so, how?  Will the findings on program impacts be looked at in the context of the type/ quality/ duration of the PD?

The IDEA National Assessment Implementation Study (report release planned for late 2010) is documenting district use of CEIS dollars, as well as sources and strategies of support for district RtI implementation. For the Evaluation of RtI Practices for Elementary School Reading, IES will document as a matter of context the training that school staff have received to support their implementation of RtI practices. Variation in training received may permit subgroup estimates of effects of RtI practices by differences in staff training, but this would be a correlational rather than an experimental analysis, since the training itself will not be assigned randomly to schools.

10.  Lastly, the evaluation seems to assume that a well-implemented RTI model will decrease the number of students identified for special education--that those students who weren’t keeping up with their typically-developing peers solely because of poor instruction would no longer be identified as needing special education because of RTI.  Does IES have any indication that special education enrollment has declined due to RTI?  Is this one of the questions the evaluation intends to answer?  If so, how?

The purpose of this evaluation is to test the hypothesis that a well-implemented RTI model will decrease the number of student identified for special education, however it is not an assumption. The CITS design will permit the study team on the current evaluation to estimate impacts of school adoption of RtI practices on rates of identification for special education.

Specific Questions

Page A-4, item 4 in numbered list – how will ED identify whether a school is “making decisions based on these data about students’ response to interventions”?

In both the screening protocol and in the forthcoming OMB package on data collection, there are questions and data collection tools designed to obtain this information. On question 5 of the screening tool, the principal or RtI coordinator answers a series of questions about how the school and how teachers use data for making decisions. Schools that use data for making decisions will have one or more of the following practices in place: a system for measuring student progress at least once a month if not more; decision rule guidelines that often coming from an existing system such as DIBELS or AIMSweb; team meetings, comprised of teachers, psychologists, literacy specialists, and the principal, that review this data for determining student’s progress and whether the child’s program should be continued or changed; and record keeping systems that document progress of the students. In the proposed forthcoming OMB package, plans for data collection on school practices will be described in detail.

Page A-5 – Why the 5 years required for schools identified by consultants outside of the pool of states where RtI is expected? While 2 years is a bare minimum, 5 years seems like a much higher bar than the 2 year benchmark for the other RtI mature schools.

When ED prepared the original submission in December 2009, ED believed that states with existing pilot programs would have more history with RtI and hence districts could have a stronger RtI implementation than would be the case in other states without such pilots. Thus, ED established a higher benchmark of “maturity” in these other states. Since that time, ED has learned more about RtI programs in other states, and on reflection now believes a two year implementation threshold is appropriate to apply to all nominations.

Page A-11 – section A10 – Does NCEE typically consider confidential which schools are in sample for its evaluations?  And does it intend to follow that convention in this study?  If so, we recommend offering the standard ESRA confidentiality pledge on all collections even at this early stage.  Right now the language in A-10 is a vague reference to confidentiality that should either be removed or replaced with standard ESRA language.

NCEE is bound by the ESRA requirement to protect confidentiality of schools.

SEC. 183. CONFIDENTIALITY.

(b) STUDENT INFORMATION.—The Director shall ensure that all individually identifiable information about students, their academic achievements, their families, and information with respect to individual schools, shall remain confidential in accordance with section 552a of title 5, United States Code, the confidentiality

standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provisions Act (20 U.S.C. 1232g, 1232h).

To comply with all applicable privacy laws and confidentiality provisions, ED will not identify schools included in this study or in any other IES evaluation. While ED has not typically provided legislation-specific details in materials submitted as part of forms clearance packages, the following language from the (2008) Statement of Work for the evaluation contract describes relevant confidentiality provisions that members of the study team (the evaluation contractor and its subcontractors and consultants) are obligated to uphold:

The contractor shall comply with: The Privacy Act of 1974 (P.L. 93-579, 5 USC 552 a); the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations on the protection of human research participants.

In those situations where the data collected, used, disseminated and/or maintained for this contract include sensitive or individually identifiable records covered by the Privacy Act of 1974, or personally identifiable information (PII) as defined by Office of Management and Budget Memorandum OMB 07-16, May 22, 2007, the contractor shall comply with U. S. Department of Education contractor security screening procedures for any staff members with access to such data. In the event that the contractor may be required to collect, maintain, use or disseminate privacy data, the contractor shall maintain privacy safeguards as required under 5 U.S.C. 552a(m) with respect to such data:

• The contractor shall be required to establish procedures at its sites and at subcontractor sites to ensure confidentiality of data. Their systems shall be required to provide reasonable assurance that information identifying individuals is in files physically separated from other research data. The contractor and subcontractors shall be required to maintain security of the complete set of all master data files and documentation. Access to individually identifiable data shall be strictly controlled. At each site all data shall be kept in locked file cabinets during nonworking hours, and work on hardcopy data shall take place in a single room, except for data entry. Physical security of electronic data shall also be maintained. Security features that protect project data shall include password-protected accounts that authorize users to use the contractor’s and subcontractors’ information systems but only to access specific network directories and network software; user rights and directory and file attributes that limit those who can use particular directories and files and determine how they can use them; and, additional security features that the network administrators establish for projects as needed. Contractor and subcontractor employees who “maintain” (collect, maintain, use, or disseminate) such data shall comply with the requirements of the confidentiality standards in section 183 of the ESRA (20 U.S.C. 9573).

• In safeguarding personally identifiable information (PII), the contractor and its subcontractors also shall be subject to the Department’s requirements contained in the Department of Education’s Handbook for the Protection of Sensitive But Unclassified Information, OCIO-15, and the Department’s policy that the transmission of sensitive but unclassified information, including PII, through an e-mail requires that the contents be password protected in a ZIP file.

Reporting of Data Security Breaches: If any PII is disclosed inadvertently or is at risk of disclosure due to a lost, missing, or intercepted transfer, the contractor must ensure that this breach is reported to the agency immediately. Per Office of Management and Budget Memorandum M-06-19, Federal agencies have a requirement to report breaches of PII security to a Federal incident response center within one hour of discovering the incident (and the agency should not distinguish between suspected and confirmed breaches). The data security plan must be written to reflect this requirement, and the contractor must provide sufficient notification to the agency for this requirement to be met.

All persons who collect, code, or otherwise have access to raw data must be trained in confidentiality issues and shall provide signed assurances of nondisclosure of confidential information. The Contractor shall maintain these records and provide them to the Government as requested in order to comply with ED’s contractor security screening requirements.

The contractor shall also comply with the Department's IT security policy requirements as set forth in the Handbook for Information Assurance Security Policy and related procedures and guidance. The contractor shall comply with IT security requirements in the Federal Information Security Management Act (FISMA), Office of Management and Budget (OMB) Circulars, and the National Institute of Standards and Technology (NIST) standards and guidance. These requirements include the successful certification and accreditation of the system before it can be implemented.

To ensure the system can be certified and accredited, security controls shall be designed and implemented in the system consistent with the NIST 800-53 - Recommended Security Controls for Federal Information Systems identified for the impact level of the system. The impact level will be determined via the completion of the Department's inventory form and critical infrastructure protection survey. System security documentation shall be developed to document the implementation of the security controls for the system and the contractor shall be available for interviews and demonstrations of security control compliance to support the certification and accreditation process.

In addition, if the system is rated as a moderate or high impact system, vulnerability scanning and penetration testing shall be performed on the hosting facility and application by the Department's Certification Review Group as part of the certification and accreditation process. Appropriate access agreements will be reviewed and signed before any scanning or testing occurs.

ED proposes to add this text to A-10 on p. A-11, including a reference to ESRA:

All data collected through this study will be gathered in compliance with: The Privacy Act of 1974 (P.L. 93-579, 5 USC 552 a); the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; the Education Sciences Reform Act of 2002 provisions on confidentiality, 20 USC 9573; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations on the protection of human research participants.

Page B-1 – If the study requires at least 40 schools to have sufficient power, why would it recruit only “40 to 50?”  Wouldn’t it be better to recruit a few more?

ED’s plan is to recruit 40 to 50 schools in which to study the design and implementation of RtI practices in elementary schools with mature RtI programs. ED also plans to identify other elementary schools in these schools’ districts for the impact quasi-experimental analysis, either as additional treatment schools under the RDD or CITS design, or as comparison schools in the case of the CITS design (if the schools have not implemented certain RtI practices or have implemented practices more recently). Thus, to improve statistical power, the number of schools that could eventually provide sample for the evaluation would exceed 40 to 50, perhaps by a considerable margin in the case of larger school district and the requirements of the CITS design. Data collection on RtI implementation and student outcomes from the larger sample of schools will be described in the second forms clearance package submitted for OMB review.

Page B-1 – In the footnote, is ED trying to say that some schools may be recruited at the end of or even after this phase of work because it didn’t recruit enough with available data during this phase?  If it already knows this, why not recruit more during this phase?

Priority for recruitment is being given to schools with mature RtI programs. Once the number of such programs and their districts are known and the relevant district and school officials have agreed to participate in the evaluation, ED will be able to estimate with greater confidence, in the next submission to OMB, the full number of schools that would be included in the evaluation through the collection of administrative records and surveys administered to school staff.

Page B-2 - In the discussion of CITS, you mention that the study's site nomination and screening process will gather information about the minimal conditions necessary for CITS to assess the feasibility of the design. Based on research to date, how confident is ED that enough schools will meet the study’s requirements for a CITS analysis? Would it be fair to call the study at this current stage a feasibility study?

ED is confident that the RDD is feasible in a sufficient number of schools to detect the effects of RtI practices on student outcomes for students falling below critical benchmarks of reading performance. Because the data requirements for the CITS design are more stringent than that for the RDD--especially in terms of retrospective data on RtI implementation by school and comparable outcomes measures over time--ED plans to use the site recruitment phase of this evaluation to assess the feasibility of the CITS design for possible inclusion in the larger study. If both designs are feasible, ED plans to implement both. If not, ED would implement the RDD only and address a narrower range of impact questions.

Page B-2, footnote 2 – If a significant number of schools have only two years of historical data and plan to continue the RtI program - Has ED considered delaying the screening and recruitment project for a year? This would mean that all of the schools with 2 years of historical data would then have 3 years, etc – providing longer baseline periods and, in theory, better estimates. Perhaps waiting a year would yield a better cost to analytical power ratio.

ED is motivated by stakeholder interest in evaluating the large-scale effectiveness of RtI practices, which could inform future reauthorization of IDEA. ED is also constrained by the five year limit to this evaluation contract, which is scheduled to expire in March 2013. For these reasons, ED is interested in beginning site recruitment as soon as possible while reserving the option of awarding a separate, RtI-focused impact evaluation contract in the future.

Page B-3 – footnote 4, wouldn’t there have to be a minimum of a 2-year lag in implementation for the later-implementing schools to be eligible for the study?  If not, please explain how they would be useful counterfactuals.

ED agrees that a two year lag in implementation for “lagged” comparison schools will make them most useful for the analysis. ED’s plan is to include sites for the CITS analysis where it is possible to continue the analysis for at least two years after implementation of RtI to include one or more school years after the initial start up. Thus, a two year lag in implementation for lagged comparison schools would be needed. There might be circumstances where an analysis with only one year of post-implementation follow up could be useful, perhaps where an unusually strong first year RtI implementation occurred. But ED expects this will be a rare exception to the usual rule of multiple years of follow up.

Page B-5, bulleted list – Based on research to date, how likely is it that enough schools collect and maintain the types of records required?

On page B-5, ED indicates that for schools to be eligible for the study under RDD they will need to maintain benchmark test data students and records tracking the student’s status throughout the year, and establish and follow rules for assigning students to a treatment or non-treatment status. From a nationally representative study of school districts (Means, Padilla, and Gallagher. 2010)¹, district survey respondents reported on the types of information they maintained electronically in 2007-08, and whether they had three years or more of longitudinal data stored in the same format. Seventy-two percent of districts reported that they include student test scores on district-administered assessment (e.g., benchmark, diagnostic), and of those, 57 percent have longitudinal data stored for 3 years or more. ED is also aware that many districts and schools maintain electronic benchmark data on individual students by using benchmarking data systems such as Dynamic Indicators of Basic Early Literacy Skills (DIBELS) or AIMSweb. These systems have recommended cut-points for each grade level on a reading measure appropriate for the child’s grade level, as well as maintaining the student records over time. The screening protocol asks respondents to report on the systems they use to benchmark student progress and how they use this data for making decisions about the student’s instructional program (i.e., whether they use the recommended benchmark cut-points and if it is the primary source of information for making this decision).

Page B-9 – Last paragraph before B3 and first bullet in B3 – who is “we?”  We would like the package written in ED’s “voice” rather than the contractors.

ED will change the text to read:

Once a sufficient number of schools meeting study requirements has been identified, the study team will recommend to IES a list of schools/districts for inclusion in the study, indicating which can be part of one or both of the quasi-experimental analyses of RtI impacts. The study team will make these recommendations to identify a sample of schools and districts with regional diversity and a mix and range of RtI practices that can inform the decisions of schools considering how to implement RtI practices. IES will approve the selection of the final sample of schools for the study.

Page B-10 – We see a reference to “compensation” in footnote 10 as well as in the supplemental materials.  We cannot approve any reference to compensation without more detail on what ED is proposing.  One key distinction is whether such “compensation” is truly a payment, such as these school staff serving as data collectors for ED.  This is rarely the case in these types of studies, except perhaps for a single school coordinator, but if that is being proposed we would typically see some type of agreement document that the individuals must sign.  If this instead is referring to incentives for participation, which is not compensation for time spent, the language should be changed and ED should submit those incentive plans to us now.  Our concern based on past experience is that a vague reference may lead to specific “negotiations” with principals or district officials over amounts or other conditions that OMB has not approved.

The reference to “compensation” is not to incentive payments but to ED’s willingness, as part of the cost of data collection, to pay districts for the costs of district or school staff assisting with data collection. We plan to compensate districts for district employees’ or contractors’ time spent preparing detailed data extracts from administrative records. Plans for this and related data collection from study districts and schools will be submitted to OMB later in 2010.

Appendix 1: project description – Can ED clarify the role of the two contractors listed?  Our preference would be to only list the one that is interfacing directly with the districts and schools.

While MDRC is the prime contractor for the evaluation and is responsible for final deliverables and the design of the impact analysis, SRI International is also playing a key role in describing RtI implementation in study schools. Both contractors will interface directly with districts and schools, so ED considers it reasonable to identify both firms on the project description. 

1
U.S. Department of Education, Office of Planning, Evaluation, and Policy Development, Use of Education Data at the Local Level From Accountability to Instructional Improvement, Washington, D.C., 2010.

6