u. s. dEPARTMENT OF eDUCATION
institute of education sciences
National Center for Education Evaluation and regional assistance
This memo responds to OMB’s questions on the proposed collection, “Evaluation of Response to Intervention Practices for Elementary School Reading (School and Staff Surveys)”, which OMB shared with IES via email and a conference call on March 30, 2012.
General Comments and Questions
1. We’d like you walk us through the study designs for each research question.
RESPONSE: Here are the questions from Part A of the justification (p. 3) and the corresponding designs:
What is the average impact on academic achievement of providing intensive secondary reading interventions to elementary school children who have been identified as at risk for reading difficulties compared with children just above the cut point for providing intervention?
This question will be addressed through a Regression Discontinuity (RD) design, for which 143 schools have been recruited that are likely to be eligible for the impact analysis in at least one grade between 1 and 3. Data collection on Grade 1 and 2 reading outcomes for this design needs to occur in spring 2012.
How do academic outcomes, including reading achievement and special education identification, vary with elementary schools’ adoption of Response to Intervention practices for early grade reading?
This question will be addressed through a Comparative Interrupted Time Series (CITS) design. Of the 143 Response to Intervention schools recruited for the RD design, up to 93 may be eligible for the CITS design as well. This summer the study team will finalize plans to sample comparison schools for this design; additional data collection would occur in early fall 2012 but would not require new data collection on student reading outcomes.
How do Response to Intervention practices for early grade reading vary across schools and how are they related to academic outcomes?
This question will be addressed through data collection on reading assessment, instruction, and intervention practices in both treatment (Response to Intervention/RtI) and comparison schools. We expect there to be variation, not only in practices between treatment and comparison schools, but in practices between different RtI schools, and between Tier 1 and Tier 2 practices in each RtI school.
2. Have the instruments been cog labed?
RESPONSE: Yes. Results from the pilot tests are included with this memo, and changes proposed to survey questions are tracked in the accompanying versions of the instruments.
3. Interventionist Survey – it appears that this survey is identical to the Teacher survey (including the section that describes who should complete it). Should the teacher survey differ from the interventionist one?
RESPONSE: It appears that the correct (and distinct files) were posted http://edicsweb.ed.gov/browse/browsecoll.cfm?pkg_serial_num=4734 (and, in particular http://edicsweb.ed.gov/browse/downldatt.cfm?pkg_serial_num=4734), but not to http://www.reginfo.gov/public/do/PRAViewIC?ref_nbr=201112-1850-002&icID=200370. We provided the distinct Appendix 3 (Interventionist Survey) to you by email on 3/30/2012.
4. Teacher Survey – a) On question A4 about type of certification, are you sure that this question will get you the responses you seek reliably? This question is asked consistently on NCES SASS and other related surveys and we were wondering if you consulted them on the development of this question.
RESPONSE: We revised the response to A4 to conform to question 33a in the 2007-8 SASS teacher survey.
b) This survey uses the term ‘reading IEP.’ Is that how the field refers to an IEP with a goal for reading?
RESPONSE: Yes. To avoid confusion, we replaced “reading IEP” with “IEP with reading goals”.
5. Who exactly is completing the school survey? The instrument itself asks for the most knowledgeable person. Isn’t it conceivable that this person is the interventionist?
RESPONSE: The survey will be addressed to the principal or school administrator. We have revised the instructions to read, “This survey should be completed by the school principal in consultation with the person or persons who are most knowledgeable about your school’s overall program of reading instruction and about the intervention services provided to children who are performing below grade-level expectations in reading. Only one school survey should be submitted for each school.”
6. Questions b20 and b21 first option starts with ‘Wait a few weeks…’ Why is this clause included? How would a responder answer the question if the person immediately consulted with peers (the latter part of the option)? A1a. By “what” do you mean “what program?”
RESPONSE: We revised b20 so the responses read…
Assign student to a reading intervention group focused on targeted reading skills, in addition to providing core instruction
Continue offering instruction in the general education classroom and collect additional progress monitoring data
Consult with peers to find approaches to address the needs of the student before considering an assignment to a reading intervention group Refer for diagnostic testing
Refer to special education
Other [_______________________]
and the response to b21 so they read…
Assign student to a different or more intensive reading intervention group focused on targeted reading skills
Continue offering the intervention and collect additional progress monitoring data
Consult with peers to find approaches to address the needs of the student before considering an assignment to a different or more intensive intervention
Refer student for diagnostic testing
Refer student to special education
Other [_______________________]
7. Will there be any cost data collected as part of this evaluation?
RESPONSE: As we discussed in our call on 3/30/12, it is difficult to gather relevant cost data for this sample of schools, because much of the RtI training for experienced schools may have been received several years in the past. Nonetheless, some questions ask the principal about the provision of funding to support RtI implementation (D1j) or a lack of resources as a barrier to RtI implementation (D11b,f,g,h). Future IES studies that fund training to support program implementation should be able to document the costs of training received in study schools over the period when data collection occurs.
2. Specific Comments and Questions on the School Survey
2a. (Question A1a) By “what” do you mean “what program?”
RESPONSE: Yes. We have revised the text to read, “What commercial reading program does your school currently use?”
2b. (Question A8) This is a pretty long and awkward question that might [present] comprehension issues.
RESPONSE: We revised this question to read, “In which grades, if any, does your school schedule core reading instruction to group students in different classes according to their overall reading level? MARK ALL THAT APPLY.” The responses are Kindergarten, Grade 1, Grade 2, Grade 3, Grade 4, and Grade 5. We also revised A9 to read, “Are there students from different grades taught reading in the same class?”
2c. (Question A10)
i. It might make a bit more sense from a design perspective to flip the order of the answer categories, unless you think most people will answer “Not at all.”
ii. Are these the only methods? If not, add an “other” category.
RESPONSE: We have made these changes.
2d. (Question B2)
i. Third answer category should say “electronically”, not “electronic”, to maintain parallelism.
RESPONSE: We have made this change.
ii. Is there some order to these data types? This matrix is pretty hard to read.
RESPONSE: The data types are ordered according to the timing of tests during the school year (student universal screening occurring first).
iii. What if no data is available? You ask this in the next question, but it seems more appropriate here.
RESPONSE: We added a “No data available” option in the last column.
2e. (Question B3)
i. The formatting of this question (info box, question, info box) seems cluttered and might confuse the respondents.
ii. Other method doesn’t really make sense, even with the info box. Can you come up with a pithier way of saying this?
RESPONSE: We removed the second box and last column, and revised the question to read as follows:
Which school staff, if any, are responsible for entering student performance data into the student data system?
|
No Data Entered |
Teachers |
Other School Staff |
District, State, or Contractor Staff |
2f. (Questions B8-17) This formatting is way too difficult to understand. I want to read down the columns instead of across the rows.
RESPONSE: We transposed the columns and rows.
2g. (Question B18) Should there be a “not important at all” answer choice?
RESPONSE: We revised the choices to read “Very important”, “Somewhat important”, and “Not important”.
2h. (Question B20) Where are these answer choices coming from? Is this a complete domain of “first steps?”
RESPONSE: While the responses cover a range of interventions under the heading of “reading intervention group” as well as various options for evaluating the child’s performance, we added an “other” option as the last response item for this question (and also for B21).
2i. (Question C3) Are these the only possible types of data?
RESPONSE: We added an “other” option.
2j. (Question D1) Why this order of sub-questions?
RESPONSE: The logical sequence of responses moves from written standards, to technology, to personnel.
2k. (Question D2) Would it be appropriate to add an awareness question here? It might help you determine if the Rs answers to the following questions are informed…
RESPONSE: As we discussed in our call on 3/30/12, we think asking about awareness of RtI would have more value for a nationally representative study, such as the ECLS-K, than for this purposeful sample of experienced RtI and comparison schools.
2l. (Question D6) Are you asking about an “average” special education student, or are they supposed to answer only if there is a set policy?
RESPONSE: We have revised the question to refer to students with Specific Learning Disabilities and the instruction they typically receive in Grade 1, Grade 2, or Grade 3.
3. Specific Comments and Questions on the Teacher Survey
3a. (Question C1) “Most” either needs to be operationalized or dropped.
RESPONSE: We revised this question to read, “In the grades you teach, does your school schedule core reading instruction to group students in different classes according to their overall reading level?”
3b. (Question C2) Why would the teacher (and not the principal) know this?
RESPONSE: We revised this question to read
Are approximately the same numbers of students assigned to each class you teach for the core reading block?
Yes No I do not teach more than one reading class
3c. (Question C5) The last two answer choices might be easier to read if they start with “Yes,…”
RESPONSE: We have made this change.
3d. (Question C5a) I would drop the first column header (Other Adults…Reading Block) as it confuses the question.
RESPONSE: We have made this change.
3e. (Question C10) Why are you defining “Core Reading Block” here when you have used it extensively in earlier questions?
RESPONSE: We dropped this reference, since “Core Reading Block” is defined at the beginning of section C.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |