Responses to OMB Questions

Date	January 16, 2008
To	Karen Matsuoka, OMB Gil Garcia, IES Amy Feldman, IES
From	REL-SE SIOP Study Team
Subject	Response to OMB Questions on 200709-1850-002, SIOP

The following are responses to the questions raised by OMB on Supporting Statement A for the REL-SE SIOP Study. We address each question separately below, and have revised Supporting Statement A accordingly. Shortly before receiving the questions from the OMB, the study team decided to expand our proposed recruitment pool, based on a review of school demographics. Changes in the proposed recruitment pool entailed minor additional changes in both parts of the Information Collection Request document; we are therefore re-submitting the entire OMB package: Statements A and B, and all attachments.

1. Please provide a little more information on the two ongoing studies, such as what you know about grade levels they target, whether they are experimental designs, and whether they will meet What Works Clearinghouse standards?

The REL-SE SIOP team is aware of two current studies that involve the SIOP Model. The subjects in one study are 7^th grade students (Impact of the SIOP Model on Middle School Science Language Learning) while the other study spans K-3^rd grade (SAILL). Neither study focuses on the grade levels of the proposed study that is the subject of this Information Collection Request (grades 4 and 5).

The Impact of the SIOP Model on Middle School Science Language Learning study uses an experimental design (three-group random assignment of 25 schools, two treatment groups and a control group) and focuses on middle school science instruction. In the SAILL study, 1192 students were randomly assigned to treatment or control groups within language of instruction groups (ESL or Bilingual Education)—a two-by-two design.

It appears that both the above studies, as well as ours, could meet the criteria for inclusion in a What Works Clearinghouse effectiveness review/topic or intervention report on "randomized controlled trial or a quasi-experimental design with evidence of equating between treatment and comparison groups" (WWC 2007 Appendix A4 Methodology, English Language Learners topic report).

We are attaching documents describing these two studies for further elaboration (these documents will be the new Supporting Statement A-Appendix B).

[Inserted in Statement A, item 1, pages 8-9.]

2. If this intervention is very popular and widely available in the State, how will you ensure that the control group is a valid control group?

Although SIOP is very popular and Pearson Achievement Solutions has made training readily available, the cost of providing training to teachers (see question #5 below) has prevented most districts and schools from providing high quality training in the SIOP approach to date. In recruiting schools, we propose to include only schools that could provide a suitable comparison with the proposed treatment. That is, one of two types of schools: First, schools where teachers have not yet received training in SIOP; second, schools in which teachers have received a very small “dose” of the treatment would be considered suitable. Schools in which teachers have received a large “dose” of the treatment would be unlikely to be interested in participating in the study as their teachers would not be motivated to repeat training they have already received. The Teacher Survey includes questions aimed at identifying teachers who have received any SIOP training prior to the study or receive it during the study year (“crossover” of comparison group members). There are probably several levels of SIOP exposure currently in the field in Georgia:

1. Districts identified enhancing preparation of mainstream teachers’ ability to work with English Language Learner (ELL) students as a key area for professional development, have sufficient financial resources to address this need, and have provided training for:

1. ESOL (Title III) staff at the district level: Districts that have adopted this strategy may have arranged for Title III staff to receive a 3-day training and some follow-up workshops. The Title III staff then provides periodic support to mainstream teachers in the district (in the form of in-service workshops, for example). This—train-the-trainer—model of SIOP training is considered less desirable (but more affordable) than the simplest treatment model of teachers receiving the 3-day training or than the model offered in our proposed study. Teachers in schools who had received this sort of train-the-trainer treatment would still be suitable for either the comparison or treatment group.

2. ESOL (full or partially Title III-funded) staff at the school level: Districts and schools that have adopted this strategy are still following a train-the-trainer model for training their mainstream teachers in SIOP, and it is still considered less desirable. In these schools, the ESOL staff may provide limited amounts of training on SIOP components to mainstream teachers, or alternatively, may not even be expected to train their fellow teachers at all but instead may be expected to shoulder the load of enhancing educational services independently of the mainstream teachers (as per usual and customary implementation of the Georgia ESOL pull-out model of instructional support for English Language Learners). We feel that in most schools that have used either of these approaches, the administrators and teachers who would be interested in participating in the study would make that decision based on their judgment of the effectiveness of having mainstream teachers trained directly. Schools that express interest in participating would, thus, also be suitable candidates for inclusion in the study.

2. Districts identified low performance of ELLs as a reason for not meeting Adequate Yearly Progress (AYP) goals and have begun to dedicate financial resources to solving the problem. Efforts have taken several forms:

1. Funding has been made available for SIOP training for ESOL staff only (see a.2 above).

2. Funding has been made available for SIOP training for all interested staff: In schools like this, typically administrators and teachers will not be interested in participating in the study, and we will not include them.

3. Districts have identified enhancing services to ELLs as an area of need but have not been able to dedicate financial resources to providing training for mainstream teachers and/or have such low percentages of ELLs in their schools that they do not qualify for Title III funding (or qualify for less than a FTE-worth of funding), and so have not funded any SIOP training. In such districts and schools, there are two types of schools:

1. Schools in which no staff has received any SIOP training: These schools are typically very eager to participate in the study. If assigned to comparison group, they would provide an ideal “uncontaminated” comparison.

2. Schools in which some staff has received SIOP training: These schools are typically eager to participate in the study, but the staff who has already received training may not be interested in participating in the study, leaving only staff who resemble those in group c.1 above.

In summary, we believe that staff who have already received high-quality SIOP training will be less likely to participate in the study, leaving only staff who have not received treatment like what we are proposing in this study, and the comparison group would thus be relatively uncontaminated. In addition, we propose to administer the SIOP protocol as part of the classroom observation in all classrooms (treatment and comparison), which will help us detect SIOP-like instructional behaviors, and the Teacher Survey includes questions asking teachers about their exposure to SIOP training.

[Inserted in Statement A, Overview—Study Approach section, pages 5-7.]

3. Typically, RELs relying on state, district or school-level administrative data describe the planned procedures and acknowledge that a state employee will have burden imposed in retrieving the data. Please clarify your plans and associated burden related to the use in the study of administrative data.

The REL-SE Senior Policy Research Analyst for Georgia, Dr. Kimberly Anderson, also serves as SIOP Study Manager. She has worked closely with Georgia Department of Education Associate Superintendent for Assessment and Accountability, Dr. Chris Domaleski, and Mrs. Levette Williams, Director of Data Collections and Reporting, to establish a procedure for the request and transfer of data. The proposed procedure is as follows: the study team will work through Dr. Anderson to make data requests of the Georgia Department of Education. Dr. Domaleski and Mrs. Williams will then assign particular junior staff to retrieve the data requested, as they determine appropriate. Data will be provided to Dr. Anderson, who will then pass it on to the study team. Dr. Anderson will be the liaison for arranging any additional correspondence between study team members and the Georgia Department of Education staff.

The estimated burden on Georgia Department of Education staff is as follows:

1. Hours that will be spent by Georgia Department of Education Data Collections Division Operations Analyst/Technology/Management personnel to provide data that the team requests. Annual salaries for these staff range from $34,533 to $60,487 (average = $47,510); this translates to roughly $22.84/hour. Staff will be asked to:

Provide percentages and numbers of ELLs per school as well as number of 4th and 5th grade teachers per school in Georgia for sampling and recruitment purposes. This information would be needed during the recruitment phase of the study. Estimated time required to fulfill requests: 14 hours.

Provide information about the nature of the database containing staff information that we would use to supplement information requested in the Teacher Survey (e.g., teacher job codes). This information would be needed during the instrument design phase of the study. Estimated time required to fulfill requests: 0.25 hours.

Estimated total time required: 14.25 hours

Cost: 14.25 hours x $22.84/hour = $325.47

1. Hours that will be spent by the Georgia Department of Education Assessment Specialist/Research and Data Analysis personnel to provide the data that we request. Salaries for these staff range from $30,792 to $80,760 (average = $55,776); this translates to roughly $26.82/hour. Staff will be asked to:

Answer questions about the structure of data reported (e.g., vertical scaling of scores, interpretation of scores). This information would be needed once only. Estimated time required to fulfill request: 0.25 hours.

Provide de-identified student assessment data for the period requested. This information would be needed once only. Estimated time required to fulfill request: 6 hours.

Estimated total time required: 6.25 hours

Total Cost: 6.25 hours x $26.82/hour = $167.63

[Inserted in Statement A, item 12, pages 19-20 and Table 2 on page 21.]

3. In addition, most REL studies have associated some burden with teachers and/or schools facilitating classroom observations. Why does this REL see it differently?

We estimate that facilitating classroom observations could entail a modest amount of a school administrative staff person’s time, but we have used the average teacher salary ($30.18/hour) as it could also be done by a teacher assigned as the study liaison. Estimated time required to fulfill request is approximately 10 minutes per teacher and 15 minutes per school:

[(10 minutes/teacher x 616 teachers) + (15 minutes/school x 88 schools)] x $30.18

We have added this information to the burden calculation [Table 2, page 16].

[7480 minutes = 124.67 hours x $30.18 = $3762.44]

[7480 minutes/88 schools = 85 minutes/school = 1.47 hours/school respondent]

Estimated total time required: 124.67 hours

Total Cost: 124.67 hours x $30.18 = $3762.44 (at $42.75 per respondent)

[Inserted in Statement A, Item 12, page 20, and Table 2 on page 21.]

3. How much would it cost a school to implement this intervention?

The approximate cost to a school of providing the intervention model employed in this study would be $2,031.25 per participant [teacher or administrator]. This estimate is based on the 2007-2008 school year contract price negotiated with Pearson Achievement Solutions.

[Inserted in Statement A, Overview—Study Approach section, page 7.]

3. IES may not cite the Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA). Please remove references to that statute.

We have removed references to that statute from the document.

[Deleted from Statement A, item 10, page 16.]

3. The SIOP study school contact information has an OMB control number on it and is a collection to all schools, so the burden associated with this IC should be included in the burden estimates.

We estimate that obtaining school contact information will entail some time on the part of the principal or principal’s designee. Most likely, the principal will take approximately 10 minutes at a regular staff meeting to present information about the study (the research team will provide informational materials for this purpose), distribute informational materials to teachers who express interest, and invite teachers to sign up to participate in the study. Using average principals’ salary ($93,573/year, or $44.99/hour) as the basis for an estimate of burden, we estimate that the additional burden associated with completing the school contact information sheet will be $5518.56:

Assuming approximately ((15 minutes/school x 88)) x $44.99/hour = $989.78

[22 hours x $44.99/hour = $989.78]

Estimated total time required: 22 hours

Total Cost: 22 hours x $44.99/hour = $989.78 (at $11.25 per principal)

[Inserted in Statement A, Item 12, page 20 and Table 2 on page 21.]

3. Can ED confirm whether any kind of parental consent form is needed to access the student assessment data from state agencies?

The student assessment data that we propose to obtain will have been stripped of all identifiers that could lead to the identification of individual subjects. The Georgia Department of Education Data Collections Divisions Operations Analyst/Technology Management has assured our project manager that this process will be utilized.

Based on the published decision-making documents of the Institutional Review Boards (IRBs) at both The University of North Carolina at Greensboro (UNCG) and Abt Associates Inc., the following criteria [from 45 CFR part 46] were used in assessing whether or not parental consent would be required to access student assessment data from the state agency:

1. If the activity is a systematic investigation, designed to develop or contribute to generalizable knowledge [45 CFR 46.102(d)] [Yes], and

b. If the activity is research, involving obtaining information about living individuals

[45 CFR 46.102(f)] [Yes], and

1. If the research involves intervention or interaction with the individuals [45 CFR 46.102(f)(1),(2)] [No, because the students will just be participating in normal school activities led by their teachers], and

1. If the information will be individually identifiable (i.e., the identity of the subject is or may readily be ascertained by the investigator or associated with the information [45 CFR 46.102(f)(2)] [No], then

The research is not research involving human subjects, and 45 CFR part 46 does not apply; thus, parental consent will not be required for either students’ participation in the study or for the use of their de-identified assessment data.

[Inserted in Statement A, item 2, pages 10-11.]

3. Can ED specify why there is not a consent form for parents to sign to allow their child to be part of an evaluation? All prior IES studies included a parental consent form.

Students in schools that are participating in the study will not be asked to do anything other than what is a normal part of the school’s adopted instructional approaches and curricula. Other IES studies may have required direct assessment of students for which parental consent would be necessary, but this proposed study (SIOP) involves no direct assessment of students beyond that conducted by schools in accordance with their state accountability mandates.

As noted above, the UNCG and Abt Associates IRBs agree that students in the SIOP study would not meet the criteria for Human Subjects such that they would be covered by 45 CFR part 46.

[Inserted in Statement A, item 2, page 9.]

7

R

egional Educational Laboratory