11-21-07 memo

Responses to OMB questions

Subject: 200709‑1850‑001, Random Assignment Evaluation of ... English

Language Learners

1. Is BPA conducting the evaluation independently from the REL staff developers of the intervention or is the evaluation a joint effort (as implied in the introduction to Part A)?

Response to #1

BPA is an independent evaluator for the study of Pacific CHILD.

BPA staff will not participate in the implementation of the intervention. In the past, BPA staff attended the intervention development meetings as observers to learn about the intervention to inform our evaluation design. BPA’s interactions with the intervention development team have been limited to obtaining information for the evaluation such as the focus of the evaluation, measures and data collection procedures.

However, BPA and PREL collaborate where appropriate. For example, we will work together on recruitment efforts. We will also work together to coordinate communication with the local school districts and schools to minimize the burden on the schools and the potential for confusion. BPA staff also sought feedback on BPA's study instruments from PREL staff, our technical working group, and IES staff, although all final decisions on the design of these instruments are made by BPA.

2. Please clarify planned incentives in light of the OMB agreement with NCEE.

Response to #2

Our proposed incentive structure is in line with NCEE guidelines. We will provide gift cards of $10 value for control teachers in American Somoa and the CNMI and a $20 value for control teachers in Hawaii. The differences in gift card amounts reflect cost of living differences between entities.

3. Typically, RELs rely on state, district or school‑level administrative data. Describe the planned procedures and acknowledge that a state employee will have burden imposed in retrieving the data. Please clarify your plans and associated burden related to the use in the study of administrative data.

Response to #3

Our planned procedures are as follows:

(1) As part of the recruitment process, we will prepare an agreement that specifies that the study requires the entity administration's assistance in accessing school-level administrative data and that the entity agrees to assist with the data request.

(2) PREL will contact the entity administration to obtain a written request including the specifications for the data required for the study, along with a request for a working-level staff meeting. (PREL’s well-established long-term working relationships with SEAs in the entities will help facilitate the process of obtaining written agreements for BPA to access school-level administrative data. However, all school-level administrative data will go directly to BPA). At the working meeting between the entity data staff and the PREL/BPA study team, the group will work out the details of the data transfer arrangements. We expect this meeting to take approximately 45 to 60 minutes, plus some email correspondence prior to and following the meeting.

(3) To minimize the burden on state employees, we will request entities transfer school-level administrative data in two or three batches, timed according to the administration's schedule within the broader timeframe of the study. According to our experience, the actual data transfer process should require a relatively short time for each data extract (1-2 hours of an experienced data person to write a simple code for extracting the data fields from the system, run the program and save it in a secure data transport format, and mail the data). The burden generally comes from competing time constraints. Even though processing our request may not be technically difficult, the administration's data staff may not be able to set aside time easily because of their regular obligations. Our being flexible about the timing (i.e., asking the data staff to process our request when they have time) should alleviate much of the burden. For example, rather than providing a specific, tight deadline date, we plan to give at least a 4-8 week window for processing our data requests. The study team will also pay for all postage and/or conference call charges that are associated with accessing the data.

(4) If the entity administration still cannot assume the burden of responding our data request, we will offer assistance in the form of our staff time.

4. Why do you anticipate difficulty in collecting complete student level variables? How important are these variables to your modeling?

Response to #4

We do not anticipate any unusual difficulty in collecting complete student-level data necessary for modeling. While there may be some data collection challenges that are specific to this study with regard to student-level data, the level of risk of such challenges is what we would expect in any data collection process for this type of information.

Not all student-level information may be readily or uniformly available in all entities, especially in American Samoa or the CNMI. For example, some student data—such as free/reduced lunch status or ELL status—may not be available; if available, these data may not be consistently recorded. The availability and reliability of such student-level information will be closely checked. We expect basic student-level data, such as students’ grade level and gender, will be available, and we will use those uniformly available data as baseline covariates in estimating the full-sample effects. When a student-level variable is not available, we will seek classroom-level or school-level information as proxies.

5. To what does "data collected for public use" under Part A, Item 16, refer? Does the REL plan to produce a microdata product? If so, what is the disclosure avoidance and review plan?

Response to #5

As required by the REL contracts with ED, we will submit a disclosure analysis plan to our project officer, who will submit it to the IES Disclosure Review Board. This plan will likely include low levels of data swapping, as used in NCES public use datasets. Once our plan is approved, we will carry it out with our data and submit all files back to the COR and DRB. After that, any future work with the dataset (ours or the public’s) will require restricted use licenses.

6. The REL schedule and a statement in Part B about "ongoing" recruiting in Hawaii implies that recruiting activities are occurring prior to OMB approval, which is not permissible. Please clarify the status of recruiting activities to date.

Response to #6

The REL schedule was based on our anticipated OMB approval date.  The schedule has shifted as we have waited for OMB approval.  To accurately reflect our meaning, the identification of potential target schools for the recruitment pool by the study team in Hawaii is underway. The identification of target Hawai'i schools is ongoing.  We spoke to the Hawai'i State Superintendent to identify potential target schools that would meet our selection criteria. At this time, we expect to have 60-70 target schools from which to recruit.

In American Samoa and CNMI, the majority of schools in the entity are identified as target schools. The total number of elementary schools is only 23 in American Samoa, and 12 in the CNMI. The majority of these schools are identified as target schools. Based on conversations with the Director of Education in American Samoa and the Commissioner of the Public School System (PSS) in the CNMI, we narrowed the potential target schools to approximately 15 schools in American Samoa and 10 schools in the CNMI.

7. The study's sampling plan seems to imply that recruiting will be of almost 100% of eligible schools in CNMI and 3/4ths in American Samoa. This is an incredibly high sampling rate, particularly if any other IES studies or surveys are planned for these areas in the next couple of years.

7.1. What is the practical impact of so many schools being in the study from each area? For example, will the high prevalence lead to more contamination risk?

Response to #7.1

Many entities in the western Pacific are small and have few schools, many of which are small. To attain the necessary numbers of schools, teachers and students to attain the necessary estimates for MDEs (this study is a school-level random assignment), we need to recruit all or most of the schools in these smaller entities.

We are aware that there are risks that program group teachers may transfer to control schools in entities where we use most of eligible schools in our study. However, we will ask participating schools to commit to keeping teacher assignments for the duration of the study, and we have received support from the entities’ department level administration that teachers will not be transferred to other schools (unless there are good reasons, such as family reasons). We will also explain to teachers that they are being asked to stay in the same school for two years. Beyond making the expectation clear and securing school- and entity-level support, we will document any transfers of teachers or students, so that we can assess the extent of the contamination and consider any adjustments we might need to make in the analyses. We accounted for the loss of teachers/schools in our power analyses and are confident that we will still be okay with some attrition.

There are no other IES sponsored studies ongoing or planned for American Samoa or CNMI.

7.2. To what degree did the REL consider a broader recruiting region (i.e., > outside of its geographic boundaries, as some other RELs have done)?

Response to #7.2

Our original plan was to include four other entities (Guam, Palau, Pohnpei, and the Republic of the Marshall Islands) in the study. After much discussion and advice from IES, we decided not to include them in the study primarily due to the lack of comparable standardized tests of student achievement in these entities, as well as the difference in the policy environment (for example, three of the entities were not subject to the No Child Left Behind requirements). Guam was excluded from the study because it is implementing a curriculum that is incompatible with the intervention.

We did not explore any regions beyond the Pacific region. The REL Pacific region contains 10 geo-political entities, whose affiliation with the U.S. ranges from statehood to compacts of free association. The Pacific region is culturally, socially, and geopolitically very different from all other U.S. regions. An important objective of the current study is to test a professional development program that was designed to address the challenges of this particular region. In addition, almost no rigorous randomized control studies in education have ever been undertaken in the REL Pacific region (the only exceptions not conducted outside of Hawai’i), a typically under-studied region. Therefore, we deliberately wanted to concentrate on the Pacific region.

Furthermore, the Pacific region itself is geographically vast. It is spread across more than 4.9 million square miles of ocean, several time zones and the International Date Line. This makes any study involving more than one entity very costly. To expand the study outside the already expansive geographical area would not be practical or cost efficient from the point of view of resources available to the study.

7.3 To what degree is this study's sample design coordinated with any other near‑term NCEE or NCES studies?

Response to #7.3

We are not aware of any other NCEE or NCES studies planned for the near term that are similar in scope. There are smaller studies to be implemented by REL Pacific, with funding from IES, as part of the same REL contract (fast response studies under the REL Task 1.2, for example). These studies will be carefully coordinated with the proposed study. Currently there is no overlap in the samples or focus for these studies.

8. The study's sampling plan seems to have discrepant numbers. For example, how can the total number of eligible schools in CNMI be 10 (table 1), but the sample target be 11 (table 2)?

Response to #8

The correct number is 10 for CNMI. (At one point, the number of schools we would target was 11, but later found that this schools was ineligible because it was planning to use a curriculum incompatible with the intervention; subsequently, we corrected the sample target to 10.) We apologize for the error in the table.

9. Please provide copies of the proposed MOUs and materials related to any "other contractual arrangements."

Response to #9

This statement refers to Part B: "We will use Memoranda of Understanding (MOUs) (see Appendix M) and other contractual arrangements to create a strong framework of agreements and mutual obligations that make it difficult for any participating school or district to drop out of the study."

As noted in the application, a copy of a sample MOU was attached as Appendix M to the original application. This sample MOU is attached again to this response. At this point, we have not determined whether or what additional contractual arrangements will be necessary. This will be determined on the case-by-case basis during the recruitment process. That is, at this point, no other contractual arrangements or materials have been developed.

10. Can you please provide more detailed information on the cost (in dollars and time) of implementing the program?

Response to #10

REL Pacific estimates the total price per school for implementing Pacific CHILD for two years to be $43,300, or $21,650 a year. This includes the time over two years for two ten-day annual institutes, six three-day mini institutes and observations of and feedback to each teacher twice a month for two hours throughout the school year. If a school were to purchase the Pacific CHILD professional development without the on-going twice-monthly in-classroom work with each teacher, the total price per school for two years is estimated to be $12,300, or $6,150 a year for the two ten-day annual institutes and six three-day mini institutes.