NW S07 - SIG Rural Schools OMB Supporting Statement Part A

CONTRACT: ED-IES-12-C-0003

Supporting Statement for OMB Clearance of a Survey of Principals of Rural Schools Receiving School Improvement Grants and Using the Transformation Model

Section A

August 16, 2013

Revised September 6, 2013

Revised October 8, 2013

Revised Oct 16, 2013

Revised December 13, 2013

Revised December 30, 2013

Contents

Section A: Justification 1

A1. Circumstances Necessitating Collection of Information 1

A2. How, by Whom, and for What Purpose Information Is to Be Used 3

A3. Use of Automated, Electronic, Mechanical or Other Technological Collection Techniques 5

A4. Efforts to Identify Duplication 6

A5. Sensitivity to Burden on Small Entities 7

A6. Consequence to Federal Program or Policy Activities if the Collection Is Not Conducted or Is Conducted Less Frequently 7

A7. Special Circumstances 7

A8. Federal Register Announcement and Consultation 7

A9. Payment or Gift to Respondents 8

A10. Confidentiality of the Data 9

A11. Additional Justification for Sensitive Questions 10

A12. Estimates of Hour Burden 10

A13. Estimate for the Total Annual Cost Burden to Respondents or Record Keepers 11

A14. Estimates of Annualized Costs to the Federal Government 11

A15. Reasons for Program Changes or Adjustments 11

A16. Plans for Tabulation and Publication 11

A17. Approval to Not Display the Expiration Date for OMB Approval 15

A18. Exception to the Certification Statement 16

References 17

List of tables

Table 1. Administration times 11

Table 2. Timeline 15

The following attachments are stored in the “NW S07” folder in the “REL Northwest Memos and Correspondence” library, with the exceptions of Attachments D and E, which we will add later since the Federal Register notices cannot be submitted until this document is approved by IES:

Attachment A: ESRA

Attachment B: Copy of Survey and Online Survey Screen Shots

Attachment C: Intro Letter Invitation and Follow-Up Emails

Attachment D: 60-Day Federal Register Notice (to be added after the notice is submitted)

Attachment E: 30-Day Federal Register Notice (to be added after the notice is submitted)

Attachment F: Pilot Survey Report

Attachment G: Affidavits of Nondisclosure

Supporting Statement for OMB Clearance of a Survey of Principals of Rural Schools Receiving School Improvement Grants and Using the Transformation Model

Section A: Justification

A1. Circumstances necessitating collection of information

This data collection is authorized by the Educational Sciences Reform Act (ESRA) of 2002 (see attachment A). Part D, Section 174(f)(2) of ESRA states that as part of its central mission and primary function, each regional educational laboratory “shall support applied research by … developing and widely disseminating, including through Internet-based means, scientifically valid research, information, reports, and publications that are usable for improving academic achievement, closing achievement gaps, and encouraging and sustaining school improvement, to schools, districts, institutions of higher education, educators (including early childhood educators and librarians), parents, policymakers, and other constituencies, as appropriate, within the region in which the regional educational laboratory is located.”

REL Northwest and our partner Policy Studies Associates (PSA), the authors of this study, are interested in how rural schools implement School Improvement Grants (SIGs). These competitive federal grants have provided funds to two cohorts of low-performing schools to increase student achievement. The grants require schools to implement one of four school improvement models: the transformation model, which involves a variety of reforms; the turnaround model, which involves primarily replacing staff; the restart model, which involves becoming a charter school; and the closure model, which closes the school.

This descriptive study collects and reports on data from rural school principals across the nation that received federal SIGs in Cohort 1 and used the transformation model. This information collection is essential to a study by the Regional Educational Laboratory (REL) Northwest, which serves five Northwest states (Alaska, Idaho, Montana, Oregon, and Washington). Most rural Northwest schools with SIG funding (96 percent) used the transformation model to improve. REL Northwest and its partner PSA are collecting survey data about the implementation of the transformation model to inform regional and national supports for future school improvement efforts. Data collection will occur once in spring 2014. IES has approved the study plan.

Statement of need and background information

To date, there is little systematic evidence about the implementation of the SIG program. In the case of rural SIG settings, this lack of research evidence is particularly problematic. Rural schools often face steep challenges when trying to implement the kinds of staff replacement and on-site professional development practices promoted by the SIG program (Klein, 2010). This study addresses these challenges by examining the level of implementation of the SIG transformation model in rural regions, identifying the organizations—states, districts, universities, and other entities—that assist schools in these implementation efforts, and exploring perceived gaps in assistance.

SIG funding mechanisms are not new. Congress put them in place as part of the No Child Left Behind Act (NCLB) of 2001 (2002), the reauthorization of the Elementary and Secondary Education Act (ESEA). However, in 2009 the American Recovery and Reinvestment Act (ARRA) substantially increased funding for these grants. With this increase, the U.S. Department of Education issued new guidance, dramatically changing SIG requirements (U.S. Department of Education, Office of Elementary and Secondary Education, 2011).

Under this guidance, federal funds pass to states as usual, but states are now charged with assisting schools in implementing one of four school improvement models (U.S. Department of Education, Office of Elementary and Secondary Education [OESE], 2011): (1) transformation, replacing the principal and implementing a number of strategies to improve the school; (2) turnaround, replacing the principal and 50 percent of staff members and providing operational flexibility needed to implement a variety of school improvement strategies; (3) restart, reopening as a school operated by a charter or education management organization; and (4) closure, closing the school and ensuring that students transfer to a high-performing school.

Because the transformation model requires less disruption and staff member replacement than other models, some consider it the most flexible of the four (Klein, 2010). Perhaps as a result of this flexibility, most grantee schools—and 96 percent of rural schools nationally—have chosen this model (Hurlburt, Le Floch, Therriault, & Cole, 2011; Hurlburt, Therriault, & Le Floch, 2012). Therefore, our study focuses on the transformation model in rural settings. The transformation model requires the following 11 activities (U.S. Department of Education, OESE, 2011):

1. Replace the principal who led the school prior to the transformation model

2. Use rigorous, transparent, and equitable evaluation systems for teachers and principals that take into account data on student growth, as well as other factors

3. Identify and reward school leaders, teachers, and other staff members who improved student outcomes, and identify and remove those who did not

4. Provide staff with ongoing, high-quality, job-embedded professional development

5. Implement strategies designed to recruit, place, and retain staff

6. Use data to identify and implement a new instructional program

7. Promote continuous use of student data in order to inform and differentiate instruction

8. Establish schedules and strategies that provide increased learning time

9. Provide ongoing mechanisms for family and community engagement

10. Use operational flexibility (such as staffing, calendars/time, and budgeting) to improve student outcomes

11. Ensure that the school receives ongoing, intensive technical assistance and related support from the LEA, the SEA, or an external organization

Study design, data collection, and timing

This descriptive study collects and analyzes principals’ reports of SIG transformation model implementation in their rural schools. It uses a one-time survey of the 211 Cohort 1 principals of rural schools implementing the SIG transformation model (attachment B). The survey will occur in spring 2014. It also incorporates existing data from the SIG baseline database available on the U.S. Department of Education website, including each school’s National Center for Education Statistics (NCES) locale code (i.e., the geographic characteristics of the school), percentage of students eligible for free or reduced-price lunch (i.e., poverty levels among students), and percentages of non-White and Native American students. The study uses the data to address the following research questions:

1. How do principals of rural SIG transformation schools rate the extent to which their school implemented the required activities in the transformation model?

2. What do principals report were the challenges to implementation of each activity?

3. What types of technical assistance providers assisted schools, which particular transformation activities did they assist with, and how did the principals rate the sufficiency of this assistance?

4. Do principal ratings of implementation, challenges, and technical assistance providers vary by school geographic or demographic characteristics?

A2. How, by whom, and for what purpose information is to be used

To make good decisions about ongoing school transformation efforts and future similar school improvement initiatives, the nation’s education leaders need more information about rural SIG implementation of the transformation model. More specifically, the data from this study will be used by REL Northwest and our state contacts to provide information to federal policymakers during ongoing efforts to reauthorize ESEA. These policymakers need such information as they consider changing or retaining language around the requirements for the continued use of SIGs under ESEA, which are likely to continue to provide funds to schools for school improvement, as well as language incorporating elements of the ESEA waivers involving school improvement efforts (U.S. Department of Education, 2012). We will also use these data to provide information to state policymakers in states with similar legislation around school improvement. Furthermore, we will use these data to inform education practitioners, particularly those who provide support for school improvement. These practitioners need the data to identify gaps in technical assistance for rural schools seeking to improve and uncover opportunities to provide additional and/or different services to schools.

Why is it crucial for these policymakers and practitioners to consider changes in policies and practices for transforming rural schools? Despite the fact that the transformation model requires less disruption and staffing changes, many education leaders anticipate that rural schools, in particular, will have difficulty meeting the requirements of the transformation model. For example, recruiting and retaining high-quality staff members is known to be more difficult in rural areas (Hammer, Hughes, McClure, Reeves, & Salgado, 2005; Monk, 2007). In addition, finding technical assistance providers who can work with rural schools can be challenging (Corbett, 2011); transforming school schedules to allow for extended learning time can be difficult due to the longer bus rides and higher transportation costs (Sandel & Bhat, 2008); and rural parents can be mistrustful of outside technical assistance providers—such as state employees or university professors—who seem determined to change their schools (Owens, Richerson, Murphy, Jageleweski, & Rossi, 2007). However, there is little empirical information about the implementation of the transformation model in rural schools.

This study will provide education policymakers and practitioners with crucial information, including: specifying the extent to which the transformation activities were implemented and the challenges to implementation and identifying which activities were supported by technical assistance providers and how sufficient principals found this support. The study will do this primarily through a specially designed survey of the 211 rural school principals implementing SIG transformation models in Cohort 1 (funded from 2010 through 2013, with an option to carry over funds to 2014). We have designed this survey (attachment B) based on our review of the literature on SIG and rural schools and on our staff members’ past work on SIG evaluation.

The first section of the survey asks principals when they became an administrator at their school and if they are familiar with SIG implementation at their school. If the principal is new to the school and does not know about the SIG implementation, the survey will prompt the principal to provide contact information for the prior principal or for a designated proxy, who is identified as a staff member who was employed during the SIG funding period and served in one of the following roles:

• A school-level administrator, such as an assistant principal, who participated in the school’s SIG

• A teacher who was responsible for assisting with the implementation of the school’s SIG

• A district administrator responsible for school improvement

The second section of the survey asks principals to rate their school’s implementation of the 11 required transformation model activities using a four-point scale that reflects the stages of implementation developed by Fixsen, Blase, Naoom, and Wallace (2009). Transformation activities include:¹

1. Replace the principal who led the school prior to commencement of the transformation model

2. Use rigorous, transparent, and equitable evaluation systems for teachers and principals that take into account data on student growth, as well as other factors

3. Identify and reward school leaders, teachers, and other staff members who improved student outcomes, and identify and remove those who did not

4. Provide staff with ongoing, high-quality, job-embedded professional development

5. Implement strategies designed to recruit, place, and retain staff

6. Use data to identify and implement a new instructional program

7. Promote the continuous use of student data in order to inform and differentiate instruction

8. Establish schedules and strategies that provide increased learning time

9. Provide ongoing mechanisms for family and community engagement

10. Use operational flexibility (such as staffing, calendars/time, and budgeting) to improve student outcomes

11. Ensure that the school receives ongoing, intensive technical assistance and related support from the LEA, the SEA, or an external organization

We ask about other school improvement activities in which rural schools may engage (through an open-ended item), and we also ask principals to indicate which activities they believe were essential to core efforts to improve the school.

Next, the principals will be asked to select one or more challenges for each activity. We will allow principals to select multiple responses for this survey item, since some items may be related to one another. For example, “insufficient funding to implement activities” may mean that staff members are overworked and, therefore, also have “insufficient staff time to implement activities.” Principals may select both responses if appropriate. They will also have the option of selecting “NA (no challenges)” or “NA (did not implement this activity).” Finally, in an open-ended item, principals may list other challenges to implementing the activities.

The next section of the survey asks principals the extent to which they agree that technical assistance from other entities was sufficient to help the school implement the required transformation activities. Participants will rate the sufficiency on a four-point agree/disagree scale. The choice of entities includes their district, their state education agency, a partner in higher education, or another technical assistance provider. This list is based on the entities that other studies have found assisted with SIG most often (Corbett, 2011; Scott, McMurrer, McIntosh, & Dibner, 2012). If a principal selects “another technical assistance provider,” the survey will prompt the principal to fill in the name of the organization that provided the technical assistance. The principals will also be able to elaborate on their responses regarding the sufficiency of the assistance from each entity.

A3. Use of automated, electronic, mechanical, or other technological collection techniques

Initially, we will invite principals to complete an online survey (with electronic submission of responses through Survey Gizmo). If return rates are less than 50 percent after the second reminder to complete the survey, we will offer an electronic gift card to increase the response rate. We will do this for three reasons. First, our pilot showed that offering an incentive (an Amazon electronic gift card) after several reminders increased the response rate from 44 percent to 77 percent. Several research studies support the use of incentives (Armstrong, 1975; Church, 1993; James, & Bolstein, 1992), particularly for online surveys of principals (Jacob & Jacob, 2012). Principals who completed the survey before the second reminder will be sent the incentive retroactively.

Those that do not respond after three reminders will have the option of completing a paper version of the survey that they may email, fax, or mail back to our partner, PSA. Either version will take about the same amount of time to complete (20 minutes). PSA will host a website where respondents can access the link to the online survey. Attachment C contains the advance mail notification letter, the invitation to participate in the survey, and the follow-up notification email/letters.

We have decided to first invite principals to respond to the survey online and then, after three reminders, to allow nonrespondents a paper option in case principals preferred this format. Kaplowitz, Hadlock, and Levine (2004) found that a web survey application achieved similar response rates to a mailed hard copy questionnaire when both were preceded by an advance mail notification. In addition, Millar and Dillman (2011) found that a combined use of multiple response-inducing techniques (web and mail) produced the best response rates.

Study results will be available via an online link to the study on the IES and REL Northwest websites. Participating principals and their districts will be invited to participate in webinars to discuss the results of the study. In these webinars, participants will share ideas about implementing the SIG transformation model and similar school improvement efforts. They will also discuss the supports provided by technical assistance providers and how to leverage additional supports to address challenges to school improvement. This participation will be entirely voluntary and is presented as a way of motivating principals to participate.

A4. Efforts to identify duplication

This one-time study collects unique information that is not available elsewhere and is not being collected by other studies. To date, most discussions of rural SIG have combined all rural schools regardless of their other geographic and demographic characteristics and have not delved deeply into the implementation of the required activities in the transformation model (e.g., Dee, 2012; Rosenberg, 2011; Scott, 2012; Scott et al., 2012). While these past studies offered important information about SIG, the current study fills a need for information about rural schools’ implementation of the required activities in the transformation model. Ongoing national studies of SIG, such as Hurlburt and colleagues (2011, 2012), focus on all types of schools and on the four school turnaround models broadly, not on characteristics of rural schools and the required activities in the transformation model. These studies do not provide information about how states and other technical assistance providers might support specific school transformation activities in rural settings.

The American Institutes for Research and Mathematica are currently conducting an in-depth descriptive study of 25 SIG schools, including 10 in rural areas. While this study will provide an in-depth look at some rural schools, it relies on less than 5 percent of rural SIG schools and will not generalize across rural SIG schools.

A5. Sensitivity to burden on small entities

This project collects data from principals of schools, which by definition are small entities. We have minimized the burden for respondents by (1) requesting information that is generally available to school leadership, including principals, other administrators, and teacher leaders directly involved in SIG implementation, and (2) requesting the minimum amount of information required to answer the study questions. Answering these questions will provide policymakers and practitioners with information needed to make good decisions about the SIG transformation model and similar school improvement efforts. For example, items about the geographic and demographic characteristics of the schools are not included in the survey because this information is available through the SIG baseline database.

As shown in section “A12. Total Annual Cost Burden to Respondents,” we estimate a 20-minute total time burden on each respondent. In addition, we will allow principals to designate a proxy, if it helps minimize the burden on the school.

A6. Consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently

This one-time data collection has the potential to inform federal, state, and local agency programs and policies related to turning around low-performing schools both through future SIG allocations and other efforts such as NCLB waivers. Without this study, policymakers and their supporters would have little information about the unprecedented $3 billion that went into SIG in 2009 through the ARRA—about six times what Congress had allocated the previous year (Scott et al., 2012). Of these funds, more than $300 million went to rural transformation in the first cohort of schools (Hurlburt et al., 2011). In addition, the types of efforts required by federal guidance to turn around low-performing schools may be similar in the future. This study informs this future work on other school improvement efforts that are similar to SIG.

A7. Special circumstances

There are no special circumstances. The request fully complies with the regulations.

A8. Federal Register announcement and consultation

Federal Register announcement

ED will publish Federal Register Notices to allow both a 60-day and 30-day public comment period. The REL will assist ED in addressing any public comments received.

There has been no comments received under the 60 day comment period notice.

Consultations outside the agency

Prior to this request for OMB approval, the research team received expert feedback on this survey and study from researchers outside the agency through REL Northwest’s Technical Working Group (TWG). The two TWG members who contributed to this study were distinguished experts in their fields and extremely knowledgeable about research design and methods: Tom Dee of Stanford University and Hans Bos of American Institutes for Research. These TWG members provided written feedback on study plans and/or reports and met via telephone with the research team to clarify their feedback and provide additional recommendations.

A9. Payment or gift to respondents

As described in section A3, if return rates are less than 50 percent after the second reminder to complete the survey, we will offer an electronic gift card to increase the response rate. We will do this for three reasons. First, during the pilot, response rates were low until we offered a gift card from Amazon. Upon the first request, only two principals (22 percent) completed the survey. After two reminders, four principals (44 percent) had completed the survey and one had partially completed it. PSA staff decided to provide an Amazon gift card as an incentive. Several principals responded positively to the incentive. The principal, who had only partially completed the survey, finished the survey. Two others also completed the pilot survey for a response rate of 77 percent. More details about the pilot are contained in attachment F.

Second, we reviewed the literature and found several reviews of survey research that showed incentives increased response rates (Armstrong, 1975; Church, 1993; James, & Bolstein, 1992). In addition, a recent experimental study of the impact of incentives on the return rate of a principal survey (Jacob & Jacob, 2012) influenced our thinking. In this experimental survey of 1,177 principals, offering an incentive had a statistically significant positive impact on the return rate: 48 percent of principals offered an incentive with the online survey returned the survey compared to 22 percent who were not offered the incentive.

Third, REL guidance recommends the use of incentives for groups that are surveyed frequently (Sloan, Ingels, & Burghardt, 2012). Because our participants are principals, they are likely to be surveyed frequently. Research shows that 50 percent of principals, in general, receive four or more survey requests per year and 25 percent receive more than seven per year (Jacob, Scott, & Bowers, 2008).

Therefore, based on the results of our pilot, the research literature, and REL guidance on survey incentives, we will offer an incentive to principals. In order to avoid offering the incentive unnecessarily, we will first ask principals to complete the survey without offering an incentive. Then, if at the second reminder the return rate is less than 50 percent (as it was in the pilot), then we will offer a $20 Amazon gift card. Principals, who already completed the survey without an incentive, will receive the $20 Amazon gift card retroactively as a thank you. We chose $20 as the amount of the gift card because this is the amount recommended in REL guidance (Sloan, Ingels, & Burghardt, 2012).

In addition to the gift card, participants will receive a link to the published study and invitations to several webinars to discuss the study. While this is not a payment or gift, it will allow participants to have a voice in the discussion of study results. During the webinar, participants will share information about implementing SIG and similar school improvement efforts. They will also discuss the supports provided by technical assistance providers and how to leverage additional supports for schools to address challenges to school improvement. We believe this opportunity will motivate people to complete the survey.

A10. Confidentiality of the data

REL Northwest and PSA will be following the new policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183: “All collection, maintenance, use, and wide dissemination of data by the Institute” are required to “conform with the requirements of section 552 of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act (20 U.S.C. 1232g, 1232h).” These citations refer to the Privacy Act, the Family Educational Rights and Privacy Act, and the Protection of Pupil Rights Amendment. Subsection (c) of section 183 referenced above requires the Director of IES to “develop and enforce standards designed to protect the confidentiality of persons in the collection, reporting, and publication of data.” Subsection (d) of section 183 prohibits the disclosure of individually identifiable information, and makes any publishing or communicating of individually identifiable information by employees or staff a felony.

REL Northwest and PSA will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Information from participating institutions and respondents will be presented at aggregate levels in reports. Unique identification numbers will be assigned to each participating school and used to identify all survey responses. The ID number/name association files will be kept secure in a confidential file separate from the data analysis file. No information identifying respondents will be included in the study data files or reports. Data from the online survey software system will be downloaded and deleted from the online system within one week after the data collection ends. These data files will then be stripped of any identifying information. Information will be reported in aggregate so that individual responses are not identifiable. All identification lists will be destroyed at the end of the project.

The research team is trained to follow strict guidelines for soliciting consent, administering data collection instruments, and preserving respondent confidentiality. All members of the research team will have successfully completed the Collaborative Institutional Training Initiative (CITI) course in the Protection of Human Research Subjects through Liberty IRB. A copy of the affidavit of nondisclosure is provided in attachment G and will be signed by each researcher who will have access to the data.

All study materials will include the following language:

Per the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district or individual. We will not provide information that identifies you or your district to anyone outside the study team, except as required by law. Any willful disclosure of such information for nonstatistical purposes, without the informed consent of the respondent, is a class E felony.

A11. Additional justification for sensitive questions

No questions of a highly sensitive nature are included in the survey or interview, and we have received Institutional Review Board approval for this study.

A12. Estimates of hour burden

We estimate the total reporting burden associated with this data collection to be 61.58 hours and $2,609.90 (table 1). This includes 63 principals designating a proxy and 169 survey respondents completing the survey (i.e., 80 percent of 211). More details about these activities are included below in table 1. We estimated a principal’s hourly wage using May 2012 Bureau of Labor Statistics National Occupational Employment and Wage Estimates (http://www.bls.gov/oes/current/oes_nat.htm).

Designating a proxy

We anticipate that approximately 63 principals (i.e., 30 percent of 211) may designate a proxy to fill out the survey because they are either new to the school or too busy to fill out the survey. We based this estimate on the national average of 20 percent principal attrition yearly (Battle, 2010), plus an additional 10 percent of principals who may be too busy. We anticipate that the designation of proxies will add about 5 minutes to the burden on schools. Designating a proxy adds 5.25 hours to the burden and $222.50.

Survey

We will invite all 211 principals or designated proxies of our target population (i.e., the Cohort 1 rural SIG transformation schools) to participate in the survey. The target response rate for these individuals is 80 percent, and the approximate time required for each respondent to complete the online survey is 20 minutes on average, based on the pilot and on a similar survey (Scott, Davis, & Krasnoff, 2012). The total hours for the survey are 56.33, and the total cost is $2,387.41.

The most recent Schools and Staffing Survey of principals by the National Center of Education Statistics had a response rate of 79 percent for both public school and Bureau of Indian Education principals (Battle, 2009). This report did not disaggregate the response rates for rural school principals. But, we suspect that response rates would be somewhat lower for rural school principals. For example, the response rate of the rural district administrators’ survey was 74 percent (Zhang, 2008). Therefore, because we set our target at 80 percent, we will use several methods to maximize response rates and address nonresponse, such as offering an incentive as described in A9 and the use of online data collection described in A3.

Table 1: Administration times

Reporting method	Number of respondents	Average time (hours)	Total burden (hours)	Hourly rate	Estimated cost
Designating a proxy	63	5 minutes	5.25	$42.38	$ 222.50
Survey	169	20 minutes	56.33	$42.38	$2,387.41
Total			61.58		$2,609.90

A13. Estimate for the total annual cost burden to respondents or record keepers

There are no direct start-up costs to respondents.

A14. Estimates of annualized costs to the Federal Government

The total estimated cost for this study is $373,250. Although the data collection occurs only once, these costs cover a two-year study period; therefore, the annual cost of the study is $186,625. We estimated these costs using the amounts budgeted in the REL Northwest Year 2 plan and approved by IES. These costs include planning the study, administering the survey, collecting and cleaning data (including the use of an incentive), and analyzing and reporting results. In addition to the evaluation costs, there are personnel costs of several Federal employees involved in the oversight and analysis of information collection that amount to an annualized cost of $5,000 for Federal labor.

A15. Reasons for program changes or adjustments

This is a new study.

A16. Plans for tabulation and publication

We will tabulate and analyze the data by the four research questions. We will address the first three questions by providing the frequencies of survey responses for each transformational activity by level of implementation, challenges, and technical assistance provider. We will also explore the first three research questions using open-ended items on the survey. The fourth research question requires inferential statistics. Plans for tabulation and analysis are described in more detail below.

Research question 1: How do principals of rural SIG transformation schools rate the extent to which their school implemented the required activities in the transformation model?

We will address the first research question primarily by calculating frequencies of survey responses for each transformation activity by the extent of implementation, (i.e., N/A, planning/developing, early or partial implementation, or full implementation). The categories of levels of implementation are defined more fully in the survey (attachment B). We will also provide descriptive statistics about the percentages of principals that rate these activities as essential to their core efforts to improve schools.

We will use one open-ended item to add details about the implementation of activities related to school improvement. The item asks principals to elaborate on their responses regarding transformation model activities that are fully implemented in their school and essential to core efforts to improve the school. To examine the item, we will develop codes inductively (Mayring, 2000). To develop codes, the two study authors will review approximately a third of the data independently and create codes. They will then compare these codes and agree on final codes. Next, they will both independently code the remaining data and compare results. If they cannot agree on the code for a particular piece of data, a third researcher will determine the final code.

Research question 2: What do principals report were the challenges to implementation of each activity?

As in research question one, we will address this question primarily through frequencies of survey responses about challenges. Because some activities may pose multiple challenges, principals may select all challenges that apply (i.e., insufficient funding, insufficient staff expertise, insufficient staff time, insufficient technological capacity/equipment, lack of teacher support, lack of district support/guidance, other).

Principals will also be given the opportunity to elaborate on their responses regarding challenges. To conduct an exploratory analysis of this qualitative data, we will develop codes inductively (Mayring, 2000). To create codes, the two study authors will review approximately one third of the data independently. They will then compare these codes and agree on final codes. Next, they will both independently code the remaining data and compare results. If they cannot agree on the code for a particular piece of data, a third researcher will determine the final code.

Research question 3: What types of technical assistance providers assisted schools, which particular transformation activities did they assist with, and how did the principals rate the sufficiency of this assistance?

We will provide two types of descriptive statistics to explore this question. First, we will provide frequencies for the principals reporting that their school received assistance on each of the transformation activities for particular types of providers (i.e., districts, states, universities, and other). Then, we will provide descriptive statistics representing the degree to which principals whose schools received this assistance agreed that the assistance was sufficient.

To provide more detail about technical assistance providers, we will examine the open-ended response that asks principals to provide the name(s) of their technical assistance organization(s) We will list the top 10 organizations in the main body of the report and the complete list of organizations in the technical appendix.

Finally, open-ended items about each type of technical assistance provider will allow principals to elaborate on their responses regarding the sufficiency of the technical assistance from each type of provider. We will analyze this qualitative data using deductive coding (Mayring, 2000). As described previously, two study authors will review the data independently, then compare and agree on final codes. Disagreements, if any, will be settled by a third researcher.

Research question 4: Do principal ratings of implementation, challenges, and technical assistance providers vary by school geographic or demographic characteristics?

To examine variation in stages of implementation and in assistance by the geographic and demographic characteristics of schools, we will use the Kruskal Wallis H test. The Kruskal Wallis H test is a nonparametric test that allows the comparison of mean ranks of more than two categorical groups. We have chosen to use this test because, in our study, the dependent variables of interest (our survey responses about the stages of implementation and about sufficiency of assistance) are ordinal and the independent variables of interest (geographic and demographic characteristics of schools) are nominal. We will conduct four series of Kruskal Wallis H tests:

1. Implementation by geographic characteristics of schools, (i.e., by locale codes representing the type of rural area a school is in, such as a rural remote area versus a distant town)

2. Sufficiency of assistance by geographic characteristics of schools

3. Implementation by demographic characteristics of schools (i.e., by whether the school serves primarily white, Latino, or Native American students)

4. Sufficiency of assistance by demographic characteristics of schools

We discuss each of these series of analyses in more detail below.

To examine variations in reported implementation by geographic characteristics, we will conduct the H tests for each item’s response category by each of the six locale codes (31, 32, 33, 41, 42, and 43). However, any cells with fewer than five responses will be eliminated from the analysis or will be collapsed if possible, (e.g., strongly agree and agree could be combined and strongly disagree and disagree could be combined). In each analysis, the ordinal Likert scale responses for the survey item will serve as the dependent variable. The independent or “grouping” variable will include the six locale codes. If the H test is statistically significant, we will perform a post hoc Dunn’s test to determine which specific groups were different from one another. We will display mean ranks, degrees of freedom, H test statistics, and Dunn’s test statistics for each analysis. To determine statistical significance we will use the Benjamini-Hochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995).

To examine variations in reported assistance by geographic characteristics, we will also conduct H tests for the items about assistance. For these items, we will consider assistance from each type of provider (district, state, university, and other technical assistance providers) separately. We expect that in many instances this may result in cells with fewer than five responses. Those items with small cell sizes will be eliminated from the analysis. In each analysis, the ordinal Likert scale responses for the survey item will serve as the dependent variable. The independent or “grouping” variable will include the six locale codes. If the H test is statistically significant, we will perform a post hoc Dunn’s test to determine which specific groups were different from one another. We will display mean ranks, degrees of freedom, H test statistics, and Dunn’s test statistics for each analysis. To determine statistical significance we will use the Benjamini-Hochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995).

To examine variations in reported implementation by demographic characteristics, we will classify the schools in four ways: those that are more than 50 percent Latino (28 schools), those that are more than 50 percent Native American (50 schools), those that are more than 50 percent White (68 schools), and those that do not have a single racial majority (64 schools).² We will use Kruskal Wallis H tests to examine variations in the data on implementation by student demographic characteristics. We will conduct the H tests for each item’s response category by each of the four demographic categories, although any cells with fewer than five responses will be eliminated from the analysis. In each analysis, the ordinal Likert scale responses for the survey item will serve as the dependent variable. The independent or “grouping” variable will include the four demographic categories. If the H test is statistically significant, we will perform a post hoc Dunn’s test to determine which groups differed from one another. We will display mean ranks, degrees of freedom, H test statistics, and Dunn’s test statistics for each analysis in a technical appendix. To determine statistical significance we will use the Benjamini-Hochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995).

To examine variations in reported assistance by demographic characteristics, we will also use Kruskal Wallis H tests. As in the previously described analysis of implementation, we will consider assistance from each type of provider (district, state, university, and other technical assistance providers) separately. We expect that in many instances we will need to eliminate analysis for cells with fewer than five responses. In each analysis, the ordinal Likert scale responses for the survey item will serve as the dependent variable. The independent or “grouping” variable will include the four demographic categories. If the H test is statistically significant, we will perform a post hoc Dunn’s test to determine which groups differed from one another. We will display mean ranks, degrees of freedom, H test statistics, and Dunn’s test statistics for each analysis in a technical appendix. To determine statistical significance we will use the Benjamini-Hochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995).

Publications

For this study, we will create two publications. A What’s Happening? report will be targeted to our national practitioner and policymaker audience, particularly state agencies, districts, universities, and other technical assistance providers involved in supporting school improvement efforts similar to those in SIG transformation schools. This report will contain graphic summaries of the study and its findings.

A Stated Briefly report will summarize findings that may appeal to regional and state organizations that are interested in improving schools but that do not provide direct technical assistance to schools—particularly those organizations that set school improvement policies and guidance. The Stated Briefly report should help these organizations set policy and create guidance that supports aspects of transformation that fewer participants reported were fully implemented, as well as potentially reducing or remediating such reported challenges.

Both What’s Happening and Stated Briefly reports will be available on the REL Northwest website and the IES website. As stated earlier, geographic and demographic analyses will not be conducted when fewer than five respondents are included. This will reduce the risk of deductive disclosure.

Timeline

The timeline for the study, including the publications, is presented in table 2.

Table 2. Timeline

Timeline	Milestone	Date for deliverable
January 2013–April 2013	Develop data collection protocol and instruments	NA
April 2, 2013	Submit draft study plan to TWG, which includes study methodology, data collection activities, and analysis plan	NA
April 30, 2013	Submit draft study plan to IES, revised in response to TWG feedback	Draft plan to IES April 30, 2013
July 24, 2013	Submit final study plan to IES	Final plan to IES June 14, 2013
October 2013	OMB submission of 60-day Federal Register Notice	NA
March–May 2014	Data collection	NA
June 2014	Data analysis	NA
July 2014	Write reports	NA
September 2014	Submit draft What’s Happening? report to IES	Draft What’s Happening? report, 12 months after approval
November 2014	Submit final What’s Happening? report to IES	Final What’s Happening? report, 13–14 months after approval
December 2015	Submit draft Stated Briefly to IES	Draft Stated Briefly, 15 months after approval
January 2015	Submit final Stated Briefly to IES	Final Stated Briefly, 16 months after approval

A17. Approval to not display the expiration date for OMB approval

This is not applicable. The instrument will display the expiration date for OMB approval.

A18. Exception to the certification statement

We do not seek any exceptions to the certification statement.

References

Armstrong, S.J. (1975). Monetary incentives in mail surveys. Public Opinion Quarterly, 39(1), 111–116.

Battle, D. (2009). Characteristics of public, private, and Bureau of Indian Education elementary and secondary school principals in the United States: Results from the 2007–08 Schools and Staffing Survey (First Look, NCES 2009-323). Retrieved from U.S. Department of Education, National Center for Education Statistics website: http://nces.ed.gov/pubs2009/2009323.pdf

Battle, D. (2010). Principal attrition and mobility: Results from the 2008–09 principal follow-up survey (First Look, NCES 2010-337). Retrieved from U.S. Department of Education, National Center for Education Statistics website: http://nces.ed.gov/pubs2010/2010337.pdf

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1), 289–300.

Church, A.H. (1993). Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly, 57(1), 62–79.

Corbett, J. (2011). Lead turnaround partners: How the emerging marketplace of lead turnaround partners is changing school improvement. Lincoln, IL: Academic Development Institute, Center on Innovation and Improvement.

Dee, T. (2012). School turnarounds: Evidence from the 2009 stimulus (NBER Working Paper No. 17990). Cambridge MA: National Bureau of Economic Research.

Fixsen, D.L., Blase, K.A., Naoom, S.F., & Wallace F. (2009). Core implementation components. Research on Social Work Practice, 19(5), 531–540.

Hammer, P.C., Hughes, G., McClure, C., Reeves, C., & Salgado, D. (2005). Rural teacher recruitment and retention practices: A review of the research literature, national survey of rural superintendents, and case studies of programs in Virginia. Nashville, TN: Edvantia, Appalachia Educational Laboratory.

Hurlburt, S., Le Floch, K.C., Therriault, S.B., & Cole, S. (2011). Baseline analyses of SIG applications and SIG-eligible and SIG-awarded schools (NCEE 2011-4019). Retrieved from U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance website: http://ies.ed.gov/ncee/pubs/20114019/pdf/20114019.pdf

Hurlburt, S., Therriault, S.B., & Le Floch, K.C. (2012). School Improvement Grants: Analyses of state applications and eligible and awarded schools (NCEE 2012-4060). Retrieved from U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance website: http://ies.ed.gov/ncee/pubs/20124060/pdf/20124060.pdf

Jacob, R.T., & Jacob, B. (2012). Pre-notification, incentives, and survey modality: An experimental test of methods to increase survey response rates of school principals. Journal of Research on Educational Effectiveness, 5(4), 401–418.

Jacob, R.T., Scott, L., & Bowers, A. (2008, April). Evaluation and implementation of web-surveys in school-based settings. Paper presented at the annual meeting of the American Educational Research Association, New York, NY.

James, J.M., & Bolstein, J. (1992). The effect of monetary incentives and follow-up mailings on the response rate and response quality in mail surveys. Public Opinion Quarterly, 54(3), 346–361.

Kaplowitz, M.D., Hadlock, T.D., & Levine, R. (2004). A comparison of web and mail survey response rates. Public Opinion Quarterly, 68(1), 94–101.

Klein, A. (2010, May 21). School turnaround models draw bipartisan concern. Education Week, 29(33). Retrieved from http://www.edweek.org/ew/articles/2010/05/21/33turnaround.h29.html

Mayring, P. (2000). Qualitative content analysis. Forum: Qualitative Social Research, 1(2). Retrieved from http://www.qualitative-research.net/index.php/fqs/article/view/1089/2386

Millar, M.M., & Dillman, D.A. (2011). Improving response to web and mixed-mode surveys. Public Opinion Quarterly, 75(2), 249–269.

Monk, D.H. (2007). Recruiting and retaining high-quality teachers in rural areas. Future of Children, 17(1), 155–174.

No Child Left Behind Act of 2001. (2002). Pub. L. No. 107-110, 115 Stat. 1425.

Owens, J.S., Richerson, L., Murphy, C.E., Jageleweski, A., & Rossi, L. (2007). The parent perspective: Informing the cultural sensitivity of parenting programs in rural communities. Child & Youth Care Forum, 36(5/6), 179–194.

Rosenberg, S.J. (2011). The challenges in implementing school improvement grant models in rural high schools (Master’s thesis). Duke University, Durham, NC.

Sandel, K., & Bhat, S. (2008). Financing and sustaining out-of-school time programs in rural communities. Retrieved from Finance Project website: http://www.financeproject.org/publications/ostruralyouth.pdf

Scott, C., Davis, A., & Krasnoff, B. (2012). Oregon School Improvement Grants: Status report 2011–2012. Portland, OR: Education Northwest.

Scott, C., McMurrer, J., McIntosh, S., & Dibner, K. (2012). Opportunities and obstacles: Implementing stimulus-funded School Improvement Grants in Maryland, Michigan, and Idaho. Washington, DC: Center on Education Policy.

Scott, G. (with Sirois, E., Spicer, S., Arsenault, J., King, M., Sorbello, S., & Steel-Lowney, B.). (2012). School Improvement Grants: Education should take additional steps to enhance accountability for schools and contractors (GAO-12-373). Washington DC: U.S. Government Accountability Office.

Sloan, M., Ingels, J., & Burghardt, J. (2012). Analytical and technical support for advancing education evaluation: How to put together an OMB supporting statement. Princeton, NJ: Mathematica Policy Research.

U.S. Department of Education. (2012, June 7). ESEA flexibility (Updated ed.). Retrieved from http://www.ed.gov/esea/flexibility/documents/esea-flexibility.doc

U.S. Department of Education, Office of Elementary and Secondary Education. (2011). Guidance on fiscal year 2010 School Improvement Grants under section 1003(g) of the Elementary and Secondary Education Act of 1965. Retrieved from http://www2.ed.gov/programs/sif/sigguidance02232011.pdf

Zhang, Y. (2008). Some perspectives from rural school districts on the No Child Left Behind Act. Retrieved from Center on Education Policy website: http://www.cep-dc.org/displayDocument.cfm?DocumentID=308

1 In order to ensure that each survey item represents only one SIG transformation activity, we split several activities into multiple survey items. To ensure that the principal is knowledgeable about all activities, we eliminated those activities that are district rather than principal responsibilities, such as principal replacement and evaluation. We also eliminated the activity, “Ensure that the school receives ongoing, intensive technical assistance and related support from the LEA, the SEA, or an external organization” because we ask principals about this assistance in a separate section of the survey.

2 One school had missing data for student ethnicity in the SIG baseline database.