NW S07 - SIG Rural Schools OMB Supporting Statement Part B

CONTRACT: ED-IES-12-C-0003

Supporting Statement for OMB Clearance of a Survey of Principals of Rural Schools Receiving School Improvement Grants and Using the Transformation Model

Section B

August 16, 2013

Revised September 6, 2013

Revised October 8, 2013

Revised October 16, 2013

Revised December 13, 2013

Contents

Section B: Collection of Information 1

B1. Respondent Universe and Sampling Methods 1

B2. Procedures for the Collection of Information 2

B3. Methods To Maximize Response Rates and Deal With Nonresponse 4

B4. Tests of Procedures or Methods to be Undertaken 5

B5. Individuals Consulted on Statistical Aspects of the Design 5

References 6

List of tables

Table 1. Characteristics of rural SIG transformation schools vary in Cohort 1……………………..2

The following attachments are stored in the “NW S07” folder in the “REL Northwest Memos and Correspondence” library:

Attachment B: Copy of Survey and Online Survey Screen Shots

Attachment C: Intro Letter Invitation and Follow-Up Emails

Attachment F: Pilot Survey Report

Supporting Statement for OMB Clearance of a Survey of Principals of Rural Schools Receiving School Improvement Grants and Using the Transformation Model

Section B: Collection of information

B1. Respondent universe and sampling methods

The Regional Educational Laboratory (REL) Northwest, with assistance from Policy Studies Associates (PSA), plans to examine how rural schools implement School Improvement Grants (SIGs). These competitive grants have provided two cohorts of low-performing schools with federal funding for school improvement and require schools to implement one of four school improvement models: the transformation model, which involves a variety of reforms; the turnaround model, which involves primarily replacing staff; the restart model, which involves becoming a charter school; and the closure model, which closes the school. Collecting data from rural SIG schools is essential to this study, which seeks to inform regional and national supports for future school improvement efforts.

REL Northwest was established under the 2012–2017 Regional Education Laboratories Program. The current authorization for the program is under the Education Sciences Reform Act of 2002, Part D, Section 174, (20 U.S.C. 9564), administered by the Institute of Education Sciences’ (IES’) National Center for Education Evaluation and Regional Assistance. IES has approved this study plan that collects principal (or proxy) survey data about the implementation of the transformation model under SIGs.

The universe for this descriptive study is relatively small (i.e., 211 schools across the nation). In order to have a large enough number of respondents to examine the data disaggregated by school characteristics, such as geographic location and student demographics, this study surveys the entire universe of rural Cohort 1 SIG schools (N = 211) and, therefore, does not use any sampling procedures. We identified these 211 Cohort 1 rural schools across the nation that implemented the SIG transformation model using the SIG baseline database and the National Center on Education Statistics preliminary 2011 data. Hurlburt, Le Floch, Therriault, and Cole (2011) compiled the SIG baseline database based on publicly available information. The SIG baseline database is available on the U.S. Department of Education website. We defined rural schools as those given locale codes 31 through 43 by the National Center for Education Statistics (NCES)¹.

Because SIG funding goes to schools serving large numbers of low-income students (i.e., those eligible for free or reduced-price lunch), rural SIG transformation schools have high levels of poverty; however, these schools vary considerably by ethnicity (table 1).

Table 1. Characteristics of rural SIG transformation schools vary in Cohort 1

	Number of schools	Percentage non-White	Percentage Native American	Percentage free or reduced-price lunch
31 Town, Fringe	6	41%	0%	47%
32 Town, Distant	33	60%	5%	68%
33 Town, Remote	16	75%	18%	81%
41 Rural, Fringe	48	70%	7%	72%
42 Rural, Distant	62	55%	16%	71%
43 Rural, Remote	46	77%	72%	78%
31–43 All Schools	211	65%	24%	72%

Source: Author’s analysis of SIG baseline database.

The schools are spread throughout the United States. Of the 50 states, 42 had Cohort 1 rural SIG transformation schools. California and three southern states had the largest number of rural SIG transformation schools: 19 schools in Florida, 12 in California, 12 in South Carolina, and 10 in North Carolina. The remaining states had fewer than 10 rural SIG transformation schools.

These survey data have not been collected previously. We expect a response rate of 75 percent. More details about the response rate are included in B3. The survey is included in attachment B.

B2. Procedures for the collection of information

This one-time data collection by REL Northwest and PSA seeks responses from the universe of 211 rural SIG schools in Cohort 1 implementing the transformation model. Therefore, we will not use any sampling methods. Our procedure for collecting information includes several steps: a pilot survey, an initial invitation letter, the actual survey, and up to five follow-up contacts.

Between October 23 and November 8, 2013, we conducted a pilot of the survey instrument with seven principals from rural SIG schools that used the transformation model and received Cohort 1 funding. This pilot allowed us to analyze pilot participant responses item by item and revise the survey to ensure that all items and instructions were relevant and easily understood. The pilot also allowed us to test and revise our data collection procedures. A memo describing the pilot is included in attachment F, and more information about the pilot is in section B4. Cohort 1 principals who participated in the pilot will have the opportunity to update their online pilot survey responses when the actual survey is administered.

After we receive OMB clearance, we will begin actual data collection. To conduct the survey, we will first send an introductory letter to superintendents and school turnaround leaders in state departments of education, superintendents of districts, and principals of rural SIG transformation schools (attachment C). The letter emphasizes the importance of the study. To provide further incentive to complete the survey, the letter informs recipients that those who respond to the survey will receive a link to the published study and will be invited to participate in multiple webinars to discuss the study results. The letter will also inform respondents that REL Northwest/PSA will email principals an online survey link within one week.

Finally, the letter informs recipients that principals who are unfamiliar with SIG at their current schools will only fill out the beginning of the survey. This portion of the survey asks if the principal is familiar with SIG implementation at his or her school. If the principal is not familiar with SIG implementation, the survey provides instructions for designating a proxy to complete the survey and ends without asking any remaining survey items. The letter also explains that current principals who were at their schools during the time period being studied will be allowed to designate a proxy if it will help the school complete the survey promptly. They can do so by responding to the email.

One week after sending the introductory letter, we will email an invitation to Cohort 1 rural SIG school principals inviting them to complete the online survey. The online survey will be available via Survey Gizmo, a survey administration software program. We will use procedures allowing respondents to log in and out of their survey as often as needed to complete it. We will send principals two reminders to complete the survey using this online format.

If return rates are less than 50 percent after the second reminder to complete the survey, we will offer a gift card to increase the response rate. We will do this for three reasons. First, our pilot showed that offering an incentive (an Amazon electronic gift card) after several reminders increased the response rate from 44 percent to 77 percent. Several research studies support the use of incentives (Armstrong, 1975; Church, 1993; James & Bolstein, 1992), particularly for online surveys of principals (Jacob & Jacob, 2012). Principals who completed the survey before the second reminder will be sent the incentive retroactively.

After three reminder emails, nonrespondents will be offered a Word doc version of the survey in case they prefer responding to this version of the survey. They may return the Word document via fax or email. If participants wish, they may request a paper document in the mail, and we will provide a paper document that is saddle stitched to avoid the pages becoming separated. All versions should take approximately 20 minutes to complete, based on our survey pilot. We will maintain all survey data in our password-protected servers under secure conditions. We will conduct up to five follow-up emails and/or telephone calls to nonrespondents to achieve at least an 80 percent response rate. If the principal is willing we will conduct the survey via phone, in order to reach the 80 percent response rate.

To clean the survey data, we will import all data into SPSS software. Our initial cleaning efforts will focus on identifying principals for whom we have missing data. We expect missing data to be minimal since the online version of the survey will prompt principals to fill in any missing responses before they submit the survey. However, some principals may opt to take the paper version, so these surveys may contain missing data. If this is the case, we will contact principals to fill in these missing data. Although we will encourage principals to complete all items on the survey and will explain its importance, their participation is voluntary. We understand that some principals may decline to respond to all of the survey items.

Once the missing data are retrieved, we will continue our data processing. Our survey data-processing procedures will include a visual review of any word document surveys, data editing as necessary, and a review of initial frequencies to ensure that responses are within acceptable ranges and there are no data entry errors.

B3. Methods to maximize response rates and deal with nonresponse

Our target response rate is 80 percent. We have set this target based on other similar studies. For example, the most recent Schools and Staffing Survey (SASS) of principals by the National Center of Education Statistics had a response rate of 79 percent for both public school and Bureau of Indian Education principals (Battle, 2009). We suspect that response rates would be somewhat different for rural school principals. For example, PSA assisted with data collection for a survey of rural district administrators, which achieved a response rate of 74 percent (Zhang, 2008). Our population is principals in low-performing rural schools. These principals may have more job stress and be less likely to respond than those in nonrural, higher performing schools. Therefore, because we set our target at 80 percent, we will use several methods to maximize response rates and address nonresponse.

First, we have minimized the burden for respondents by (1) requesting information that is generally available to school leadership, including principals, other administrators, and teacher-leaders directly involved in SIG implementation and (2) requesting the minimum amount of information required to answer the study questions.

Second, as discussed in B2, we will send an initial introductory letter to superintendents and school turnaround leaders in state departments of education, superintendents in districts, and principals in selected rural SIG transformation schools. Several other studies have found these introductory letters improved survey response rates (Cook, Heath, & Thompson, 2000; Kaplowitz, Hadlock, & Levine, 2004). We will also send reminder emails as described previously. If needed after multiple reminders, we are prepared to administer the survey by phone.

Third, we will offer an incentive to principals who have not completed the survey after two reminders. Several research reviews show that incentives often increase response rates (Armstrong, 1975; Church, 1993; James, & Bolstein, 1992). More importantly, a recent experimental study of an online principal survey showed that principals’ response rates, in particular, were statistically significantly higher when principals were offered an incentive (Jacob & Jacob, 2012).

Finally, as discussed previously, principals who are new to their schools or who are too busy to complete the survey may designate a proxy. At some SIG schools, district or teacher leaders are very involved with SIG implementation (Scott, McMurrer, McIntosh, & Dibner, 2012). These individuals may have more time to respond to the survey and, therefore, increase return rates.

B4. Tests of procedures or methods to be undertaken

Survey items in this data collection are adapted from a previously conducted survey as part of an evaluation of SIG in Oregon (Scott, Davis, & Krasnoff, 2012). In addition, as described previously, we conducted a pilot of the survey instrument from October 23 through November 8, 2013. We randomly selected nine principals of rural SIG schools that received their SIG funds in Cohort 1. Seven principals responded by November 8, 2013. The pilot followed the survey procedures described in this document for the actual study. Additionally, pilot participants were asked to comment on whether the survey correspondence clearly conveyed the purpose of the study and whether the survey instructions and items were clear and easy to follow. Cohort 1 principals who participated in the pilot will have the opportunity to update their online pilot survey responses when the actual survey is administered. A more detailed description of the pilot is included in attachment F.

B5. Individuals consulted on statistical aspects of the design

Two technical experts will play an important role in providing insight and guidance in support of this study of the rural SIG transformation model. They include:

• Tom Dee, Professor, Graduate School of Education, Stanford University, 650.723.6847

• Hans Bos, Vice President, Education, American Institutes for Research, 707.701.3004

Both individuals are members of REL Northwest’s Technical Working Group (TWG). The main responsibility of TWG members is to provide written feedback on study plans and/or study reports and to meet as a group via telephone with the research team to clarify feedback and provide additional recommendations. The TWG members also are available to advise the researchers as questions come up, speak with REL staff about future research project ideas, and generally lend their expertise as needed. All TWG members are experts who are distinguished in their fields and extremely knowledgeable about research design and methods.

References

Armstrong, S.J. (1975). Monetary incentives in mail surveys. Public Opinion Quarterly, 39(1), 111–116.

Battle, D. (2009). Characteristics of public, private, and Bureau of Indian Education elementary and secondary school principals in the United States: Results from the 2007–08 Schools and Staffing Survey (First Look, NCES 2009-323). Retrieved from U.S. Department of Education, National Center for Education Statistics website: http://nces.ed.gov/pubs2009/2009323.pdf

Church, A.H. (1993). Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly, 57(1), 62–79.

Cook, C., Heath, F., & Thompson, R.L. (2000). A meta-analysis of response rates in web- or Internet-based surveys. Educational and Psychological Measurement, 60(6), 821–836.

Hurlburt, S., Le Floch, K.C., Therriault, S.B., & Cole, S. (2011). Baseline analyses of SIG applications and SIG-eligible and SIG-awarded schools (NCEE 2011-4019). Retrieved from U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance website: http://ies.ed.gov/ncee/pubs/20114019/pdf/20114019.pdf

Jacob, R.T., & Jacob, B. (2012). Pre-notification, incentives, and survey modality: An experimental test of methods to increase survey response rates of school principals. Journal of Research on Educational Effectiveness, 5(4), 401–418.

James, J.M., & Bolstein, J. (1992). The effect of monetary incentives and follow-up mailings on the response rate and response quality in mail surveys. Public Opinion Quarterly, 54(3), 346–361.

Kaplowitz, M.D., Hadlock, T.D., & Levine, R. (2004). A comparison of web and mail survey response rates. Public Opinion Quarterly, 68(1), 94–101.

Scott, C., Davis, A., & Krasnoff, B. (2012). Oregon School Improvement Grants: Status report 2011–2012. Portland, OR: Education Northwest.

Scott, C., McMurrer, J., McIntosh, S., & Dibner, K. (2012). Opportunities and obstacles: Implementing stimulus-funded School Improvement Grants in Maryland, Michigan, and Idaho. Washington, DC: Center on Education Policy.

Zhang, Y. (2008). Some perspectives from rural school districts on the No Child Left Behind Act. Retrieved from Center on Education Policy website: http://www.cep-dc.org/displayDocument.cfm?DocumentID=308

1 NCES has developed locale codes to describe a school’s location ranging from “large urban” to “remote rural.” The codes are based on the physical location represented by an address that is matched against a geographic database maintained by the Census Bureau.