Pilot Report

Attachment F: Pilot Survey Report

REL Northwest conducted a pilot test of the survey between October 23 and November 8. The initial invitation to complete the online survey went out to the sample of nine principals on October 23, 2013. Principals were instructed to complete the survey by October 30, 2013. After the initial email invitations were sent, two of the emails were undeliverable and one principal declined to participate. Through follow-up calls to schools and districts, REL Northwest discovered that one email was undeliverable because the school had closed and the other because the principal was no longer at the school. We then drew two new schools for our sample. Staff sent invitations to the principals of these two new schools, as well as to the new principal of the third school.

By October 28, two days before the pilot test deadline, only two of the nine principals in the pilot study had responded. We then provided two reminders to complete the survey. First, we sent an email reminder to the seven principals to complete the survey and extended the submission deadline to November 1. Second, we also called schools to urge principals to complete the survey and provide feedback, but it had little success directly reaching principals. In most cases, REL Northwest staff spoke with school secretaries and left voicemail messages for principals.

By October 31, one day before the extended deadline, four principals had completed the survey (a return rate of 44 percent) and one had partially completed it. None of these principals had provided feedback as requested. REL Northwest staff decided to follow up with the principals who had already completed the survey to ask them for their feedback and to provide $50 Amazon gift cards as an incentive. (Note that this amount is slightly higher than the $20 we will offer as an incentive during the actual survey, because the principals in the pilot not only completed the survey but also provided feedback, which took more time than the actual survey will take). In addition to emailing responding principals, we also emailed the nonresponding principals offering the incentive and asking them to complete the survey and provide feedback via email by November 8. The principal who had partially completed the survey was asked to complete it and to provide feedback.

Several principals responded positively to the incentive. Of the five principals who had completed or started the survey when we decided to offer the incentive, three provided feedback by email or by phone. In addition, the principal who had only partially completed the survey, finished it. Two others completed the survey a few days later, one of whom provided helpful feedback.

REL Northwest staff closed the survey on November 8. Seven of nine principals responded to the survey pilot test, for a response rate of 77 percent. Four of the seven principals provided the following comments about their experience taking the survey:

1. The SIG-related jargon was not a problem.

2. The response options were complete and appropriate.

3. One principal was confused about how to answer the questions about receiving technical assistance from a university, as they received assistance from an organization affiliated with a university.

4. One principal felt that the survey should have provided opportunities to share other school improvement activities. For example, work addressing literacy, school climate, and mainstreaming of special needs students are important school improvement efforts, but there was not an opportunity in the survey to highlight such work.

5. Two principals had difficulty finding the time to complete the survey. The length of the survey was not burdensome but their other responsibilities came first.

Principal comments one and two required no revisions to the survey. Principal comment three required no revision, since the survey already allowed the principals to respond about the technical assistance form the organization affiliated with a university under the “other” category. To address principal comment four, we moved an open-ended item from the end of the survey to near the beginning, which gives principals a more direct opportunity to enter information about other school improvement activities. Principal comment five (drawn from two principals) required no response; however, we have added an incentive to complete the survey, which we believe will increase response rates, as it did in the pilot survey.