OMB Submission Part A 5-17-07

Supporting Statement for Paperwork Reduction Act Submissions

REL-Northwest Regional Needs Assessment Survey of Superintendents, Principals, and Teachers in a Five-State Area

OMB Control No: 1850-New, EDICS# 3237

Request for OMB Review – Part A

OMB Form 83-I and Supporting Statement for Data Collection

Submitted by:

Regional Educational Laboratory-Northwest

Portland, Oregon

May 2007

INTRODUCTION

The Regional Educational Laboratory Northwest (REL-Northwest) is requesting approval to collect information from a random sample of educators in the Pacific Northwest Region, including educators in the states of Alaska, Idaho, Montana, Oregon, and Washington. We are requesting a clearance to collect initial survey information in Year 2 of our contract (2007) and again in Year 4 (2009).

The goal of the initial needs assessment survey is to establish a road map by which to plan programs and set a meaningful research agenda to address state and regional educational needs. The goal of the follow-up survey in Year 4 is to assess similar questions from the same population, but with the express purpose of re-evaluating changes in the priorities and to plan future research and action. Therefore, the supporting statement below, describes the overarching process we will use for the project and details the survey planned for 2007, with reference to commonalities between it and the survey planned for Year 4 (2009).

A. Justification

A1. Rationale for Collecting Information

The proposed information collection is mandated in the Education Sciences Reform Act of 2002 (ESRA), Public Law 107-279, Section 174(f)(3) which states that “each regional educational laboratory awarded a contract under this section shall support applied research, development, wide dissemination, and technical assistance activities by developing a plan for identifying and serving the needs of the region by conducting a continuing survey of the educational needs, strengths, and weaknesses within the region.”

Further justification is found in Section 1.1—Regional Education Needs Analysis, Training and Technical Assistance Response Unit of the Institute for Education Sciences’ 2005 Statement of Work/Scope of Work for the Regional Educational Laboratory contracts (ED-05-R-0006) implementing the above quoted section of ESRA which states that “the contractor shall develop a Needs Analysis, Training and Technical Assistance Response Unit responsible for collecting information on the region’s needs for school improvement. The contractor shall assess regional needs regularly through surveys and contractor-developed outreach strategies to solicit comments from teachers and from district and state policymakers and administrators for pressing concerns that need attention.”

Outreach strategies used in addition to the surveys include hearings conducted in Years 1 and 3 in each of the five states REL-Northwest serves. The purpose of the hearings is to provide a forum for educators and education policy-makers to identify their most pressing informational needs. Outcomes from these hearings feed directly into the items on the Year 2 and Year 4 survey instruments. The hearings help REL-Northwest identify they common information needs across the region, prioritize these needs and establish a research program to address them. Another form of input is through REL-Northwest’s CRARS system (Contacts, Referrals, Activities, and Reporting System). This system keeps the Lab in touch with educators who have received Lab services and provides a mechanism whereby educators can provide feedback to the Lab about services received and request additional services from REL-Northwest. This system also allows REL-Northwest a way to solicit input on what additional services and information educators need.

A third justification is found in the implementation requirements from the REL Contract (as stated in the “Regional needs survey in Years 2 & 4” section of the Revised Fast Response Plan, Task 1.1 – April 12, 2006): “REL-Northwest will conduct a regional needs assessment survey in Years 2 and 4 for a random, representative sample of the region’s educators – including teachers, principals, and superintendents – to provide a systematic, quantitative set of perceived needs for evidence that can be prioritized on the basis of their ratings.” This information will be used as a component of the needs analysis and the survey results will help to describe regional needs.

In addition to fulfilling the requirements described above, there are several practical reasons for conducting the surveys. First, schools, districts and state education agencies are working in an environment that is changing relatively quickly as requirements of NCLB phase in and accountability plans are amended to reflect new flexibility or new restrictions in specific areas (e.g., science standards and assessments are required to be in place starting in 2008). Second, NCLB-required performance objectives are being raised substantially and at regular intervals toward a goal of 100% proficiency by 2014. Third, the surveys are being used for program planning. The Year 2 survey will inform the REL-Northwest research agenda enabling REL-Northwest to meet the most pressing informational needs of regional educators. The second survey in Year 4 serves as a check on this agenda and allows REL-Northwest to be responsive if informational priorities change.

A2. Purpose and Use of Information Collected

This will be a new data collection. Respondents will be district superintendents, principals, and teachers from Alaska, Idaho, Montana, Oregon and Washington. The survey is designed to elicit information from Northwest educators regarding the key challenges their schools face in improving student learning and the types of research evidence that will assist them to best address these challenges. Surveying will take place in Years 2 and 4 to gather information to follow-up discussions with administrators and policy-makers held previously in each of the five states REL-Northwest serves.

The information will also be used to help set the research agenda and program planning for the REL-Northwest. REL-Northwest will use the findings to: 1) Identify promising practices that can be evaluated to determine their impact on student learning and the extent to which they can be successfully adapted for diverse educational environments, 2), Identify issues which call for rigorous policy analyses that can guide states in developing sound educational policies to address the major challenges schools face and 3) Inform policy discussions among state and local education leaders and state policy makers.

A3. Use of Electronic Data Collection Techniques

The educators selected for the survey will be issued a printed survey and provided a link to a secure website should they wish to complete the survey electronically rather than on paper. The content of the survey instruments for either method of responding will be identical. Each potential respondent will be assigned a unique identification number. This number will appear on the printed surveys. Educators opting to complete the survey online will enter the identification number to access the survey database. Identification/PIN numbers will be used to check-in responses and determine the need for follow-up contacts. Unique ID numbers will be removed from the survey database at the conclusion of the data collection period and prior to commencing any tabulations or analysis.

Parallel surveys for teachers, principals, and district superintendents have been developed. Time estimates were based on the amount of time it took respondents to complete a paper and pencil version of the study. Because the surveys are identical, it was estimated the online version will take about the same amount of time to complete, estimated to be between 15 and 20 minutes.

Giving respondents the option of completing the survey by mail or electronically via the web allows them to choose the method that will be most convenient for them to use and thus potentially lowers the burden and maximizes the potential for each respondent to complete the survey. Moreover, data collected electronically reduces the amount of time needed for data entry and minimizes error associated with data entry. The electronic version of the survey will have fillable fileable capability, meaning surveys can be completed online and submitted electronically.

We estimate between 25% and 50% of surveys completed will be completed online.

A4. Efforts to Identify Duplication

REL-Northwest has systems in place to allow for on-going dialogue with regional contacts in various roles and settings to alert the Lab to any data collections by state agencies, institutions of higher education and professional associations. These contacts have not revealed any similar efforts. At the Year 1 hearings when REL-Northwest explained to educators and policy-makers that findings from the discussion groups would be used to inform this data collection effort, no one in attendance was aware of similar studies either in the planning stages or currently underway. Nor have educators suggested that similar studies were underway through the CRARS system described above.

Secondary research conducted by REL-Northwest has revealed that minimal research has been done to accurately identify the information needs of Northwest educators and/or project these needs to the population of educators in this region.¹ We are not aware of any data collection efforts focused on evidentiary needs of educators in the Northwest. Thus, we will not be duplicating information that federally funded programs are already reporting.

A5. Minimizing Burden for Small Schools

Information will be collected from educational professionals, including teachers, principals, and district superintendents working in various school districts in a five-state region. We recognize that participating in a survey can place a burden on small schools and districts because of few personnel and many demands on time. Since we are using a random sampling technique the impact on teaching staff will be proportionate to school size, and so will not place proportionately more burden on staff of smaller schools.

We have designed the surveys to reduce respondent burden by developing a parsimonious questionnaire focused only on the essential information needed, and easy-to-use questions that ask respondents to prioritize, by assigning “points,” to four items within each of 11 topic areas. We have also limited open-ended questions to short answers to reduce respondent burden and yet allow for information or ideas the educators have that are not covered by the questions in the 11 topic areas.

A6. Consequences of Not Collecting the Information as Proposed

If the proposed information is not collected it will be difficult, if not impossible, to determine the priorities Northwest educators have for evidentiary needs and thus difficult to set a research agenda for developing information targeted most effectively to improving educational programs that will ultimately improve student achievement. In addition, it is important to survey each group of educators—teachers principals, and superintendents—to achieve a balanced picture of priorities and needs.

A7. Special Circumstances

No special circumstances exist for this data collection.

A8. Soliciting Public Comments

The Institute for Education Sciences (IES) will place the announcement concerning this information collection in the Federal Register. Public comments on the design of the collection, the survey questions used, and the collection process will be solicited at that time.

There were hearings with educators and education policy makers in each of the five states before the survey was designed. Educators were encouraged to share and discuss their needs and experiences on a variety of educational topics unique to their state. The hearings were not designed to be Q & A between participants and moderators as is normal in a focus group setting, nor were structured questions used as is typical in a survey project. A review of notes from the five hearings revealed patterns of information common to all states REL-Northwest serves. These patterns were used to develop the initial set of items that were then cognitively tested.

A9. Payment or Gifts to Respondents

No incentives will be provided in conjunction with this data collection effort.

A10. Assurance of Confidentiality

REL-Northwest and Gilmore Research follow the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). REL-Northwest and Gilmore Research will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Information from participating institutions and respondents will be presented at aggregate levels in reports. Information on respondents will be linked to their institution but not to any individually identifiable information. No individually identifiable information will be maintained by the study team. All institution-level identifiable information will be kept in secured locations and identifiers will be destroyed as soon as they are no longer required.

All members of the study team having access to the institution-level data have been certified by REL-Northwest’s Institutional Review Board as having received training in the importance of confidentiality and data security.

A11. Sensitive Questions

The questions asked on this survey are not considered to be of a sensitive nature. Since participation is completely voluntary and respondents are randomly selected, respondent risk is very slight. The study asks only for opinions, so there are no right or wrong answers. No one other than the respondent knows he/she has been selected for the survey so there is neither risk of retaliation for responses provided or risk of pressure to participate or not participate in the study. This conclusion is confirmed by an exemption from IRB approval.

A12. Estimates of Hour Burden for Collecting Data

This survey seeks to collect data from teachers of core subjects, principals, and school district superintendents in Alaska, Idaho, Montana, Oregon and Washington. The estimated number of survey returns is based on prior REL-Northwest experience conducting paper surveys with these populations, but with more intensive follow up to non-responders than in the past. The survey is designed to meet OMB Standard 1.3 which states:

“Agencies must design the survey to achieve the highest practical rates of response, commensurate with the importance of survey uses, respondent burden, and data collection costs, to ensure that survey results are representative of the target population so that they can be used with confidence to inform decisions.” ²

Estimated response rates are 80% for each population. The hourly burden shown in the table below is based on the expected number of responses and from timing a paper and pencil pretest with three principals and four teachers. This test had a mean response time of 16 minutes. Time was added to take into consideration the amount of time required to open and read advance letters, survey instructions and reminder postcards as well as to fold, seal and mail the survey in the return envelope provided. The estimated total time per survey was 20 minutes from start to finish including reading instructions and placing the completed survey in the pre-paid business reply envelope. We estimate the time it will take the respondent to complete the survey online including logon/logoff time will be equivalent to the paper and pencil version.

Separate, but parallel surveys have been designed for teachers, principals, and superintendents. Estimates of total hours burden for all surveys are in the following table.

	Total Study Population	Sample Size	Expected Number of Responses (80%)	Number of Collections	Burden Hours per Respondent	Total Annual Burden Hours	Estimated Cost Burden*
Teachers of Core Subjects	65,189	2,500	2,000	1	.333	666	$ 19,314
Principals	4,688	1,500	1,200	1	.333	400	$ 15,600
Superintendents	862	825	660	1	.333	220	$ 8,800
Total	70,739	4,825	3,860			1,286	$ 43,714
*Based estimated hourly rates derived from annual median salary information obtained from CNNmoney.com. $29/hr for teachers, $39/hr for principals, and $40/hr for superintendents.

A13. Cost Burden for Respondents

There are no start-up costs or operation or maintenance costs to respondents and their school districts associated with this information collection.

A14. Annualized Cost to Federal Government

The total annualized cost of $280,795 includes the costs of developing survey items and methodology, drawing the sample of 4,825 educators and collecting and analyzing the data. Capital and start-up costs were determined by the research vendor under contract to conduct this study. Costs for printing and postage are pass-through costs. Costs for survey design, telephone calls to obtain sample lists, data entry, coding, cleaning, tabulation and analysis are estimates provided by the vendor and are based on prevailing industry rates.

Additionally, costs include project administration and oversight by REL-Northwest senior staff and an estimate of the hourly wages for respondents to complete the survey based on the estimated burden hours. Cost estimations are based on prior experience fielding similar self-administered surveys with the targeted survey populations.

COST CATEGORY
Senior REL-Northwest staff time: Dr James Leffler, Lucy Barnett and Richard Greenough	$36,200
Respondent hourly wages*	$43,714
Total capital/startup cost including printing and mailing pre-survey notification letters, surveys and cover letters, web survey programming, data entry, coding, cleaning, tabulation and analysis	$200,881
TOTAL SURVEY COST	$280,795
*Hourly rates assumed = $40 for superintendents, $39 for principals, $29 for teachers

A15. Program Changes or Adjustments

No changes or adjustments were noted.

A16. Plans for Data Analysis and Reporting

Superintendents Sample Frame

The number of districts with administrative superintendents (sources: NCES Core of Common Data and SEA websites) is as follows: Alaska: 53, Idaho: 114, Montana: 202, Oregon: 197, Washington: 296, for a total of 862. All district superintendents will be invited to complete a survey. Some districts share the same superintendents which reduces the size of the total pool from 862 to 825.

Principals Sample Frame

The number of principals in the population, by state, is estimated based on 2004 data (source: state provided lists of schools and administrators) is as follows: Alaska: 426, Idaho: 601, Montana: 487, Oregon: 1,150, Washington: 2,024, and the regional total: 4,688.

To select principals for the study, 300 schools will be randomly selected from school lists provided by each state. Schools without principals will be removed from the list before the random selection occurs.

Core Teachers Sample Frame

The number of teachers of core subjects in the population, by state from SY2004-05 (source: NCES, Common Core of Data, preliminary release) is estimated to be: Alaska: 4,424, Idaho: 8,477, Montana: 6,124, Oregon: 15,800, Washington: 30,363, and the regional total: 65,189.

For core teachers, REL-Northwest proposes using the SEA provided teacher lists that are available in Oregon, Washington, and Idaho. Prior experience with these lists has proven them to be reliable sample sources. In Montana and Alaska, where reliable contact information for teachers is more difficult to obtain, schools selected for participation in the study will be contacted individually to obtain lists of core teachers from which to choose study participants.

Core teachers will be selected using a two-stage stratification method. First, approximately 300 schools will be selected at random for each state. Teachers of core subjects in each of these schools will then be pooled and 500 teachers will be randomly selected to participate in the survey.

With this sample design we anticipate being able to generalize to teachers, principals and superintendents at both the regional and the state level. The data will not be examined below the state level for two reasons:

1. All districts have only one superintendent and many have only one principal. Thus, we cannot tie responses to the district level without violating IRB confidentiality requirements.

2. We believe the “Paperwork burden” would expand exponentially if it is necessary to obtain a representative sample by district with no gain in region-level information. The focus of this data collection is to guide regional level planning and strategy. While we may not be able to stratify to the district level, we will be able to detect patterns in evidentiary needs by subgroups such as minority enrollment, free-reduced lunch eligibility, and school size. We will know which districts are within each subgroup and so will be able to make some “educated guesses” about the needs in these districts.

In terms of general data analysis, we will examine the perceived importance of various types of evidence to inform educational practices measured through use of a point allocation system. Data from the surveys will be collated and variable distributions will be examined to identify outliers and inform the choice of analytic techniques. Mean scores will be tallied for each item that receives a point allocation and items will be rank ordered within each group and overall, based on mean scores.  The scores will provide not only a rank order, but also give some idea of the magnitude of importance educators in each role group (teachers, principals, and superintendents) place on each item. 

Items rated on a 1 to 10 scale will be reported as frequencies, mean ratings and collated into groups of high/medium/low. Cross-tabulation tables will be generated using WinCross software.  Anticipated descriptive statistics include frequencies, mean, median, mode, and standard deviations, and to make comparisons among groups we will use chi-square, student-t, z-tests of proportions, cluster analysis, and correlation techniques as appropriate to address our research questions and planning goals. Examples of tables that might be used are provided on the following pages.

Research products include a draft research report with separate analysis for each survey population for review and comment by REL-Northwest staff and a final report incorporating comments.

TEACHER SURVEY Q1a. Identifying the problem areas that impact student learning the most
	TOTAL	STATE					RURALITY		SCHOOL LEVEL			SCHOOL MINORITY
		AK	ID	MT	OR	WA	Urban	Rural	Elem.	Middle	High	Low	Intermediate	High
Total Responding	1,500	300	300	300	300	300
Top 3 (Net)	53%
Major problem (10)	16%
9	22%
8	14%
Middle 4 (Net)	38%
7	14%
6	9%
5	11%
4	4%
Bottom 3 (Net)	9%
3	4%
2	5%
Not a problem (1)	0%
Mean	8.23%
Standard Deviation	1.17%
Standard error	0.46%
Z-tests for proportions will be used to test for significant differences between subgroups.

TEACHER SURVEY Q3a. Improving School Attendance

Average

STATE

RURALITY

SCHOOL LEVEL

SCHOOL MINORITY

AK

ID

MT

OR

WA

Urban

Rural

Elem.

Middle

High

Low

Intermediate

High

Total Responding

300

300

300

300

300

Strengthening connections between home and school

6.39

Dealing with the effects of family culture on school attendance

4.72

Creating school as a welcoming place for students and families

2.74

Developing a safety net of support systems for students that promote improved attendance

3.91

Student T-tests will be used to compare means across subgroups

Timeline for Data Collection and Reporting
Contract Month	Project Task
December 8, 2006	Publish project documents on Federal Register (60 day)
December – February 2007	Federal Register survey posting duration
February – March 2007	Federal Register survey posting (30 day)
May 2007	OMB Approval
September 2007	Prepare for data collection by validating mailing addresses, printing pre-notification letters, survey instruments and preparing the mail out to Superintendents Collect teacher lists from for schools in Montana and Alaska.
October 15, 2007	Mail pre-notification letters to Superintendents
October 21, 2007	Mail out surveys and cover letters to Superintendents
October 22, 2007	Begin receiving data from Superintendents
October 28, 2007	Send reminder postcard to non-responding Superintendents Prepare for data collection by validating mailing addresses, printing pre-notification letters, survey instruments and preparing the mail out to Principals and Teachers
November 11, 2007	Mail out replacement survey and cover letter to non-responding Superintendents.
December 1, 2007	Send final (Priority Mail) survey packet
December 3 - 5, 2007	Make follow-up phone calls to Superintendents
December 21, 2007	Close data collection period for Superintendents
January 7, 2008	Mail pre-notification letters to Teachers and Principals
January 11, 2008	Mail out surveys and cover letters to Principals and Teachers
January 12, 2008	Begin receiving data from Principals and Teachers
January 18, 2008	Send reminder postcard to non-responding Principals and Teachers
February 1, 2008	Mail out survey and cover letter to non-responding Principals and Teachers
February 15, 2008	Send final (Priority Mail) survey packet
February 19-21, 2008	Make follow-up phone calls to Principals (and Teacher if possible)
March 3, 2008	Close data collection field period for Principal and Teachers
January – April, 2008	Edit, enter and clean data from paper surveys, generate descriptive statistics and other analyses
March – May 31, 2008	Prepare initial draft of reports and publications
June 2008	Submit final reports and publications

A17. Display of OMB Certificate

The OMB clearance number will be displayed on all surveys.

A18. Exceptions to Certification Statement

No exceptions were noted.

1 Main sources of data other than surveys are: SEA data on Adequate Yearly Progress and School/District Improvement determinations; SEA data on achievement results; NCES Common Core of Data statistics on student demographic trends (free-reduced lunch rates, minority enrollment rates, LEP enrollment rates); state forums and Sentinel Surveillance System for feedback from education leaders on emerging issues.

2 Office of Management and Budget. “Standards and Guidelines for Statistical Surveys.” September 2006. Section 1.3. p. 8.

9