Prenotification Letters

4

.

SUPPORTING STATEMENT

FOR PAPERWORK REDUCTION ACT SUBMISSION

B. Collection of Information Employing Statistical Methods

Explain sampling methods.

The primary data gathering instruments designed to employ statistical methods are the needs assessment surveys, focus group protocols, and interview protocols. No incentives, financial or otherwise, will be offered to participants.

Describe statistical survey methodology.

REL Midwest’s needs assessment methodology will include a carefully integrated program of quantitative surveys and qualitative research to inform its regional work.

Key themes such as testing, education technology, English language learners, and school programs intended to improve behavior, character, and responsibility will be addressed over the course of the five-year contract period primarily through the following primary quantitative and qualitative approaches.

The quantitative research with local and state educational leaders will take place in Years 1, 3, and 4.. A survey will be constructed and disseminated in Year 1. Many questions developed for the basic instrument in Year 1 will be repeated in Years 3 and 4 in order to track responses over time. (REL Midwest expects some questions to be adjusted over time due to changing policies or emerging trends and hypotheses but plan to hold the core of the quantitative work steady in order to effectively track changes in opinion.) Additionally, an oversample of teachers in Year 4 will provide opportunities to enrich the data sets with fresh perspectives.

In Years 2 and 4, the emphasis will be on qualitative exploration of key themes, through which REL Midwest can gain a deeper understanding of the dynamics, explore the impacts of new information on responses, and develop hypotheses that can be tested through the following year’s quantitative research.

While the methods described above represent the major data collection sources, several other sources will serve to substantiate findings from these methods. Other sources include real time analysis of ongoing commentary, unsolicited requests from the field (stored in the needs and responses database), information on needs and priorities from webcasts, and feedback from training workshops and other technical assistance activities. The identification, assessment, and analysis will be within and across stakeholder groups to the extent the samples and response rates allow.

Describe procedures for collecting information.

Data Collection

The quantitative questionnaire/survey is designed for state and local educational leaders only. The survey will only be administered to those who have either a professional background and current experience in K-12 education settings, or an intensive involvement in education advocacy. Parents will not be administered this survey.

The survey sample will include a total of 2,800 participants, consisting of 100 state level educators and policymakers per state and 300 district and school level education stakeholders per state. A description of the state and district level participants, as well as the methods to be used in randomly sampling state and district participants are described below.

State-Level Sample. The state level sample is a functional group. Each state-level sample will be designed to yield approximately 100 interviews per state, resulting in approximately 700 state-level interviews across the entire REL Midwest region. After extensive background research and planning, REL Midwest has identified and compiled a functional state-level sample that consists of the following categories:

State legislators that sit on one or more education committees

State legislators’ education staff

State Department of Education Professionals and State Board Members

State and local Chamber of Commerce Executives

REL Midwest obtained state-level contact information through web searches, telephone calls to various agencies, and through our longstanding relationships with states in the region. The method for obtaining contact lists for each respondent category is described below. In most cases, lists include respondent name, position, telephone, email address, and mailing address.

State legislators that sit on one or more education committees: Database available through the Education Commission of the States (ECS). A subscription is necessary to access the database.

State legislators’ education staff: Database available through the Education Commission of the States (ECS). A subscription is necessary to access the database.

State Department of Education Professionals: In two states, a database containing information for all DOE staff was available online. In the five remaining states, REL Midwest contacted a representative at each Department of Education, informed them of the Needs Assessment study, and requested a list of all DOE staff. Once the DOE staff list was obtained, REL Midwest selected DOE staff members to include in the state-level database by position title.

State and local Chamber of Commerce Executives: Obtained online, through each state’s Chamber of Commerce website.

The state education leadership sample will be divided into subgroups, which will be defined by respondent type (legislators, Department of Education staff, Chamber of Commerce executives, etc.). Via a random selection process, each subgroup will divide the sample into representative “minisamples” or replicates. Standard best practices of replicate control measures will be employed to ensure highest possible response rates. (We will thoroughly “work” each replicate before opening a subsequent one. In this way, the integrity of the sample will be maintained by allowing only a representative sample to be available for dialing). Quotas will be set for the total number of interviews to be completed in each state. In this way, each state has an adequate number of responses to be individually analyzed.. State subgroups will be proportionately represented in each replicate and in the total sample for each state. These quotas are set to ensure adequate state-level representation of the smaller subgroups in the final sample of completed interviews. When reporting results for the region, state-level results will be weighted to proportionately represent the regional population using the selected strata.

District-Level Sample. Each state also will have 300 interviews conducted at the district level, which will be divided into three groups: superintendents and assistant superintendents, school board presidents, and K-12 principals.

The district-level leadership sample will be purchased through MDR, a division of Dunn and Bradstreet. MDR conducts a census of educational institutions nationwide twice a year and updates their directories of superintendents, principals, and teachers to include names, position titles, geographic data, physical work address, e-mail address, and phone numbers. The MDR data base covers every district and school in the country.

Similar to the state-level sample, the district leadership sample will be divided into subgroups, which will be defined by respondent type (principals, superintendents and school board presidents). To ensure regional representation, the district-level survey sampling will begin with stratifying each state in the region based on type of locale, percentage of students who receive free or reduced-price lunch, and student minority status. Districts will be randomly and proportionally selected from these strata. For each state, quotas will be set for the number of interviews to be completed to ensure that the number of completed interviews in each state is large enough for analysis. Within each state, our three subsamples are proportionately represented. Across the seven states, a quota is in place to ensure a significant number for each state, regardless of population size. Stratifying the sample in this way will ensure that we will be able to reliably analyze and report on opinions of defined positions both within a state and across the Midwest region. The number of district-level interviews by subgroup are as follows:

	Illinois	Iowa	Indiana	Minnesota	Ohio	Michigan	Wisconsin
Principals	198	193	222	203	214	226	209
Superintendents	51	54	39	48	43	37	45
School Board Presidents	51	54	39	48	43	37	45
Total	300	300	300	300	300	300	300

In following the randomization procedures described above, REL Midwest will select a sample that is an approximate microcosm of the state and district samples as defined above. The number of interviews completed per subsample will correspond to the size of that population for each state as shown above. The margin of error will be based upon a 95% level of confidence. When reporting results for the region, results of the state and district samples will be weighted to proportionately represent the regional population using the selected strata.

A different random sample will be generated for each data collection period (e.g., Year 2 and Year 4). Because REL Midwest and Public Agenda are generating an independent random sample each time they conduct surveys and focus groups, burden will be minimized by maintaining a low probability that they will speak with the same participants within a set of organizations multiple times over the five-year contract period.

REL Midwest will work with Princeton Data Source to administer the CATI and online surveys. Princeton Data Source (PDS) is an affiliate of Princeton Survey Research Associates International, located in Princeton, N.J. and Washington D.C. PDS specializes in providing high-quality research by telephone, mail and online from a state-of-the-art facility in Fredericksburg, Virginia. Founded in 1999, PDS provides data for clients who need independent, reliable information about the attitudes, knowledge and behavior of the general public, as well as of defined subgroups.

Qualitative data collection will be conducted with a cross section of education stakeholders, including state and local policymakers, K-12 teachers and administrators, and parents. The focus group moderators and interviewers are either staff members of Public Agenda or consultants that Public Agenda have worked with extensively in the past. These moderators have extensive experience in working with stakeholders on education policy topics. The focus group questionnaire for parents contains open-ended questions that are qualitatively different than those designed for teachers, administrators, and policymakers. These questions are geared toward parents that have diverse levels of experience working as professionals or volunteers at the district and school level.

Table 4 provides subgroup sample sizes, response rates, and target completion rates.

Table 4. Participants’ Subgroup Sample Sizes, Response Rates, and Target Completions

Subgroup	Estimate Population Size	Sample Selected	Expected Response Rate	Target Completions
State level survey respondents	3,500	700	80%	Stop data collection at 700
District level survey respondents	15,000	2,100	80%	Stop data collection at 2,100
Interview participants	3,500 (same as state-level sample)	28-30	80%	Selective sample: Stop at 28
Focus group participants	15,000 (same as district-level sample)	84	80%	Selective sample: Stop at 84
Total	18,500	2,912	80%	2,912

During OMB review of the survey and focus group protocol in Year 1, a total of four to five informal focus groups will be conducted with state policymakers and local educators. Focus groups will be designed to probe into the thinking of respondents and uncover the dynamics underlying priority areas that were identified in needs sensing reports from 2005 and focus groups with the REL Midwest Board in early 2006. Specifically, five to six major questions will be explored in each group (with sub-questions and free-form probes to be administered as well). In addition to question themes, Public Agenda may employ other proven focus group techniques such as asking respondents to make lists, create report cards, sort pictures, draw, or participate in a minidebate. Additional formal focus groups may be conducted during Year 1, depending on when OMB approves the survey and focus group protocol.

Data analysis.

For the quantitative analyses, Public Agenda will thoroughly clean datasets through use of a computer validation program. Upon completion of data checks, Public Agenda will run simple summary statistics and look at emerging relationships. Public Agenda analyzes the quantitative survey data by applying a mix of descriptive statistics, cross-tabulations, t-tests and correlations as relevant to the question structure and number of responses. The frequency of each response category for total respondents, selected target subpopulations, and other meaningful points of analysis are identified. Additional t-tests and correlations are run depending on the question structure. Analysis of the quantitative survey will explore significant outcomes by parceling results by state, stakeholder group, and other demographic characteristics. Relationships within and between groups will be explored using tests of significance when applicable.

Public Agenda’s inductive approach to qualitative analysis and reporting incorporates systematic methods of managing data through reduction, organization, and connection (Dey, 1993; LeCompte, 2000). Public Agenda begins by summarizing interview and focus group transcripts, highlighting the common and unique experiences that diverse stakeholder groups report. Individual interview summary reports are then compared and synthesized to identify common themes and divergent cases. Descriptive findings are included in the final report to add depth and context and to enhance understanding of various stakeholders’ beliefs reported in the quantitative survey.

The findings from both focus groups and leadership interviews will help to illuminate broader research topics and answer questions about how and why education leaders hold certain beliefs. Insights from focus groups and interviews also can contribute to the design of future surveys and track changes in public attitudes in a timely and cost-efficient manner.

Reports from qualitative focus group and interview data will be designed to be descriptive rather than predictive, revealing a range of attitudes and beliefs about the issues that emerge and the perceptions that drive them.  Analyses will be written up as narratives that detail the findings and incorporate direct quotes when appropriate.

Describe rates of response.

REL Midwest and Public Agenda anticipate an 80% response rate and plan to take several steps to ensure that they meet their target. REL Midwest’s plan includes a very extensive pretest in order to more closely predict response rates to the studies. REL Midwest’s multimodal contact strategy includes multiple attempts to reach each individual sampled. This approach is designed to increase response rates—to give respondents many opportunities to participate in the survey either by phone or e-mail. As stated earlier, up to five to seven attempts will be undertaken to reach an individual, and this technique has been shown to increase overall response rates. Once those attempts have been exhausted in a particular replicate of sample, a new replicate will be issued. In addition, survey participants will receive a prenotification letter that includes a clear statement of the purpose and benefits of this research. When possible, letters will contain appropriate state department of education insignia, and a letter of encouragement to participate from the chief state school officer. Copies of the pre-notification letter and letter of encouragement are provided in Appendix A and B.

Discuss tests of procedures.

Public Agenda adheres to the Total Design Method of data collection, the highest professional standard of quality for survey research, and will use established design principles to ensure that the survey questionnaire functions as intended.

After OMB approval is granted, Public Agenda will again pretest the survey with individuals who are representative of the population to be surveyed, using cognitive-testing procedures in which draft versions of questionnaires are tested in individual interviewing sessions by specially trained interviewers. Following the iterative pretest process, Public Agenda will conduct a pilot study of no more than 100 subjects (including pretest respondents). Steps will be taken to promote intra-subject validation, and the questionnaire will be revised to reduce confounding mode effects.

Public Agenda will develop and test the interview and focus group protocol by conducting four focus groups on discrete topics, with fewer than nine participants. These focus groups will occur in diverse settings with principals, school board members, community and business leaders, and educational advocates (such as state DOE staff and legislators’ education staff). Results from these focus group sessions will be used to inform revisions to the formal interview and focus group questions, and to inform the Year 1 Needs Sensing Final Report.

Describe any consulting services that will be used.

REL Midwest proposes to continue to assess the needs of its region in collaboration with Public Agenda, one of the well-respected survey, research, and communications firms in the country. Continuous needs sensing and reporting will ensure that the applied research and technical assistance activities conducted are indeed on target and meet the specific needs of the Midwest region.

Public Agenda

Founded in 1975, Public Agenda is a nonpartisan, nonprofit organization providing unbiased, highly rigorous research that bridges the gaps between the public and those who formulate policy. Perhaps best known for its decades of thorough work in the education realm, Public Agenda’s research on topics ranging from foreign policy to religion, and from parenting to immigration brings greater clarity and understanding to complex issues. Public Agenda’s two-fold mission is to help American leaders better understand the public’s point of view and to help citizens know more about critical policy issues so they can make thoughtful, informed decisions.

Public Agenda has three decades of experience conducting large-scale research with expertise in all major opinion research methodologies, including surveys, focus groups, expert interviews, leadership surveys, tracking surveys, and one-on-one interviews.

Princeton Data Source LLC

Princeton Data Source LLC is a state-of-the-art telephone interviewing facility located in Fredericksburg, Va. With 90 CATI-equipped interviewing positions, PDS conducts surveys for a wide variety of clients across the country. The focus at PDS is on quality. The facility was built to PDS specifications and is outfitted with the latest in computer and telephone technology to smooth the daily work of telephone interviewing. Using SPSS-MR Quancept CATI software, PDS produces excellent fieldwork 52 weeks a year.

PDS meets and exceeds client expectations every day, whether interviewing members of the general public or members of specific professional groups, such as doctors or journalists. As required by the project, interviewing is conducted both during the day and at night – seven days a week. The field house conducts both long-term fieldwork under contract, as well as short-term polling assignments. PDS clients include the Pew Research Center, Bloomberg News Service, Kaiser Family Foundation and the Commonwealth Fund. PDS regularly conducts surveys of the general population, plus various elite populations. With the rise of the internet as an option for surveys using previously identified individuals, PDS has successfully conducted multimode surveys of lawyers, doctors, journalists, think-tank staff, government officials, teachers, school superintendents and many others.

Prenotification Letter

Dear :

REL Midwest and Public Agenda would like to invite you to become part of a select group of educators, education experts and other stakeholders to participate in a survey designed to understand your views on the challenges faced by school districts in your state. The results of this survey will be used to help gain a fundamental understanding of which education programs, practices, policies and approaches are effective.

The survey is brief; your responses are completely confidential and will be combined and reported in statistical form only.

To take the survey online, please go to [INSERT WEB ADDRESS]

If you prefer to be interviewed by telephone, please call our interviewing center toll-free at 1-888-310-0547

In order to ensure that each person takes the survey once you will be asked for the ID code below:

ID CODE: [INSERT ID CODE]

REL Midwest is part of a federally funded network of 10 regional educational laboratories, and exists to bring the latest and best research and proven practices into school improvement efforts. Serving the seven states of Illinois, Indiana, Iowa, Michigan, Minnesota, Ohio, and Wisconsin, REL Midwest provides policymakers and practitioners with resources based on the highest quality evidence as defined by scientifically valid research principles. You can find out more at www.learningpt.org

Public Agenda has over 30 years of experience in engaging the public in productive and meaningful dialogue, conducting qualitative and quantitative public opinion studies, and producing high quality citizen education materials. Over the last decade, Public Agenda has acquired an unmatched understanding of the viewpoints different groups bring to public school reform and our reputation for fairness, balance, and quality has stood the test of time. Our research is cited by groups with very different perspectives on reform ranging from The American Enterprise Institute, Manhattan Institute and Fordham Foundations to the American Federation of Teachers, Public Education Network (PEN) and Learning First. You can find out more information about Public Agenda at www.publicagenda.org

We hope you will take a few minutes to share your perspectives and respond to our questions. Your participation in our survey is completely voluntary. If you have any questions about Public Agenda or this research project, please contact Jon Rochkind at 212-686-6610, ext. 43 or send an email to jrochkind@publicagenda.org If you have technical questions or have difficulty with the web form, please call Samantha Daugherty at 1-888-310-0547or e-mail her at samantha.daughtery@princetondatasource.com.

Sincerely:

Letter of Encouragement

Dear :

Recently, REL-Midwest and Public Agenda contacted you to become part of a select group of educators, education experts and other stakeholders to participate in a survey designed to understand your views on the challenges faced by school districts in your state.

We understand that your time is very valuable, yet we hope that you can participate in this important project; the results of which will be used to help gain a fundamental understanding of which education programs, practices, policies and approaches are effective.

The survey is brief; your responses are completely confidential and will be combined and reported in statistical form only.

To take the survey online, please go to [INSERT WEB ADDRESS]

If you prefer to be interviewed by telephone, please call our interviewing center toll-free at 1-888-310-0547

In order to ensure that each person takes the survey once you will be asked for the ID code below:

ID CODE: [INSERT ID CODE]

REL Midwest is part of a federally funded network of 10 regional educational laboratories, and exists to bring the latest and best research and proven practices into school improvement efforts. Serving the seven states of Illinois, Indiana, Iowa, Michigan, Minnesota, Ohio, and Wisconsin, REL Midwest provides policymakers and practitioners with resources based on the highest quality evidence as defined by scientifically valid research principles. You can find out more at www.learningpt.org

Public Agenda has over 30 years of experience in engaging the public in productive and meaningful dialogue, conducting qualitative and quantitative public opinion studies, and producing high quality citizen education materials. Over the last decade, Public Agenda has acquired an unmatched understanding of the viewpoints different groups bring to public school reform and our reputation for fairness, balance, and quality has stood the test of time. Our research is cited by groups with very different perspectives on reform ranging from The American Enterprise Institute, Manhattan Institute and Fordham Foundations to the American Federation of Teachers, Public Education Network (PEN) and Learning First. You can find out more information about Public Agenda at www.publicagenda.org

We hope you will take a few minutes to share your perspectives and respond to our questions. Your participation in our survey is completely voluntary. If you have any questions about Public Agenda or this research project, please contact Jon Rochkind at 212-686-6610, ext. 43 or send an email to jrochkind@publicagenda.org If you have technical questions or have difficulty with the web form, please call Samantha Daugherty at 1-888-310-0547or e-mail her at samantha.daughtery@princetondatasource.com.

Sincerely:

NCREL Regional Needs Assessment Budget
Staff - % of annual salary				2006
Permanent Full-Time Project Staff
Director of Research					25%
Research Associate					25%
Executive Editor/Web master				13%
Technology Manager					13%
Assistant Editor					13%
Director, Education Insights				8%
Director of Public Engagement				25%
Manger of Public Engagement				25%
Vice President of Communications				4%
Director of Communications/Writer				4%
President						8%
Executive Assistant					8%
Administrator/Contract Manager				17%
Receptionist					8%
Sub-total						194,872
Benefits at 25%					48,718
Total						243,590
Consultants @ $1,000 per day
Regional Manager					30,000
Research consultant					10,000
Copy editor/proofreader				3,000
Online research consultant				5,000
Sub-total						48,000
Focus Groups
Focus Groups @ $3,000 per				27,000
Transcribing @ $260 per				4,160
Roper						500
Sub-total						31,660
Survey
Survey						10,000
Software						5,000
Sub-total						15,000
Travel @ $1,500 per trip				13,500
Production and Publication				10,000
Interactive Online Distribution				10,000
Miscellaneous					1,000
Sub-total direct costs				372,750
Overhead @ 20%					74,550
of direct costs
Rent/Utilities/RE taxes* @				35,000
17% of annual budget
Technology* @ 17% of				7,000
annual budget
Total						489,300
*This percentage represents the ratio of the project budget to the annual rent and technology budgets.

4