OMB Part B revised 10.14.09

Evaluation of the IDEA Personnel Development Program

Statement for Paperwork Reduction Act Submission

PART B: Collection of Information Employing
Statistical Methods

Contract ED-04-CO-0059/0022

October 14, 2009

Prepared for

Institute of Education Sciences

U.S. Department of Education

Prepared by

Westat

Contents

Page

Part B: Collection of Information Employing Statistical Methods 1

B.1 Respondent Universe and Sampling Methods 1

B.2 Information Collection Procedures 5

B.3 Methods to Maximize Response Rates 7

B.4 Test of Procedures 7

B.5 Individuals Consulted on Statistical Aspects of Design 8

Part B: Collection of Information Employing Statistical Methods

B.1 Respondent Universe and Sampling Methods

Participants in the IHE Study

The IHE Study will focus on all courses of study¹ for which funding was requested in the FY 2006 or FY 2007 PDP grant competitions. Those 2006 applications that represent the same course of study as a 2007 application will be surveyed only once. Respondents will be the proposed Project Directors for the grant or their designees. This constitutes the respondent universe. Exhibit B-1 shows that the respondent universe consists of 450 courses of study. Exhibit B-1 categorizes the universe by year of application, funding status, and focus area. These courses of study come from 12 competitions: six focus areas for each of 2 years. One hundred-eighty five of the applications were funded; 265 were not funded.

Exhibit B-1. Course of Study Sample^a

	FY 2006		FY 2007		FY 2006 & 2007
Priority and Focus area	Funded	Non-funded	Funded	Non-funded	Funded	Non-funded
Leadership	20	19	24	51	44	70
High-incidence	12	42	22	14	34	56
Minority	11	16	14	40	25	56
Related	11	22	15	18	26	40
Low-incidence	13	2	17	25	30	27
EI and EC	11	5	15	11	26	16
Totals	78	106	107	159	185	265

^aThis table provides the number of funded and non-funded courses of study for FY 06 and FY 07. Applications for courses of study that applied in both 2006 and 2007 have been removed from the 2006 sample.

Sampling Procedures and Size of the Sample for the IHE Study

We propose to survey the 450 courses of study in the respondent universe. Although sampling could be used to reduce the burden on IHEs, it is not feasible here because we wish to provide descriptive statistics for each of 24 cells (two funding statuses times two years times six competitions). As a part of the survey, respondents will be asked to list changes made to their courses of study since the time of their application. We will ask respondents to provide documentation for a randomly selected subset of these changes.

Each documented change will be reviewed by a panel of three experts and will be rated on a 5-point scale, with 1 representing very low quality, 2 representing low quality, 3 representing moderate quality, 4 representing high quality, and 5 representing very high quality. Analysis of a preliminary review of 100 applications suggests that documents will be available in the categories and numbers shown in Exhibit B-2. In order to minimize the burden on respondents, we will sample changes from funded institutions as shown in the Exhibit. It is anticipated that non-funded institutions will report fewer changes. However, due to the larger number of non-funded institutions in the sample, we base our estimates on the expectation that we will be able to collect approximately the same number of changes from the funded and non-funded institutions. Where less than 33 changes are reported for a particular category, we will request all of the documents. Where more than 33 changes are reported, 33 will be requested. We estimate that approximately 120 funded will be asked to provide documentation of changes, with an average of 1.4 changes sampled per course of study. One hour is the estimated time needed per respondent to collect and ship these materials

Exhibit B-2. Document Collection Sample

Sampling of changes made to funded courses of study – a priori estimates
Change category	Courses of study (COS) with change	Average number of changes per COS	Number of changes in universe	Number of changes sampled	Number of changes documented	Average number of changes documented per COS
New courses or classes	40	2.0	80	33	30	0.8
New training units, modules, fieldwork	120	2.0	240	33	30	0.3
New recruitment plans	100	1.0	100	33	30	0.3
Reorganized or relocated course of study	30	1.0	30	30	27	1.0
New mentoring program	22	1.2	26	26	23	1.2
New faculty	15	1.1	17	17	15	1.1
New credentials	15	1.1	17	17	15	1.1
Total universe	150	3.4	510
Total sample	120			189	170	1.4

Each of the 170 funded course of study changes will be reviewed by 3 experts for a total of 510 reviews. Each review is estimated to take one hour. We will need 3 reviewers for each of the six competitions for a total of 18 reviewers. Based on these estimates of the numbers of course of study changes and the numbers of reviewers, each reviewer will need approximately 56 hours to review the materials assigned to them. Adding 10 hours per reviewer for training, we will need each reviewer to commit 66 hours to the process. The Council for Exceptional Children is experienced in the recruitment of reviewers, so this task will be easily accomplished.

Participants in Study of the National Centers

The Study of the National Centers will focus on the 12 funded Centers with start dates ranging from 2001 to 2008. This represents the population of Centers funded in that time period. Respondents will be Project Directors for the Centers or their designees. Exhibit B-3 lists the Centers targeted for participation, the names of the grantees, and the start-end dates of the grants. No sampling is proposed at the grant level. Sampling will be conducted at the product/services level based on inventories of products and services submitted by staff from each National Center.

Exhibit B-3. List of Centers for the Evaluation of the IDEA Personnel Development Program

Center	Start-end dates
Center for Training Personnel to Provide Evidence-Based Educational Services to Students with Autism Spectrum Disorders (ASD). The goal of the Center is to increase the capacity of local school districts to meet the needs of students with ASD through direct training of service providers, course materials for pre-service personnel, and dissemination of information and materials.	1/03-12/07
National IHE Faculty Enhancement Center (IRIS). The Center is designed to create and implement a comprehensive national resource for faculty teaching pre-service courses in general education school administration, school counseling, and school nursing, resulting in new graduates who meet the needs of students with disabilities.	8/01-6/06
IDEA and Research for Inclusive Settings: The Center for Training Enhancements (IRIS-II). The overarching goal is to enhance the preparation of the nation’s current and future professionals in several disciplines who are or will be working in inclusive educational settings.	10/06-9/11
Center for Improving Teacher Quality. The goal of the Center is to work with states to develop exemplary models for building statewide systems of training and improved licensure and certification for teachers of students with disabilities.	11/02-10/07
NIUSI Principal Leadership Academies Initiatives. The National Institute for Urban School Improvement, Principal Leadership Academies Initiative will assist 400+ principals nationwide to develop and implement inclusive schools, ensuring that students with and without disabilities meet academic standards set by their states. NIUSI-PLAN combines face-to-face and on-line learning and on-site coaching.	11/06-10/11
National Center on High Quality Personnel in Inclusive Early Childhood Settings. The goal of the Center is to increase the number of high quality personnel serving children in inclusive preschool settings by improving the quality and accessibility of rigorous, on-going professional development, training, and technical assistance opportunities. The Center assists states in identifying and leveraging existing fiscal, human, and material resources to facilitate the development of an integrated, cross-agency state plan for professional development.	10/06-9/11
National Early Childhood Training Enhancement Center. The goal is to increase the competence of early childhood professional development providers in designing, delivering, and evaluating evidence-based approaches to professional development and increase the knowledge and skill-based learning of early childhood practitioners, families, and professional development providers.	10/07-9/12

Exhibit B-3. List of Centers for the Evaluation of the IDEA Personnel Development Program (continued)

Center	Start-end dates
Center for Early Childhood Education/Early Intervention Personnel Preparation. The purpose of the Center is to collect, synthesize, and analyze information related to certification and licensure requirements for personnel working with infants, toddlers, and preschoolers who have special needs and their families; the quality of training programs that prepare these professionals; and the supply and demand of professionals representing all disciplines providing early intervention and early childhood special education services.	1/03-12/07
Center to Inform Personnel Preparation Policy and Practices in Special Education. The goal of the Center is to inform special education teacher preparation policy and practice by examining and recommending to IHEs, SEAs, and LEAs those policies and practices that improve retention of beginning special education teachers, such as high quality induction, mentoring, and collaborative teaching and to recommend implementation strategies for polices and practices that provide beginning special and regular education teachers with the knowledge and skills to effectively support students with disabilities in different classroom settings.	1/08-6/10
National Center for Leadership in Visual Impairment. NCLVI houses a consortium of universities; it serves as a conduit through which federal monies for leadership will flow to support doctoral candidates. Objectives are to develop a collaborative model for producing leadership personnel in special education with an emphasis on VI and to increase the numbers of doctoral graduates available for positions in higher education teaching and research, public policy, administration at national state and/or local levels, curriculum development, and supervision and/or general research.	10/04-8/09
Interdisciplinary Training in Analysis of Large-Scale Databases. The Center is designed to recruit, select, and retain doctoral students with diverse academic backgrounds and experiences and to provide them with a comprehensive array of doctoral-level coursework and experiences related to the design, implementation, and analysis of large-scale datasets.	10/06-9/11
Carolina Interdisciplinary Large-Scale Policy Research Training. Pre-doctoral and post-doctoral Fellows from relevant disciplines attended an intensive 8-week session to gain knowledge and competencies needed to improve their skills in conducting large-scale research. They subsequently engaged in internships, conducting research and analysis on large-scale special education data sets.	10/02-9/06

Sampling Procedures and Size of the Sample for the Study of the National Centers

The Study of the National Centers will focus on the 12 funded Centers with start dates ranging from 2001 to 2008. This represents the population of Centers funded in that time period. No sampling is proposed at the grant level. Sampling will be conducted at the product/services level based on inventories of products and services submitted by Center staff. Westat proposes selecting a stratified random sample of 10 products/services from each Center. The two strata are (1) products and services nominated by the Center as signature works and (2) non-nominated products and services. This stratification is proposed because some products may be central to the Center’s goals, whereas others may be peripheral. For example, a Center engaged in personnel preparation may have developed materials for recruiting students, a legitimate product or group of products. However, we would not want recruitment materials to displace a signature work, such as the courses offered through the personnel development program, when the sample of products is selected. We also considered sampling based on level of effort but did not believe that the Centers could generate sufficiently accurate cost estimates for each individual product or service without a large increase in burden.

The nominated and non-nominated samples for each Center will be selected with equal probability from each stratum, with a maximum of three signature works. If the number of signature works is insufficient to reach the desired sample size using equal probability, we propose selecting all of the signature works and adjusting the probability for the non-signature works to reach the desired total sample size of 10.

Westat proposes using a random number generator to sample products/services from the enumerated lists submitted by the Centers in each of the strata described above. Once products have been sampled, Center staff will be asked to submit materials associated with the selected products or services. The materials will be reviewed by members of the Expert Review Panel for quality, relevance, and usefulness. Once the quality, relevance, and usefulness scores have been calculated, Westat will generate weights for each product or service, based on the probability of selection. In analysis, the results will be weighted to generate statistics that are representative of all the products within a Center.

B.2 Information Collection Procedures

IHE Study

The proposed IHE Study includes two components: (1) a web survey of the funded and non-funded applicants to the FY 2006 and FY 2007 PDP grant competition and (2) an expert panel review of changes to courses of study documented by the funded applicants.

Westat staff will review applications from the FY 2006 and FY 2007 PDP competitions to define the course of study proposed for each application. This course of study information will be included in the survey. Project Directors will be provided a link to their individualized web survey. A paper form of the survey (Appendix A) will be available, when requested. Telephone and email reminders will be sent to encourage response. Data from the web survey will be automatically available as a database. Survey data from the paper version will be entered into the same database.

As part of the survey, respondents to the IHE Survey will be asked to list courses that are “new” or “significantly modified” since the time of their PDP grant application. They will also asked to list other changes and are given examples of substantive changes in the instructions. There will, of course, be some variation in the levels of change respondents choose to report, but the respondents will be the initial judges of what changes are substantive. Once Westat has the data from all respondents, staff will be able to examine the nature and range of responses and make some informed judgments about what constitutes a substantive change in the context of all the changes reported. Because the quality and magnitude of changes made by the funded applicants will be assessed by the IHE Study review panels, Westat will, if necessary, filter out non-substantive changes both before sampling the changes that respondents will be asked to document and after receiving documentation associated with the sampled changes. This filtering will not be focused on the quality of the change, because a measure of quality is an important result of the review panels’ work. Instead, the filtering will be done to eliminate changes that are not at a level of magnitude or of sufficient importance to a project to constitute a substantive change. Such non-substantive changes need not be reviewed because, regardless of their quality, they will be of too little consequence to be relevant to the evaluation’s descriptive analyses.

A randomly selected subset of approximately 120 Project Directors from funded courses of study will be requested to submit documentation of some of the substantive changes to their course of study (Appendix B). Westat staff will catalog and store these materials. Copies of these materials will be sent to a panel of experts for a quality review. The rubrics for the quality review will include indicators linked to the goals, objectives, and selection criteria outlined in the PDP grant priorities. The panel of experts will rate each indicator on a 5-point scale, with anchors at ratings of 1 (very low level of quality), 3 (moderate level of quality), and 5 (very high level of quality).

Study of the National Centers

The Study of the National Centers will include a sequential set of data collection activities. After reviewing winning proposals submitted by the grantees operating the Centers, researchers will conduct telephone interviews with staff from each of the 12 Centers under evaluation. Next, an inventory will be used to catalog the cumulative accomplishments of the Centers through detailed lists of the Center’s products and services to date, including estimates of the cost of each. Data from these lists will be the basis for selecting a sample of each Center’s products for review by the National Centers Review Panel. Once the products have been sampled, we will send a Product Request Form to each Center asking it to submit all available materials relevant to the sampled products or services. In addition, for each sampled product, Center staff will be asked to provide descriptive information to assist members of the expert panel in understanding the purpose of the product or service. These data collection activities are described below in more detail.

The evaluation team proposes semi-structured telephone interviews with staff from each of the 12 Centers being evaluated. During these interviews, members of the evaluation team will interview Center directors and other key staff to understand the grant recipient’s goals, objectives, staffing structure, and operations. These data will provide necessary context for understanding the Center’s products and services as well as the costs associated with those products. Importantly, during the calls, Westat staff will also review the instructions for completing the Inventory of Products and Services with the Center director or a designee. A copy of the Inventory will be sent to Center directors prior to the interview. Interviews will last roughly 2 hours, with two members of the evaluation team participating in each. A copy of the interview protocol is included in Appendix C.

After each interview, evaluation staff will write up the interview notes and prepare an integrated summary of the information from the Center’s grant proposal and telephone interview that combines information into a single narrative for each Center. The descriptions will be standardized across Centers, to the extent possible. We recognize, however, that the Center missions vary greatly, and we want to ensure that the evaluation staff will produce detailed descriptions of each center as well as a cross-site report that will document similarities and differences across the Centers. The relevant Center narratives will ultimately be provided to the members of the National Centers Review Panel selected to conduct each review, along with specific information regarding the products and services they will review. Prior to sending the narratives to members of the Review Panel, we will ask Center staff to read the narratives to ensure that they are accurate and complete.

The evaluation team will use the Inventory of Products and Services completed by Center staff to catalog the cumulative accomplishments of the Centers through detailed lists of each Center’s products and services to date. The Inventory of Products and Services will request a brief description of each product or service, the medium used (e.g., CD, seminar), audience or consumer, purpose, activities, start and end dates, and rough cost information for each product/service. In addition to providing descriptive information on all the products and services generated by the National Centers using PDP funds over the past 8 years, data from these lists will be the basis for selecting a sample of each Center’s products for review by the National Centers Review Panel. A copy of the Inventory of Products and Services is included in Appendix D.

From the inventories, we will sample up to 10 products or services from the signature works or other Center products. Once the products have been sampled, evaluation staff will send a Products Request Form to the Centers, asking them to submit all available materials relevant to the sampled products or services. For each product, Center staff will be asked to provide a brief description of (1) evidence of need for the product, (2) reasoning behind the product or service design, (3) principal actors, (4) the timeframe in which each was developed and distributed, and (5) any evaluative data specifically related to each product and service. A copy of the Products Request Form is included in Appendix E.

B.3 Methods to Maximize Response Rates

IHE Study

To ensure a high rate of participation in the IHE study, Westat recommends that a letter from ED, signed by both an NCEE and an OSEP official, be sent to the Project Directors informing them of the evaluation. Funded institutions will be reminded of their obligation to participate (20 U.S.C. 1221e-3 and 3474). In addition, in order to achieve acceptable response rates from the non-funded applicants in the IHE study, we propose to provide a payment of $30 to non-funded institutions for completing the survey. For the data from the non-funded applicants to meet the response rates standards for rigorous research, 80% of this group will need to respond. Thus, a payment will be instrumental in order for the study to reach this response rate given that the non-funded applicants will not be currently funded through this grant program and have no obligation or requirement to complete the survey or submit materials.

Study of the National Centers

To ensure 100 percent participation in the study of the National Centers, Westat recommends that a letter from ED, signed by both an NCEE and an OSEP official, be sent to the Center directors informing them of the evaluation. This letter will be followed by an introductory letter from Westat. If necessary, the directors can be reminded of their obligation to participate (20 U.S.C. 1221e-3 and 3474).

B.4 Test of Procedures

IHE Study

The feasibility of collecting materials from program applicants was tested in April, 2008. In November, 2008, potential IHE survey items were tested with applicants to the FY 2005 program. A field test of the IHE survey with nine IHEs from FY 2005 competitions was completed in January 2009. In particular, this test run included procedures for recruiting sites. A paper version of the IHE survey was used for data collection during the field test. Information derived from the field test was critical for further shaping the design and implementation of the study and refining the data collection instruments prior to full-scale implementation in Fall 2009.

Study of the National Centers

Westat pilot tested the interview protocol, Inventory of Products and Services, and Product Request Form with two of the National Centers and made minor revisions based on feedback from participants.

B.5 Individuals Consulted on Statistical Aspects of Design

These data collection plans were developed by Westat and its subcontractor, the Council for Exceptional Children (CEC). The research team is led by Tom Fiore, project director. Other members of the evaluation team who worked on the design include: Camilla Heid, Elaine Carlson, David Judkins, Hyunshik Lee, and Tom Munk from Westat and Richard Mainzer from CEC. The NCEE project officer, John Rice, also played a central role in data collection plans. Contact information for these individuals is provided below.

John Rice

U.S. Department of Education

202-219-3394

Tom Fiore

Westat

919-474-0349

Camilla Heid

Westat

301-294-4413

Elaine Carlson

Westat

301-251-4277

David Judkins

Westat

301-315-5970

Hyunshik Lee

Westat

301-610-5112

Tom Munk

Westat

919-474-8158

Richard Mainzer

CEC

703-264-9408

1A course of study is defined as the set of courses at an IHE designed, at least in part, to prepare students to perform the professional or paraprofessional role that was the focus of a particular PDP grant application; this set of courses typically results in some type of degree, certificate, or endorsement.