OMB PartA revised 10.14.09

In 2003, the Office of Management and Budget (OMB) conducted an assessment of the Personnel Preparation to Improve Services and Results for Children with Disabilities Program (Personnel Development Program)^¹ using the Program Assessment Rating Tool (PART). OMB assigned the program a rating of “results not demonstrated.” One reason cited by OMB for this PART rating was that an independent evaluation of the program had not been conducted.^² In response to the OMB rating, the U.S. Department of Education (ED) awarded a 4-year contract to conduct an evaluation of the Personnel Development Program (Individuals with Disabilities Education Improvement Act, Title I, Part D, Subpart 2, Section 664; PL No 108-446). The evaluation will be conducted by the National Center for Education Evaluation (NCEE) in the Institute of Education Sciences (IES). It will include two separate studies, each focusing on different recipients of program funding. One study will evaluate the National Centers, and the other study will evaluate courses of study^³that prepare personnel to serve children with disabilities at Institutions of Higher Education (IHEs). Findings from the studies will be released in a report in fall 2011.

A.1 Explanation of Circumstances That Make Collection of Data Necessary

Program Background

The United States has experienced a long-standing shortage of personnel to meet the educational needs of children with disabilities (U.S. Department of Education, DANS, 2004). Since the late 1950s, the federal government has invested funds in preparing personnel to serve children with disabilities. The IDEA Personnel Development Program (PDP) provides grants to institutions of higher education to prepare teachers, related service personnel, and leadership personnel to serve children with disabilities. The majority of grant funds are used for stipends to students, who must meet a service obligation in exchange for funding. A secondary function of the PDP is to provide indirect support for personnel preparation through research, technical assistance, or personnel preparation resources. This work is completed through a series of National Centers funded under the program.

Overview of the Evaluation

IHE Study

The PDP evaluation includes two separate components. The first is a study of institutions of higher education (IHEs) that have applied for funds to directly prepare personnel under the PDP. It is designed to (1) collect descriptive data from all the funded and non-funded applicants to the FY 2006 and FY 2007 competitions (185 funded and 265 non-funded) and (2) document changes to the applicants’ courses of study. The changes to programs of applicants funded by the PDP will be rated for quality by a panel of experts. Throughout this submission, we refer to this survey and the review of materials from funded and non-funded IHE applicants as the IHE Study.

A web survey of Project Directors is planned for fall 2009 with items addressing the course of study: (1) status; (2) focus; (3) entry and completion requirements; (4) grant support for students; (5) changes to the course of study since the time of the application; (6) enrollment and completion information; (7) standardized exit exam scores; (8) allocation of PDP grant funds; and (9) information about formal data collection from program completers. Documentation of a sample of changes made to funded courses of study will be rated by an expert panel for quality. These may, for example, include (1) syllabi, CVs, and assessments from newly created or substantially modified courses; (2) materials documenting new training units, modules, or fieldwork; (3) new recruitment plans; (4) reorganized or relocated courses of study; (5) new mentoring programs; (6) curriculum vitae of new faculty members; and (7) new credentials resulting from candidate efforts in the course of study (see Exhibit B-2). Three members of the IHE Expert Review Panels will review the documents representing each change and rate the contributions made to the quality, relevance, and usefulness of the specific personnel preparation course of study.

Study of the National Centers

The second component of the evaluation is a Study of the National Centers funded under the program. This component of the evaluation is designed to (1) document the products and services generated by the National Centers; (2) produce a rough estimate of their costs; and (3) rate the quality, relevance, and usefulness of a sample of those products and services. The Study of the National Centers will include all 12 of the Centers funded between 2001 and 2008.

Data collection will be conducted in 2009 and 2010, as close as possible to the end date for each Center’s funding. Following initial telephone interviews with Center staff, an inventory will be completed by Center staff that will catalog the cumulative accomplishments of each Center. Data from the inventory of products and services will be the basis for selecting a sample of each Center’s products for review by the National Centers Review Panel, which will comprise individuals with expertise relevant to the work of each Center. Centers may designate up to 10 percent of their products as signature works, which will be sampled in a separate stratum. Once the products have been sampled, we will ask Center staff to provide descriptive information about each and to submit all available materials relevant to the sampled products or services. Three experts will review each product or service for quality, relevance, and usefulness. Results of the ratings will be reported for each Center in two metrics: the percentage of products rated of high quality, high relevance, or high usefulness; and mean quality, relevance, and usefulness scores.

A.2 How the Information Will Be Collected, by Whom, and For What Purpose

Research Questions

The following research questions will be addressed for the IHE Study.

What are the characteristics of courses of study funded by the PDP (i.e., admission criteria, field experiences, exit exam results, etc.)?
How many new (i.e., previously nonexistent) courses of study are created with grant funding?
What becomes of courses of study that do not receive funding in a given year (e.g., continue operation, cease to exist, scale back enrollment, etc.)?
What proportion of non-funded applicants subsequently re-applies for grant funding and what is the result?
How do grantees use the funds from the PDP?

How do new recipients of a grant (i.e., IHEs with no previous program funding) use new funding compared to previous recipients of funding from the program? How does this differ by grant focus area?
How many new classes are created or new professors hired by funded courses of study? What are the nature of these classes and qualifications of these professors? How does this differ by grant focus area?

How many new individuals do grant recipients enroll in their courses of study and how many complete the course or drop out?
What is the quality of modifications to funded courses of study (or new classes within existing courses of study) since funding from the PDP?

The following three research questions will be addressed for the Study of the National Centers:

What materials have been developed and how much technical assistance has been provided by the National Centers with funds awarded from the PDP?
What has been the cost for the development and dissemination of these materials and services?
What are the quality, relevance, and usefulness of the materials and technical assistance provided by the National Centers using funds awarded from the PDP?

Respondents

Data to address these study questions will be collected from three types of respondents: funded IHE Project Directors, non-funded IHE Project Directors, and funded directors of National Centers.

Funded IHE Project Directors. Project Directors for PDP-funded grants are university faculty who are receiving monies from ED under the 2006 or 2007 competitions to prepare personnel who will serve children with disabilities. They will be asked to provide descriptive information for their existing courses of study, including their (1) status; (2) focus; (3) entry and completion requirements; (4) grant support for students; (5) changes to the course of study since the time of the application; (6) enrollment and completion information; (7) standardized exit exam scores; (8) allocation of PDP grant funds; and (8) information about formal data collection from program completers Documentation of a sample of the changes to the funded course of study listed by the Directors will be requested.

Non-funded IHE Project Directors. Project Directors for non-funded-PDP grants are university faculty who applied for, but did not receive, monies from ED under the 2006 or 2007 PDP competitions. They will be asked to provide descriptive information for their existing courses of study, in the same categories as the funded IHE Project Directors.

Funded Directors of National Centers. Directors of the National Centers are typically university faculty with 4- to 5-year grants from ED to assist in the preparation of personnel who serve children with disabilities through research, technical assistance, or personnel preparation resources provided to IHEs or local education agencies (LEAs). Center Directors will be asked to provide background information on their Center, including relevant activities pre-dating the Center, initial goals and objectives, current goals and objectives, organizational structure, and sustainability of the project after the grant period. In the Inventory of Products and Services, Center Directors will be asked to provide a description of each product and service developed using grant funds, the presentation medium, approximate cost, and beginning/end date for availability. In the Product Request Form, Center Directors will be asked to provide further information on sampled products or services, including evidence of need, reasoning behind the design, and evaluative information. In addition, the Center Director will be asked to submit all documentation and materials relevant to the sampled products and services.

Instruments

The proposed evaluation includes five data collection instruments: (1) an IHE Survey of funded and non-funded applicants as well as leadership and combined priority applicants, (2) an IHE Materials Collection Form, (3) an Interview Protocol for National Center Directors, (4) an Inventory of Products and Services that were developed by the National Centers, and (5) a Products Request Form for the National Centers. This section describes each instrument, its respondents, as well as the link between the instruments and study questions listed previously. Copies of the actual instruments and cover letters are included in appendices A through H.

IHE Survey

Respondents to the IHE Survey will be all individuals bid as Project Directors on a 2006 or 2007 PDP grant application, whether or not the application was funded. Skip patterns within the web-based survey will navigate applicants to relevant items only. The IHE Survey will provide information to address most of the study questions, including: (1) courses of study characteristics, (2) how many new courses of study were created with grant funding, (3) what became of non-funded courses of study, (4) how the use of grant funds differed for new and recurring grantees, (5) the number and type of new classes created or new professors hired by funded courses of study, (6) the number of new individuals who enrolled in and completed funded courses of study, and (7) the characteristics of students before and after funding. A draft of the survey is found in Appendix A. Four versions of the questionnaire are provided – one for funded applicants to the Leadership priority, one for funded applicants to the Combination priority (all non-doctoral programs), one for non-funded Leadership applicants and one for non-funded Combination applicants.

IHE Materials Collection Form

A sample of the materials that document changes to courses of study after the time of application will be collected from some of the Project Directors who complete the survey. An expert review of these documents will be used to answer the study question regarding the nature and quality of new classes, new professors, and other changes to funded courses of study. A sample Materials Collection Form is found in Appendix B.

Interview Protocol for National Center Directors

Each of the 12 Directors of the National Centers will also be asked to complete a telephone interview. The information from this collection will be used to facilitate the expert review of products and services sampled from each Center. Reviewers will receive an abstract on each Center to help provide context for the products and services being reviewed. In this semi-structured interview, respondents will be asked to describe the ways in which the work of the Center is a continuation of work done previously; how the work of the Center has changed since initial funding and what prompted any changes; obstacles and challenges that the Center has faced in completing the proposed work; future direction of the Center, if work is ongoing; ways the Center’s work has been sustained, if funding has expired; Center products, services, or activities used the least/most extensively; and an overview of the Center’s organizational structure and staffing. The interview protocol can be found in Appendix C.

Inventory of Products and Services

Each of the 12 Directors of the National Centers will be asked to complete an Inventory of Products and Services. This instrument will be used to respond to two study questions, “What materials have been developed and how much technical assistance has been provided by the National Centers with funds awarded from the Personnel Preparation Program?” and “What has been the cost for the development and dissemination of these materials and services?” This inventory can be found in Appendix D.

Products Request Form for the National Centers

Each of the 12 Directors of the National Centers will be asked to complete a Products Request Form for the items sampled from their Inventory. This instrument will be used to obtain background information, including evidence of need, reasoning behind the design, and evaluative information and actual products or artifacts of the services being reviewed by the expert panel. The form and the ensuing expert review will be used to address the following study question: “What is the quality, relevance, and usefulness of the materials and technical assistance provided by the National Centers using funds awarded from the Personnel Preparation Program?” The request form can be found in Appendix E.

Exhibit A-1 below summarizes the data collection activities planned for the evaluation. Note that each data collection occurs only once, although in the case of the Study of the National Centers, the collection occurs at different points in time, depending on the year in which the Center was funded.

Exhibit A-1. Data Collection Activities

Instrument	Respondent	Year 2008	Year 2009	Year 2010	Key data
IHE Study
Survey	All Project Directors for FY06 and 07 applications		X		For 2008-2009: course of study’s status, focus, entry and completion requirements, grant support for students; allocation of PDP grant funds; information about formal data collection For 2005-2009: changes since the time of the application; enrollment and completion numbers; standardized exit exam results; consequences of a failure to win PDP funding for non-funded applicants.
Materials Collection	Project Directors for funded FY06 and 07 applications			X	Documentation requested for quality changes, added training units, etc.. Teacher CVs, syllabi, and assessments requested for all new and significantly modified courses.
Centers Study
Telephone Interview	Directors of National Centers		X	X	Goals and objectives, primary audiences, organizational structure.
Inventory of Products and Services	Directors of National Centers		X	X	Products/services, presentation medium, audience, purpose, cost, start/end dates.
Products Request Form	Directors of National Centers		X	X	Products, documentation of services, reasoning behind design, evidence of need for product or service, evaluative information.

A.3 Use of Improved Information Technology to Reduce Burden

We will administer the IHE Survey on the web, so it is easily accessible to Project Directors. Burden will be reduced with the use of complex skip patterns and pre-loading of information from grant applications. When needed, the survey will also be available in hard copy. For the Study of the National Centers, the Inventory is a Word document, and the Products Request Form is a partially pre-filled Word document that will be completed electronically.

A.4 Efforts to Identify and Avoid Duplication

The majority of the information to be collected through these instruments does not currently exist within ED or other agencies. The data that do exist are being collected through extant data collection from the Office of Special Education Program and will be used in addition to the data collected through the instruments. For example, lists of applicants (called competition slates), the applications themselves, Annual Performance Reports written by grantees, and student data reports collected by Westat for OSEP will all be used to help answer the research questions, as shown in Exhibit A-4.

A.5 Efforts to Minimize Burden on Small Business or Other Entities

No small businesses will be involved as respondents. Every effort has and will be made to minimize the burden on IHE faculty and Center staff. As noted below, the IHE Survey will take an average of 1 hour. Many of these respondents are ED funded, so the time will not have a significant economic impact on them or their institutions. The IHE materials collection (1 hour) will be limited to a subset of Project Directors. While the total burden on each of the Directors of the National Centers is 15 hours, there are very few respondents (12) so the total burden for the study of the National Centers is relatively small, and they are all ED funded.

A.6 Consequences of Less-Frequent Data Collection

Each data collection will occur only once. If the data collection is not completed, OMB, administrators, policymakers, and the public will not know whether this program is performing effectively; that is, if the products and services of grantees are commensurate with helping institutions of higher education produce qualified and well-trained personnel.

A.7 Special Circumstances Requiring Collection of Information in a Manner Inconsistent with Section 1320.5(d)(2) of the Code of Federal Regulations

There are no special circumstances associated with this data collection.

A.8 Federal Register Comments and Persons Consulted Outside the Agency

In accordance with the Paperwork Reduction Act of 1995, the IES published a notice in the Federal Register announcing the agency’s intention to request an OMB review of data collection activities. The first notice was published in the Federal Register, Volume 74, pages 8783 – 8784 on February 26, 2009, and provided a 60-day period for public comments.

The data collection instruments were developed by the evaluation research team led by Westat, with assistance from the Council for Exceptional Children (CEC) and consultant Bob Floden, under the direction of the IES COR. The feasibility of collecting materials from program applicants was tested in April, 2008. In November, 2008, potential IHE survey items were tested with applicants to the FY 2005 program. In January, 2009, the IHE surveys were pilot tested with nine applicants from the FY 2005 program. This pilot informed our time estimate; and the comments from the pilot test respondents were addressed in the revised instruments. In fall 2008, the interview protocol and inventory of products and services for the Study of the National Centers was pilot tested with two directors of National Centers, and comments were incorporated into the instruments.

In addition, a Technical Work Group (TWG) met in summer 2008 to discuss design and measurement issues. Their input led to substantial changes in the design and data collections. Members included:

Guido Imbens, Harvard University;
Dan Black, University of Chicago;
Mary Brownell, University of Florida;
Robert Floden, Michigan State University;
Dan Reschly, Vanderbilt University;
Jeff Smith, University of Michigan; and
Deborah Speece, University of Maryland.

A.9 Payments to Respondents

We propose to provide a payment of $30 to non-funded institutions for completing the survey. For the data from the non-funded applicants to meet the response rates standards for rigorous research, 80% of this group will need to respond. Thus, a payment will be instrumental in order for the study to reach this response rate given that the non-funded applicants will not be currently funded through this grant program and have no obligation or requirement to complete the survey or submit materials. There will be no payments made to the funded applicant respondents in the IHE Study or the respondents in the Study of the National Centers as both these groups will currently receive OSEP grant funding.

A.10 Assurance of Confidentiality

For the IHE study, no raw data or aggregate data that identifies the grantee or institution will be shared with OSEP project officers or any other Department staff. Survey respondents will be randomly assigned a 3-digit ID for completion of the survey. One item will request information about the respondent’s course of study, including the ED grant number, but no other personal information will be included in the survey or stored in the survey database. Data will only be shared outside of the immediate evaluation team through the construction by Westat of a separate restricted-use data file. Only the randomly assigned ID will remain in the restricted-use data file; all other applicant identifiers, such as the ED grant number, will have been removed in the construction of the file.

Once survey administration is complete, all files containing identifying information will be stored separately from those containing survey responses. Only selected members of the evaluation team will have access to files containing information that could be used to link survey data with identifiable respondents. Project staff will adhere to the regulations and laws regarding the confidentiality of individually identifiable information. All contractor staff members working on this effort with access to the data are required to sign a confidentiality pledge. Westat staff will aggregate data for descriptive analyses, which will be shared with the Department and will ultimately be included in the evaluation’s final report. For the IHE Study, a letter to participants will read as follows:

“Westat will protect the confidentiality of all information collected for the study and will use it for evaluation purposes only. No information that identifies any study participant will be released, except as required by law. Information from participating institutions and respondents will be presented at aggregate levels in reports. All institution-level identifiable information will be kept in secured locations, and identifiers will be destroyed as soon as they are no longer required.”

No promise of confidentiality will be made to the National Centers because the evaluation requires that the report name individual Centers and pair the names with results from the Center inventory and the results from expert panel review. For the Centers Study, a letter to participants will read as follows:

“For this data collection, Westat will report results for each Center. Data about your Center will be available to the Department of Education and may appear in evaluation reports.”

A.11 Questions of a Sensitive Nature

The questions included on the data collection instruments for this study do not involve sensitive topics.

A.12 Estimates of Respondent Burden

Exhibit A-2 includes our estimate of the reporting burden for respondents. In all, 726 responses will be required from 462 respondents (450 IHEs, 12 Centers), for a burden of 846 hours or 50,760 minutes, an average of 110 minutes per respondent.

Exhibit A-2. Estimates of Respondent Burden

Instrument	Respondent	Anticipated number completed	Hours per completion	Burden in hours	Burden in Dollars
IHE Study		(a)	(b)	a x b
IHE Survey	Project Director	450	1	450	$15,300
IHE Materials Collection	Project Director/ IHE Staff	240	1	240	$8,160
Study of the National Centers
Interview Protocol for Center Directors	Directors of National Centers	12	2	24	$816
Center Inventory of Products and Services	Staff of National Centers	12	8	72	$2,448
Center Products Request Form	Staff of National Centers	12	5	60	$2,040
Total burden		726		846	$28,764

NOTE: Assumes an hourly faculty rate of $34 per hour.

A.13 Estimates of the Cost Burden to Respondents

There are no annualized capital/startup or ongoing operation and maintenance costs associated with collecting the information. In addition to their time, which is estimated in Exhibit A-2, there will be photocopying costs for some of the respondents. At $0.11 per page, we estimate an average cost of $33 (300 pages) per Center and an average cost of $2.20 (20 pages) for each funded IHE. Some materials will be submitted electronically, so we consider this a high-end estimate. Costs for shipping materials will be assumed by the contractor.

A.14 Estimates of Annualized Government Costs

The total cost to the federal government for the Evaluation of the IDEA Personnel Preparation Program is $2,804,871, and the annual cost is $411,674 in FY 2008, $357,890 in FY 2009, $1,397,900 in FY 2010, and $637,407 in FY 2011. The annualized cost is $701,218.

A.15 Changes in Hour Burden

This is a first time submission.

A.16 Time Schedule, Publication, and Analysis Plan

Time Schedule

The schedule shown below in Exhibit A-3 displays the sequence of activities required to conduct these information collection activities and includes key dates for activities related to instrument design, data collection, analysis, and reporting.

Exhibit A-3. Time Schedule

Activities and deliverables	Date
Instrument design	Summer 2008 – Winter 2008-2009
Pilot test of Center instruments	September-November 2008
Pilot test of IHE survey	January 2009
Interviews with Center Directors	August 2009-April 2010
Conduct IHE Survey	Fall 2009
Collect IHE materials	January-February 2010
Expert review of IHE collected materials	March-May 2010
Collect Center materials
Expert review of Center materials	June-August 2010
Draft report	January 2011
Final report and restricted-use data file	September 2011

Publication and Analysis Plan

This section describes the analysis and reporting plans for data from each of the study’s five instruments. Exhibit A-4 shows how the survey items and other data sources will address each of the research questions proposed for the IHE Study and Study of the National Centers.

Exhibit A-4. Research Questions and Corresponding Survey Items

Research question	IHE Survey Items				Other Instruments
	Combined Priority Funded	Combined Priority Non-funded	Leadership Priority Funded	Leadership Priority Non-funded
IHE Study
Course of study status
What are the characteristics of the funded courses of study (i.e., admission criteria, field experiences, exit exam results, etc.)?	2-14 17-18, 20-22	5-16, 19-23	2-9, 12, 14-16	5-11, 14-17
How many new (i.e., previously nonexistent) courses of study are created with grant funding?	1		1
What becomes of courses of study that do not receive funding in a given year (e.g., continue operation, cease to exist, scale back enrollment, etc.)?		1-4, 18, 19		1-4, 13, 14
What proportion of non-funded applicants subsequently re-apply for grant funding, and what is the result?					Competition slates; Applications
Use of PDP funds
How do grantees use the funds from the PDP?	12, 13, 19	15	7, 8, 13	10	APRs; Student data reports
How do new recipients of a grant (i.e., IHEs with no previous program funding) use new funding compared to previous recipients of funding from the program? How does this differ by grant focus area?	12, 13, 19		7, 8, 13		APRs; Student data reports; Applications; Competition slates
How many new classes are created or new professors hired by funded courses of study? What are the nature of these classes and qualifications of these professors? How does this differ by grant focus area?	15, 16	17, 18	10, 11	12, 13	IHE Materials Collection Form and expert review; APRs
Students enrolled in courses of study
How many new individuals do grant recipients enroll in their courses of study and how many complete the course or drop out?	17	19	12	14	Student data reports; APRs
Course of Study Quality
What is the quality of new courses of study (or new classes within existing courses of study) funded by the PDP?	15	17	10	12	IHE Materials Collection Form and expert review
Study of the National Centers
What materials have been developed and how much technical assistance has been provided by the National Centers with funds awarded from the PDP?					Center Inventory of Products and Services: columns C and D
What has been the cost for the development and dissemination of these materials and services?					Center Inventory of Products and Services: column K
What are the quality, relevance, and usefulness of the materials and technical assistance provided by the National Centers using funds awarded from the PDP?					Center Product Request Form and expert review

IHE Survey

Data collected through the IHE Survey will be used to describe the use of federal funds to support the preparation of personnel to serve children with disabilities in several realms. Based on the survey, the final report will describe:

Characteristics of funded courses of study;
The uses of federal funding by grantees;
The association between funding and the status of courses of study;
The association between funding and admissions, enrollment, and completion of courses of study;
Changes made to courses of study subsequent to funding and the quality of those changes.

Once the data from the web-based survey have been collected, coded, and cleaned, relevant descriptive statistics (e.g., means, medians, ranges, and variance estimates) will be generated for each item. The data will subsequently be analyzed for major subgroups, such as grant priority and focus area (i.e., leadership, high-incidence disability, low-incidence disability, related services, early intervention, and minority institution), year of application, and previous funding history. Each of the data tables will be accompanied by a corresponding table with standard error estimates. Chi-squares, ANOVAs, and t-tests will be used to test for statistically significant differences across subgroups. The results from analysis of the IHE Survey data will be included in the study’s final report to describe the characteristics of funded courses of study (and, to a more limited degree, non-funded courses of study), uses of federal PDP funds, and the association between funding and changes in the course of study. Exhibit A-5 provides a sample table for data from the IHE Survey.

IHE Materials Collection Form

The IHE Materials Collection Form will be used to obtain documents to rate the quality of changes to the course of study through an expert review process. It will specifically focus on the nature and quality of new classes, new professors, and other changes to funded courses of study. Three members of the IHE Review Panel will rate the quality of changes made to each course of study using scoring rubrics. Once all of the completed rubrics have been received, the evaluation team will use the mean overall rating from the three panel members to generate the final quality ratings. Results of the ratings will be reported overall and by major subgroup (e.g., priority and focus area, funding year, and previous grant history). Exhibit A-6 provides an example of a summary table that could be prepared for communicating the results of the material review. The data from the review process will be integrated with results from the IHE Survey to describe the characteristics of funded and non-funded courses of study and the quality of changes subsequent to funding.

Exhibit A-5. Use of PDP Funds for 2008-2009

Category	Average percent of funds spent per category 2008-09
Funding for candidates
Funding for faculty
Development of curriculum or curriculum elements
Other

Exhibit A-6. Quality Rating Distribution for Changes to the Funded Course of Study

Course of study change	Level of quality
Course of study change	Mean quality rating (1-5)	Variance	Range
Overall mean rating for all changes to COS
Overall ratings of newly created or modified courses
Syllabi from newly created or modified courses
Teacher curriculum vitae from newly created or modified courses
Assessments from newly created or modified courses
New modules, training units, or fieldwork
Reorganized or relocated course of study
New mentoring program
New faculty
New certification area

Center Telephone Interview

The data collected in telephone interviews with Center staff will be used to provide contextual information for members of the expert review team and will also be used in the final report to describe each Center. Immediately following each interview, the contractor will write up interview notes following the format of the interview protocol. Prior to the meeting of the expert review panel, the notes will be used to prepare a 2-page narrative that describes each Center’s goals, structure, and operations. These Center-specific narratives will illustrate how a Center prioritizes and organizes its work.

Center Inventory of Products and Services

The first purpose of the Inventory of Products and Services will be to catalog and describe the cumulative accomplishments of the Centers. The second purpose will be to generate a sampling frame that can be used to select a sample of products/services to be reviewed and rated by the expert panels.

The final report will include rich descriptive information from the Inventory, for example, the variety of products and services generated by each Center, (e.g., class modules or conferences) the media used, the content areas addressed, and the relative costs for different types of products and services. This will be combined with data from the Telephone Interview to paint a detailed portrait for each of the 12 Centers, including alignment of the Center’s goals and objectives with its products and services.

Center Product Request Form

Using the Product Request Form, Centers will be asked to provide materials for each of the products or services sampled for review as well as background information to support a comprehensive understanding the product or service. Three members of the National Centers Review Panel will rate each product/service for quality, relevance, and usefulness using criteria linked to ED’s priority for the Center. Once all nine of the completed rubrics for a product or service have been received (three for quality, three for relevance, and three for usefulness), the evaluation team will distribute them to all the product’s reviewers. Members of the National Centers Review Panel will reconvene for a second meeting in summer of 2010. At that time, they will meet in their three-person teams to discuss any inherent challenges in the review and describe and justify their preliminary ratings. Based on these discussions, reviewers will have an opportunity to modify their preliminary ratings and generate a final rating for each criterion and for the overall quality, relevance, and usefulness scores.

The evaluation team will use the mean overall rating from the three panel members to generate the final quality, relevance, and usefulness ratings for the product. Results of the ratings will be reported for each Center in two metrics: the mean quality, relevance, and usefulness scores and the percentage of products rated of high quality, high relevance, or high usefulness. We propose classifying products with an average score of 4 or higher on a 5-point scale to be of high quality, relevance, or usefulness. Quality, relevance, and usefulness ratings will be broken out by signature works and non-signature works. Exhibits A-7 through A-9 provide an example of summary tables that could be prepared for communicating the overall results of the evaluation. The data from the review process will be integrated with descriptive information from the Telephone Interview and Inventory of Products and Services to describe both Center-specific and cross-Center results.

Exhibit A-7. Analysis of Quality

		Products nominated by Center				Products selected at random			Overall
	Total products inventoried	Number	Proportion of level of effort (1-100)	% High quality (1-100)	Mean quality (1-5)	Number	% High quality (1-100)	Mean quality (1-5)	Number	% High quality (1-100)	Mean quality (1-5)
Center for Training Personnel to Provide Evidence-Based Educational Services to Students with Autism Spectrum Disorders
National IHE Faculty Enhancement Center (IRIS)
IDEA and Research for Inclusive Settings: The Center for Training Enhancements (IRIS-II)
Center for Improving Teacher Quality
NIUSI Principal Leadership Academies Initiatives
National Center on High Quality Personnel in Inclusive Early Childhood Settings
National Early Childhood Training Enhancement Center
Center for Early Childhood Education/Early Intervention Personnel Preparation
Center to Inform Personnel Preparation Policy and Practices in Special Education
National Center for Leadership in Visual Impairment
Interdisciplinary Training in Analysis of Large-Scale Databases
Carolina Interdisciplinary Large-Scale Policy Research Training
Total

Exhibit A-8. Analysis of Relevance

		Products nominated by Center				Products selected at random			Overall
	Total products inventoried	Number	Proportion of level of effort (1-100)	% High relevance (1-100)	Mean relevance (1-5)	Number	% High relevance (1-100)	Mean relevance (1-5)	Number	% High relevance (1-100)	Mean relevance (1-5)
Center for Training Personnel to Provide Evidence-Based Educational Services to Students with Autism Spectrum Disorders
National IHE Faculty Enhancement Center (IRIS)
IDEA and Research for Inclusive Settings: The Center for Training Enhancements (IRIS-II)
Center for Improving Teacher Quality
NIUSI Principal Leadership Academies Initiatives
National Center on High Quality Personnel in Inclusive Early Childhood Settings
National Early Childhood Training Enhancement Center
Center for Early Childhood Education/Early Intervention Personnel Preparation
Center to Inform Personnel Preparation Policy and Practices in Special Education
National Center for Leadership in Visual Impairment
Interdisciplinary Training in Analysis of Large-Scale Databases
Carolina Interdisciplinary Large-Scale Policy Research Training
Total

Exhibit A-9. Analysis of Usefulness

		Products nominated by Center				Products selected at random			Overall
	Total products inventoried	Number	Proportion of level of effort (1-100)	% High usefulness (1-100)	Mean usefulness (1-5)	Number	% High usefulness (1-100)	Mean usefulness (1-5)	Number	% High usefulness (1-100)	Mean usefulness (1-5)
Center for Training Personnel to Provide Evidence-Based Educational Services to Students with Autism Spectrum Disorders
National IHE Faculty Enhancement Center (IRIS)
IDEA and Research for Inclusive Settings: The Center for Training Enhancements (IRIS-II)
Center for Improving Teacher Quality
NIUSI Principal Leadership Academies Initiatives
National Center on High Quality Personnel in Inclusive Early Childhood Settings
National Early Childhood Training Enhancement Center
Center for Early Childhood Education/Early Intervention Personnel Preparation
Center to Inform Personnel Preparation Policy and Practices in Special Education
National Center for Leadership in Visual Impairment
Interdisciplinary Training in Analysis of Large-Scale Databases
Carolina Interdisciplinary Large-Scale Policy Research Training
Total

A.17 Display of Expiration Date for OMB Approval

The Institute of Education Sciences is not requesting a waiver for the display of the OMB approval number and expiration date on the data collection instruments. All data collection instruments will display the expiration date for OMB approval.

A.18 Exceptions to Certification Statement

This submission does not require an exception to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).

1 With the FY 2006 competition, the program was retitled to the Personnel Development Program (PDP)

2 http://www.whitehouse.gov/omb/expectmore/summary/10001039.2003.html, last accessed November 6, 2008.

3 For purposes of the evaluation, a “course of study” is defined as the specific subunit of a school or department within an IHE that a PDP grant creates or enhances. For example, a course of study might be a master’s degree program to prepare early childhood educators or a training sequence leading to national certification on orientation and mobility. The course of study is the unit of analysis as opposed to the IHE because IHEs may house several courses of study, each of which may or may not receive funding from the PDP.

File Type	application/msword
File Title	Abt Double-Sided Body Template
Author	Abt Associates Inc
Last Modified By	Authorised User
File Modified	2009-10-14
File Created	2009-10-14