GPRA Section A Replace

U.S. Department of Education

National Writing Project

GPRA Annual Performance Indicators

Section A

Office of Management and Budget

Clearance Package Supporting Statement

And Data Collection Instruments

April 8, 2008

TABLE OF CONTENTS

Introduction 1

SECTION A. Justification

A.1 Circumstances Making the Collection of Information Necessary 1

A.2 Purposes and Uses of the Data 2

A.3 Use of Technology To Reduce Burden 2

A.4 Efforts To Identify Duplication 2

A.5 Methods To Minimize Burden on Small Entities 2

A.6 Consequences of Not Collecting Data 3

A.7 Special Circumstances 3

A.8 Federal Register Comments and Persons Consulted Outside the
Agency 3

A.9 Payments or Gifts 3

A.10 Assurances of Confidentiality 3

A.11 Justification of Sensitive Questions 5

A.12 Estimates of Hour Burden 5

A.13 Estimates of Cost Burden to Respondents 5

A.14 Estimate of Annual Cost to the Federal Government 6

A.15 Program Changes or Adjustments 6

A.16 Plans For Tabulation and Publication of Results 6

A.17 Approval To Not Display the OMB Expiration Date 8

A.18 Explanation of Exceptions 8

SECTION B. Collections of Information Employing Statistical Methods

B.1 Respondent Universe and Sample Size 9

B.2 Procedures for Collection of Information 10

B.3 Methods for Maximizing Response Rate and Dealing With
Nonresponse 25

B.4 Tests of Procedures and Methods 26

B.5 Consultations on Statistical Aspects of the Design 26

Appendix A. Elementary and Secondary Education Act (ESEA) of 1965, as Amended, Title 2, Part C, Subpart 2

Appendix B. Government Performance and Results Act of 1993 (GPRA), Section 4

Appendix C. Rubric #1: NWP Institute Folders

Appendix D. Rubric #2: NWP Teacher Packets

SUPPORTING STATEMENT

FOR PAPERWORK REDUCTION ACT SUBMISSION

INTRODUCTION

The National Writing Project (NWP) is a directed grant program funded through the U.S. Department of Education (ED) to improve the quality of student writing and learning in grades K-16 by training teachers to teach writing. The NWP is a single grantee that supports many types of local teacher training programs, including summer training institutes.

ED is seeking approval to:

• Collect and review syllabi and supporting materials from NWP summer institute sites

• Collect lists of institute attendees

• Collect and review pre- and post-institute lesson plans and supporting materials from teachers who attended NWP summer institute training sessions and returned to the classroom

This package describes the data collection and review process for two new annual performance measures, developed pursuant to the Government Performance and Results Act of 1993 (GPRA), for the NWP program. These data are necessary to assess the performance of the NWP program in meeting its stated goals and objectives. The data collection will occur in two phases. Although all ED grantees are required to provide performance data on an annual or periodic basis, the respondents for this data collection are participants in the NWP grant; therefore, their participation in the data collection is voluntary.

Each year, the NWP receives a single federal grant. The NWP, in turn, provides subgrants to about 190 sites operated by colleges and universities. These sites provide support services to teachers during the school year and offer intensive summer institutes. Summer institutes are the focus of this reporting effort. An average of 25 teachers attend each summer institute. While a census of all institutes and teachers would be preferable, reviewing summer institute training syllabi and their participants’ teaching materials for all 190 sites for GPRA purposes would be extremely labor intensive. Thus, samples of NWP sites and teachers will be drawn. Further, the sampling of teachers from among the full set of teachers attending summer institutes in a given year would lead to a widely dispersed (across sites) set of sampled teachers. This would require staff to deal with a relatively large number of sites to obtain information from as few as one teacher per site. In addition to high data collection costs, this would most likely increase nonresponse rates. For these reasons, a two-stage sample design is being used, where a sample of 200 teachers will be drawn from within a set of sampled 40 sites.

SECTION A. JUSTIFICATION

A1. Circumstances Making the Collection of Information Necessary

Information for the National Writing Project (NWP) GPRA Annual Performance Indicators is being collected in compliance with the Elementary and Secondary Education Act (ESEA) of 1965, as amended, Title 2, Part C, Subpart 2, Sections 2331-2332 U.S.C. 6701-6702 (provided in appendix A), the Government Performance and Results Act of 1993 (GPRA), Section 4 (1115) (provided in appendix B), and the Education Department General Administrative Regulations (EDGAR), 34 CFR 75.253. EDGAR states that recipients of discretionary grants must submit a final performance report demonstrating that substantial progress has been made toward meeting the approved objectives of the project. In addition, discretionary grantees are required to report on their progress toward meeting the performance measures established for the ED grant program.

The overall goal of the NWP program is to improve the quality of student writing and learning. Two new measures have been developed to report on the performance of NWP’s process to meet this goal (i.e., training teachers to teach writing). The two GPRA measures that have been established for this program, which are covered by this request are:

1. The percentage of NWP summer institute training syllabi deemed to be of high quality by an independent review panel of qualified experts.

2. The percentage of NWP summer institute training session participants who improve the quality of the writing assignments given, as demonstrated through an independent review of lesson plans by a panel of qualified experts.

A2. Purposes and Uses of the Data

The purpose of this data collection is to obtain the data necessary to assess the program’s performance by determining the program’s progress toward meeting its goals.

The data will be used for program monitoring and will be included in ED’s annual program performance reporting.

A3. Use of Technology To Reduce Burden

This data collection will not involve electronic data submissions. Sites will need to provide lists of the approximately 25 teachers attending their summer institutes and materials for the Institute Folders. Teachers will need to submit copies of lesson plans and assignments using writing that they have given to the students they teach. If sites or teachers prefer to send materials electronically, they will be permitted to do so; however, it will not be required.

A4. Efforts To Identify Duplication

There is no duplication of reporting. The information requested for this reporting is not collected or reported elsewhere.

A5. Methods To Minimize Burden on Small Entities

The data collection does not involve small businesses or other small entities.

A6. Consequences of Not Collecting Data

Annual or periodic performance reporting is stipulated in GPRA 1993, Section 4. Currently, the NWP grantee provides limited information annually on the number of summer institutes that adhere to the NWP model. The proposed data collection will provide additional information not currently collected about program performance and reported by the grantee. The data will be collected and reported annually.

A7. Special Circumstances

There are no special circumstances that would require the collection to be conducted in a manner inconsistent with OMB guidelines.

A8. Federal Register Comments and Persons Consulted Outside the Agency

The 60-day notice was published in the Federal Register on February 5, 2008 on page 6714. A copy of this notice is attached as Exhibit A-1. We did not receive any comments.

Members of the National Writing Project main office located at the University of California at Berkeley have been contacted regarding this project on several occasions including a conference call on October 10, 2007. The participants from the NWP main office on the conference call were:

• Richard Sterling, Executive Director

• Judy Buchanan, Deputy Director

• Paul LaMahieu, Director, Research and Evaluation

• Elyse Eidman-Aadahl, Director, National Programs and Site Development

A9. Payments or Gifts

There are no payments or gifts to grantees or NWP participants in support of the data collection.

A10. Assurances of Confidentiality

Activities to ensure confidentiality of individuals and their attributes will be conducted in accordance with the Privacy Act of 1974, which safeguards individuals against invasion of personal privacy by (1) permitting them to determine what personal records are collected, maintained, used, or disseminated; (2) preventing personal records from being used for purposes other than those to which they agreed; and (3) giving individuals access to their records and allowing them to correct or amend those records. Project staff will adhere to the regulations and laws regarding the confidentiality of individually identifiable information. All contractor staff members working on this reporting effort with access to the data are required to sign a confidentiality pledge.

Exhibit A-1. Federal Register Notice

Materials received from sites will most likely contain the name of the summer institute. Project staff will apply a numeric identification label to the materials and then obscure any identifying information such as the name of the institution. This will ensure that reviewers are unable to identify the site, thus protecting the site and making sure reviews are based solely on the content. The identification label will allow project staff to keep track of materials and follow up with sites that have not submitted the requested materials.

Teachers who are selected to participate will be assured that the materials they provide will be kept confidential and that neither they nor their schools will be identified in any reports. Written assurances will be provided in advance letters. Lesson plans and supporting materials will be labeled with a numeric identification, and any information that could identify the teacher, his/her school, or the NWP summer institute he/she attended will be obscured. The information will be entered into a management system, which will permit project staff to identify the site and whether the lesson plans and materials are pre-institute or post-institute. By tracking teacher-provided materials, project staff will be able to follow up with teachers who have not provided materials and to link pre- and post-institute materials during analysis. The expert panel members will not have access to the information that identifies teachers and sites.

A11. Justification of Sensitive Questions

There are no questions of a sensitive nature.

A12. Estimates of Hour Burden

Exhibit A-2 below presents a summary of estimated response burden for each data collection activity in terms of both total estimated hours and total estimated cost. This is an annual reporting cycle, with data being collected in two phases. During the first phase of the data collection cycle, the sources of data will be:

• Summer institute syllabi and supporting materials

• Summer institute teacher attendance lists

• Teacher pre-institute lesson plans and supporting materials (4 examples)

During the second phase of the data collection cycle, the only source of data will be teacher post-institute lesson plans and supporting materials (4 examples).

A13. Estimate of Cost Burden to Respondents

There are no additional respondent costs associated with this data collection other than the hour burden estimated in item A12.

Exhibit A-2. Annual Burden Estimates, by Data Source.

Data Source	Estimated Number of Respondents	Estimated Annual Burden per Responses (in Hours)	Total Estimated Annual Burden (in Hours)	Total Estimated Annual Cost (in Dollars)
Summer Institute syllabi and supporting materials	40	1 hour	40	$2,0001
Summer Institute teacher attendance lists	40	½ hour	20	$1,0001
Teacher pre-institute lesson plans and supporting materials	200	1 hour	200	$5,0002
Subtotal for phase 1	280	2 ½ hours	260	$8,000
Teacher post-institute lesson plans and supporting materials	200	1 hour	200	$5,0002
Subtotal for phase 2	200	1 hour	200	$5,000
Total	480	3 ½ hours	460	$13,000

¹ Based on an estimated hourly rate of $50 for administrators.

² Based on an estimated hourly rate of $25 for teachers.

A14. Estimate of Annual Cost to the Federal Government

The estimated cost to the federal government of conducting the performance reporting for NWP is based on the government’s contracted cost of the data collection and related study activities along with personnel cost of government employees involved in oversight. The estimated cost to the Federal Government is approximately $150,000 each year.

A15. Program Changes or Adjustments

The annual reporting burden in item 13 of the OMB 83-I shows 460 hours. This is a new reporting requirement for NWP that is in addition to current reporting requirements for the program. The grantee will continue to report annually information on the number of sites adhering to the NWP model and its own stated goals and objectives using the Department’s approved ED 524-B Annual Performance Report form.

A16. Plans for Tabulation and Publication of Results

The data collected through the expert panel reviews of the Institute Folders and Teacher Packets will be entered into a data entry system that will mirror the rubrics used by the expert panel members. Each rubric score generated by panel members will be entered into the system. The system will track information based on previously assigned IDs for sites and teachers. Although blind to reviewers, the ID will also indicate whether the score is for a pre-institute or a post-institute set of teacher materials. The data entry system will not contain any information that could identify an individual site or teacher. The system will facilitate data verification and will ensure data are recorded accurately.

Analyzing the Data from Rubric # 1 Institute Folders

Since each Institute Folder (i.e., the institute syllabus and supporting materials) will be reviewed by two expert panel members, there will be two sets of scores for each of the 40 sampled sites. Based on the rubric for the Folder review, sites can receive a maximum of 100 points from each reviewer. This means an institute would have a maximum combined score of 200 points. (Prior to the expert panel review meeting, the benchmark for high quality will be established in collaboration with ED’s NWP program office.) For example, the benchmark (or cut-off point) for high quality could be set at 80 out of 100 points, or 80 percent of the allowable points. If this were to be the case, then the percentage of institutes that have a combined score (from the two reviews) of 160 points or higher would be calculated and reported.

Analyzing the Data from Rubric # 2 Teacher Packets

The analysis of Teacher Packet scores will be similar to the analysis of the Folder scores. Each Teacher Packet will be reviewed by two panel members, so there will be two scores for both the pre-institute and post-institute materials. A combined score will be calculated based on the scores from the two reviewers for both the pre-institute and the post-institute materials. A simple change score will be calculated to determine whether there was an increase in the quality of materials between the two time periods. For example, if the pre-institute combined score for a Teacher Packet was 150 points and the post-institute combined score was 175 points, the change score would be 25 points. If the difference between the two scores is greater than zero, then the quality of the writing assignments given between the two time periods will be counted as having improved. If the change from pre- to post- is zero or a negative number, then no improvement will be recorded.¹

There are no plans for publication beyond reporting results to the Department’s Budget Service for compliance with GPRA.

Table A-1 shows the schedule for the data collection and performance reporting for 2009. A similar schedule will be followed for subsequent years.

Table A-1. Project schedule for 2009 GPRA reporting

Activity	Expected Completion Date
Receive OMB approval	6/5/08
Sample summer institute sites	6/6/08
Notify sites of selection	6/9/08
Begin collecting summer institute materials	6/23/08
Begin collecting lists of teachers participating in summer institutes	6/23/08
Begin sampling teachers	6/25/08
Begin collecting pre-institute materials from teachers	7/28/08
Complete collection of lists of teachers participating in summer institutes	8/1/08
Complete sampling of teachers	8/10/08
Complete collection of summer institute materials	8/25/08
Complete collection of pre-institute materials from teachers	10/21/08
Begin collecting post-institute materials from teachers	12/1/08
Complete collection of post-institute materials from teachers	1/30/09
Conduct review of materials by expert panel members	3/16 – 3/20/09
Begin data entry	3/23/09
Complete data entry and cleaning	4/15/09
Begin analyzing data	4/16/09
Complete analysis	5/15/09
Submit results memo	6/15/09

A17. Approval To Not Display the OMB Expiration Date

The OMB number and expiration date will be displayed on all correspondences and materials sent to NWP site directors and teachers.

A18. Explanation of Exceptions

There are no exceptions to the certification statement.

1 We will review the distribution of scores on pre-lesson plans to determine if teachers already scored very high on the pre-institute lesson plan review. This may indicate a ceiling effect and may alter how the Department uses these data. For example, the Department may decide to exclude teachers who receive a maximum score on pre-institute lesson plan assessment from the analysis.

2