GPRA Section B Replace

U.S. Department of Education

National Writing Project

GPRA Annual Performance Indicators

SECTION B

Office of Management and Budget

Clearance Package Supporting Statement

And Data Collection Instruments

April 8, 2008

TABLE OF CONTENTS

Introduction 1

SECTION A. Justification

A.1 Circumstances Making the Collection of Information Necessary 1

A.2 Purposes and Uses of the Data 2

A.3 Use of Technology To Reduce Burden 2

A.4 Efforts To Identify Duplication 2

A.5 Methods To Minimize Burden on Small Entities 2

A.6 Consequences of Not Collecting Data 3

A.7 Special Circumstances 3

A.8 Federal Register Comments and Persons Consulted Outside the
Agency 3

A.9 Payments or Gifts 3

A.10 Assurances of Confidentiality 3

A.11 Justification of Sensitive Questions 5

A.12 Estimates of Hour Burden 5

A.13 Estimates of Cost Burden to Respondents 5

A.14 Estimate of Annual Cost to the Federal Government 6

A.15 Program Changes or Adjustments 6

A.16 Plans For Tabulation and Publication of Results 6

A.17 Approval To Not Display the OMB Expiration Date 8

A.18 Explanation of Exceptions 8

SECTION B. Collections of Information Employing Statistical Methods

B.1 Respondent Universe and Sample Size 9

B.2 Procedures for Collection of Information 10

B.3 Methods for Maximizing Response Rate and Dealing With
Nonresponse 25

B.4 Tests of Procedures and Methods 26

B.5 Consultations on Statistical Aspects of the Design 26

Appendix A. Elementary and Secondary Education Act (ESEA) of 1965, as Amended, Title 2, Part C, Subpart 2

Appendix B. Government Performance and Results Act of 1993 (GPRA), Section 4

Appendix C. Rubric #1: NWP Institute Folders

Appendix D. Rubric #2: NWP Teacher Packets

SECTION B. Collections of Information Employing Statistical Methods

B.1 Respondent Universe and Sample Size

NWP has about 190 sites with an average of 25 teachers attending each summer institute. Given these numbers, reviewing summer institute training syllabi and their participants’ teaching materials for all 190 sites would be extremely labor intensive. Thus, samples of NWP sites and teachers will be drawn. Further, the sampling of teachers from among the full set of teachers attending summer institutes in a given year would lead to a widely dispersed (across sites) set of sampled teachers. This would require staff to deal with a relatively large number of sites to obtain information from as few as one teacher per site. In addition to high data collection costs, this would most likely increase nonresponse rates. For these reasons, a two-stage sample design is recommended. A two-stage sample design involves drawing a sample of teachers from within a set of sampled sites.

Sites participating in the grantee’s own evaluation will be excluded from selection.

Stage One – Site Sample

The first-stage sampling will select 40 NWP sites. Because the number of teachers who participate in the summer institutes varies considerably across the sites, a probability proportional to size (PPS) sample will be used. With the PPS sampling technique, the probability of selecting an NWP site will be proportional to the number of teachers who participate in the summer institute at that site. Using PPS in the first stage of sampling ensures that teachers from larger sites have the same probability of getting into the sample as teachers from smaller sites. The advantage of using PPS at the first stage, in conjunction with a simple random sample (SRS) in the second stage (described below), is that this strategy will ensure that each teacher will have the same overall chance of selection.

The timing of site selection is dependent on two other activities being completed. The site sampling should take place immediately following OMB approval and receipt of the list of NWP sites with the anticipated number of teachers who will attend at each site. Stage 1 sampling will use anticipated numbers of teachers rather than actual numbers of teachers attending each summer institute. This stage of sampling addresses size differences across the NWP sites and it is important to acquire size information about each site to inform this stage of sampling. Because sites begin at different times throughout the summer (and thus would not have actual lists of attendees at the time of Stage 1 sampling), it is necessary to collect and use anticipated numbers from all sites.

Stage Two – Teacher Sample

The second stage of sampling will take place as soon as each summer institute begins and will be based on the list of teachers who actually attend the institute. Although teachers go through an involved application process, it is possible that some teachers who apply and are accepted into a summer institute ultimately may not participate in the summer training session. To reduce the potential for selecting a teacher who does not participate in the summer institute, teachers will be sampled from the list of participating teachers.

For the purpose of this study, we further define participating teachers who are eligible for the sample as those institute attendees arriving from a classroom and returning to a classroom. This stipulation is needed because some summer institute attendees may be writing specialists within a school district and may not be coming from a classroom. In parallel, some attendees will be coming from a classroom but may not be returning to a classroom—they may be delegated to deliver professional development to home district teachers in a specialist role on the strength of the institute experience.

A reserve sample will be drawn to ensure that the necessary materials are requested from five eligible teachers from each site. An additional two teachers (i.e., those sampled in excess of the five needed) will be sampled and held in reserve. In the event it is determined that a teacher selected for this GPRA measure reporting does not meet the criterion of coming from and returning to the classroom, he/she will be replaced with a teacher from the reserve list. This should ensure that we have a sample of five teachers from each site but not more than 200 from across the 40 sites.

Ideally, an NWP site’s website or application process should inform candidates of the possibility that a maximum of five of them will be randomly selected for the writing assignment sample. If an early notice is not feasible at the application stage, teachers will be informed when letters are sent notifying them of their selection and requesting their participation.

Thus, the second stage of sampling will result in a sample of 200 teachers who attended the sites selected in the first stage. This stage will use an SRS method to select five teachers at each of the 40 sites. Teacher sampling will take place on a site-by-site (or rolling) basis as institutes begin their summer training session. The next section describes the process and the timing for collecting the information needed to conduct the sampling. An additional two teachers will be randomly sampled and placed on the reserve sample list for the site. The reserve sample will offset the overall estimate that 30 percent of the teachers who attend the summer institute do not return to the classroom.

B.2 Procedures for Collection of Information

NWP is a single federal grantee that currently provides subgrants to about 190 summer institute sites operated by colleges and universities. On average, 25 teachers attend each summer institute. While a census of all institutes and teachers would be preferable, reviewing summer institute training syllabi and their participants’ teaching materials for all 190 sites for GPRA purposes would be extremely labor intensive. Thus, samples of NWP sites and teachers will be drawn. Sampling teachers from among the full set of teachers attending summer institutes would lead to a widely dispersed set of sampled teachers. As a result it would require staff to deal with a relatively large number of sites in order to obtain information from as few as one teacher. This would increase the data collection costs considerably. Thus, a two-stage sample design is being proposed.

The proposed information collection will use expert panelists who have known expertise in the field of writing and teaching writing to review and score materials provided by NWP sites and teachers who attended summer institutes. They will assess the quality of the NWP summer institutes. Using an expert panel is the most cost-efficient method to review materials from the greatest number of sites for GPRA reporting. This method also ensures objectivity since the grantee will not be responsible for collecting and reviewing the materials.

Statistical Methodology for Stratification and Sample Selection

The calculations for the two-stage NWP GPRA sample design are described using the following equations.

For the overall probability of selecting teacher j in site i, :

Let i denote a site, i = 1,…,N

where N is the number of summer training sites.

Let j denote a teacher attending site i, j = 1,…,t

where t is the teacher enrollment at site i.

Then T, the total number of teachers attending summer training institutes is given by

.

The number of teachers to sample per site is defined as m (the same number at all sampled sites) and the number of institutes to sample is defined as n. A total of M = m × n teachers will be sampled. Under this sample design, the probability p_i of selecting site i is

(PPS) .

Then, based on the sample described above, the probability of selecting teacher j from selected site i is

(SRS) .

The overall probability of selecting teacher j in site i is the product of these probabilities

= = = = which is equal for all teachers.

Estimation procedure. The two-stage design with PPS in the first stage and SRS in the second stage will result in teachers being self-weighted.

Degree of accuracy. For this data collection, not every NWP participant is asked to provide writing assignments, so results provide an estimate of the percentage of participants who improve the quality of the writing assignments given rather than the actual percentage. The level of precision is conveyed by providing the survey estimate plus or minus its margin of error. The margin of error is a function of the population size, the sample size, the estimated percentage, and the confidence level.

For the GPRA measure on the quality of NWP summer institute syllabi, the sample is 40 sites out of a population of 165 sites. (Approximately 25 of the 190 sites are expected to be excluded due to their participation in the grantee’s own evaluation.) Table B-1 shows the margin of error estimates for the analysis of institute syllabi quality at the institute level with a 95 percent level of confidence. Using the figures in Table B-1, a confidence interval for an estimated percentage can be computed. The confidence interval around the percentage reported is a statement about the range that the “true” percentage would fall in if repeated samples of 40 sites were taken from a population of 165 sites. As an example, the margin of error for a 20 percent estimate is 9.2 percent. A 95 percent confidence interval for the 20 percent estimate would be 20 percent ± 9.2 percent, meaning that if the data collection were conducted 100 times the “true” percentage in the population would fall between 10.6 percent and 29.4 percent in 95 of the 100 data collections.

Table B-1. Margin of error estimates for institute syllabi quality

Percent estimate	Margin of error
10% or 90%	6.9%
20% or 80%	9.2%
30% or 70%	10.5%
40% or 60%	11.2%
50%	11.5%

For the GPRA measure on improvement in writing assignment quality, the sample consists of 200 teachers out of a population of approximately 4,125 teachers (or 25 teachers multiplied by 165 sites). Table B-2 provides the margin of error estimates given the population size, the sample size, and a 95 percent level of confidence. The table can be interpreted as follows: if the percentage estimate is 70 percent, then the 95 percent confidence interval for the percentage is 70 percent ± 5.2 percent, or from 64.8 percent to 75.2 percent.

Table B-2. Margin of error estimates for improvement in writing assignment quality

Percent estimate	Margin of error
10% or 90%	3.4%
20% or 80%	4.5%
30% or 70%	5.2%
40% or 60%	5.6%
50%	5.7%

Unusual problems. We do not anticipate any unusual problems.

Power Projections. This does not apply.

Periodic data collection. NWP summer institute sites and teachers will be sampled annually. For this inaugural data collection, sites will be sampled and asked to provide syllabi and supporting materials in the summer of 2008. Sites will also be asked to provide lists of attendees. Teachers will be sampled and asked to provide pre-summer institute lesson plans and supporting materials at the start of the summer 2008 training. Teachers will be contacted beginning in late 2008 and asked to submit post-institute lesson plans and supporting materials.

Data Collection Summary

The first stage of data collection will be to obtain a list of NWP sites for the summer 2008 and the anticipated number of teachers to be enrolled at each site. Forty sites will be sampled using the PPS sampling method. These 40 sites will be contacted to obtain a copy of the course syllabi and supporting materials for the summer 2008 training session. A draft letter to sites appears in Exhibit B-1. Each site will be asked to provide the course syllabi and one or two other documents that reflect the content of the institute and are necessary to determine the quality of the NWP summer institute training syllabi. These materials will be compiled into an Institute Folder for each site.

The Institute Folder will include course goals/description, course assignments, project descriptions, writing requirements, and/or reading requirements/institute anthology. Information about what to include in the Institute Folder is included in the letter (see Exhibit B-1).

In addition to materials for the Institute Folder, at the start of their summer session, each site will be asked to compile a list of the teachers attending the training. From this list, a simple random sample (SRS) of five teachers will be drawn. Since the second part of the assessment for GPRA relies on a review of teacher lesson plans pre- and post- attendance at the summer institute training session, it is essential that teachers be coming from and returning to the classroom so they have the necessary materials. A reserve sample of two teachers will also be drawn for each site in the event a sampled teacher does not meet the necessary criterion.

ED’s contractor, Synergy, will be responsible for all data collection activities.

Exhibit B-1

Draft Letter to Summer Institute Site Directors

Dear Summer Institute Director:

I am writing to inform you that the National Writing Project (NWP) Summer Institute at <<summer institute site name>> has been randomly selected to participate in an important new data collection that will provide the U.S. Department of Education (ED) and Congress with needed information about the NWP program. In compliance with the Government Performance and Results Act of 1993 (GPRA), ED is required to collect and report on the performance of all programs funded with federal dollars. As you may know, the NWP program at <<summer institute site name>> is partially funded through a federal grant to the National Writing Project, located at the University of California at Berkeley.

ED is requesting that <<institute name>> prepare and submit the following materials to its contractor, Synergy.

• An Institute Folder containing materials developed for the summer 2008 training session that describe what will take place at the summer institute. The folder should contain the syllabus for the training session and one or two supporting documents that reflect the content of the institute. These should be existing documents that reflect factors such as the course goals/description, course assignments, project descriptions, writing requirements, and/or reading requirements/institute anthology. These documents should not include grant applications, annual reports, or statistical information.

• A list of teachers attending the summer 2008 training session. This list will be requested at the start of the summer institute and should include only those teachers in attendance. Any teacher who applied but decided not to attend should not be included on the list. (A sample of teachers will be drawn, contacted separately, and asked to provide copies of lesson plans and writing assignments.)

The information being collected is meant to strengthen data reporting for the NWP and thus help measure the NWP program’s progress toward meeting its overall goal of improving the quality of student writing and learning. Two new measures have been developed to report on various aspects of the quality of NWP’s process to meet this goal (i.e., training teachers to teach writing). The two new GPRA measures for the NWP program covered by this request are:

• The percentage of NWP summer institute training syllabi deemed to be of high quality by an independent review panel of qualified experts

• The percentage of NWP summer institute training session participants who improve the quality of the writing assignments given, as demonstrated through an independent review of lesson plans by a panel of qualified experts

Exhibit B-1 continued

The information collected will be reviewed by a panel of writing experts identified by ED. Although the participation of <<summer institute site name>> in this data collection is voluntary, without your help ED will not have the information it needs to monitor the program and report that information to Congress. All information you provide will be kept strictly confidential. Individual sites and teachers will not be identified in any reports about the program.

Thank you for your cooperation and participation in providing this important information to Congress for the NWP program. If you have any questions, please contact <<contact name>> at Synergy, our contractor. He/She can be reached at xxx-xxx-xxxx or <<email address>>. You may also reach me at (202) 260-3548 or at Margarita.Melendez@ed.gov.

Sincerely,

Margarita L. Meléndez

NWP Program Manager

U.S. Department of Education

Enclosure

OMB #: xxxxxxxxx

Exp.:

Teachers will be contacted and asked for their cooperation as part of the continual self-improvement that is a hallmark of the NWP. (Draft letters to teachers appear in Exhibits B-2A and B-2B.) Initially, sampled teachers will be asked to provide four writing assignment examples from the fall of the 2007-08 school year (the school year immediately preceding their attendance at the institute, i.e., pre-institute). To determine improvement in the quality of writing assignments given by teachers over time, teachers will be contacted again beginning in late 2008 and asked to provide four examples from the fall of the 2008-09 school year (i.e., post-institute). Teachers who have more than four writing assignments in the sampled time period will be given specific instructions on how to select assignments to be submitted for review. It is expected that four examples of writing assignments over the two-month period will provide adequate representation of the level of quality of assignments given without overburdening teachers or expert panel members.

The four pre-institute and four post-institute writing assignment examples that a teacher submits will be part of the pre- and post-Teacher Packets. It is understood that assignments are not dispensed to students in a vacuum; assignments emerge from teacher instructional themes or units. Accordingly, the teachers will be asked to supply the necessary context for understanding the assignment, such as unit plans. Importantly, it is not necessary for the teacher to supply entire units—just adequate context for panelists to make informed judgments regarding assignment quality. Teachers will be asked to identify the grade level and subject for each writing assignment.

Exhibit B-2A

Draft Letter to Teachers

(pre-institute materials)

Dear <<Teacher’s Name>>:

I am writing to inform you that you have been randomly selected to participate in an important new data collection that will provide needed information about the National Writing Project (NWP) program to the U.S. Department of Education (ED) and Congress. In compliance with the Government Performance and Results Act of 1993 (GPRA), ED is required to collect and report on the performance of all programs funded with federal dollars. The <<summer institute site name>> is partially funded through a federal grant to the National Writing Project, located at the University of California at Berkeley.

The <<summer institute site name>> identified you as a participant in its 2008 summer training institute. To help with this important information collection, I am requesting that you provide four (4) examples of assignments you gave your students that required writing. For purposes of this data collection, a “writing assignment” is defined as any assignment in which 50 percent or more of the task requires a writing response. Further, the assignments can be parts of lesson plans or unit plans that identify the assignments using writing. These should be assignments given between October 1, 2007, and November 30, 2007. The enclosed instructions provide details on how to select assignments and supporting materials. If you did not teach during the 2007-08 school year, or you are not planning on returning to the classroom for the 2008-09 school year, please contact <<contact name>> at Synergy, our contractor.

Your participation is very important as you are one of only 200 teachers selected nationwide for this data collection. The information being collected is meant to strengthen data reporting for the NWP and, thus, help measure the NWP program’s progress toward meeting its overall goal of improving the quality of student writing and learning. ED has established the following measure to report on the quality of various aspects of NWP’s process to meet this goal:

The percentage of NWP summer institute training session participants who improve the quality of the writing assignments given, as demonstrated through an independent review of lesson plans by a panel of qualified experts.

The lesson plans and supporting materials submitted by participants will be reviewed by a panel of experts in writing content and pedagogy who have been identified by ED. Although your participation is voluntary, without your help ED will not have the information it needs to monitor the program and report that information to Congress. All information you provide pursuant to this request will be kept strictly confidential. Individual teachers and schools will not be identified to expert panel members, or in any reports about the program. Should you choose not to participate, you will not be penalized in any way. You may also withdraw from the data collection at any time.

Thank you for your cooperation and participation in providing this important information to Congress for the NWP program. We will contact you again in the winter of 2008-09 to request examples of fall 2008 lesson plans and writing assignments.

Exhibit B-2A (continued)

Please send the materials in the postage-paid envelope provided. By sending the requested materials, you are agreeing to be part of this important data collection regarding the National Writing Project. If you have any questions, please contact <<contact name>> at Synergy. <<He/She>> can be reached at xxx-xxx-xxxx or <<email address>>. You may also reach me at (202) 260-3548 or Margarita.Melendez@ed.gov.

Sincerely,

Margarita L. Meléndez

NWP Program Manager

U.S. Department of Education

Enclosures

OMB #: xxxxxxxxx

Exp

Exhibit B-2A (continued)

Instructions for Selecting Assignments

1. Look at your lesson plans and/or unit plans for the period October 1, 2007, through November 30, 2007.

2. Identify the assignments you gave to your students in which 50 percent or more of the task required writing. These can be parts of lesson plans or unit plans that identify the assignments using writing. Do not include writing samples from students even if they were used as part of the assignment.

3. Count and number the assignments involving writing across all classes that you teach.

4. Selecting assignments:

1. If four or fewer assignments were given, then select all.

2. If more than four assignments were given during the period, order the assignments by date given, and use the table below to determine which assignments to select.

Number of assignments	Select assignment number:
5	1, 2, 4, 5
6	1, 2, 5, 6
7	1, 3, 5, 6
8	1, 3, 6, 8
9	1, 4, 6, 9
10	1,4, 7, 10
11	1, 5, 7, 11
12	1, 5, 8, 12
13 or more	Call Synergy contact

5. On the cover sheet provided for each assignment selected, fill in the grade level, subject area, class or course level (e.g., special education, gifted, honors, AP, IB, ESL, remedial), timeframe for assignment, and assessment method. Provide portions of lesson or unit plans that describe the assignments’ timeframe and assessment method only if this information is not apparent in the assignments themselves.

6. Return the materials in the postage-paid envelope.

OMB #: xxxxxxxxxxx

Exp.

Exhibit B-2A (continued)

Assignment Cover Sheet

Instructions: Please attach this sheet to the front of each selected assignment and provide the following information.

Grade Level(s):

Subject area:

Class/Course level:

Timeframe for assignment:

Assessment method:

OMB #: xxxxxxxxxxx

Exp.

Exhibit B-2B

Draft Letter to Teachers

(post-institute materials)

Dear <<Teacher’s Name>>:

During summer of 2008, you participated in an important data collection conducted by the U.S. Department of Education (ED) regarding the National Writing Project (NWP). At that time, you provided examples of writing assignments that you gave to your students during the fall of 2007. We indicated then that we would be contacting you again in winter of 2008-09 to obtain examples of assignments that required writing given in fall of 2008.

This information is being requested in compliance with the Government Performance and Results Act of 1993 (GPRA), which stipulates that ED is required to collect and report on the performance of all programs funded with federal dollars. You may recall that NWP at <<summer institute site name>> was partially funded in 2008 with federal funds through a grant to the National Writing Project, located at the University of California at Berkeley.

At this time, I am requesting that you provide four (4) examples of assignments you gave your students that required writing. For purposes of this data collection, a “writing assignment” is defined as any assignment in which 50% or more of the task requires a writing response. Further, the assignments can be parts of lesson plans or unit plans that identify the assignments using writing. These should be assignments given between October 1, 2008, and November 30, 2008. The enclosed instructions provide details on how to select assignments and supporting materials if you have more than four assignments that meet the criteria. These materials will be reviewed by a panel of writing experts and used to respond to the GPRA measure by reporting on the “percentage of NWP summer institute training session participants who improve the quality of the writing assignments given, as demonstrated through an independent review of lesson plans by a panel of qualified experts.”

Your continued participation is appreciated and important to the continued success of the NWP program. Although your participation is voluntary, without your help ED will not have the information it needs to monitor the program and report that information to Congress. All information you provide will be kept strictly confidential. Individual teachers and schools will not be identified to expert panel members or in any reports about the program. You will not be penalized should you choose to withdraw from the data collection.

Thank you for your continued cooperation and participation in providing this important information to Congress for the NWP program. If you have any questions, please contact <<contact name>> at Synergy, our contractor. He/She can be reached at xxx-xxx-xxxx or <<email address>>. You may also reach me at (202) 260-3548 or Margarita.Melendez@ed.gov.

Sincerely,

Margarita L Meléndez

NWP Program Manager

U.S. Department of Education

Enclosures

OMB #: xxxxxxxxx

Exp.:

Exhibit B-2B (continued)

Instructions for Selecting Assignments

1. Look at your lesson plans and/or unit plans for the period October 1, 2008, through November 30, 2008.

2. Identify the assignments you gave to your students in which 50 percent or more of the task required writing. These can be parts of lesson plans or unit plans that identify the assignments using writing. Do not include writing samples from students even if they were used as part of the assignment.

3. Count and number the assignments involving writing across all classes that you teach.

4. Selecting assignments:

1. If four or fewer assignments were given, then select all.

2. If more than four assignments were given during the period, order the assignments by date given, and use the table below to determine which assignments to select.

Number of assignments	Select assignment number:
5	1, 2, 4, 5
6	1, 2, 5, 6
7	1, 3, 5, 6
8	1, 3, 6, 8
9	1, 4, 6, 9
10	1,4, 7, 10
11	1, 5, 7, 11
12	1, 5, 8, 12
13 or more	Call Synergy contact

5. On the cover sheet provided for each assignment selected, fill in the grade level, subject area, class or course level (e.g., special education, gifted, honors, AP, IB, ESL, remedial), timeframe for assignment, and assessment method. Provide portions of lesson or unit plans that describe the assignments’ timeframe and assessment method only if this information is not apparent in the assignments themselves.

6. Return the materials in the postage-paid envelope.

OMB #: xxxxxxxxxxx

Exp.

Exhibit B-2B (continued)

Assignment Cover Sheet

Instructions: Please attach this sheet to the front of each selected assignment and provide the following information.

Grade Level(s):

Subject area:

Class/Course level:

Timeframe for assignment:

Assessment method:

OMB #: xxxxxxxxxxx

Exp.

Identification of Expert Panel Members

Once all of the Institute Folders and Teacher Packets have been collected, a panel of experts in the field of writing and writing instruction will be assembled.

The expert panel will consist of up to ten individuals who are recognized as experts in the field of writing, particularly from the perspective of teaching writing. Panel members will be selected based on two primary qualities: content expertise as a practitioner in writing and teaching writing and the capacity to work on deadline given the volume of material to be reviewed, which is important given the five intense days of the review. Previous expert panel experience would be a plus, but it is not a requirement. In summary, panelists will be well-qualified writing specialists with experience in writing instruction and will understand writing across the curriculum and national standards for teaching writing.

The 2009 expert panel members will be drawn from Maryland, Virginia, and the District of Columbia.

Expert Panel Review Process

A panel of experts will be assembled and tasked with reviewing syllabi and supporting materials from the sampled summer institutes and applying a rubric to determine their quality. The expert panel members will also be given a separate task of reviewing teachers’ lesson plans and writing assignments and applying a rubric to determine the quality of the writing assignments given to students by their teachers.

The panel will meet for 5 weekdays to review and score the Institute Folders and Teacher Packets.

Rubrics for Use by the Expert Panel

Rubrics will be used to support the expert panel’s review work. Rubrics will be used to train expert reviewers on the two tasks and will allow the expert panelists to record their judgments in a consistent and reliable fashion. The experts will exercise independent assessments about the materials they review and will be asked to make judgments about the overall quality of submissions. The use of rubrics for expert panel review of Institute Folders and writing assignments in Teacher Packets means that there will be a working plan for responding to the NWP GPRA measures. Rubrics are essential elements to the work of the expert panels.

Appendix C contains the rubric the writing experts will use to review and score the Institute Folder materials. The rubric allows expert panelists to determine the quality of the materials contained in the Institute Folder. In cases where the two independent scores are highly divergent, it will be necessary to examine those differences and discuss them with the two reviewers. If there is more than a 15 point discrepancy in scores, the Folder would be subject to further discussion. It is anticipated that 10 percent of the Folders will require this examination.

After completing their review of the 40 site Institute Folders, panel members will assess the teacher materials included in the Packets. For each sampled teacher, two Teacher Packets will be prepared. One packet will contain examples of pre-institute lesson plans and supporting materials. The second packet will contain post-institute lesson plans and supporting materials. All materials will be coded to protect source (institute) identification as well as to ensure teacher and school anonymity. Grade level, subject and course information for each writing assignment will be identified and included on a cover sheet to facilitate review. Expert panelists will review and blindly rate the writing assignment examples from teachers in terms of quality, using a rubric designed specifically for the task. By the phrase, blindly rate, we mean that the panelists will review assignments without knowledge of whether the assignments were given pre- or post-institute, or which institute the teacher attended.

Appendix D contains the rubric the experts will use to review and score the writing assignments and supporting materials in the Teacher Packets. Divergent scores for Teacher Packets will be handled in a similar manner as the Institute Folders.

B.3 Methods for Maximizing the Response Rate and Dealing With Nonresponse

There are two key aspects to maximizing the number of sample members from whom data are collected: minimizing the number of sample members lost through attrition and completing data collection with the maximum number of sample members who are retained in the sample.

To minimize attrition, we will obtain contact information for attendee-teachers, including the names of the schools and districts where they teach. We will inform teachers of the importance of this data collection for program monitoring purposes and that Congress may use this information to make future funding decisions for the NWP program.

A 90 percent response rate is anticipated for sites, and an 80 percent response rate is anticipated for teachers, both pre-institute and post-institute. To ensure the highest response rate possible, the following steps will be taken:

• Mail advance letters addressed to individuals (site directors or teachers) and signed by an ED official

• Provide a helpline so site directors and teachers can ask questions

• Promise confidentiality

• Have the NWP grantee support the information collection by requesting site directors and teachers to cooperate

• Minimize time between contacts with teachers

• Follow up with site directors and teachers by mail and telephone

We will follow up with any sites that do not respond to our initial request for data. We will also follow up with teachers who do not initially provide pre-institute materials or post-institute materials. Every effort will be made to obtain pre-institute materials; teachers who fail to provide pre-institute materials will not be contacted during the data collection period for post-institute materials.

B.4 Tests of Procedures and Methods

We conducted a pretest of the Institute Folder and Teacher Packet rubrics between March 3 and March 14, 2008. Four individuals with expertise in writing and teaching writing reviewed syllabi from nine NWP Summer Institutes conducted in 2007. These individuals also reviewed nine writing assignments representing a variety of grade levels and disciplines. The purpose of the pretest was to determine the usability and reliability of both rubrics.

The results of the pretest showed that the rubrics are easy to use. The pretesters felt that the Institute Folder rubric is an effective tool for measuring the quality of the Institute in accomplishing the NWP goals. The pretesters felt the indicators are clearly stated and reflect important components for assessing the quality of the Institute. The pretesters also felt that the Teacher Packet rubric is a very effective instrument for measuring the quality of writing assignments and felt it is appropriate for all grade levels.

The latitude given to pretesters in determining scores does affect the reliability of the instruments. For both rubrics, there was considerable variation in scores across pretesters for the same materials. The rubric scoring procedures were not changed in response to the variation in scores because it was determined that the view points of experts are important to the process of assessing quality. We did not want to constrain the view points of experts with a prescriptive scoring procedure. Additionally, it is possible that reliability will improve with a single, combined training where panelists can benefit from being together so any questions can be raised and addressed group-wide.

The only modification to the rubrics based upon the pretest feedback was to incorporate the deduction of up to 5 points from the total score for poor presentation of materials including grammatical and typographical errors. The pretesters found these errors particularly egregious for professionals who are delivering a program designed to teach writing.

B.5 Consultations on Statistical Aspects of the Design

Westat has prepared the sampled design. Kim Standing (301-294-3943) is the project leader for the NWP GPRA Data Collection, and Annie Lo (301-738-8374) is the lead statistician.

2