OMB-ED QandA May 2008

Response to OMB comments on

200804-1855-001: National Writing Project

GPRA Annual Performance Indicators

1. Could ED develop a scale for respondents to use when allocating the number of points within each rubric (for example, 20 points means fully meets criteria, 15 points means meets the majority but not all criteria, etc.)?

We have revised the rubrics adding a scale for the experts to use when evaluating the Institute Folders and Teacher Packets. Please see revised rubrics.

2. Is the survey intended to provide annual estimates of change? If so, what level of change is reasonably expected to occur from year to year?

Yes, we intend to conduct the expert panel review on an annual basis and therefore change from year to year can be assessed. At the present time we are not in a position to define targets for annual change. This type of information has not been collected for this program before, and we are not aware of a similar program study from which to extrapolate in determining targets and annual estimates of change. We plan to look at the baseline data and determine the appropriate targets for the program’s GPRA measures for subsequent years. If the baseline percentages are very high, then we expect to establish maintenance scores. Otherwise, we expect to establish reasonable growth estimates in program performance.

3. In Supporting Statement B2, the discussion of under “degree of accuracy” does not indicate what level of precision ED requires for its key estimates (only what level will be obtained with the planned sample for various hypothetical estimates). Please discuss, relating this answer back to your answer in #2 above.

Given the lack of historical information on which to establish reasonable estimates as stated in #2 above, ED accepts the margins of errors for the key estimates, which will be no larger than 12 percent for the institute-level estimates and no larger than 6 percent for the teacher-level estimates.

Considering the dimensions and corresponding indicators below, assess the quality of the institute in accomplishing the NWP goals as demonstrated through the Institute Folder.

The NWP Institute Goals:

1. To improve student writing and learning in K-16 classrooms

2. To extend the uses of writing that is transferable across grades and subject areas

3. To provide an effective development plan

4. To build on the role of successful classroom teachers.

Dimension	Indicators	Points
Course Design	The expert panel will take into consideration the extent to which the institute’s course design:  is grounded in what current research indicates is a good model for teaching writing to adult learners  builds upon the expertise of classroom teachers from different grades and subject areas  provides adequate time for teachers to acquire, practice, and reflect on new concepts and skills 17 – 20 points Very high quality 13 – 16 points High quality 9 – 12 points Moderate quality 5 – 8 points Low quality 1 – 4 points Very low quality 0 points Not enough evidence to judge	/ 20
Instructional Approaches	The expert panel will take into consideration the extent to which the institute’s instructional approaches used in the course:  are based on best practices for teaching writing to adult learners  reflect current research or professional wisdom on approaches for teaching teachers  provide opportunities to demonstrate newly-learned strategies across grade and subject areas 25 – 30 points Very high quality 19 – 24 points High quality 13 – 18 points Moderate quality 7 – 12 points Low quality 1 – 6 points Very low quality 0 points Not enough evidence to judge	/ 30

Dimension	Indicators	Points
Course Content	The expert panel will take into consideration the extent to which the institute’s course content:  draws on accepted theories on how to teach writing  includes research-informed instructional strategies for embedding and extending K-16 student writing  teaches methods of assessment that align with current best practices in K-16 writing assessment  teaches institute participants how to use assessment findings to guide instruction in K-16 student writing and learning  helps participants develop a quality professional development plan to improve K-16 writing in their home district and schools 25 – 30 points Very high quality 19 – 24 points High quality 13 – 18 points Moderate quality 7 – 12 points Low quality 1 – 6 points Very low quality 0 points Not enough evidence to judge	/ 30
Methods of Assessment	The expert panel will take into consideration how well the institute’s methods of assessment:  provide feedback to participants about their own command of writing  provide feedback to participants on how to improve student writing and learning 17 – 20 points Very high quality 13 – 16 points High quality 9 – 12 points Moderate quality 5 – 8 points Low quality 1 – 4 points Very low quality 0 points Not enough evidence to judge	/ 20
Presentation	The expert panel will take into consideration the overall presentation of materials including typographical and grammatical errors. Deduct up to 5 points for poor presentation.
	Total Points	/ 100

Rubric #2: NWP Teacher Packets

Considering the following dimensions, assess the quality of the writing assignments in the Teacher Packet.

Grade Level(s):

Subject area:

Class/Course level:

Timeframe:

Assessment method:

	Points
Given the grade level and subject area, assess the extent to which you agree with the following:
The writing activities are cognitively appropriate 16 – 20 points Strongly agree 11 – 15 points Agree 6 – 10 points Disagree 1 – 5 points Strongly disagree 0 points Not enough evidence to judge	/ 20
The assignments reflect the use of effective instructional approaches for teaching writing 24 – 30 points Strongly agree 16 – 23 points Agree 8 – 15 points Disagree 1 – 7 points Strongly disagree 0 points Not enough evidence to judge	/ 30

	Points
Given the grade level and subject area, assess the extent to which you agree with the following:
The assignment parameters are stated such that students will have a clear understanding of the expectations for successfully completing the assignment 16 – 20 points Strongly agree 11 – 15 points Agree 6 – 10 points Disagree 1 – 5 points Strongly disagree 0 points Not enough evidence to judge	/ 20
The time frames are adequate to accomplish the assignments 9 – 10 points Strongly agree 6 – 8 points Agree 3 – 5 points Disagree 1 – 2 points Strongly disagree 0 points Not enough evidence to judge	/ 10
The assessment methods align with current best practices in K-16 writing assessment 16 – 20 points Strongly agree 11 – 15 points Agree 6 – 10 points Disagree 1 – 5 points Strongly disagree 0 points Not enough evidence to judge	/ 20
Deduct up to 5 points for poor presentation, including typographical and grammatical errors.
Total	/100