Supporting statement for Wave 1 for 2009

NATIONAL ASSESSMENT OF

EDUCATIONAL PROGRESS

Volume 1

SUPPORTING STATEMENT

FOR

WAVE 1 OF 2009 SUBMITTAL

(PART OF 2008–2010 SYSTEM CLEARANCE PROPOSAL

OMB# 1850-0790)

Student Grade 4 Operational Core Questions

Student Grade 4 Operational Reading Questions

Student Grade 4 Operational Mathematics Questions

Student Grade 8 Operational Core Questions

Student Grade 8 Operational Reading Questions

Student Grade 8 Operational Mathematics Questions

March 12, 2008

Explanation and Burden Information for This Submittal

This document contains supplemental information pertaining to the 2008–2010 NAEP System Clearance proposal. The System Clearance package was submitted in January 2007 and approved in May 2007 with OMB # 1850-0790. The terms of clearance for OMB approvals state that each subsequent submittal activity under the System Clearance is to be submitted to OMB.

This submittal contains burden information and the actual background questionnaires for the following components of the 2009 assessments:

• Student Grade 4 Operational Core

• Student Grade 4 Operational Reading

• Student Grade 4 Operational Mathematics

• Student Grade 8 Operational Core

• Student Grade 8 Operational Reading

• Student Grade 8 Operational Mathematics

These specific questionnaires are the first group (Wave 1) of questionnaires submitted for approval for usage in 2009. Two additional groups of questions (Waves 2 and 3) will be submitted as parts of the overall 2009 OMB submittal package.

Wave 2 will include questions for student operational science (grades 4, 8, and 12); student operational reading and mathematics at grade 12; student pilot civics, U.S. history, and geography (grades 4, 8, and 12); and student pilot reading and mathematics (grades 4, 8, and 12).

Wave 3 will include questions for student special studies, including the National Indian Education Study (grades 4 and 8) and the Extended Background Questions for the socioeconomic status study (grades 4, 8, and 12). Questionnaires for Teachers (grades 4 and 8), Schools (grades 4, 8, and 12), and Students with Disability and English Language Learners (SD-ELL) will also be part of the Wave 3 submittal.

2009 Wave 1 Burden Information

The Wave 1 submittal contains the core (demographic) and subject-specific student background questions for the grades 4 and 8 operational reading and mathematics assessments.

Student response burden is .25 hours (15 minutes) for the total background question sections. At grade 4, the estimated total number of students for the reading and mathematics operational assessments is 344,000 resulting in 86,000 hours of student burden. At grade 8, the sample sizes for both reading and mathematics will be the same as at grade 4. Thus, the estimated total number of students for the grade 8 reading and mathematics operational assessments is 344,000 resulting in 86,000 hours of student burden. The estimated total number of students participating in the grades 4 and 8 operational reading and mathematics assessments is 688,000 with a resulting student burden of 172,000 hours.

(Note that the Core questions are separate for the OMB submission, but their burden is not computed separately. All students take core questions which are part of the overall 15 minute background question burden. The core questions were approved in the System Clearance submittal in 2007 but are included in the Volume II questionnaires because they will be administered in 2009.)

See the following table for a summary of Wave 1 burden estimates.

Wave 1

Estimated Burden for NAEP 2009 Assessments Contained in This Submittal

by Grade Level

Components	# of Students	Student Burden (in hours)	# of Teachers	Teacher Burden	# of Schools	School Burden	SD/ELL (# of school personnel)	SD/ELL Burden	Total Burden
				(in hours)		(in hours)		(in hours)	(in hours)
Grade 4 Students
*Core
Reading Operational	170,625	42,656							42,656
Math Operational	173,375	43,344							43,344
Totals	344,000	86,000							86,000
Grade 8 Students
*Core
Reading Operational	170,625	42,656							42,656
Math Operational	173,375	43,344							43,344
Totals	344,000	86,000							86,000
Totals	688,000	172,000							172,000

* NOTE: Core questions are separate for the OMB submission, but their burden is not computed separately. All students take core questions which are part of the overall 15 minute background question burden. The core questions were approved in the System Clearance submittal in 2007 but are included in the Volume II questionnaires because they will be administered in 2009.

Refer to Appendix A for a chart with the estimated burden for all three Waves (by grade level) of the 2009 submittals.

Overview of NAEP 2008–2010 Assessments

The following broad overview of the 2008–2010 NAEP assessments was included as part of the initial systems clearance submittal. The National Assessment Governing Board (the Governing Board) determines NAEP policy and assessment schedule, and future Board decisions may result in changes to some aspects of an assessment (e.g., which subjects are assessed in which years). However, overall methodology and assessment process will remain constant. In the 2009 assessment year, questionnaires will be administered to students at grades 4, 8, and 12; to teachers at grades 4 and 8; and to school administrators at grades 4, 8, and 12.

The 2009 assessments consist of the following:

• Operational assessments in reading, mathematics, and science in grades 4, 8, and 12;

• Pilot assessments for civics, U.S. history, and geography at grades 4, 8, and 12; pilot assessments in reading and mathematics at grades 4, 8, and 12; and

• Special studies, including (1) American Indian/Alaska Native students, at grades 4 and 8, as part of the National Indian Education Study; and (2) a pilot of extended background questions as part of the development of a new socioeconomic status variable at grades 4, 8, and 12.

How, by Whom, and for What Purpose the Data Will be Used

In the original request for system clearance, NCES asked for approval of the instruments to be used to gather data from the 2008–2010 national and state assessments. This submittal applies to the first set of questionnaires that will be submitted for the 2009 assessments. This first set of questions contains student core questions (grades 4, 8, and 12), and student reading and mathematics subject-specific questions (grades 4 and 8).

Given that the purpose of NAEP is to gather data on the achievement of students in the subject areas assessed for use in monitoring education progress, and because of the program's increasing visibility, it is incumbent on the program to develop the most reliable and valid instruments possible. To do so, NAEP has employed four strategies in recent assessment cycles:

1. Small-scale pilot testing of new materials and test administration techniques;

2. Pilot testing items to determine which items best measure the constructs under consideration;

3. Field testing^* of operational assessments to accommodate the mandated six-month reporting; and

4. Full-scale operations.

Background questionnaire development follows the same pattern as that of cognitive item development, although we tend to pilot fewer items with less duplication and use the resulting data to refine the questions. Guidance for what is asked is provided by the Governing Board. NCES develops the questionnaires, which the Governing Board then approves for submission to OMB in a two-stage process: (1) prior to pilot testing, and (2) again after NCES and its contractors make selections for the operational assessment based on pilot data. The questions are designed to provide

• the information for disaggregating data according to categories specified in the legislation,

• contextual information that is subject specific (e.g. reading, mathematics) and has an impact and known relationship to achievement, and

• policy relevant information specified by the Governing Board.

In all assessment years, questionnaires are generally administered to students at grades 4, 8, and 12; teachers at grades 4 and 8; and school administrators at grades 4, 8, and 12. SD/ELL (Students with Disabilities/ English Language Learner) questionnaires are completed by teachers or administrators of students identified as learning disabled or as English Language Learners.

Development Information for the 2009 Components Contained in Wave 1 Submittal

Operational Reading and Mathematics: Grades 4 and 8

Wave 1 contains the subject-specific background questions for the operational reading and mathematics assessments at grades 4 and 8. Both the reading and mathematics assessments are structured to fit the common booklet model. At each grade the assessment will use two 25-minute cognitive item blocks followed by two sections of background questions. Assessment accommodations will be allowed at each grade. The reading and mathematics assessments will be administered at the national/state level at grades 4 and 8.

Development activities for the reading student questionnaires primarily entailed developing new items to capture the issues associated with the new assessment frameworks. Student reading items were also converted into student mathematics items to achieve comparability across subject areas and address gaps in issue coverage. The 4th- and 8th-grade student reading items were also assessed according to the results of the 2007 pilot test of these questionnaires. Item performance determined any changes made to items. The 4th- and 8th-grade student reading questionnaires are proposed as final for the 2009 operational administration of NAEP.

Development activities for the mathematics student questionnaires at grades 4 and 8 entailed updating dated technology items and modifying newly developed student reading items. Student reading items were converted into student mathematics items to achieve comparability across subject areas and address gaps in issue coverage. The 4th- and 8th-grade student mathematics items were assessed according to the results of the 2007 pilot test of these questionnaires. Item performance determined any changes made to items.

The final selection of background items for 2009 reflects the mandate that the background questionnaires meet the requirements of the Governing Board’s policy, particularly with regard to the overall time burden involved for respondents.

APPENDIX A

2009 Overall Burden Estimate Chart (3 Waves)

by Grade Level

Subjects	# of Students	Student Burden (in hours)	# of Teachers /	Teacher Burden	# of Schools	School Burden	SD/ELL (# of school personnel)	SD/ELL Burden	Total Burden
				(in hours)		(in hours)		(in hours)	(in hours)
4th Grade – Wave 1-Operational (Mathematics & Reading)	344,000	86,000							86,000
Wave 2 - Operational (Science); Pilot (Civics, History, Geography, Reading, Mathematics)	163,375	40,844							40,844
Wave 3 - Student Special Studies*(NIES, EBQ); Teacher (Operational & Pilot); School (Operational and Pilot); SD-ELL Questionnaires	11,300	1,575	27,299	9,009	9,100	4,550	18,154	5,991	21,124
4th Grade Totals	518,675	128,419	27,299	9,009	9,100	4,550	18,154	5,991	147,968
8th Grade – Wave 1-Operational (Mathematics & Reading);	344,000	86,000							86,000
Wave 2 - Operational (Science); Pilot (Civics, History, Geography, Reading, Mathematics)	164,375	41,094							41,094
Wave 3 - Student Special Studies* (NIES, EBQ); Teacher (Operational & Pilot); School (Operational and Pilot); SD-ELL Questionnaires	11,000	1,500	27,336	9,021	9,112	4,556	18,178	5,999	21,075
8th Grade Totals	519,375	128,594	27,336	9,021	9,112	4,556	18,178	5,999	148,169
12th Grade -Wave 1	NA	NA							NA
Wave 2-Operational (Mathematics, Math Trend, state math, Reading, Rdg Trend, state reading,Science); Pilot (Reading and Math,Civics, Hist, Geog) Spec Studies ( Sc. Hot/ICT, )	118,667	29,667							29,667
Wave 3 - Student Special Studies* (EBQ for SES); School (Operational and Pilot); SD-ELL Questionnaires	7,500	625	395	130	132	66	4,416	1,457	2,278
12th Grade Totals	126,167	30,292	395	130	132	66	4,416	1,457	31,945
1 HSTS (High School Transcript Study)						388
2 E-Filing (Electronic Filing)						6,970
Grand Totals	1,164,217	287,304	55,029	18,160	18,343	16,530	40,748	13,447	328,082

* Note: The burden estimates for students in Wave 3 are comprised of 15 minutes for NIES questions and 5 minutes for EBQ questions.

¹ HSTS - A subset of the sample of schools will also be sampled to participate in the HSTS.

² E-Filing - A subset of the sample of schools where school personnel complete e-filing activities.

* Based on the results of recent assessment cycles, field testing to accommodate 6-month reporting may no longer be needed.

2009 Wave 1 Supporting Statement 2