Wave 2 2009 - NAEP supporting statement psrt A

NATIONAL ASSESSMENT OF

EDUCATIONAL PROGRESS

Volume 1

SUPPORTING STATEMENT

FOR

WAVE 2 OF 2009 SUBMITTAL

(PART OF 2008–2010 SYSTEM CLEARANCE PROPOSAL

OMB# 1850-0790)

Student Grade 4 Operational Science Questions

Student Grade 4 Pilot Reading Questions

Student Grade 4 Pilot Mathematics Questions

Student Grade 4 Pilot U.S. History Questions

Student Grade 4 Pilot Civics Questions

Student Grade 4 Pilot Geography Questions

Student Grade 8 Operational Science Questions

Student Grade 8 Pilot English/Language Arts Questions

Student Grade 8 Pilot Mathematics Questions

Student Grade 8 Pilot U.S. History Questions

Student Grade 8 Pilot Civics Questions

Student Grade 8 Pilot Geography Questions

Student Grade 12 Operational Core Questions

Student Grade 12 Operational Science Questions

Student Grade 12 Operational English/Language Arts Questions

Student Grade 12 Operational Mathematics Questions

Student Grade 12 Pilot U.S. History Questions

Student Grade 12 Pilot Civics Questions

Student Grade 12 Pilot Geography Questions

May 15, 2008

Explanation and Burden Information for This Submittal

This document contains supplemental information pertaining to the 2008–2010 NAEP System Clearance proposal. The System Clearance package was submitted in January 2007 and approved in May 2007 with OMB #1850-0790. The terms of clearance for OMB approvals state that each subsequent submittal activity under the System Clearance is to be submitted to OMB.

This submittal contains burden information and the actual background questionnaires for the following components of the 2009 assessments:

Student Grade 4 Operational Science Questions

Student Grade 4 Pilot Reading Questions

Student Grade 4 Pilot Mathematics Questions

Student Grade 4 Pilot Civics Questions

Student Grade 4 Pilot U.S. History Questions

Student Grade 4 Pilot Geography Questions

Student Grade 8 Operational Science Questions

Student Grade 8 Pilot English/Language Arts Questions

Student Grade 8 Pilot Mathematics Questions

Student Grade 8 Pilot Civics Questions

Student Grade 8 Pilot U.S. History Questions

Student Grade 8 Pilot Geography Questions

Student Grade 12 Operational Core Questions

Student Grade 12 Operational Science Questions

Student Grade 12 Operational English/Language Arts Questions

Student Grade 12 Operational Mathematics Questions

Student Grade 12 Pilot Civics Questions

Student Grade 12 Pilot U.S. History Questions

Student Grade 12 Pilot Geography Questions

These specific questionnaires are the second group (Wave 2) of questionnaires submitted for approval for usage in 2009. One additional group of questions (Wave 3) will be submitted as part of the overall 2009 OMB submittal package.

The Wave 1 submittal included the core (demographic) and subject-specific student background questions for the grades 4 and 8 operational reading and mathematics assessments.

Wave 3 will include questions for student special studies, including the National Indian Education Study (grades 4 and 8) and the Extended Background Questions for the socioeconomic status study (grades 4, 8, and 12). Questionnaires for Teachers (grades 4 and 8), Schools (grades 4, 8, and 12), and Students with Disabilities and English Language Learners (SD-ELL) will also be part of the Wave 3 submittal, as will the High School Transcript Study (HSTS).

2009 Wave 2 Burden Information

The Wave 2 submittal contains the subject-specific student background questions for the operational science assessments at grades 4, 8, and 12; the operational English/Language Arts and mathematics assessments at grade 12; the pilot reading and mathematics assessments at grades 4 and 8; and the pilot civics, U.S. history, and geography assessments at grades 4, 8, and 12. The grade 12 core (demographic) background questions are also included. (Note that the Core questions are separate for the OMB submission, but their burden is not computed separately. All students take core questions which are part of the overall 15 minute background question burden. The core questions were approved in the System Clearance submittal in 2007 but are included in the Volume II questionnaires because they will be administered in 2009.)

Student response burden is .25 hours (15 minutes) for the total background questions sections. At grade 4, the estimated total number of students for the subjects included in the Wave 2 materials is 164,375 resulting in 41,094 hours of student burden. At grade 8, the estimated total number of students for the subjects being submitted is 165,375 resulting in 41,344 hours of student burden. At grade 12, the estimated total number of students for the subjects included in the Wave 2 materials is 116,667 resulting in 29,167 hours of student burden.

The estimated total number of students participating in the grades 4, 8, and 12 operational and pilot assessments is 446,417 with a resulting student burden of 111,604 hours. (Note: Total estimates have changed slightly from Wave 1 total estimates as more specific sample size information has become available. There have been slight increases in the Science HOTS/ICT and grade 12 reading and mathematics operational estimates.)

See the following table for a summary of Wave 2 burden estimates*.

* Refer to Appendix A for a chart with the estimated burden for all three Waves (by grade level) of the 2009 submittals.

Wave 2

Estimated Burden for NAEP 2009 Assessments Contained in This Submittal—By Grade Level

Components	# of Students	Student Burden (in hours)	# of Teachers	Teacher Burden	# of Schools	School Burden	SD/ELL (# of school personnel)	SD/ELL Burden	Total Burden
				(in hours)		(in hours)		(in hours)	(in hours)
Grade 4 Students
Science Operational	151,375	37,844							37,844
Science Probe (ICT/HOTS)	4,000	1,000							1,000
Civics Pilot	1,000	250							250
US History Pilot	1,000	250							250
Geography Pilot	1,000	250							250
Reading Pilot	3,000	750							750
Mathematics Pilot	3,000	750							750
Totals	164,375	41,094							41,094
Grade 8 Students
Science Operational	151,375	37,844							37,844
Science Probe (ICT/HOTS)	4,000	1,000							1,000
Civics Pilot	1,000	250							250
US History Pilot	1,500	375							375
Geography Pilot	1,500	375							375
Reading Pilot	3,000	750							750
Mathematics Pilot	3,000	750							750
Totals	165,375	41,344							41,344
Grade 12 Students
Science Operational	11,000	2,750							2,750
Science Probe (ICT/HOTS)	4,000	1,000							1,000
Reading Operational	51,000	12,750							12,750
Mathematics Operational	47,667	11,917							11,917
Civics Pilot	1,000	250							250
US History Pilot	1,000	250							250
Geography Pilot	1,000	250							250
Totals	116,667	29,167							29,167
Overall Totals	446,417	111,604							111,604

Overview of NAEP 2008–2010 Assessments

The following broad overview of the 2008–2010 NAEP assessments was included as part of the initial System Clearance submittal. The National Assessment Governing Board (the Governing Board) determines NAEP policy and assessment schedule, and future Board decisions may result in changes to some aspects of an assessment (e.g., which subjects are assessed in which years). However, overall methodology and assessment process will remain constant. In the 2009 assessment year, questionnaires will be administered to students at grades 4, 8, and 12; to teachers at grades 4 and 8; and to school administrators at grades 4, 8, and 12.

The 2009 assessments consist of the following:

• Operational assessments in reading, mathematics, and science in grades 4, 8, and 12;

• Pilot assessments for civics, U.S. history, and geography at grades 4, 8, and 12; pilot assessments in reading and mathematics at grades 4 and 8; and

• Special studies, including (1) American Indian/Alaska Native students at grades 4 and 8 as part of the National Indian Education Study; and (2) a pilot of extended background questions as part of the development of a new socioeconomic status variable at grades 4, 8, and 12.

How, By Whom, and for What Purpose the Data Will Be Used

In the original request for system clearance, NCES asked for approval of the instruments to be used to gather data from the 2008–2010 national and state assessments. This submittal applies to the second set of questionnaires that will be submitted for the 2009 assessments. This set of questions contains operational and pilot student questions. Specifically, science (operational grades 4, 8, and 12), reading and mathematics (operational grade 12 and pilot grades 4 and 8), and social studies (U.S. history, civics, and geography pilots at grades 4, 8, and 12) are contained in the Volume II submittal.

Given that the purpose of NAEP is to gather data on the achievement of students in the subject areas assessed for use in monitoring education progress, and because of the program's increasing visibility, it is incumbent on the program to develop the most reliable and valid instruments possible. To do so, NAEP has employed four strategies in recent assessment cycles:

1. Small-scale pilot testing of new materials and test administration techniques;

2. Pilot testing items to determine which items best measure the constructs under consideration;

3. Field testing^* of operational assessments to accommodate the mandated six-month reporting; and

4. Full-scale operations.

Background questionnaire development follows the same pattern as that of cognitive item development, although we tend to pilot fewer items with less duplication and use the resulting data to refine the questions. The questions are designed to provide

• the information for disaggregating data according to categories specified in the legislation,

• contextual information that is subject specific (e.g., reading, mathematics) and has an impact and known relationship to achievement, and

• policy relevant information specified by the Governing Board.

In all assessment years, questionnaires are generally administered to students at grades 4, 8, and 12; teachers at grades 4 and 8; and school administrators at grades 4, 8, and 12. SD/ELL (Students with Disabilities/ English Language Learner) questionnaires are completed by teachers or administrators of students identified as learning disabled or as English Language Learners.

Development Information for the 2009 Components Contained in Wave 2 Submittal

Operational Science: Grades 4, 8, and 12

The items proposed for inclusion in the background questionnaires for the 2009 NAEP Science Assessment were developed based on a review of existing NAEP items from the 2000 and 2005 science background questionnaires, the 2008 science pilot items, and other NCES surveys including the Education Longitudinal Study of 2002 (ELS:2002). A number of new items were also developed based on the recommendations made at the Science Experts Workshop.

The science items that were piloted in 2008 have been modified for the 2009 operational assessments to achieve standardization on the background questionnaire items across subjects (i.e., reading and mathematics). The content of the science items has remained the same between the 2008 pilot and 2009 operational versions. The changes that have been made to the 2009 operational items are, mainly, standardizing response options (following the same response sets that were used in reading and mathematics when applicable) and providing a time frame in the item stem for each item. Minor item rewriting has also been done for clarification purposes.

Mathematics: Operational (Grade 12) and Pilot (Grades 4 and 8)

Operational

Development activities entailed updating dated technology items and modifying newly developed student reading items for administration in the student mathematics questionnaires. Student reading items were turned into student mathematics items to achieve comparability across subject areas and to address holes in issue coverage. The student mathematics items were assessed according to the results of the pilot testing, which was administered in 2007 for grades 4 and 8 and in 2008 for grade 12.

Pilot

Three major activities for item development took place for the 2011 mathematics pilot assessment: (1) developing new items targeted to measure the aspects of these 2011 highlighted issues; (2) updating, modifying, and adding technology-related items; and (3) modifying newly developed student reading items for administration in the student mathematics questionnaires.

New item development was based on certain aspects of issues and sub-issues—Availability and Use of Instructional Resources (Resources), Organization of Instruction (Instruction), Teacher Preparation (Teacher Prep), Role of Technology in Instruction (Technology), Student Engagement with Mathematics (Student Engagement), and Teaching and Learning of Algebra (Algebra)—that are addressed as important factors for student learning and have been made into student questionnaire items. Item development for modified or newly developed technology questions included (a) types of technology devices that students use, (b) types of learning activities using technology devices, and (c) interaction between students and teachers using technology. Some of the student reading items served as the basis for student mathematics items to achieve comparability across subject areas.

Reading (English/Language Arts): Operational (Grade 12) and Pilot (Grades 4 and 8)

Operational

The bulk of activities during the development year entailed developing, reviewing, pilot testing, and evaluating subject-specific items for the 2009 operational administration of NAEP Reading Student Background Questionnaires. Additional pilot testing on the grade 12 reading items, administered in 2008, guided the item review and selection at grade 12.

Development activities for the reading student questionnaire primarily entailed developing new items to capture the issues associated with the new assessment frameworks. Student reading items were also turned into student mathematics items to achieve comparability across subject areas and to address holes in issue coverage.

Pilot

The standardized rotation schedule was applied to the student reading questionnaires for the 2011 pilot assessment. In addition to the rotation schedule, factors considered were item priority (whether an item is essential and should be asked in every administration), parallelism with questionnaires of other subject areas (i.e., the 2011 student mathematics questionnaires), face validity, and strength of an item’s or issue’s constituent base.

Social Studies: Pilot (Grades 4, 8, 12)

Development activities for the student questionnaires for U.S. history, civics, and geography primarily entailed updating and modifying items from the questionnaires in the last assessment for each of these subject areas. Although maintaining trend on as many items as possible is desirable, standardization across subject areas has been one of the major efforts in recent questionnaire development. Thus, in updating and modifying previously existing items for the 2009 pilot assessments, we made major efforts to achieve standardization across the three social studies subjects and across reading and mathematics.

Besides issue coverage, factors considered for item development included item priority—whether the content is essential and should be asked in every administration—some parallelism with 2009 student questionnaires for reading and mathematics, face validity, and strength of an item’s or issue’s constituent base.

In selecting the 2010 background questionnaire items for social studies, response frequencies were reviewed for each item based on the data collected from the last assessments (i.e., 2006 assessments for U.S. history and civics and the 2001 assessment for geography). That statistical review determined which items could be moved forward to the 2010 assessments and what kinds of modification were needed to improve the items. The modification typically occurred in changes of item wording for clarification purposes, adding follow-up questions, adding details in the item stem, providing a time frame in the item stem, or changing response options. Changes in response options occurred in two cases: (1) to standardize types of response options used across the questionnaires and (2) to segregate relatively large percentages by adding more categories.

One design consideration for the social studies questionnaires is the fact that U.S. history, civics, and geography are often taught together under the general rubric of social studies at the elementary and middle school grades. Furthermore, schools use different labels when referring to the same social studies subject areas. We revised certain items to reflect this reality, while at the same time making sure that each questionnaire was somewhat distinctive from the others. To achieve this goal, we not only included a unique set of items in each of the questionnaires, but for activities described in the items that could occur in any social studies class, the social studies subject was referred to as “social studies or history” in the U.S. history questionnaire, “social studies or civics” in the civics questionnaire, and “social studies or geography” in the geography questionnaire.

APPENDIX A

2009 Overall Burden Estimate Chart (3 Waves)

by Grade Level (revised 4/08)

Subjects	# of Students	Student Burden (in hours)	# of Teachers	Teacher Burden	# of Schools	School Burden	SD/ELL (# of school personnel)	SD/ELL Burden	Total Burden
				(in hours)		(in hours)		(in hours)	(in hours)
4th Grade Wave 1—Operational (mathematics & reading)	344,000	86,000							86,000
Wave 2—Operational (science); Pilot (civics, history, geography, reading, mathematics)	164,375	41,094							41,094
Wave 3—Student Special Studies* (NIES, EBQ); Teacher (Operational & Pilot); School (Operational and Pilot); SD-ELL Questionnaires	11,300	1,575	27,299	9,009	9,100	4,550	18,154	5,991	21,124
4th Grade Totals	519,675	128,669	27,351	9,026	9,117	4,559	18,189	6,002	148,255
8th Grade Wave 1—Operational (mathematics & reading);	344,000	86,000							86,000
Wave 2—Operational (science); Pilot (civics, history, geography, reading, mathematics)	165,375	41,344							41,344
Wave 3—Student Special Studies* (NIES, EBQ); Teacher (Operational & Pilot); School (Operational and Pilot); SD-ELL Questionnaires	11,000	1,500	27,336	9,021	9,112	4,556	18,178	5,999	21,075
8th Grade Totals	520,375	128,844	27,388	9,038	9,129	4,565	18,213	6,010	148,457
12th Grade—Wave 1	NA	NA							NA
Wave 2—Operational (mathematics, reading, science); Pilot (civics, history, geography) Special Studies ( Science Hot/ICT )	116,667	29,167							29,167
Wave 3—Student Special Studies* (EBQ for SES); School (Operational and Pilot); SD-ELL Questionnaires	7,500	625	395	130	132	66	4,416	1,457	2,278
12th Grade Totals	124,167	29,792	395	130	132	66	4,346	1,434	31,422
1 HSTS (High School Transcript Study)
2 E-Filing (Electronic Filing)						388
						6,984
Grand Totals	1,164,217	287,304	55,134	18,194	18,378	16,561	40,748	13,447	328,134

* Note: The burden estimates for students in Wave 3 are comprised of 15 minutes for NIES questions and 5 minutes for EBQ questions.

¹ HSTS—A subset of the sample of schools will also be sampled to participate in the HSTS.

² E-Filing—A subset of the sample of schools where school personnel complete e-filing activities.

* Based on the results of recent assessment cycles, field testing to accommodate 6-month reporting may no longer be needed.

2009 Wave 2 Supporting Statement 2