Justification for Wave I

NATIONAL ASSESSMENT OF

EDUCATIONAL PROGRESS

Volume 1

SUPPORTING STATEMENT

FOR

WAVE 1 OF 2008 SUBMITTAL

(PART OF 2008-2010 SYSTEM CLEARANCE PROPOSAL

OMB# 1850-0790)

Student Grade 8 Operational Arts Questions: Visual Arts, Music

School Grade 8 Operational Questionnaire (School Characteristics and Policies, Arts, Charter School)

Student Grade 12 (Pilot Reading, Pilot Mathematics)

Student with Disabilities (SD) and English Language Learner (ELL) Questionnaires

June 15, 2007

Explanation and Burden Information for This Submittal

This document contains supplemental information pertaining to the 2008-2010 NAEP System Clearance proposal (submitted in January 2007). The terms of clearance for OMB approvals state that each subsequent submittal activity under the System Clearance is to be submitted to OMB.

This submittal contains burden information and the actual background questionnaires for the following components of the 2008 assessments:

• Student Grade 8 Operational Arts (Visual Arts and Music)

• School Grade 8 (School Characteristics and Policies, Arts, Charter School)

• Student Grade 12 reading pilot and student Grade 12 mathematics pilot

• SD-ELL (Student Disabilities/English Language Learner)

These specific questionnaires are the second group (Wave 1) of questionnaires submitted for approval for usage in 2008. The first groups of questions were submitted as part of the System Clearance submittal in January 2007 and included: student core questions (grades 4, 8, 12); reading and mathematics subject-specific background questions (grades 4 and 8); long-term trend (LTT) core, reading and mathematics questions (ages 9, 13, 17). A subsequent (Wave 2) submittal will include: pilot Science student, teacher and school questionnaires, and Reading and Mathematics pilot teacher and school questionnaires.

Estimated Respondent Burden for 2008 Assessments

Contained in this Submittal

By Grade

2008

Subjects	# of Students	Student Burden (in hours)	# of Teachers /	Teacher Burden	# of Schools	School Burden	SD/ELL (# of school personnel)	SD/ELL Burden
				(in hours)		(in hours)		(in hours)
8th Grade Arts Operational	6,000	1,500	NA	NA	120	60	210	70
12th Grade –ReadingPilot Math Pilot	6,000 6,000	1,500 1,500					210 210	70 70
Totals	18,000	4,500			120	60	630	210

Overview of NAEP 2008-2010 Assessments

The following broad overview of the 2008-2010 NAEP assessments was submitted as part of the initial systems clearance submittal. The National Assessment Governing Board determines NAEP policy and assessment schedule, and future Board decisions may result in changes to some aspects of an assessment (e.g., which subjects are assessed in which years). However, overall methodology and assessment process will remain constant. In the 2008 assessment year, questionnaires will be administered to students at grades 4, 8, and 12; to students at ages 9, 13, and 17 for LTT; to teachers at grades 4 and 8; and to school administrators at grades 4, 8, and 12.

The 2008 assessments consist of:

• national operational assessments in the Arts (Visual Arts and Music) at grade 8 and long-term trend assessments at ages 9, 13, and 17.

• pre-calibration field test assessments for reading and mathematics at grades 4 and 8.

• Pilot assessments in science at grades 4, 8, and 12; pilot assessments in reading and mathematics at grade 12; LTT pilot assessments for mathematics at ages 9, 13, and 17; bridging studies for reading at grades 4 and 8; and a special study on incentives for grade 12 reading.

How, by Whom, and for What Purpose the Data Will be Used

In the original request for system clearance, NCES asked for approval of the instruments to be used to gather data from the 2008-2010 national and state assessments. This submittal applies to the second set of questionnaires (Arts - Grade 8, Reading and Mathematics - Grade 12) that will be submitted for the 2008 assessments. The first set of questions contained student core questions (grades 4, 8, and 12), reading and mathematics student subject-specific questions (grades 4 and 8), and student long-term trend (ages 9, 13 and 17).

Given that the purpose of NAEP is to gather data on the achievement of students in the subject areas assessed for use in monitoring education progress, and because of the program's increasing visibility, it is incumbent on the program to develop the most reliable and valid instruments possible. To do so, NAEP employs four strategies:

1. Small-scale pilot testing of new materials and test administration techniques;

2. Pilot testing items to determine which items best measure the constructs under consideration;

3. Field testing of operational assessments to accommodate the mandated six-month reporting; and

4. Full-scale operations.

Questionnaire development follows the same pattern as that of cognitive item development, although we tend to pilot fewer items with less duplication and use the resulting data to refine the questions. Guidance for what is asked is provided by the National Assessment Governing Board. NCES develops the questionnaires, which the Governing Board then approves for submission to OMB in a two-stage process. The Governing Board approves the questionnaires prior to pilot testing, and then again after NCES and its contractors make selections for the operational assessment based on pilot data. The questions are designed to provide the information for disaggregating data according to categories specified in the legislation, to provide contextual information that is subject specific (e.g. reading, mathematics) and has an impact and known relationship to achievement, and to provide policy relevant information specified by the Governing Board.

Design Information for the 2008 Operational, Pre-Calibration, and Pilot Subjects Contained In This Submittal

Operational components

The NAEP 2008 assessment will contain two operational components.

1. Operational LTT assessment at ages 9, 13, and 17. The LTT assessment is administered at different times of the year for different age groups. Both the reading and mathematics LTT assessments will be administered to a nationally representative sample of students at each age level. The operational LTT assessments will be a combination of the cognitive blocks carried forward from 2004 and new items that were pilot tested in 2004.

2. Operational arts assessment at grade 8. The arts assessment will have two distinct components: music and visual arts. Each is timed separately and is not structured to fit the “common booklet” model¹. Therefore, each booklet is administered separately and within its own session. The arts assessment is administered during the standard NAEP assessment window (end of January through beginning of March). In both cases the assessment will use cognitive blocks immediately followed by two background question sections. The visual arts and music assessments will be administered to a nationally representative sample of students. The operational music and visual art assessments will be a re-administration of selected cognitive blocks administered in 1997.

Pre-calibration components.

To support six-month reporting of NAEP 2009 reading and mathematics assessment results at grades 4 and 8, a subset of the 2009 assessment instruments will be administered in 2008 and “pre-calibrated.” The reading and mathematics pre-calibration assessments will be administered to a nationally representative sample of students, with targets and spiraling plans designed to ensure that at least 2,000 students receive each new cognitive block (all comprehension blocks for reading and three blocks at each grade in mathematics). The purpose of the pre-calibration process is to perform differential item functioning (DIF) analyses, to obtain starting values of the item parameters for the 2009 operational scaling, and to make item treatment decisions prior to the 2009 operational scaling. These data are designed to facilitate the six-month reporting in 2009, not to provide data for item revisions, or for use in 2008 reporting.

1. Reading pre-calibration at grades 4 and 8. The reading pre-calibration will be composed of items selected from those pilot tested in 2007. Since a new reading framework will be implemented in 2009, there will be no items carried forward from the 2007 reading operational assessment. Once the reading items are selected, reading comprehension passages and items will be placed in a balanced incomplete block (BIB) design. Typically, a reduced BIB is used in pre-calibration assessments. However, because the entire reading assessment is new and the vocabulary-only half-blocks are being introduced, the test design will vary somewhat from the traditional design. Books will be assembled according to the “common booklet” model as they would appear in the 2009 assessment. This will result in a more efficient booklet bundle map.

2. Mathematics pre-calibration at grades 4 and 8. The mathematics pre-calibration will be composed of items selected from those pilot tested in 2007 and those items retained from the 2007 operational assessment. Items pilot tested in 2007 will be selected to replace operational items released after the 2007 assessment. Once these items are selected, they will be placed in a reduced BIB design. To expedite the printing and review process and to reduce scoring costs, but still adhere to strict statistical requirements, the set of booklets used for the pre-calibration will be a subset of the test booklets that will be used for the 2009 operational assessment. Books will be assembled according to the “common booklet” model as they will appear in the 2009 assessment. This will result in a more efficient booklet bundle map.

Pilot-test components.

To support future assessments, NAEP pilot tests items to replace items that are released. In cases when new frameworks are being introduced (e.g., 2009 science at all grades and mathematics at grade 12), the majority of the items are developed and pilot tested. Pilot testing of items is intended to provide item-level data that is used for two purposes: (1) to revise and improve the items and (2) to select items for use in future operational assessments.

i. LTT mathematics pilot tests at ages 9, 13 and 17. New items will be developed and pilot tested to replace items that will be released following the 2008 LTT assessment. New items for the 2012 LTT operational assessment will be selected from the pools of items pilot tested in 2008. The pilot-test LTT items will be assembled into 15-minute blocks, and placed in booklets structured like the current LTT assessment, so that the pilot books can be in the same session as the operational books.

ii. Reading pilot tests at grade 12. Given the introduction of a new reading framework for the 2009 assessment, a new grade 12 reading assessment will be developed and pilot tested in 2008 for operational administration in 2009. None of the items from previous grade 12 reading assessments will be administered in 2009. Pilot tests in 2008 will be built using the common booklet model. However, because of the introduction of vocabulary only half-blocks, the test design will vary somewhat from the traditional design.

3. Mathematics pilot tests at grade 12. Given the introduction of a revised Grade 12 Mathematics Framework for the 2009 assessment, a new grade 12 mathematics assessment will be developed and pilot tested in 2008. Even though the 2009 Grade 12 Mathematics assessment will be the first under a new framework, NCES has decided to carry over some items from either the 2000 or 2005 Mathematics assessment that map to the new framework.

4. Science pilot tests at grades 4, 8 and 12. A new framework is set to be implemented with the 2009 Science Assessment. The 2008 Science Pilot will be used to provide information that will help revise, improve, and select the items and tasks that will be part of the 2009 operational assessment. Even though the 2009 Science Assessment will be the first under a new framework, NCES has decided to carry over some items from the 2005 Science assessment that map to the new framework. Pilot tests in 2008 will be built using the common booklet model. However, because of the introduction of new hands-on-tasks (HOTs) and interactive computer tasks (ICTs), the test and sample design will vary somewhat from the traditional design.

Trend study components.

NAEP will typically conduct trend studies to determine if significant changes in assessment conditions and/or procedures which may impact differentially upon student performance. In 2008, NCES will conduct a “bridging” study in reading at grades 4 and 8 to facilitate a state-level trend study in 2009. The study will administer “braided” books to students, which contain one block from the old assessment and one block from the new assessment. Order and context effects would be examined using this information.

SD-ELL (Student with Disabilities/English Language Learners)

These questionnaires are completed by school personnel who are most knowledgeable about students identified as SD or ELL. NAEP encourages the inclusion of all students who can meaningfully participate in the assessment, including those with disabilities and those with limited English proficiency. In order to obtain a complete picture of educational progress for all students, it is important to collect supplemental information on students in the sample who have been identified as having a disability or are English language learners. Students with disabilities include those who have an individual education plan (IEP), Section 504 plan, or equivalent documentation.

The flowchart format utilized in the 2007 version reduces the number of questions required to complete the questionnaires, thus reducing overall burden. Since schools may have several students identified as SD and/or ELL, this revision of the question format is aimed at reducing overall school burden.

For the 2007-08 LTT, the SD and ELL questionnaires from the 2007 assessment will be used with only slight modifications to address administrative issues encountered in the field. This version of the questionnaires is included with Wave 1 to assure ample time for production and printing in advance of the October 2007 LTT Age 13 assessment.

It is anticipated that additional structural changes will be made to the SD and ELL questionnaires for the 2008 Arts assessment and the various pilot/field tests. The substance of the questionnaires will not be significantly altered, only the presentation and response options. This second revised version will be submitted with Wave 2.

Additional studies.

NAEP frequently includes additional studies, as was discussed in the System Clearance submittal, in regular assessments to investigate content issues (e.g., Meaning Vocabulary Study in 2007), delivery options (e.g., various technology-based assessments), linking to other NCES surveys (e.g., Early Childhood Longitudinal Study, Kindergarten Cohort (ECLS-K) Linking Study in 2007), or reporting variables (e.g., socio-economic status (SES) Indicator Study in 2007).

1 The “common booklet” model in NAEP consists of two 25-minute cognitive blocks followed by two background questions sections. For the arts assessment, each of the cognitive blocks has different time limits; therefore, the time for each arts booklet differs.

2008 Wave 1 Supporting Statement 2