NAEP 2011 Wave I Supporting Statement

NATIONAL ASSESSMENT OF

EDUCATIONAL PROGRESS

Volume I

SUPPORTING STATEMENT

FOR

WAVE 1 SUBMITTAL FOR 2011

(PART OF 2011–2013 SYSTEM CLEARANCE PROPOSAL

OMB# 1850-0790)

Student Grade 4 Core Questions

Student Grade 4 Reading Questions

Student Grade 4 Mathematics Questions

Teacher Grade 4 Questions (Background, Education, and Training (BET), Reading, Mathematics)

School Grade 4 Questions (School Characteristics and Policies (SCP), Reading, Mathematics, Charter School)

Student Grade 8 Core Questions

Student Grade 8 Reading Questions

Student Grade 8 Mathematics Questions

Student Grade 8 Science Questions

Teacher Grade 8 Reading Questions (Background, Education, and Training (BET), Classroom Organization and Instruction (COI) Reading)

Teacher Grade 8 Mathematics Questions (Background, Education, and Training (BET), Classroom Organization and Instruction (COI) Mathematics)

Teacher Grade 8 Science Questions (Background, Education, and Training (BET), Classroom Organization and Instruction (COI) Science)

School Grade 8 Questions (School Characteristics and Policies (SCP), Reading, Mathematics, Science, Charter School)

Student Grade 12 Core Questions

Student Grade 12 Economics Questions

March 31, 2010

Contents

1. Explanation and Burden Information for This Submittal 3

1. Explanation and Burden Information for This Submittal 3

2. 2011 Wave 1 Burden Information 3

3. Overview of NAEP 2011 Assessments 3

4. How, By Whom, and For What Purpose the Data Will Be Used 3

5. Estimates of Costs to the Federal Government 3

6. Information Pertaining to the 2011 Assessments: 3

Student Questionnaires 3

Teacher Questionnaires 3

School Questionnaires 3

APPENDIX A: Committee Lists 3

1. Explanation and Burden Information for This Submittal

This document contains supplemental information pertaining to the 2011–2013 National Assessment of Educational Progress (NAEP) System Clearance proposal. The system clearance package was submitted in November 2009 and approved in April 2010 (OMB #1850-0790 v.26). The terms of clearance for OMB approvals state that each subsequent submittal activity under the system clearance is to be submitted to OMB with a 60-day federal register notice waver.

This submittal contains burden information and the background questionnaires (also referred to as noncognitive questions) for the following components of the 2011 NAEP assessments:

Grade 4

Student Grade 4 Core Questions

Student Grade 4 Reading Questions

Student Grade 4 Mathematics Questions

Teacher Grade 4 Questions: Background, Education, and Training (BET); Reading; Mathematics

School Grade 4 Questions: School Characteristics and Policies (SCP); Reading; Mathematics; Charter School

Grade 8

Student Grade 8 Core Questions

Student Grade 8 Reading Questions

Student Grade 8 Mathematics Questions

Student Grade 8 Science Questions

Teacher Grade 8 Reading Questions: Background, Education, and Training (BET); Classroom Organization and Instruction (COI) Reading

Teacher Grade 8 Mathematics Questions: Background, Education, and Training (BET); Classroom Organization and Instruction (COI) Mathematics

Teacher Grade 8 Science Questions: Background, Education, and Training (BET); Classroom Organization and Instruction (COI) Science

School Grade 8 Questions: School Characteristics and Policies (SCP); Reading; Mathematics; Science; Charter School

Grade 12

Student Grade 12 Core Questions

Student Grade 12 Economics Questions

These specific questionnaires are the initial group (Wave 1) of questionnaires submitted for approval for usage in 2011. A second group of 2011 questionnaires (Wave 2) will be submitted later this year. Wave 2 will contain student, teacher, and school questionnaires for the writing assessment; student, teacher, and school questions for the National Indian Education Study (NIES); special study questions; and students with disabilities (SD) and English language learner (ELL) worksheets.

2. 2011 Wave 1 Burden Information

The Wave 1 submittal contains the grades 4, 8, and 12 core questions; the reading, mathematics, science, and economics student questions; the grade 4 teacher and school questions; and the teacher and school grade 8 reading, mathematics, and science questions. Some special studies will also be administered in 2011 and their background questions will be included in Wave 2. Final decisions regarding the 2011 special studies are not available at this time, but will be included in the Wave 2 submittal.

The Wave 1 burden information is listed by grade (see chart on page 6) and includes the following categories of questions:

Students - Students in 4th, 8th and 12th grades complete assessment booklets that commonly contain two 25-minute cognitive blocks, followed by two background (noncognitive) question sections, which require a total of 15 minutes to complete. The first background question block contains core questions, many of which are related to demographic information. The second background block contains subject-specific questions.

Teachers - The teachers of 4th- and 8th-grade students participating in NAEP will be asked to complete questions about their teaching background, education, training, and classroom organization and instruction. Fourth-grade teacher burden is estimated to be 30 minutes while 8th- and 12th-grade teacher burden is 20 minutes. Fourth-grade teachers often have multiple subject-specific sections to complete, which results in the additional burden estimate.

Principals/Administrators - The school administrators in the sampled schools will be asked to complete a questionnaire. The school core questions are designed to measure school characteristics and policies that research has shown are highly correlated with student achievement. In addition, there is a section with subject-specific questions focusing on curriculum and instructional services. Finally, there is a section with charter school questions, to gather information for schools which identify themselves as charter schools.

In addition to the questionnaires, burden information is provided for the following:

E-Filing and Pre-Assessment Visit - Survey sample information is collected from schools in the form of lists of potential students who may participate in NAEP. This sample information can be gathered manually or electronically at the school, district, or state level. If done at the school or district level, some burden will be incurred by school personnel. The Pre-Assessment Visit is the opportunity for the NCES contractor field staff to meet with the school personnel to review procedures and logistics for the upcoming assessment. The pre-assessment visit requires one hour and the e-filing burden is also estimated at an hour. However, the e-filing process is only done in a subset of schools (approximately 38% in recent years), which is taken into account when computing the e-filing burden.

The total combined burdens for the Wave 1 materials are:

Respondent Category	Number of Respondents	Number of Hours of Burden
Student	897,540	224,386
Teacher	62,884	26,563
School	15,741	29,583
Totals	976,165	280,532

See the following tables for a summary of Wave 1 burden estimates.

Wave 1 Questions - Single Survey Burden Times

Grade 4
Student Grade 4 Core + Reading Questions	15 minutes
Student Grade 4 Core + Mathematics Questions	15 minutes
Teacher Grade 4 Questions	30 minutes
School Grade 4 Questions	30 minutes
Grade 8
Student Grade 8 Core + Reading Questions	15 minutes
Student Grade 8 Core + Mathematics Questions	15 minutes
Student Grade 8 Core + Science Questions	15 minutes
Teacher Grade 8 Reading Questions	20 minutes
Teacher Grade 8 Mathematics Questions	20 minutes
Teacher Grade 8 Science Questions	20 minutes
School Grade 8 Questions	30 minutes
Grade 12
Student Grade 12 Core + Economics Questions	15 minutes

Estimated Burden for NAEP 2011 Assessments Contained in This Submittal (Wave 1)

By Grade Level

3. Overview of NAEP 2011 Assessments

The following broad overview of the 2011 NAEP assessments was included as part of the 2011–2013 System Clearance submittal. The National Assessment Governing Board (the Governing Board) determines NAEP policy and assessment schedule, and future Governing Board decisions may result in changes to some aspects of an assessment (e.g., which subjects are assessed in which years). However, the overall methodology and assessment process will remain consistent. In the 2011 assessment year, questions will be administered to students at grades 4, 8, and 12; to teachers at grades 4 and 8; and to school administrators at grades 4, 8, and 12.

The 2011 data collection activities described in this Wave 1 submission consist of the following:

• National, state, and urban district assessment in reading at grades 4 and 8;

• National, state, and urban district assessment in mathematics at grades 4 and 8;

• National and state in science at grade 8;

• Pilot assessments for 2013 reading and mathematics at grades 4 and 8; and

• Pilot assessment for 2012 economics at grade 12.

Committees that helped define the cognitive and background questions are listed in Appendix A.

4. How, By Whom, and For What Purpose the Data Will Be Used

The following broad overview of the use of the data from the 2011 NAEP assessments is abbreviated from the 2011–2013 System Clearance submittal. The purpose of NAEP is to collect and report assessment data on student achievement in the subject areas assessed for use in monitoring education progress. In addition to reporting overall results of student performance and achievement, NAEP also reports student performance results for various subgroups of students and on various educational factors. Guidance for what is asked in the questions is set by the Governing Board. NCES is responsible for developing the questions and for selecting the final set of questions. The questions are designed to (a) provide the information for disaggregating data according to categories specified in the legislation, (b) provide contextual information that is subject specific (e.g. reading, mathematics) and has an impact and known relationship to student achievement, and (c) provide policy-relevant information specified by the Governing Board.

In the original request to the Office of Management and Budget (OMB) for system clearance, NCES requested approval of the data-gathering instruments to be used in the 2011–2013 national and state assessments. This submittal applies to the first set of questions containing the questions for the students, teachers, and school administrators that will be submitted as part of the 2011 assessments. A second submittal, with the remainder of the questions for the 2011 assessments, will be submitted in approximately two months. An expanded description of the purposes of the data is provided in the System Clearance submittal.

5. Estimates of Costs to the Federal Government

Administration Cost Estimates

The following estimates apply to the 2011 administration costs:

• $8.1 million for the printing, packaging, and distribution phases of the administrations.

• $20.5 million for the cost of the 2011 field supervisors and assessment administrators to go into schools to administer the assessment, including travel expenses.

• $470,000 for web support and maintenance related to the 2011 assessments.

Given that Wave 1 (this first submittal of 2011 questions which includes core, reading, mathematics, and science-related questions) and Wave 2 (including writing, NIES, special studies, and SD- and ELL-related worksheets) are in the field at the same time, the costs are broken out across the two waves in 2011 as follows:

• the bulk of the printing, packaging, and distribution costs occur in Wave 1, which comprises 80–90% of the printing volume;

• the field supervision and assessment costs are also spread over the entire administration, with the bulk associated with the Wave 1 materials; and

• for web support and maintenance, the majority of the costs are associated with the computer-based writing assessment, which will be included in the Wave 2 material.

Thus, the approximate breakdown of the total $29.07 million for 2011 is allocated as follows:

• Wave 1 – $23.80 million

• Wave 2 – $5.27 million

6. Information Pertaining to the 2011 Assessments:

Student Questionnaires

Core

The core questions for 2011 are comprised of a subset of prior core questions.

Reading

The reading background questions will be a subset of the questions administered in the 2009 operational assessments. New frameworks for reading were implemented in 2009 and the questions will continue the trend for 2011. The reading pilot assessments administered in 2011 will also use the same 2009 reading background questions.

Mathematics

Like reading, the mathematics background questions will a subset of the questions administered in the 2009 operational assessments. Also, the 2011 pilot mathematics assessments will use the same 2009 mathematics background questions.

Science

The science background questions will be the same as the questions administered in the 2009 operational assessment. Note that no pilot assessment will be given in science in 2011.

Economics

The economics background questions (pilot assessment only) will be a subset of the questions administered in the 2006 assessment.

Teacher Questionnaires

Teachers of grades 4 and 8 students participating in the main NAEP assessments are each administered a questionnaire. No teacher questionnaire will be administered at grade 12. At grade 4, a single teacher questionnaire will be administered across all subjects assessed. At grade 8, teacher questionnaires will be administered for each subject-specific area. Teacher questionnaires are also available online for completion by teachers, if desired. Historically, approximately 10% of teachers complete the questionnaire online.

The NAEP 2011 teacher questionnaires are composed of multiple parts: background, education, and training (BET); and one or more subject-specific classroom organization and instruction (COI) components. Given the structure of elementary school, the grade 4 teacher questionnaire contains multiple subjects in the subject-specific section. The grade 8 teacher questionnaires focus on a single subject, resulting in multiple forms of questionnaires.

The following table depicts the layout for the 2011 teacher questionnaires included in the Wave 1 submittal:

Teacher Questionnaires for the NAEP 2011 Assessments

Grade	Type	Section 1	Section 2	Section 3
4	Reading, Mathematics	BET	COI (Reading)	COI (Mathematics)
8	Reading	BET	COI (Reading)
8	Mathematics	BET	COI (Mathematics)
8	Science	BET	COI (Science)

BET – Background, education, and training

COI – Classroom organization and instruction

School Questionnaires

A single school questionnaire will be administered at each grade. Each grade-specific questionnaire will include a section on school characteristics and policies (SCP), a section for each subject being assessed at that grade (at grade 12, pilot economics will not be included), and a section with charter school questions as the last section in the school questionnaire. Note that the charter school section is only completed if the school is a charter school. School questionnaires, similar to teacher questionnaires, are also offered in an online version. Historically, approximately 10% of school questionnaires are completed online.

The following table depicts the layout for the 2011 school questionnaires included in the Wave 1 submittal:

School Questionnaires for the NAEP 2011 Assessments

Grade	Section 1	Section 2	Section 3	Section 4	Section 4	Section 5
4	SCP	Reading	Mathematics	Charter school
8	SCP	Reading	Mathematics	Science	Writing*	Charter school

SCP – School characteristics and policies

* Writing will be included in Wave 2 of the 2011 NAEP submission.

APPENDIX A: Committee Lists

NAEP Background Variable Committee

Name Affiliation

Patricia Alexander University of Maryland

Arthur Applebee University at Albany

Claudia Buchman Ohio State University

Lizanne Destefano University of Illinois

Robert Hauser University of Wisconsin-Madison

Kathleen Heid Penn State University

Henry Levin Columbia University

Peter Levine University of Maryland

Linda Levstik University of Kentucky

Samuel Lucas University of California-Berkeley

Senta Raizen West-Ed

Andrew Sum Northeastern University

William Walstad University of Nebraska-Lincoln

NAEP Reading Committee

Name Affiliation

Patricia Alexander University of Maryland

Christine Carriere Carl Von Linne School

Mary Beth Curtis Lesley University

Patsy Dunton Maine Department of Education

Arzie Galvez Office of Curriculum, Instruction and School Support,

Los Angeles, CA

Evan Lefsky Lake County Schools

Pamela Mason Harvard Graduate School of Education

Joe McGonegal Catholic Memorial High School

Margaret McKeown University of Pittsburgh

Katie Oliver Morgan Park Middle School

Timothy Shanahan University of Illinois

Alfred Tatum University of Illinois at Chicago

Lisa White Plymouth Public Schools

Junko Yokota National-Louis University

NAEP Mathematics Committee

Name Affiliation

David J. Brancamp Nevada Department of Education

Sareeta Carter Benjamin Banneker Academic High School

Herb Clemens Ohio State University

Marcia Cole District of Columbia Public Schools 

Carl Cowen Indiana University-Purdue University, Indianapolis

Bradford R. Findell Ohio Department of Education

Diana Rivera Garcia Departamento De Educación Puerto Rico

Anne Gonzales South Gate Middle School

Kathleen Heid Penn State University

Judy Kinley Morris Elementary School

Carole Phillip Alice Deal Middle School

Penny Roberts Greenville Elementary School

Daren Starnes The Lawrenceville School

JT Sutcliffe St. Mark’s School of Texas

Elizabeth Sweeney Boston Public Schools

NAEP Science Committee

Name Affiliation

Alicia Cristina Alonzo Michigan State University

Charles W. Anderson Michigan State University

Susan Craft Hanahan High School

George Deboer American Association for the Advancement of Science

Alex Decaria Millersville University

Ibari Igwe Paul Public Charter School

Crystal Lovell Lawrence Township Public Schools

Ellen Mingione Delaware Department of Education

Amy Pearlmutter Littlebrook Elementary School

Senta Raizen West-Ed

Cary Sneider Portland State University

Mary Thandi Buthelezi Wheaton College

David White Vermont Department of Education

Gerald Wheeler National Science Teacher Association

NAEP Writing Committee

Name Affiliation

Arthur Applebee University at Albany, SUNY

Diane August Center for Applied Linguistics

Margretta Browne Montgomery County Public Schools

Elyse Eidman-Aadahl University of California

Nikki Elliot-Schuman Office of Superintendent of Public Instruction, WA

Rayna Goldfarb Abraham Lincoln High School

Carol Jago California Reading and Literature Project, UCLA

Charles Macarthur University of Delaware

Michael Mccloskey Johns Hopkins University

Norma Mota-Altman San Gabriel High School

Sandra Murphy University of California Davis

Drew Sterner Tamanend Middle School

Victoria Young Texas Education Agency

NAEP Economics Committee

Name Affiliation

Kris Bertelsen St. Charles High School

Stephen Buckles Vanderbilt University

Steven L. Cobb Center for Economic Education, Denton, TX

Jaime Festa-Daigle Lake Havasu High School

Julie Heath University of Memphis

Richard Macdonald St. Cloud State University

Andrea Morgan Oregon Department of Education

Kevin Smith Renaissance High School

William Walstad University of Nebraska-Lincoln