Justification

NATIONAL ASSESSMENT OF

EDUCATIONAL PROGRESS

Volume I

SUPPORTING STATEMENT

FOR

WAVE 2 SUBMITTAL FOR 2011

(PART OF 2011–2013 SYSTEM CLEARANCE PROPOSAL

OMB# 1850-0790)

Student Grade 4 Knowledge and Skills Appropriate (KaSA) Study Mathematics Questions

Student Grade 4 National Indian Education Study (NIES) Questions

Teacher Grade 4 KaSA Study for Puerto Rico Questions

Teacher Grade 4 NIES Questions

School Grade 4 KaSA Study for Puerto Rico Questions

School Grade 4 NIES Questions

Student Grade 8 Writing Computer-Based Assessment (WCBA) Questions

Student Grade 8 KaSA Study Mathematics Questions

Student Grade 8 Mathematics Computer-Based Study (MCBS) Questions

Student Grade 8 NAEP-TIMSS (Trends in International Mathematics and Science Study) Mathematics and Science Questions

Student Grade 8 NIES Questions

Teacher Grade 8 Writing Questions: Background, Education, and Training (BET), Classroom Organization and Instruction (COI) Writing

Teacher Grade 8 KaSA Study for Puerto Rico Questions

Teacher Grade 8 NIES Questions

Teacher Grade 8 Reading/Writing Questions: BET, COI Reading, COI Writing

School Grade 8 Writing Questions: School Characteristics and Policies (SCP), Writing, Charter School

School Grade 8 KaSA Study for Puerto Rico Questions

School Grade 8 NIES Questions

Student Grade 12 Writing Computer-Based Assessment (WCBA) Questions

School Grade 12 Writing Questions

July 21, 2010

Contents

1. Explanation and Burden Information for This Submittal 3

2. 2011 Wave 2 Burden Information 4

3. Overview of NAEP 2011 Assessments Contained in This Submittal 8

4. How, by Whom, and for What Purpose the Data Will Be Used 10

5. Estimates of Costs to the Federal Government 11

6. Information Pertaining to the 2011 Questionnaires in This Submittal 12

Student Questionnaires 12

Teacher Questionnaires 13

School Questionnaires 15

APPENDIX A: Committee Lists 16

1. Explanation and Burden Information for This Submittal

This document contains supplemental information pertaining to the 2011–2013 National Assessment of Educational Progress (NAEP) System Clearance proposal. The system clearance package was approved in April 2010 (OMB #1850-0790, v.26). The terms of clearance for OMB approvals state that each subsequent submittal activity under the system clearance is to be done so with a 60-day federal register notice waver.

This submittal contains burden information and the non-cognitive questionnaires for the following components of the 2011 NAEP assessments:

Grade 4

Student Grade 4 Knowledge and Skills Appropriate (KaSA) Study Mathematics

Student Grade 4 National Indian Education Study (NIES)

Teacher Grade 4 KaSA Study for Puerto Rico

Teacher Grade 4 NIES

School Grade 4 KaSA Study for Puerto Rico

School Grade 4 NIES

Grade 8

Student Grade 8 Writing Computer-Based Assessment (WCBA)

Student Grade 8 KaSA Study Mathematics

Student Grade 8 Mathematics Computer-Based Study (MCBS)¹

Student Grade 8 NAEP-TIMSS (Trends in International Mathematics and Science Study) Mathematics and Science

Student Grade 8 NIES

Teacher Grade 8 Writing: Background, Education, and Training (BET), Classroom Organization and Instruction (COI) Writing

Teacher Grade 8 KaSA Study for Puerto Rico

Teacher Grade 8 NIES

Teacher Grade 8 Reading/Writing Questions: BET, COI Reading, COI Writing

School Grade 8 Writing: School Characteristics and Policies (SCP), Writing, Charter School

School Grade 8 KaSA Study for Puerto Rico

School Grade 8 NIES Questions

Grade 12

Student Grade 12 Writing Computer-Based Assessment (WCBA)

School Grade 12 Writing

These specific questionnaires are the second group (Wave 2) of questionnaires submitted for approval for use in 2011. An initial group of 2011 questionnaires (Wave 1), approved in June 2010, contained the relevant student, teacher, and school questionnaires for the reading, mathematics, science, and economics assessments for 2011. As noted in the 2011–2013 Systems Clearance submittal, NAEP contains three kinds of questions: “cognitive” assessment questions, which measure student knowledge of an academic subject; “non-cognitive” assessment questions, which gather construct-related information not directly reflecting subject knowledge, such as student motivation or effort; and “background” questions which gather factual information such as demographic variables.

2. 2011 Wave 2 Burden Information

The Wave 2 submittal contains the grades 8 and 12 writing non-cognitive (computer-delivered) questions; the NIES student, teacher, and school questions; and the non-cognitive questions for the MCBS and KaSA special studies.²

The Wave 2 burden information is listed by grade (see charts on pages 6 and 7) and includes the following categories of questions:

Students - Students in 4th, 8th and 12th grades complete assessment booklets that commonly contain two 25-minute cognitive blocks (the writing computer-based blocks are 30 minutes each), followed by two sections of questions (one background and one non-cognitive), which require a total of 15 minutes to complete. The background section contains core questions, many of which are related to demographic information. The non-cognitive section contains subject-specific non-cognitive questions.

Teachers - The teachers of 4th- and 8th-grade students participating in NAEP will be asked to complete questions about their teaching background, education, training, and classroom organization and instruction. The 4th-grade KaSA and NIES teacher burden is estimated to be 20 minutes, and 8th-grade writing, NIES, mathematics (KaSA or NAEP-TIMSS-related), science (NAEP-TIMSS-related), and reading/writing teacher burdens are all also estimated to be 20 minutes. No teacher questionnaires will be given to teachers of students in the MCBS study.

Principals/Administrators - The school administrators in the sampled schools will be asked to complete a questionnaire. The initial section of questions is designed to measure school characteristics and policies that research has shown are highly correlated with student achievement. In addition, there is a section with subject-specific questions focusing on curriculum and instructional services. Most school questionnaires (though not KaSA administered in Puerto Rico or NIES) have a section with charter school questions, to gather information for schools that identify themselves as charter schools. The burden for school questionnaires is estimated to be 30 minutes.

In addition to the questionnaires, burden information is provided for the following:

E-Filing and Pre-Assessment Visit - Survey sample information is collected from schools in the form of lists of potential students who may participate in NAEP. This sample information can be gathered manually or electronically at the school, district, or state level. If done at the school or district level, some burden will be incurred by school personnel. The pre-assessment visit is the opportunity for the NCES contractor field staff to meet with the school personnel to review procedures and logistics for the upcoming assessment. The pre-assessment visit requires one hour and the e-filing burden is also estimated at an hour. However, the e-filing process is only done in a subset of schools (approximately 38 percent in recent years), which is taken into account when computing the e-filing burden.

SD and ELL (Students with Disabilities and English Language Learners) worksheets are completed by teachers or administrators of students identified as SD and/or as ELL. The SD and ELL worksheets are part of this Wave 2 submittal, thus the burden for school personnel who complete them is included with Wave 2. The students whose information is reflected in the SD and ELL worksheets are included in the burden for the entire 2011 assessments (both Wave 1 and Wave 2).

The total combined burdens for the Wave 2 materials are:

Respondent Category	Number of Respondents	Number of Hours of Burden
Student	111,500	27,875
Teacher	16,115	5,372
School	6,945	13,057
SD and ELL	18,340 (school personnel)	31,965
Totals	152,900	78,269

See the following tables for a summary of Wave 2 burden estimates.

Wave 2 Questions - Single Survey Burden Times

Grade 4
Student Grade 4 Mathematics (KaSA) Questions	15 minutes
Student Grade 4 NIES Questions	15 minutes
Teacher Grade 4 NIES Questions	20 minutes
Teacher Grade 4 KaSA Study Questions	20 minutes
School Grade 4 NIES Questions	30 minutes
School Grade 4 KaSA Study Questions	30 minutes
Grade 8
Student Grade 8 Core + WCBA Questions	15 minutes
Student Grade 8 Core + Mathematics (KaSA) Questions	15 minutes
Student Grade 8 Core + Mathematics (MCBS) Questions	15 minutes
Student Grade 8 Core + Mathematics (NAEP-TIMSS) Questions	15 minutes
Student Grade 8 Core + Science (NAEP-TIMSS) Questions	15 minutes
Student Grade 8 NIES Questions	15 minutes
Teacher Grade 8 WCBA Questions	20 minutes
Teacher Grade 8 Mathematics (NAEP-TIMSS) Questions	20 minutes
Teacher Grade 8 Science (NAEP-TIMSS) Questions	20 minutes
Teacher Grade 8 NIES Questions	20 minutes
Teacher Grade 8 KaSA Study Questions	20 minutes
Teacher Grade 8 Reading + Writing Questions	20 minutes
School Grade 8 NIES Questions	30 minutes
School Grade 8 KaSA Study Questions	30 minutes
School Grade 8 Writing Questions	30 minutes
Grade 12
Student Grade 12 Core + WCBA Questions	15 minutes
School Grade 12 Writing Questions	30 minutes

Note: The students and the teachers of students participating in the NAEP-TIMSS study will complete either mathematics or science questionnaires. These questionnaires are the same as the mathematics and science questionnaires (student and teacher) submitted in the Wave 1 materials.

Teachers of students in the mainland U.S. KaSA study will complete the same questionnaires as the mathematics teachers of students in the operational or pilot assessments. These questionnaires were submitted with the Wave 1 materials. The teachers and administrators in the Puerto Rico KaSA study will complete the questionnaires administered in the 2007 NAEP assessment administered in Puerto Rico.

Estimated Burden for NAEP 2011 Assessments Contained in This Submittal (Wave 2)

By Grade Level

3. Overview of NAEP 2011 Assessments Contained in This Submittal

The following broad overview of the 2011 NAEP assessments was included as part of the 2011–2013 System Clearance submittal. The National Assessment Governing Board (the Governing Board) determines NAEP policy and assessment schedule, and future Governing Board decisions may result in changes to some aspects of an assessment (e.g., which subjects are assessed in which years). However, the overall methodology and assessment process will remain consistent. In the 2011 assessment year, questions will be administered to students at grades 4, 8, and 12; to teachers at grades 4 and 8; and to school administrators at grades 4, 8, and 12.

The 2011 data collection activities described in this Wave 2 submission consist of the following:

• National assessment in WCBA at grades 8 and 12;

• Special studies in NIES (Grades 4 and 8), MCBS (grade 8 mathematics), KaSA (grades 4 and 8 mathematics), and a NAEP-TIMSS linking study (grade 8 mathematics and science);

• SD and ELL worksheets are completed by teachers or administrators of students identified as SD and/or as ELL.

Descriptions of the special studies in Wave 2 are provided below.

Knowledge and Skills Appropriate (KaSA) study for students at grades 4 and 8 in mathematics will examine if the administration of blocks of items that target the knowledge and skills at the lower-end of the performance distribution would yield lower standard errors and better measurement precision for low-performing groups of students. There will be two versions of the KaSA assessment administered during the 2011 NAEP administration window. For Puerto Rico, the assessment will include three types of books, containing either

• two KaSA blocks,

• one KaSA block and one operational mathematics block, or

• two operational mathematics blocks.

For the national sample, there will be two types of books administered as part of the KaSA assessment, containing either

• two KaSA blocks or

• one KaSA block and one operational mathematics block.

There is no need to administer additional books containing two operational NAEP mathematics blocks for the national sample, since these books will be administered as part of the operational assessment.

NAEP-TIMSS (Trends in International Mathematics and Science Study) linking study at grade 8 will link the results from NAEP to TIMSS in an effort to compare states to the international countries. The basic principle is that randomly equivalent groups of students will take blocks of NAEP and TIMSS questions in order to link the two sets of results. The data collection for the study will take place in 2011, and will involve grade 8 students in main NAEP and main TIMSS taking mathematics and science, as well as special study samples which will be administered “braided booklets” containing material from both NAEP and TIMSS. The primary purpose of this study is to estimate a link between NAEP mathematics and TIMSS mathematics, and between NAEP science and TIMSS science

This submittal only contains information for the NAEP administration window of the NAEP-TIMSS study which will take place in the January-March 2011 timeframe. The sample for the NAEP window contains 10,000 students who will be administered a ‘braided’ or mixed booklet consisting of one NAEP cognitive section (either mathematics or science), one TIMSS cognitive section (the same subject as the NAEP section), and two NAEP background sections. The students will take the same background questions as the students participating in main NAEP mathematics or science. As such, these questionnaires were approved as part of the Wave 1 materials and are part of this Wave 2 submittal to account for burden related to the NAEP-TIMSS study. The attendant burden for this study is based on the 15 minutes for the background sections for the students and the proportional teacher and school background questionnaires completed by school personnel related to this study. These burdens are incorporated into the chart on page 7.

The NAEP-TIMSS linking assessments administered in the TIMSS assessment window (April-June 2011) were approved in April 2010 under the OMB review and clearance process established by PIRLS and TIMSS (OMB# 1850-0645 v.6).

Mathematics Computer-Based Study (MCBS) will study the possibility of using an adaptive testing algorithm in NAEP. The study, administered on computer, will be conducted in mathematics at grade 8. A sample of students will first take a cognitive block of medium difficulty and, depending on the individual student performance on that block the computer will then assign the student an easy, medium, or difficult block for the second cognitive block. Another sample of students will take the same blocks via standard NAEP random assignment. Comparisons will be made between the results from these two samples of students to determine if tailoring block difficulty to student ability is a worthwhile endeavor for NAEP.

National Indian Education Study (NIES) will be conducted as part of the 2011 assessments in reading and mathematics at grades 4 and 8. The national sample includes students from public and nonpublic schools that have both large and small American Indian/Alaska Native (AI/AN) student populations. The administration of the NAEP assessment will be followed with the administration of a questionnaire specifically designed for the NIES study. Questionnaire data will be linked to NAEP performance data.

Committees that helped define the cognitive, background, and non-cognitive questions are listed in Appendix A.

4. How, by Whom, and for What Purpose the Data Will Be Used

The following broad overview of the use of the data from the 2011 NAEP assessments is abbreviated from the 2011–2013 System Clearance submittal. The purpose of NAEP is to collect and report assessment data on student achievement in the subject areas assessed for use in monitoring educational progress. In addition to reporting overall results of student performance and achievement, NAEP also reports student performance results for various subgroups of students and on various educational factors. Guidance for what is asked in the questions is set by the Governing Board. NCES is responsible for developing the questions and for selecting the final set of questions. The questions are designed to provide (1) the information for disaggregating data according to categories specified in the legislation, (2) contextual information that is subject specific (e.g. reading, mathematics) and has an impact and known relationship to student achievement, and (3) policy-relevant information specified by the Governing Board.

In the original request to the Office of Management and Budget (OMB) for system clearance, NCES requested approval of the data-gathering instruments to be used in the 2011–2013 national and state assessments. This submittal contains the second set of questionnaires that pertain to the 2011 assessments. An expanded description of the purposes of the data was provided in the system clearance submission.

5. Estimates of Costs to the Federal Government

Administration Cost Estimates

The following estimates apply to the 2011 administration costs:

• $8.1 million for the printing, packaging, and distribution phases of the administrations.

• $20.8 million for the cost of the 2011 field supervisors and assessment administrators to go into schools to administer the assessment, including travel expenses.

• $470,000 for web support and maintenance related to the 2011 assessments.

Given that Wave 1 (submitted in April 2010, the first submittal of 2011 questions that included core, reading, mathematics, and science questions) and Wave 2 (this submittal: includes writing, NIES, special studies, and SD- and ELL-related worksheets) are in the field at the same time, the costs are broken out across the two waves for 2011 as follows:

• the bulk of the printing, packaging, and distribution costs occur in Wave 1, which comprises 80–90% of the printing volume;

• the field supervision and assessment costs are also spread over the entire administration, with the bulk associated with the Wave 1 materials; and

• for web support and maintenance, the majority of the costs are associated with the computer-based writing assessment, which is included in the Wave 2 material.

Thus, the approximate breakdown of the total $29.37 million for 2011 is allocated as follows:

• Wave 1 – $23.80 million

• Wave 2 – $5.57 million

Note that NCES has requested $5 million for the following tasks associated with the NAEP-TIMSS linking study: braided assessment administration during the NAEP window, printing of all of the braided test booklets, scoring and linking of all of the NAEP blocks, and contracting the NAEP state coordinators to recruit the additional 8th grade schools where braided assessment will be administered during the TIMSS and the NAEP testing windows. However, consistent with the costs described above, only the costs associated with the printing and administration are included in the $5.57 million attributed to this submittal.

6. Information Pertaining to the 2011 Questionnaires in this Submittal

Student Questionnaires

Core

The core questions for 2011 are comprised of a subset of prior core questions. The core questions were submitted with the Wave 1 materials.

Writing (WCBA)

The writing framework is new and these writing non-cognitive questions will be offered for the first time operationally in 2011. In 2010 a subset of newly developed writing non-cognitive questions were piloted. After review by NCES and the Governing Board, the final versions of the writing questions were approved for the 2011 assessment. The writing questions were developed to be delivered via computer.

Mathematics (KaSA)

The non-cognitive questions being used in the mathematics KaSA study are the same as the questions in the operational 2011 assessments except that three additional questions were added that probe the students’ reactions to the math blocks. These questions were tested in a cognitive interview process.

Mathematics (MCBS)

The non-cognitive questions being used in the mathematics MCBS study are the same as the questions in the operational 2011 assessments except that three additional questions were added which relate to the adaptive nature of the study. These questions were tested in a cognitive interview process.

Mathematics (NAEP-TIMSS)

For the mathematics portion of the NAEP-TIMSS study the non-cognitive questions will be exactly the same as in the grade 8 mathematics operational assessment.

Science (NAEP-TIMSS)

For the science portion of the NAEP-TIMSS study the non-cognitive questions will be exactly the same as in the grade 8 science operational assessment.

NIES

The NIES questions will be the same as those given in the 2009 NIES study. The questions were largely revised for 2009 and this will now be the second iteration for the trend line of NIES questions.

Teacher Questionnaires

Teachers of students at grades 4 and 8 participating in the NAEP assessments are each administered a questionnaire. No teacher questionnaire will be administered at grade 12. At grade 4, a single teacher questionnaire will be administered across all subjects assessed. At grade 8, teacher questionnaires will be administered for each subject-specific area. Teacher questionnaires are also available online for completion by teachers, if desired (but not in Puerto Rico). Historically, approximately 10 percent of teachers complete the questionnaire online.

The NAEP 2011 teacher questionnaires are composed of multiple parts: background, education, and training (BET); and one or more subject-specific classroom organization and instruction (COI) components. Given the structure of elementary school, the grade 4 teacher questionnaire often contains multiple subjects in the subject-specific section. The majority of the grade 8 teacher questionnaires focus on a single subject, resulting in multiple forms of questionnaires. To facilitate the administration of the questionnaires to teachers who teach both reading and writing, a combined reading and writing questionnaire will be given in the small instances when both subjects are assessed at the same school.

The following table depicts the layout for the 2011 teacher questionnaires included in the Wave 2 submittal:

Teacher Questionnaires for the NAEP 2011 Assessments

Grade	Type	Section 1	Section 2	Section 3
4	Mathematics (for KaSA)	BET	COI (Mathematics)
4	National Indian Education Study (NIES)	NIES
8	Writing (WCBA)	BET	COI (Writing)
8	Mathematics (for MCBS, KaSA, NAEP-TIMSS)	BET	COI (Mathematics)
8	Science (NAEP-TIMSS)	BET	COI (Science)
8	National Indian Education Study (NIES)	NIES
8	Reading/Writing	BET	COI (Reading)	COI (Writing)

BET – Background, education, and training

COI – Classroom organization and instruction

School Questionnaires

A single school questionnaire will be administered at each grade. Related to this submittal, there will be a school questionnaire for grades 8 and 12 WCBA, a NIES school questionnaire for grades 4 and 8, and a school questionnaire for KaSA at grades 4 and 8. Historically, approximately 10 percent of school questionnaires are completed online.

The following table depicts the layout for the 2011 school questionnaires included in the Wave 2 submittal:

School Questionnaires for the NAEP 2011 Assessments

Grade	Type	Section 1	Section 2	Section 3
4	Mathematics (for KaSA)	SCP	Mathematics	Charter school (mainland U.S. but not Puerto Rico)
4	National Indian Education Study (NIES)	NIES
8	Writing (WCBA)	SCP	WCBA	Charter school
8	Mathematics (for KaSA)	SCP	Mathematics	Charter school (mainland U.S. but not Puerto Rico)
8	National Indian Education Study (NIES)	NIES
12	Writing (WCBA)	SCP	WCBA	Charter school

SCP – School characteristics and policies

APPENDIX A: Committee Lists

NAEP Background Variable Committee

Name Affiliation

Patricia Alexander University of Maryland

Arthur Applebee University at Albany, SUNY

Claudia Buchman Ohio State University

Lizanne Destefano University of Illinois

Robert Hauser University of Wisconsin, Madison

Kathleen Heid Penn State University

Henry Levin Columbia University

Peter Levine University of Maryland

Linda Levstik University of Kentucky

Samuel Lucas University of California, Berkeley

Senta Raizen WestEd, Washington, D.C.

Andrew Sum Northeastern University

William Walstad University of Nebraska, Lincoln

NAEP Mathematics Committee

Name Affiliation

David J. Brancamp Nevada Department of Education

Sareeta Carter Benjamin Banneker Academic High School, Washington, D.C.

Herb Clemens Ohio State University

Marcia Cole District of Columbia Public Schools 

Carl Cowen Indiana University-Purdue University, Indianapolis

Bradford R. Findell Ohio Department of Education

Diana Rivera Garcia Departamento de Educación Puerto Rico

Anne Gonzales South Gate Middle School, South Gate, CA

Kathleen Heid Penn State University

Judy Kinley Morris Elementary School, Des Moines, IA

Carole Phillip Alice Deal Middle School, Washington, D.C.

Penny Roberts Greenville Elementary School, Greenville, KY

Daren Starnes The Lawrenceville School, Lawrenceville, NJ

JT Sutcliffe St. Mark’s School of Texas, Dallas, TX

Elizabeth Sweeney Boston Public Schools

NAEP Science Committee

Name Affiliation

Alicia Cristina Alonzo Michigan State University

Charles W. Anderson Michigan State University

Susan Craft Hanahan High School, Hanahan, SC

George Deboer American Association for the Advancement of Science

Alex Decaria Millersville University, PA

Ibari Igwe Paul Public Charter School, Washington, D.C.

Crystal Lovell Lawrence Township Public Schools, Lawrenceville, NJ

Ellen Mingione Delaware Department of Education

Amy Pearlmutter Littlebrook Elementary School, Princeton, NJ

Senta Raizen WestEd, Washington, D.C.

Cary Sneider Portland State University

Mary Thandi Buthelezi Wheaton College, Norton, MA

David White Vermont Department of Education

Gerald Wheeler National Science Teacher Association

NAEP Writing Committee

Name Affiliation

Arthur Applebee University at Albany, SUNY

Diane August Center for Applied Linguistics

Margretta Browne Montgomery County Public Schools, Rockville, Maryland

Elyse Eidman-Aadahl University of California

Nikki Elliot-Schuman Office of Superintendent of Public Instruction, WA

Rayna Goldfarb Abraham Lincoln High School, Philadelphia, PA

Carol Jago California Reading and Literature Project, UCLA

Charles Macarthur University of Delaware

Michael McCloskey Johns Hopkins University

Norma Mota-Altman San Gabriel High School, San Gabriel, CA

Sandra Murphy University of California, Davis

Drew Sterner Tamanend Middle School, Warrington, PA

Victoria Young Texas Education Agency

NIES TRP (Technical Review Panel) Committee

Name Affiliation

Henry Braun Boston College

Bryan Brayboy Arizona State University

Robert Cook Crazy Horse Memorial, Crazy Horse, SD

Cultural Affairs Specialist/Education Outreach

Steve Culpepper University of Colorado, Denver

Susan Faircloth The Pennsylvania State University

Valeria Littlecreek Oklahoma State Department of Education

Christopher Lohse Council of Chief State School Officers

Larry Ludlow Boston College

Debora Norris Arizona Department of Education

Ted Wright Haskell Indian Nations University, Lawrence, KS

Tarajean Yazzi-Mintz Indiana University

1 Referred to as Multi-Stage Testing (MST) study in the System Clearance submission

2 The NAEP-TIMSS study contains the same non-cognitive questions as the operational mathematics and science assessments. These questionnaires were submitted with the Wave 1 materials and thus are not part of the Wave 2 submittal. However, the description and burden for the NAEP-TIMSS study is included as part of Wave 2.