Volume I NAEP 2014 Wave 1

This document contains supplemental information pertaining to the 2014-2016 National Assessment of Educational Progress (NAEP) system clearance proposal. NAEP is administered by the National Center for Education Statistics (NCES) in the Institute for Education Sciences of the U.S. Department of Education. The National Assessment Governing Board (henceforth referred to as the Governing Board) sets policy for NAEP and determines the content framework for each assessment. The system clearance package was submitted in November 2012 and approved in March 2013 (OMB #1850-0790 v.36). The terms of clearance for OMB approvals state that NCES will publish a notice in the Federal Register allowing a 30-day public comment period on the details of each collection concurrent with submission to OMB.

The 2014 submittal is divided into two waves to meet scheduling and question development requirements. The first wave contains the core,¹ social studies (civics, geography, U.S. history), and science (paper-and-pencil) descriptions, burden, and questionnaires. Wave 2 will contain technology and engineering literacy (TEL) and science ICT (interactive computer tasks) descriptions, burden, and materials, as well as information regarding school coordinator activities, including the collection of information on students with disabilities (SD) and English language learners (ELL).

This Wave 1 submittal contains burden information and survey questions (referred to as background or noncognitive questions) for the components of the NAEP 2014 assessments listed in Table 1 below.

Table 1: List of Components in Wave 1

Component	Part in Volume II
Grade 4
Student — Core, Science	Part 1a
Teacher — Background, Education, Training (BET) - Science, Classroom Organization and Instruction (COI) - Science	Part 1b
School — School Characteristics and Policies (SCP), Science, Charter School	Part 1c
Grade 8
Student — Core, Civics, Geography, U.S. History, Science	Part 2a
Teacher — BET-Social Studies (Civics, Geography, U.S. History), COI-Social Studies (Civics, Geography, U.S. History), Teacher — BET-Science, COI-Science	Part 2b
School — SCP, Social Studies (Civics, Geography, U.S. History), Science, Charter School	Part 2c
Grade 12
Student — Core, Science	Part 3a
School — SCP, Science, Charter School	Part 3b

2. Overview of Wave 1 NAEP 2014 Assessments

Much of the following broad overview of the 2014 NAEP assessments was included as part of the 2014–2016 system clearance submittal. The Governing Board determines NAEP policy and assessment schedule, and future Governing Board decisions may result in changes to some aspects of an assessment (e.g., which subjects are assessed in which years). However, the overall methodology and assessment process remains consistent. The national NAEP assessments contain two kinds of questions: cognitive (assessment questions that measure student knowledge of an academic subject) and survey (background questions that gather information on demographic as well as classroom instructional procedures). The Governing Board is responsible for selecting and approving all of NAEP’s survey questions, as well as the cognitive items.

The NAEP assessments are currently supported by an alliance of organizations under contract with NCES, within the U.S. Department of Education. The main NAEP assessment reports current achievement levels and trends in student achievement at grades 4, 8, and 12 for the nation and, for certain assessments, states and select urban districts. These assessments follow subject-area frameworks developed by the Governing Board and use the latest advances in assessment methodology. The subject-area frameworks evolve to match instructional practices.

3. Sampling Information

The 2014 Sampling Design Memo² (see appendix D) provides specific sampling information regarding the 2014 NAEP assessments. It includes information on:

Assessment types
Primary Sampling Units Selection
Stratification and Oversampling
Overlap Control of Samples
Substitute Samples
Student Sampling Information
Weighting Requirements

4. Information Pertaining to the 2014 Questionnaires in This Submittal

Student Questionnaires

Core — the first survey question block for all students contains core questions, many of which are related to demographic information.

Civics — the grade 8 operational civics subject-specific questions are existing trend questions. These questions were last administered as part of the 2010 civics assessment.

Geography — the grade 8 operational geography subject-specific questions are existing trend questions. These questions were last administered as part of the 2010 geography assessment.

U.S. History — the grade 8 operational U.S. history subject-specific questions are existing trend questions. These questions were last administered as part of the 2010 U.S. history assessment.

Science — the grades 4, 8, and 12 pilot science subject-specific questions consist of trend and newly developed questions. The new questions were pre-tested in cognitive interviews (also referred to as cog labs) in 2012. Refer to appendix C for the reports associated with the science cog labs and for a list of the new questions contained in the science pilot questionnaires (student, teacher, and school).

Membership of the committees that helped define the cognitive and survey questions pertaining to the 2014 subjects in Wave 1 is provided in this document, in appendix A. All of the assessments covered by Wave 1 will be administered by paper-and-pencil.

Teacher Questionnaires

Teacher questionnaires are comprised of multiple sections, which include:

Background, Education, and Training (BET; also referred to as the teacher core) — the operational BET questions are trend or previously piloted questions. Science pilot consists of trend and pilot questions.

Classroom Organization and Instruction (COI) — the operational COI questions in the social studies subjects are trend questions. The COI section for the science pilot questionnaires contains both existing trend and new questions.

All teacher questionnaires will be completed online in 2014. If a grade 8 teacher teaches both social studies and science, the online questionnaire system will administer the appropriate BET questions and both COI sections to that teacher.

School Questionnaires

School questionnaires are comprised of multiple sections, which include:

School Characteristics and Policies (SCP; also referred to as the school core) – One version of SCP questions will be given at each grade. The SCP questions are comprised of trend or previously piloted questions.

Social Studies (Civics, Geography, and U.S. History) – The operational school questions in the social studies subjects are trend questions that were administered in the 2010 assessments.

Science – The pilot science questionnaires contain both existing trend and new questions.

Charter School – Charter school questions are comprised of trend questions that were administered in previous assessments. Note that the charter school section is only completed if the school is a charter school.

All 2014 school questionnaires will be completed online. Table 2 depicts the composition of the 2014 school questionnaires.

Table 2: School Background Questionnaires for the NAEP 2014 Assessments
Grade	Section 1	Section 2	Section 3	Section 4
4	SCP	Science _Pilot	Charter School
8	SCP	Social Studies	Science _Pilot	Charter School
12	SCP	Science _Pilot	Charter School

Note: SCP – School Characteristics and Policies

5. Burden Information for 2014 Wave 1

The nature of NAEP is that samples of students alternate from a relatively low volume in national-level only administration years to a substantial volume increase in state-level administration years that include one or more assessments that support the reporting of results by state and certain urban districts. In 2014, NAEP will conduct national-level only assessments, so the estimated burden is much lower than in 2013 NAEP. The total combined burden for the 2014 Wave 1 materials is provided in table 3.

Table 3: Total Burden for Wave 1

Respondent Category	Number of Respondents	Total Burden Hours
Students	30,500	12,708
Teachers	2,894	988
Schools	533	267
Total	33,927	13,963

Note: School coordinator activities, such as e-filing and pre-assessment activities, along with their related burden, will be included in the Wave 2 submittal.

The burden includes the following categories:

Students – Students in eighth grade complete civics, geography, or U.S. history assessment booklets that contain two 25-minute cognitive blocks, and two survey question sections that require a total of 15 minutes to complete. The social studies subjects are all paper-based assessments.

The science pilots associated with Wave 1 will be administered in paper-and-pencil and hands-on tasks (HOTs) formats. The paper-and-pencil version contains two 25-minute cognitive blocks. The science hands-on tasks have students work with lab materials and other equipment to perform experiments. These hands-on tasks may require a total of an additional 15 minutes of student time for the cognitive portion of the assessment. The paper-and-pencil and HOTs versions will also include survey questionnaire sections.

Additional student burden is included for time to read directions and distribute test booklets. For paper-and-pencil and HOTs assessments, this additional burden is estimated at 10 minutes, thus the total burden for students is 25 minutes³ for the paper-and-pencil and HOTs assessments.

Appendix B includes a sample parental notification letter regarding NAEP. The letter will be adapted for each grade/subject combination and the school principal may edit it. However, the information regarding confidentiality and the appropriate law reference will remain unchanged.

Teachers – In 2014, teachers will be asked to complete questions about their teaching background, education, training, and classroom organization and instruction. At fourth-grade, only science teachers will complete teacher questionnaires and this burden is estimated at 20 minutes. The teachers of eighth-grade students participating in the NAEP civics, geography, U.S. history, or science assessments will complete questionnaires. The eighth-grade teacher burden is estimated to be 20 minutes for most teachers, and 30 minutes for a small percent (5%) who may teach both social studies and science. Note there is no teacher questionnaire for the teachers of students taking the twelfth-grade assessments.

Schools – The school administrators in the sampled schools will be asked to complete a school questionnaire. The school questionnaire burden is estimated to be 30 minutes at all three grades.

Table 4: Wave 1 Questionnaires - Per Respondent Burden Time

Respondent Category	Time per Respondent
Grade 4
Student Core + Science (paper and pencil, including HOTs)	15 minutes
Student Directions and Set-up	10 minutes
Teacher Questionnaire	20 minutes
School Questionnaire	30 minutes
Grade 8
Student Core + Civics	15 minutes
Student Core + Geography	15 minutes
Student Core + U.S. History	15 minutes
Student Core + Science (paper and pencil, including HOTs)	15 minutes
Student Directions and Set-up	10 minutes
Teacher Questionnaire (single subject)	20 minutes
Teacher Questionnaire (social studies and science)	30 minutes
School Questionnaire	30 minutes
Grade 12
Student Core + Science (paper and pencil, including HOTs)	15 minutes
Student Directions and Set-up	10 minutes
School Questionnaire	30 minutes

Table 5: Estimated Burden for NAEP 2014 Questionnaires Contained in This Submittal (Wave 1).

Subjects	# of Students	Avg. time per respondent [minutes)	Student Burden (hours)	# of Teachers	Avg. time per respondent [minutes)	Teacher Burden (hours)	# of Schools	Avg. time per respondent [minutes)	School Burden (hours)
4th Grade
Science pilot (P&P, HOTs)	1,500	25	625	152	20	51	38	30	19
8th Grade
Operational (US history, civics, geography); Pilot (science P&P, science HOTs)	27,500	25	11,458	2,742	20 or 30¹	937	457	30	229
12th Grade
Science pilot (P&P, HOTs)	1,500	25	625	-	-	-	38	30	19
Total	30,500	N/A	12,708	2,894	N/A	988	533	N/A	267

Total number of respondents:				33,927	Total burden hours:				13,963

Note: Due to rounding, some totals may differ slightly from sum of subtotals.

¹ It is estimated that 95% of the grade 8 teachers teach one subject and, thus, have 20 minutes of burden, while about 5% of the grade 8 teachers teach both social studies and science and, thus, have 30 minutes of burden.

6. How, by Whom, and for What Purpose the Data Will Be Used

The purpose of NAEP is to collect and report assessment data on student achievement in the subject areas assessed for use in monitoring educational progress. In addition to reporting overall results of student performance and achievement, NAEP also reports student performance results for various subgroups of students and on various educational factors. The Governing Board sets guidance for what is asked in the questions. NCES is responsible for developing the questions and for selecting the final set of questions. The questions are designed to (a) provide the information for disaggregating data according to categories specified in the legislation,⁴ (b) provide contextual information that is subject specific (e.g. reading, mathematics) and has an impact and known relationship to student achievement, and (c) provide policy-relevant information specified by the Governing Board.

In the original request to OMB for system clearance, NCES requested approval of the data-gathering instruments to be used in the 2014–2016 national- and state-level NAEP assessments. An expanded description of the purposes of the data is provided in the system clearance submittal.

7. Estimates of Costs to the Federal Government

Table 6 provides estimates for the Wave 1 portion of the 2014 administrations.

Table 6: Administration Cost Estimates (Wave 1 subjects)

Activity	Provider	Estimated Cost
Printing, packaging, and distribution phases of the administration including: prepare and package the assessment and all auxiliary materials distribute assessment booklets and materials to the test administrators for each school	Pearson (the Materials, Distribution, Processing and Scoring [MDPS] contractor)	$1.3 million
Field administration including: administration of assessments data collection from the field	Westat (the Data Collection contractor and the NAEP Support and Service Center [NSSC] contractor)	$1.6 million
Totals		$2.9 million

Appendix A: Committee Lists

NAEP Civics Standing Committee

Name Affiliation

Patricia Avery University of Minnesota, Minneapolis, MN

Steve Camicia Utah State University, Logan, UT

Michael Fischer Consultant, Encino, CA

Daphne Greene Doe Elementary School, Mountain City, TN

Lewis Huffman South Carolina Department of Education

Peter Levine Tufts University, Medford/Somerville, MA

Kimberly Eggborn Howard County Public Schools, Ellicott City, MD

Ayo Magwood Cesar Chavez Public Charter School, Washington, DC

Rick Miller Princeton Regional Schools, Princeton, NJ

Richard Ochoa Alta High School, Sandy, UT

Clarissa Peterson DePauw University, Greencastle, IN

Joseph Stewart, Jr. Clemson University, Clemson, SC

Marcie Taylor-Thoma Maryland State Department of Education, Baltimore, MD

Kim Tetzlaff Banyan Elementary School, Newbury Park, CA

Jackie Viana Hialeah Gardens Middle School, Miami, FL

NAEP Geography Standing Committee

Name Affiliation

Sarah Bednarz Texas A&M University, College Station, TX

Osa Brand National Council for Geographic Education, Washington, DC

Karen Thomas-Brown University of Michigan–Dearborn, Dearborn, MI

Charlie Fitzpatrick ESRI Schools, Arlington, VA

Donna Fontaine Westside Elementary School, Smithfield, VA

Paul Gray Russellville High School, Russellville, AR

Harry Hanna Frederick County Public School, Walkersville, MD

Briavel Holcomb Rutgers University, New Brunswick, NJ

Niem Huynh Association of American Geographers, Washington, DC

Lauren Mitterman Gibraltar Middle School, Fish Creek, WI

David Palmer Eaglecrest High School, Centennial, CO

Harris Payne Omaha Public Schools, Omaha, NE

Joe Stoltman Western Michigan University, Kalamazoo, MI

Kelly Swanson Johnson Senior High, St. Paul, MN

Fred Walk Illinois State University, Normal, IL

Sonja Yow Eastern Kentucky University, Richmond, KY

NAEP Science Standing Committee

Name Affiliation

Alicia Cristina Alonzo Michigan State University, East Lansing, MI

Carlos Ayala Sonoma State University, Rohnert Park, CA

Mary Thandi Buthelezi Wheaton College, Wheaton, IL

Susan Craft Hanahan High School, Hanahan, SC

George Deboer American Association for the Advancement of Science, Washington, DC

Alex Decaria Millersville University, Millersville, PA

Crystal Edwards Lawrence Township Public Schools, Lawrenceville, NJ

Ibari Igwe Shrewd Learning, Elkridge, MD

Michele Lombard Swanson Middle School, Arlington, VA

Ellen Mingione Delaware Department of Education, Dover, DE

Brett Moulding Utah Partnership for Effective Science Teaching and Learning, Ogden, UT

Amy Pearlmutter Littlebrook Elementary School, Princeton, NJ

Steve Semken Arizona State University, Tempe, AZ

Gerald Wheeler National Science Teacher Association, Arlington, VA

David White Consultant, Berlin, VT

NAEP U.S. History Standing Committee

Name Affiliation

Keith Barton Indiana University, Bloomington, IN

Konstantin Dierks Indiana University, Bloomington, IN

Cosby Hunt Bell Multicultural High School, Washington, DC

Linda Levstik University of Kentucky, Lexington, KY

Louisa Moffitt Marist School, Atlanta, GA

Alana Murray Loiederman Middle School, Silver Spring, MD

Caleb Perkins International Education Curriculum, Instruction, & Assessment, Olympia, WA

Kim Rasmussen Auburn Washburn Unified School District, Topeka, KS

Tina Reynolds Woodfill Elementary School, Fort Thomas, KY

Larry Starr Nebraska Department of Education, Lincoln, NE

Kathleen Anderson Steeves National Council for History Education, College Park, MD

Diana Turk New York University, New York, NY

Rebecca Valbuena Stanton School, Glendora, CA

Bruce Vansledright University of Maryland, College Park, MD

Appendix B: Parent/Guardian Notification Letter

NAEP (NAEP Assessment Year) PARENT/GUARDIAN NOTIFICATION LETTER

(School Letterhead)

(Insert Date Here)

Dear Parent or Guardian:

We are pleased to notify you that (school name) has been selected to represent schools across the nation by participating in the National Assessment of Educational Progress (NAEP). First administered in 1969, NAEP is the largest nationally representative assessment of what students know and can do in various subject areas. It is administered by the National Center for Education Statistics, within the U.S. Department of Education. NAEP is different from state assessments because it provides a common measure of student achievement across the country. The results of NAEP are released as The Nation's Report Card, which provides information about student achievement to educators, parents, policymakers, and the public.

In our school, the NAEP assessment will be given on (date) in (subject). Your child (may be/has been) selected to take the assessment. In addition to subject area questions, students will be asked some questions about themselves and their educational experience. These questions provide contextual information for the assessment, as well as information that may be related to students’ learning. If you would like to view sample subject area and student questions, please visit http://nationsreportcard.gov/parents.asp.

It will take about (assessment time) minutes for most students to participate in the assessment. The results are completely confidential (in accordance with the Confidential Information Protection provisions of Title V, Subtitle A, Public Law 107-347), and the information provided will be used for statistical purposes only. Your child’s grades will not be affected. Your child may be excused from participation for any reason, is not required to complete the assessment, and may skip any question. While NAEP is voluntary, we depend on student participation to provide an accurate measure of student achievement that will inform improvements in education. Your child will represent many other students, so participation is very important. However, if you do not want your child to participate, please notify me in writing by (date).

There is no need to study in preparation for NAEP. We do, however, ask parents to encourage their children to do their best and get plenty of rest the night before the assessment. A brochure that explains what participation in NAEP means for you and your child is available at http://nces.ed.gov/nationsreportcard/pdf/parents/2012469.pdf. Should you have questions, please contact me at (telephone number) or at (e-mail address).

We are excited that our school will be participating in NAEP and pleased that your child (may be/has been) selected. We know that (school name)'s students will help us show what our nation’s students know and can do.

Sincerely,

School Principal

1 NCES collects student question data, referred to as core questions, that are required by law (20 U.S.C. § 9622; i.e., race, ethnicity, and socioeconomic status) and provide a context for reporting student performance.

2 The 2014 Sampling Memo (dated 3-25-2013) contains information related to the grades 4 and 12 social studies assessments, which were subsequently dropped.

3 The 25-minute burden calculation is comprised of 15 minutes for survey questions and 10 minutes for set up and directions. The time for the cognitive sections is not part of the burden calculation.

4 Education Sciences Reform Act of 2002 (ESRA), National Assessment of Educational Progress (20 USC § 9622).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	NATIONAL ASSESSMENT OF
Author	joconnell
File Modified	0000-00-00
File Created	2021-01-30