Appendix C NAEP System Clearance 2014-2016

NATIONAL CENTER FOR EDUCATION STATISTICS
NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS

Appendix C
Example of Sample Design Document (2013 Assessment)

Request for System Clearance for
NAEP Assessments for 2014-2016
OMB# 1850-0790

NAEP 2014-2016 System Clearance: Appendix C

page C-1

Date:

May 24, 2012

Memo: 20131.3A/1.3B/1.3G/1.3D/1.E/1.3R

To:

William Ward, NCES
Nicole Beaulieu, ETS
Ed Kulick, ETS
David Freund, ETS
Amy Dresher, ETS
Connie Smith, Pearson
Dianne Walsh
Lauren Byrne
Debby Vivari
Rob Dymowski
William Wall
John Burke

From:

Keith Rust

Subject:

Sample Design for 2013 NAEP

I.

Lloyd Hicks
Jennifer Kali
David Ferraro
Erin Wiley
Dwight Brock
Amy Lin
Chris Averett
Kavemuii Murangi
David Hubble
Sarah Shore

Introduction

For 2013, the NAEP assessment involves the following components:
n

National assessments in reading and math at grades 4, 8, and 12;

n

State-by-state and Trial Urban District Assessment (TUDA) assessments in reading and
math for public schools at grades 4 and 8;

n

State-by-state assessments in reading and math for 13 selected states at grade 12;

n

A special study to link NAEP and the High School Longitudinal Study (HSLS), for
mathematics at grade 12;

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R

May 24, 2012

n

A special study to link NAEP and the Lexile assessment, for reading at grade 8;

n

A special study of Knowledge and Skills Appropriate (KaSA) items in mathematics, at
grades 4 and 8, including Puerto Rico;

n

A special study of Accessible Booklets, for reading at grades 4 and 8;

n

A special study of Read-Aloud Accommodations, for reading at grade 4;

n

A special study to link NAEP and PISA, for mathematics, in certain schools selected for
grade 12 NAEP;

n

Pilot tests in reading and math at grades 4 and 8;

n

A pilot test of the Technology and Engineering Literacy (TEL) Assessment at grade 8.

Below is a summary list of the features of the 2013 sample design.
1.

As in recent NAEP studies, Trial Urban District Assessment (TUDA) samples will form
part of the corresponding state samples, and the state samples will form part of the
national sample. The same twenty-one Trial Urban District Assessment (TUDA)
participants as in 2011 will be involved.

2.

At grade 8 there will be two public school samples, and two private school samples. A
relatively small PSU-based sample, known as the beta sample (for public schools) and
the epsilon sample (for private schools), will be used for the computer-based TEL pilot
assessment. The alpha samples for grades 4 and 8 public, the gamma sample for grade
12 public, and the delta samples for private schools at grades 4, 8, and 12 will be used
for the operational assessments in reading and mathematics, and for the pilot test
samples for reading and mathematics, and all special studies except for the Read-Aloud
Accommodations Study.

3.

Oversampling of private schools at grades 4 and 8 will be scaled back to the level of
2009. Response rates permitting, this will allow separate reporting, for reading and
mathematics, for Catholic, and non-Catholic schools, but not further breakdown. As in
2009 there will be no oversampling of private schools at grade 12.

4.

There will be no samples in territories, other than for Puerto Rico at grades 4 and 8 for
the KaSA study.

5.

For the first time since 2003, there is no National Indian Education Study (NIES). This
means that less extensive sampling of BIE schools is required. To ensure sound results
for AIAN students at the national level, at grades 4 and 8 BIE students will be sampled
at the same rate as students in Oklahoma, the state with the largest AIAN population.

6.

As in 2011, the Department of Defense Schools are expected to be reported as a single
jurisdiction (DoDEA), instead of the two components of domestic (DDESS) and

Page 2 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R

May 24, 2012

overseas (DoDDS). However, for design purposes, we will still sample these as two
separate entities.
7.

At grade 12, there will state-level samples for 13 states. Eleven of these also participated
at the state level in 2009. As at grades 4 and 8, state and national samples will be
integrated. The 13 states participating at the state level for reading and mathematics are
Arkansas, Connecticut, Florida, Idaho, Illinois, Iowa, Massachusetts, Michigan, New
Hampshire, New Jersey, South Dakota, Tennessee, and West Virginia. The NAEPPISA linking study will be conducted in the grade 12 sample schools in Connecticut,
Florida, and Massachusetts. These three states will conduct state-level PISA in the fall
of 2012.

8.

The Read-Aloud Accommodations study will be conducted in special sample of grade 4
public schools, to be referred to as the rho sample. This sample will be restricted to a
small number of states, and will not have any overlap schools with any of the other
2013 NAEP assessments. The sample will not be a nationally representative probability
sample of schools.

9.

The sample sizes of assessed students for these various components are shown in Table
1 (which also shows the approximate numbers of participating schools).

Note that the figures in Table 1 (and also Tables 3 and 4) do not reflect the fact that, at grade 4,
certain schools in the pencil and paper assessments may elect to include all students in the sample,
rather than the sample size targeted by NAEP. This will lead to an increase is sample size, likely to
be about five percent overall. The increase will vary by state, depending on the size of the state and
the size of its schools – obviously there is no addition in states where the design calls for the
selection of all students.
Table 1.

Target sample sizes of assessed students, and expected number of participating
schools, for 2013 NAEP
Spiral
Spiral
Indic.

Grade 4
nat'l/state math
nat'l/state reading
math Pilot
reading Pilot
math KaSA
Reading accessible booklet study
Puerto Rico KaSA
Total - alpha
Total- delta

RM
RM
RM
RM
RM
RM
PR
2
1

Jurisdictions
States (incl.
DC, BIE,
Urban
DoDEA)
districts
53
53

1

21
21

Students
Public
Private
school
school
students
students
163,000
163,000
4410
5145
3920
2940
5000
347,415

3,000
3,000
90
105
80
60
0
6,335

Page 3 of 24

Total
166,000
166,000
4500
5250
4000
3000
5000

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R
Table 1.

May 24, 2012

Target sample sizes of assessed students, and expected number of participating
schools, for 2013 NAEP (Continued)
Spiral
Spiral
Indic.

Jurisdictions
States (incl.
DC, BIE,
Urban
DoDEA)
districts

Grade 4 (Continued)
Typical max. no. students/school
Average assessed
students/school
Total schools - alpha, delta
Reading Aloud Study
Total - Rho
Typical max. no. students/school
Average assessed
students/school
Total schools - rho
Total number of students grade 4
Total number of schools grade 4
Grade 8
nat'l/state math
nat'l/state reading
math Pilot
reading Pilot
math KaSA
Reading accessible booklet study
Lexile Linking Study
Puerto Rico KaSA
Total - alpha
Total - delta
Typical max. no. students/school
Average assessed
students/school
Total schools - alpha, delta
TEL Pilot
Total - beta
Total - epsilon
Typical max. no. students/school
Average assessed
students/school
Total schools - beta, epsilon
Total number of schools grade 8
Total number of students grade 8

RA, RB,
RC
3

Students
Public
Private
school
school
students
students
60

64

44
7,900

24
265

2,100
2,100
27
27
78
349,515
7,970

RM
RM
RM
RM
RM
RM
RM
PR
2
1

53
53

1

21
21

159,080
159,080
4,410
5,145
3,920
2,940
7,840
5,000
347,415
60
50
6,950
13,500
13,500

TL
1
1

Page 4 of 24

Total

8,183
2,100

0
6,335
265

78
355,850
8,243

3,000
3,000
90
105
80
60
160
0

162,080
162,080
4,500
5,250
4,000
3,000
8,000
5,000

6,495
65
25
260
1,500

30

1,500
30

25
540
7,410
360,915

25
60
320
7,995

7,210
15,000

600
7,810
368,910

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R
Table 1.

Target sample sizes of assessed students, and expected number of participating
schools, for 2013 NAEP (Continued)
Spiral
Spiral
Indic.

Grade 12
national and state reading
National and state math
HSLS Linking Study
NAEP-PISA Linking Study
Total - gamma
Total- delta
Typical max. no. students/school
(excluding HSLS)
Average assessed
students/school
Total schools - gamma, delta

RM
RM
ML
MP
3
1

Jurisdictions
States (incl.
DC, BIE,
Urban
DoDEA)
districts
13
13
3

Students
Public
Private
school
school
students
students
41,120
38,380
6,000
18,000
103,500

1,500
1,400

Total
42,620
39,780
6,000
18,000

2,900

GRAND TOTAL STUDENTS
GRAND TOTAL SCHOOLS

II.

May 24, 2012

60

60

50
1,690

40
73

1,763

813,930
17,070

17,230
658

831,160
17,736

Assessment Types

The assessment spiral types are shown in Table 2. At grade 4 there will be five different spirals used,
with three at grade 8 and three at grade 12. Session IDs contain six digits, traditionally. The first two
digits identify the assessment “type” (subjects and type of spiral in a general way). Grade is
contained in the second pair of digits, and the session sequential number (within schools) in the last
two digits. For example, session RM 0401 denotes the first grade 4 reading and mathematics
operational assessment in a given school.

Page 5 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R
Table 2.

NAEP 2013 assessment types and IDs

ID

Type

RM

Operational
Pencil & Paper

RC

Operational–
Pencil & Paper
Operational–
Pencil & Paper
PilotComputerbased
Special Study Pencil & Paper
Special Study
– Pencil &
Paper
Special Study
– Pencil &
Paper
Special Study
– Pencil &
Paper
Special Study
– Pencil &
Paper

PR

Puerto Rico

RM
RM

TL
ML

MP

RA

RB

III.

May 24, 2012

Subjects
Reading (including pilot,
accessible booklets), math
(including pilot, KaSA)
Reading (including pilot,
accessible booklets, Lexile
linking), math (including
pilot and KaSA)

Grades

Reading, math

12

Public and
Private
Public and
private

Technology and
Engineering Literacy

8

Public and
Private

HSLS Linking Study

12

Public

PISA Linking Study

9, 10,
11

Public

In three states only

Reading Aloud Study

4

Public

In a small number of states; no
special accommodations

Reading Aloud Study

4

Public

In a small number of states;
Read Aloud Items

Reading Aloud Study

4

Public

KaSA math

4&8

Public

4

8

Schools
Public and
Private

Comments
Includes all grade 4 assessment
components, except for Puerto
Rico and Reading Aloud
Includes all grade 8
components except Puerto Rico
and TEL pilot

National sample, not in Puerto
Rico

In a small number of states;
Read Aloud Everything
Booklets will not be used in any
other jurisdiction.

Sample Types and Sizes

In similar fashion to past years (but somewhat different), we will identify six different types of
school samples: Alpha, Beta, Gamma, Delta, Epsilon, and Rho. These distinguish sets of schools
that will be conducting distinct portions of the assessment. The Rho sample is new for 2013; this
will be used for the grade 4 Read-Aloud Accommodations study which has a unique design.

1.

Alpha Samples at Grades 4 and 8

These are public school samples for grades 4 and 8. They will be used for the operational state-bystate assessments in reading and math, and contribute to the national samples for these subjects as

Page 6 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R

May 24, 2012

well. The will also be used for pilot tests and special studies, and outlined in Tables 1 and 2. There
will be alpha samples for each state, DC, DoDEA, BIE, and Puerto Rico.
The details of the target student sample sizes for the alpha samples are as follows:
n

With the exception of BIE, the target sample size for each state and grade is 6,600
students. For BIE, the sample will be selected at the same rate as Oklahoma. This is to
ensure adequate samples of American Indian students for national reporting. The rate
from Oklahoma is being used since it has the largest American Indian population of any
state. Therefore using this rate ensures that the BIE sample will not be so small as to
unduly inflate the standard errors of estimates for American Indian/Alaska Native
students, while at the same time not placing undue burden on BIE schools, for which
no jurisdiction level results are to be reported, in the absence of the NIES.

n

There will be samples for twenty-one TUDA districts. For the five largest (New York
City, Los Angeles, Chicago, Miami-Dade, and Houston), the student target sample sizes
for reading and mathematics are three-quarters the size of a state sample (2,475 per
subject). For the other fifteen districts the student target sample sizes for reading and
math are one-half the size of a state sample (1,650 per subject). An exception will be
made in the case of Detroit, where the target sample sizes will be increased in
anticipation of substantial numbers of schools closings where the particular schools
affected have not yet been identified.

Note, that above, there is a conflict between sample size requirements at the state level, and the
TUDA district level. This will be resolved as in previous years: the districts will have the target
samples indicated in B, and reflected in Table 3. For the states that contain one or more of these
districts, the target sample size indicated in A (and shown in Table 3) will be used to determine a
school sampling rate for the state, which will be applied to the balance of the state outside the
TUDA district(s). Thus the target student sample sizes, shown in Table 3, for states that contain a
TUDA district, are only ‘design targets’, and are smaller than the final total sample size for the state,
but larger than the sample for the balance of the state, exclusive of its TUDA districts. In the case of
the District of Columbia, the state sample size requirement is that all schools and students be
included. This renders moot any requirements for the DC TUDA sample, which by default consists
of all schools operated by the DCPS district (but excludes charter schools in DC, even though those
are all included in the state sample, as these are not operated by DCPS).
In Puerto Rico, the target sample size is 5,800 per grade (grades 4 and 8), with the goal of assessing
5,000 students. Only KaSA mathematics will be assessed in Puerto Rico (PR session type).

Page 7 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R

May 24, 2012

As in past state-by-state assessments, schools with fewer than 20 students in the grade in question
will be sampled at a moderately lower rate than other schools (at least half, and often higher,
depending upon the size of the school). This is in implicit recognition of the greater cost and burden
associated with surveying these schools.
Table 3 shows the target student sample sizes, and the approximate counts of schools to be selected
in the alpha samples, along with the school and student frame counts, by state and TUDA districts
for grades 4 and 8. The table also identifies the jurisdictions where we take all schools and where we
take all students.

Page 8 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R
Table 3.

May 24, 2012

Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for
the 2013 state-by-state and TUDA district assessments (alpha samples)
Grade 4

Jurisdiction
Alabama
Alaska
Arizona
Arkansas
Bureau Of Indian Education
California
Colorado
Connecticut
Delaware
District Of Columbia
DoDEA Schools
Florida
Georgia
Hawaii
Idaho
Illinois
Indiana
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri

Schools in
frame
748
365
1,179
496
135
5,846
1,013
604
115
131
108
2,097
1,242
202
368
2,343
1,101
674
729
731
789
341
887
973
1,856
952
436
1,170

Schools in
sample
117
197
120
119
16
113
122
116
95
131
108
110
107
117
128
123
116
137
141
117
120
159
115
117
121
126
114
131

Students
in frame
59,269
9,827
83,555
37,017
3,246
463,973
63,266
41,489
9,647
4,907
7,507
202,702
130,245
14,155
21,351
154,495
80,245
35,421
35,907
51,685
55,300
13,945
61,870
71,298
118,242
61,086
38,958
68,189

Grade 8
Overall
target
student
sample
size
6,600
6,600
6,600
6,600
411
6,600
6,600
6,600
6,600
4,907
7,507
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600

Page 9 of 24

**
**

Schools in
frame
484
285
769
315
109
2,818
520
306
60
79
63
1,150
548
80
200
1,603
485
387
412
391
531
211
356
482
1,042
695
290
727

Schools in
sample
112
144
118
112
13
113
116
110
60
79
63
112
106
60
103
119
107
118
126
114
119
115
109
108
114
125
108
124

Students in
frame
57,283
9,701
81,283
35,831
2,785
456,125
59,357
42,667
9,398
4,307
5,589
196,792
124,307
12,843
20,466
154,308
79,856
34,991
34,645
49,403
50,584
14,205
60,987
71,962
120,377
61,246
36,999
68,007

Overall
target
student
sample size
6,600
6,600
6,600
6,600
375
6,600
6,600
6,600
6,600
4,307
5,589
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600

*
**
**

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R
Table 3.

May 24, 2012

Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for
the 2013 state-by-state and TUDA district assessments (alpha samples) (Continued)
Grade 4

Jurisdiction
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Puerto Rico
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virginia
Washington
West Virginia
Wisconsin
Wyoming
Albuquerque
Atlanta

Schools in
frame
400
568
379
265
1,366
431
2,368
1,414
260
1,877
895
767
1,742
1,017
170
618
328
998
4,277
582
224
1,137
1,214
425
1,111
188
99
61

Schools in
sample
199
165
112
133
117
127
113
112
260
119
137
129
116
160
117
112
194
116
112
112
224
112
119
146
131
188
56
61

Students
in frame
10,863
22,085
33,851
14,495
100,453
25,731
192,295
118,129
6,995
133,358
49,300
42,827
129,550
38,842
10,437
55,228
9,380
75,934
373,404
46,508
6,419
93,610
77,826
20,875
60,997
6,849
7,594
4,264

Grade 8
Overall
target
student
sample
size
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,995
6,600
6,600
6,600
6,600
5,800
6,600
6,600
6,600
6,600
6,600
6,600
6,419
6,600
6,600
6,600
6,600
6,849
3,300
4,264

Page 10 of 24

**

**

**
**

Schools in
frame
289
339
158
137
741
203
1,335
694
188
1,090
593
414
914
407
57
298
254
565
2,165
232
122
388
606
202
633
93
42
26

Schools in
sample
151
131
88
88
110
105
109
110
188
112
128
120
109
123
57
107
153
112
112
106
122
106
115
109
118
93
42
26

Students in
frame
10,912
21,503
33,028
15,191
99,535
24,183
193,235
111,343
7,330
132,686
46,433
42,949
132,799
37,363
10,842
52,433
9,306
71,570
348,907
42,593
6,364
92,179
77,099
20,637
61,063
6,568
6,903
3,369

Overall
target
student
sample size
6,600
6,600
6,600
6,600
6,600
6,600
6,600
6,600
7,330
6,600
6,600
6,600
6,600
5,800
6,600
6,600
6,600
6,600
6,600
6,600
6,364
6,600
6,600
6,600
6,600
6,568
3,300
3,369

**

*

**

**
*
**

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R
Table 3.

May 24, 2012

Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for
the 2013 state-by-state and TUDA district assessments (alpha samples) (Continued)
Grade 4

Jurisdiction
Austin
Baltimore City
Boston
Charlotte
Chicago
Cleveland
Dallas
Detroit
Fresno
Hillsborough County
Houston
Jefferson County
Los Angeles
Miami-Dade
Milwaukee
New York City
Philadelphia
San Diego
District of Columbia PS

Schools in
frame
80
124
79
105
488
83
147
121
70
164
179
96
516
274
115
709
177
138
87

Schools in
sample
54
68
79
54
96
83
53
81
54
55
82
54
82
83
67
82
60
57
87

Students
in frame
6,862
6,292
4,106
11,245
29,942
3,573
12,932
7,584
5,823
15,161
16,525
7,617
51,516
26,903
5,807
63,731
12,098
10,040
3,369

Grade 8
Overall
target
student
sample
size
3,300
3,300
4,106
3,300
4,950
3,573
3,300
4,470
3,300
3,300
4,950
3,300
4,950
4,950
3,300
4,950
3,300
3,300
3,369

**

**

**

Schools in
frame
24
93
38
38
470
81
38
74
26
80
64
42
162
160
89
458
142
63
37

Counts for states do not reflect the oversampling for their constituent TUDA districts, nor the grade 4 ‘take all’ option.
Target student sample sizes reflect sample sizes prior to attrition due to exclusion, ineligibility, and nonresponse.
* identifies jurisdictions where all schools (but not all students) for the given grade are included in the NAEP sample.
** identifies jurisdictions where all students for the given grade are included in the NAEP sample.

Page 11 of 24

Schools in
sample
24
62
38
38
95
81
38
65
26
48
49
42
77
76
56
82
58
40
37

Students in
frame
5,360
5,501
3,808
9,852
29,469
3,550
10,113
5,001
5,457
15,020
12,738
7,030
46,818
25,573
5,371
61,278
10,970
9,691
2,357

Overall
target
student
sample size
3,300
3,300
3,808
3,300
4,950
3,550
3,300
4,240
3,300
3,300
4,950
3,300
4,950
4,950
3,300
4,950
3,300
3,300
2,357

*
**
*
**
*
*

*

**

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R

May 24, 2012

Table 4 consolidates the target student (and resulting school) sample size numbers, to show the total
target sample sizes in each state, combining the TUDA targets with those for the balance of the
state.

Page 12 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R
Table 4.

May 24, 2012

Total sample sizes, combining state and TUDA samples
Grade 4

Jurisdiction
Alabama
Alaska
Arizona
Arkansas
Bureau Of Indian Education
California
Colorado
Connecticut
Delaware
District Of Columbia
DoDEA Schools
Florida
Georgia
Hawaii
Idaho
Illinois
Indiana
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Montana

Schools
in frame
748
365
1,179
496
135
5,846
1,013
604
115
131
108
2,097
1,242
202
368
2,343
1,101
674
729
731
789
341
887
973
1,856
952
436
1,170
400

Schools
in sample
117
197
120
119
16
290
122
116
95
131
108
225
164
117
128
194
116
137
141
156
120
159
170
188
194
126
114
131
199

Students
in frame
59,269
9,827
83,555
37,017
3,246
463,973
63,266
41,489
9,647
4,907
7,507
202,702
130,245
14,155
21,351
154,495
80,245
35,421
35,907
51,685
55,300
13,945
61,870
71,298
118,242
61,086
38,958
68,189
10,863

Grade 8
Overall
target
student
sample
size
6,600
6,600
6,600
6,600
411
17,189
6,600
6,600
6,600
4,907
7,507
13,479
10,648
6,600
6,600
10,267
6,600
6,600
6,600
8,929
6,600
6,600
9,229
10,326
10,646
6,600
6,600
6,600
6,600

Page 13 of 24

**
**

Schools in
frame
484
285
769
315
109
2,818
520
306
60
79
63
1,150
548
80
200
1,603
485
387
412
391
531
211
356
482
1,042
695
290
727
289

Schools in
sample
112
144
118
112
13
241
116
110
60
79
63
213
129
60
103
190
107
118
126
139
119
115
160
140
174
125
108
124
151

Students in
frame
57,283
9,701
81,283
35,831
2,785
456,125
59,357
42,667
9,398
4,307
5,589
196,792
124,307
12,843
20,466
154,308
79,856
34,991
34,645
49,403
50,584
14,205
60,987
71,962
120,377
61,246
36,999
68,007
10,912

Overall
target
student
sample size
6,600
6,600
6,600
6,600
375
17,251
6,600
6,600
6,600
4,307
5,589
13,486
9,790
6,600
6,600
10,285
6,600
6,600
6,600
8,963
6,600
6,600
9,305
10,059
10,566
6,600
6,600
6,600
6,600

*
**
**

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R
Table 4.

May 24, 2012

Total sample sizes, combining state and TUDA samples (Continued)
Grade 4

Jurisdiction
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Puerto Rico
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virginia
Washington
West Virginia
Wisconsin
Wyoming

Schools
in frame
568
379
265
1,366
431
2,368
1,414
260
1,877
895
767
1,742
1,017
170
618
328
998
4,277
582
224
1,137
1,214
425
1,111
188

Schools
in sample
165
112
133
117
149
159
156
260
197
137
129
165
160
117
112
194
116
293
112
224
112
119
146
185
188

Students
in frame
22,085
33,851
14,495
100,453
25,731
192,295
118,129
6,995
133,358
49,300
42,827
129,550
38,842
10,437
55,228
9,380
75,934
373,404
46,508
6,419
93,610
77,826
20,875
60,997
6,849

Grade 8
Overall
target
student
sample
size
6,600
6,600
6,600
6,600
7,938
9,361
9,271
6,995
9,996
6,600
6,600
9,283
5,800
6,600
6,600
6,600
6,600
17,665
6,600
6,419
6,600
6,600
6,600
9,270
6,849

**

**

**

Schools in
frame
339
158
137
741
203
1,335
694
188
1,090
593
414
914
407
57
298
254
565
2,165
232
122
388
606
202
633
93

Sample sizes for each state do reflect the samples in the TUDA districts within the state, but not the grade 4 take-all option.
* identifies jurisdictions where all schools (but not all students) for the given grade are included in the NAEP sample.
** identifies jurisdictions where all students for the given grade are included in the NAEP sample.

Page 14 of 24

Schools in
sample
131
88
88
110
119
156
138
188
189
128
120
156
123
57
107
153
112
216
106
122
106
115
109
163
93

Students in
frame
21,503
33,028
15,191
99,535
24,183
193,235
111,343
7,330
132,686
46,433
42,949
132,799
37,363
10,842
52,433
9,306
71,570
348,907
42,593
6,364
92,179
77,099
20,637
61,063
6,568

Overall
target
student
sample size
6,600
6,600
6,600
6,600
8,008
9,456
9,315
7,330
9,973
6,600
6,600
9,355
5,800
6,600
6,600
6,600
6,600
17,792
6,600
6,364
6,600
6,600
6,600
9,321
6,568

**

*

**

**

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R

May 24, 2012

Stratification
Each state and grade will be stratified separately, but using a common approach in all cases. TUDA
districts will be separated from their state, and each part stratified separately. The first level of
stratification will be based on urban-centered type of location. This variable has 12 levels (some of
which may not be present in a given state or TUDA district), and these will be collapsed so that each
of the resulting location categories contains at least nine percent of the student population. Within
each of the resulting location categories, schools will be assigned a minority enrollment status. This
is based on the two race/ethnic groups that are the second and third most prevalent within the
location category. If these groups are both low in percentage terms, no minority classification will be
used. Otherwise three (or occasionally four) equal-sized groups (generally high, medium, and low
minority) will be formed based on the distribution across schools of the two minority groups.
Within the resulting location and minority group classes (of which there are likely to be from three
to fifteen, depending upon the jurisdiction), schools will be sorted by a measure derived from school
level results from the most recent available state achievement tests at the relevant grade. In general,
mathematics test results will be used, but where these are not available, reading results will be used.
In the few states that do not have math or reading tests at grades 4 and 8 (or where we are unable to
match the results to the NAEP school frame), instead of achievement data, schools will be sorted
using a measure of socio-economic status. This is the median household income of the 5-digit ZIP
Code area where the school is located, based on 2000 Population Census data.
Once the schools are sorted by location class, minority enrollment class, and achievement data (or
household income), a systematic sample of schools will be selected using a random start. Schools
will be sampled with probability proportional to size. The exact details of this process are described
in the individual sampling specification memos.

2.

Beta Sample

The beta sample at grade 8 is a public school sample. This sample will be used to conduct the TEL
pilot assessment. Although only a pilot test sample, this will be a nationally representative sample,
with the exception that schools that are included in the grade 8 alpha sample with certainty will not
be represented. This is because there is to be no overlap with the grade 8 alpha school sample.

Page 15 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R

May 24, 2012

The first stage of sampling for the beta sample is the selection of a sample of geographic primary
sampling units (PSUs). These will be selected using the same design as for other recent PSU-based
assessments. However, with the advent of county-level census and ACS data following the 2010
Census, a new PSU frame is being created, and stratified. A total 67 PSUs will be selected,
representing the U.S. (but not including Puerto Rico, or DODEA schools that are located outside
the 50 states and D.C.). This PSU component is needed because of the operational complexities of
administering the computer-based assessment. A select group of staff will be trained to administer
those assessments. The school stratification of the beta sample within PSUs will be by type of
location and median household income. Schools with more than 15 percent black or Hispanic
students will be sampled at twice the rate of other schools, so as to increase the student sample sizes
for these two groups.

3.

Gamma Sample

This is the public school sample at grade 12, used for the reading and mathematics assessments. It
will consist of a single sample of schools covering both national- and state-level estimates.
As in past assessments, modest oversampling of Black and Hispanic students will be undertaken in
this sample, in the balance of the country outside the 13 participating states. This will be carried out
at the school level. Each school with more than 15 percent Black and Hispanic students will be
given twice the selection probability of other schools of comparable size. This means that while
about 53 percent of the student population (including over 90 percent of the Black and Hispanic
students) are in the oversampled schools, about 70 percent of the sample students will come from
these schools.
Table 5 shows the sample size targets for each of the 13 state-specific samples. Note that the target
student sample sizes vary slightly be state. These are target sampled, rather than assessed, and have
been estimated based on experience from 2009. Of the 13 states, 11 participated at the state level in
2009, and so ineligibility, nonresponse, and exclusion rates from 2009 have been used to derive
targets for those states. For the other two states (Michigan and Tennessee), national rates from 2009
were used. The goal in each case is to assess about 2,300 students in each of the two subjects
(reading and mathematics).

Page 16 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R
Table 5.

May 24, 2012

Grade 12 school and student frame counts, expected school sample sizes, and
initial target student sample sizes for the 2013 state-by-state assessments (gamma
samples)
Grade 12

Jurisdiction
Arkansas
Connecticut
Florida
Idaho
Illinois
Iowa
Massachusetts
Michigan
New Hampshire
New Jersey
South Dakota
Tennessee
West Virginia

Schools in frame
297
245
965
210
954
408
371
1032
89
432
191
369
147

Schools in sample
101
110
116
95
125
121
107
136
77
110
140
126
92

Students in frame
32,035
41,607
176,821
19,057
149,998
37,793
67,923
126,382
15,749
97,690
8,796
67,111
18,796

Overall target
student sample size
6,200
6,750
6,600
6,250
7,250
6,850
6,250
7,400
8,350
6,500
6,500
7,400
6,650

Stratification
Unlike the other samples, the Gamma sample will have an explicit stratification at the highest level.
The schools will be stratified by state, for the thirteen state-by-state participants, and the balance of
the US, giving 14 explicit strata. In each of the thirteen states, stratification will be carried out in the
same way as for the alpha samples. The last level sorting variable will be median household income,
as there are no achievement data available at the school level for grade 12.
For the explicit stratum containing the balance of the US, the highest level of sorting will be by
Census Division. This gives 8 implicit strata (the balance of New England is too small to be an
effective stratum, and is combined with the balance of the Mid-Atlantic Division). The next stratifier
in the hierarchy is type of location, which has twelve categories. Many of the type of location strata
nested within Census divisions will be collapsed with neighboring type of location cells (this will
occur if the expected school sample size within the cell is less than 4.0). These geographic strata will
be subdivided using a dichotomous high minority status category. Schools are in the oversampling
stratum if they had more than 10 Black or Hispanic students and greater than 15 percent Black or
Hispanic students. Otherwise the school will be put in a regular sampling stratum. If the expected
sample size within these strata is less than 8.0, they will be left as is. If the expected sample size is
greater than 8.0, then each of these two categories will be subdivided into up to four substrata (two
for expected sample size up to 12.0, three for expected sample size up to 16.0, and four for expected
Page 17 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R

May 24, 2012

sample size greater than 16.0). For the category of no oversampling, the subdivision will be by state
or groups of contiguous states. For the oversampling strata, the subdivision will be by percentage
Black and Hispanic. Within these substrata, the schools are to be sorted by school type (public, BIE,
DoDEA) and median household income from the 2000 Census (using a serpentine sort within the
school type substrata).
High School Longitudinal Study (HSLS) Linking Study Sample
The sample of students for the HSLS linking study will be obtained as follows. For the whole
gamma sample, the NAEP school sample will be selected so as to maximize the overlap with the
HSLS school sample. Then at the time of student sampling, any student who is in the HSLS sample,
but is not selected for NAEP (the RM spiral), will be included in the study and assigned the ML
spiral. This is expected to be about 6,000 students from about 300 schools, after accounting for
schools that are not responding to the HSLS.
NAEP-PISA Linking Study Sample
The sample of students for this study will be obtained as follows. For all gamma sample schools in
Connecticut, Florida, and Massachusetts, samples of students will be selected from each of grades 9,
10, and 11 (from among those grades offered by the school). The procedure for selecting the
student samples will be similar to that used to select grade 12 students, and is described in the
student sampling section below. The selected students will be assigned the MP spiral appropriate for
their grade.

4.

Delta Samples

These are the private school samples at grades 4, 8, and 12 for conducting the operational
assessments in reading and math, as well as the special studies that are incorporated into the RM
spiral. The number of students targeted per schools will be 64 at grade 4, 65 at grade 8, and 60 at
grade 12 – 30 for reading, 30 for math, with 4 at grade 4 and 5 at grade 8 for pilot tests and special
studies. The sample sizes are large enough to report results by Catholic and non-Catholic at grades 4
and 8, but at grade 12 only for private schools as a whole. Approximately half the sample at each
grade will be from Catholic schools. This is very similar to the design used in 2009, at each grade.

Page 18 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R

May 24, 2012

Stratification
The private schools are to be explicitly stratified by private school type (Catholic/Other). Within
each private school type, stratification will be by Census region (4 categories), type of location (12
categories), and enrollment size. In general, where there are few or no schools in a given stratum,
categories will be collapsed together, always preserving the private school type.

5.

Epsilon Sample

This sample is analogous to the beta sample, but for private schools, at grade 8. The same PSUs will
be used as for the beta sample, and the TL session type will be conducted. There will be no
oversampling of private schools for this sample. The epsilon sample schools will not overlap with
the delta sample schools.

6.

Rho Sample

This sample will be used to conduct the grade 4 Reading Aloud Accommodations study. It will
consist of about 70 schools from a small number of states, with the selected schools each having
enough SD and ELL students so that nine students can be selected who are SD, nine who are ELL,
and nine who are neither. These students will then be randomly assigned to one of three conditions,
and assessed with a NAEP reading assessment. The three conditions are a) standard conditions; b)
read aloud the test questions, but not the stimulus passage; c) read aloud all of the assessment
material. These schools will not be included in any other NAEP 2013 sample, at any grade.

IV.

New Schools

To compensate for the fact that files used to create the NAEP school sampling frames are at least
two years out of date at the time of frame construction, we will supplement the Alpha, Gamma, and
Delta sample with new school samples at each grade.
The new school samples will be drawn using a two-stage design. At the first stage, a minimum of ten
school districts (in states with at least ten districts) will be selected from each state for public

Page 19 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R

May 24, 2012

schools, and ten Catholic dioceses will be selected nationally for the private schools. The sampled
districts and dioceses will be asked to review lists of their respective schools and identify new
schools. Frames of new schools will be constructed from these updates, and new schools will be
drawn with probability proportional to size using the same sample rates as their corresponding
original school samples.
The school sample sizes in the above tables do not reflect new school samples.

V.

Substitute Samples

Substitute samples will be selected for each of the Beta, Gamma, Delta, and Epsilon samples. The
substitute school for each original will be the next “available” school on the sorted sampling frame,
with the following exceptions:

VI.

n

Schools selected for any NAEP samples will not be used as substitutes.

n

Private schools whose school affiliation is unknown will not be used as substitutes.
Also, unknown affiliated private schools in the original samples will not get substitutes.

n

A school can be a substitute for one and only one sample. (If a school is selected as a
substitute school for grade 12, for example, it cannot be used as a substitute for either
grade 4 or grade 8.)

n

A public school substitute will always be in the same state as its original school.

Contingency Samples

The districts that are taking part in the TUDA program are volunteers. Thus it is possible that at
some point over the next few months, a given district might choose to opt out of the TUDA
program for 2013. However, it is not acceptable for all schools in such a district to decline NAEP,
as then the state estimates will be adversely affected. Thus to deal with this possibility, in each
TUDA district, subsamples of the alpha sample schools will be identified as contingency samples. In
the event that the district withdraws from the TUDA program prior the selection of the student
sample, all alpha sampled schools from that district will be dropped from the sample, with the
exception of those selected in the contingency sample. The contingency sample will provide a
proportional representation of the district, within the aggregate state sample. Student sampling in
those schools will then proceed in the same way as for the other schools within the same state.
Page 20 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R

May 24, 2012

Similarly, in the thirteen states participating at grade 12, in the gamma sample contingency samples
will be identified in those states. If the state decides to withdraw from the state-level assessment, the
contingency sample schools will be retained, as part of the grade 12 national sample.

VII.

Student Sampling

Students within the sampled schools will be selected with equal probability. The student sampling
parameters vary by sample type (Alpha, Beta, Gamma, Delta, Epsilon, and Rho) and grade, as
described below.
Alpha Sample, Grade 4 Schools (Except Puerto Rico)
n

All students, up to 70, will be selected.

n

If the school has between 71 and 90 students (inclusive), the school will be asked if it
wishes to have all students selected, or a sample of 63. If the school elects to have all
students selected, all students will be selected. If the school asks for a sample of 63
students, a systematic sample of 63 students will be selected.

n

If the school has more than 90 students, a systematic sample of 63 students will be
selected.

n

In some schools, the school may be assigned more than one ‘hit’ in sampling. In these
schools we will select a sample of size 63 times the number of hits, taking all students if
this target is greater than or equal to 63/70 of the total grade 4 enrolment.

n

All students will be assigned to assessment type RM.

Alpha Sample, Grade 8 Schools (Except Puerto Rico)
n

All students, up to 70, will be selected.

n

If the school has more than 70 students, a systematic sample of 63 students will be
selected.

n

In some schools, the school may be assigned more than one ‘hit’ in sampling. In these
schools we will select a sample of size 63 times the number of hits, taking all students if
this target is greater than or equal to 63/70 of the total grade 8 enrolment.

n

All students will be assigned to assessment type RM.

Page 21 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R

May 24, 2012

Alpha Sample, Puerto Rico Grades 4 and 8
n

All students, up to 55, will be selected.

n

If the school has more than 55 students, a systematic sample of 50 students will be
selected with no oversampling.

n

All students will be assigned to assessment type PR.

Delta Samples, Grades 4 and 8
n

All students, up to 72, will be selected.

n

If the school has more than 72 students, a systematic sample of 64 students (grade 4), or
65 students (grade 8) will be selected.

n

All students will be assigned to assessment type RM.

Beta and Epsilon Samples, Grade 8
n

In each school, a sample for computer-based testing will be selected as follows: All
students up to 30 will be selected. If there are more than 30 students enrolled, a sample
of 30 students will be selected.

n

All students will be assigned to the TL session.

Gamma and Delta Samples, Grade 12
n

All students, up to 66, will be selected.

n

If the school has more than 66 students, a systematic sample of 60 students will be
selected.

n

In some schools, the school may be assigned more than one ‘hit’ in sampling. In these
schools we will select a sample of size 60 times the number of hits, taking all students if
this target is greater than or equal to 60/66 of the total grade enrollment.

n

All students so selected will be assigned to assessment type RM.

n

For the gamma sample, any grade 12 HSLS students not selected in the above steps will
be added to the sample and assigned assessment type ML.

n

For the gamma sample, in Connecticut, Florida, and Massachusetts, in each school a
sample of students from grades 9, 10, and 11 will be selected. For grades 10 and 11, the
sample will be drawn as for grade 12, but using 30 in place of 60, and 33 in place of 66,

Page 22 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R

May 24, 2012

in the above algorithm. For grade 9, the same procedure will be used, but using figures
of 18 and 20. These students will be assigned assessment type MP.
Rho Sample, Grade 4
Details of the student sampling procedures for the grade 4 Reading Aloud Accommodations study
have yet to be finalized. However, they will be something like the following. On the basis of e-filed
student data for all grade 4 students in the school, students will be classified into four groups: SD
only, ELL only, neither SD nor ELL, and Other (including students who are both SD and ELL, and
students for whom complete data on SD and ELL status is not available). A random sample of nine
students will be selected from each of the first three of these groups. The students from each group
will be randomly assigned in equal numbers to each of the three assessment conditions.

VIII.

Weighting Requirements

Weighting activities for the 2013 NAEP assessments are not covered under the current NAEP
contracts. However, based on past experience, below is a description of the likely weighting
requirements.
The Operational Samples
These samples will have a single set of weights for each subject (reading and math at grades 4, 8, and
12) applied to reflect probabilities of selection, school and student nonresponse, any trimming, and
the random assignment to the particular subject. There will be a separate replication schemes by
grade and public/private. Such weights will also be derived for the Puerto Rico KaSA assessment.
The Pilot Test and Special Study Samples
Fully adjusted weights will not be provided for the students in the pilot test studies and special
studies. However, with the exception of the grade 4 Reading Aloud Accommodations study,
preliminary weights will be available for these samples. These will reflect the school and student
selection probabilities, but with no adjustments for nonresponse.

Page 23 of 24

Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R

May 24, 2012

The HSLS Linking Sample
Weights may be required for the linking sample, which account for the sampling procedures and
nonresponse for both the HSLS and the NAEP assessment. Clearly this will require collaboration
between the next NAEP contractor responsible for weighting, and RTI International, the contractor
for the HSLS study. Once the analysis plan for this study has been specified, a weighting plan can be
developed.

Page 24 of 24