Appendix D NAEP Main 2015 Wave 1 Sample Design Memo (Draft)

Appendix D NAEP Main 2015 Wave 1 Sample Design Memo (Draft).pdf

National Assessment of Education Progress (NAEP) 2014-2016 System Clearance

Appendix D NAEP Main 2015 Wave 1 Sample Design Memo (Draft)

OMB: 1850-0790

Document [pdf]

Download: pdf | pdf

Appendix D
NAEP 2015 Sample Design Memo

Date:

February 26, 2014

Memo: 20151.1A/1.1B/1.1G/1.1D/1.1E

To:

William Ward, NCES
Nicole Beaulieu, ETS
Ed Kulick, ETS
David Freund, ETS
Amy Dresher, ETS
Connie Smith, Pearson
Dianne Walsh
Lauren Byrne
Debby Vivari
Rob Dymowski

From:
Reviewer:

Lloyd Hicks and John Burke
Keith Rust

Subject:

Sample Design for 2015 NAEP - Draft

Jennifer Kali
David Ferraro
Erin Wiley
Dwight Brock
Amy Lin
Chris Averett
Kavemuii Murangi
David Hubble
Sarah Shore
William Wall

Introduction

For 2015, the NAEP assessment involves the following components:
A.

National assessments in reading, mathematics, and science at grades 4, 8, and 12;

State-by-state and Trial Urban District Assessment (TUDA) assessments in reading and
mathematics for public schools at grades 4 and 8;

A state-by-state assessment in science for public schools at grades 4 and 8;

No state-by-state assessments in any states at grade 12;

\\westat.com\dfs\NAEPLIB\2015\Memos\School Sampling\Public Grades 4 & 8-A\2015-m01v01a.docx

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E

February 26, 2014

Technology-based assessment (TBA) start-ups in reading, mathematics, and science at
grades 4, 8, and 12;

Pilot assessments of science interactive computer tasks (ICTs) and hybrid hands-on
tasks (HOTs) at grades 4, 8, and 12;

A special national study of Knowledge and Skills Appropriate (KaSA) items in
mathematics, at grades 4 and 8, including Puerto Rico.

Below is a summary list of the features of the 2015 sample design.
1.

The alpha samples for grades 4 and 8 public, the gamma sample for grade 12 public,
and the delta samples for private schools at grades 4, 8, and 12 will be used for the
operational assessments in reading, mathematics, science, and meaning vocabulary.
These samples will also be used for the national KaSA special studies at grades 4 and 8.

There will be two sets of relatively small PSU-based samples for the TBA start-up
assessments and for the pilot assessments of science ICTs/HOTs at grades 4, 8, and 12.
The public school samples for these assessments are known as the Beta samples and the
corresponding private samples are known as the Epsilon samples.

As in recent NAEP studies, Trial Urban District Assessment (TUDA) samples will form
part of the corresponding state samples, and the state samples will form part of the
national sample. There are twenty-one Trial Urban District Assessment (TUDA)
participants. Twenty of the twenty-one participated also participated in 2013. Milwaukee
WI is no longer participating, but Duvall County FL has been added in.

All states but Alaska, California, Colorado, Pennsylvania, and Vermont have signed on
to do science at grades 4 and 8. BIE schools will not be doing science either.

Most schools in states doing science will receive a 1:1:1 RMS spiral (1/3 reading, 1/3
math, and 1/3 science). Most schools in states not doing science will receive a 10:10:1
RMS spiral (10/21 reading, 10/21 math, and 1/21 science) to ensure that these states
are sufficiently represented in the national science sample.

All TUDAs but Albuquerque, NM, Chicago, IL, and New York City, NY – those that
constitute a relatively large proportion of their state population – will receive the 10:10:1
RMS spiral. Albuquerque, Chicago, and New York will receive the 1:1:1 RMS spiral
(note that they are in states that have signed on for science).

A small subset of schools from each jurisdiction, regardless of whether they are doing
science, will receive a spiral that includes meaning vocabulary and KaSA booklets at a
rate that will support national level reporting. This spiral, the NL spiral, will consist of
6/21 reading, 6/21 math, 6/21 science, 2/21 meaning vocabulary, and 1/21 KaSA
booklets. The subsample of schools selected for the NL spiral in each state is
proportional to the representation of each state in the nation.

Page 2 of 23

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E

February 26, 2014

There will be no samples in territories, other than for Puerto Rico at grades 4 and 8 for
the KaSA study.

Most BIE schools will receive a 10:10:1 RMS spiral and a small subset will receive the
NL spiral.

10.

All BIE schools and students will be included in the operational samples at grades 4 and
8. This is because, after a hiatus in 2013, the National Indian Education Study (NIES) is
resuming. Having all BIE students in sample is designed to provide detailed national
results for American Indian and Alaskan Native (AIAN) students in reading and
mathematics, as part of the National Indian Education Study (NIES). Because science
will be spiraled with math and reading, the science sample will be spread across almost
all BIE schools at grades 4 and 8.

11.

As in 2013, the Department of Defense Schools are expected to be reported as a single
jurisdiction (DoDEA), instead of the two components of domestic (DDESS) and
overseas (DoDDS). DoDEA schools will receive either the 1:1:1 RMS spiral or the NL
spiral.

12.

At grade 12, there will be no state-level samples.

13.

All private schools will receive a NL spiral.

14.

Oversampling of private schools at grades 4 and 8 will be done at the same level as
2013. Response rates permitting, this will allow separate reporting for reading,
mathematics, and science for Catholic and non-Catholic schools, but no further. As in
2013 there will be no oversampling of private schools at grade 12.

15.

The sample sizes of assessed students for these various components are shown in Table
1 (which also shows the approximate numbers of participating schools).

Page 3 of 23

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E
Table 1.

February 26, 2014

Target sample sizes of assessed students, and expected number of participating
schools, for 2015 NAEP
Spiral

Jurisdictions
States (incl.
DC, BIE,
Urban
Spiral Indic.
DoDEA)
districts
Grade 4
nat'l/state reading
nat'l/state math
nat'l/state science
reading - vocabulary
math KaSA
Puerto Rico KaSA
Total - alpha
Total- delta
Typical max. no. students/school
Average assessed students/school
Total schools - alpha, delta
Reading - TBA Start-up
Mathematics - TBA Start-up
Science - TBA Start-up
Science ICT Pilot
Science HOT Pilot
Total - beta
Total - epsilon
Typical max. no. students/school
Average assessed students/school
Total schools - beta, epsilon
Total number of students grade 4
Total number of schools grade 4

RS, RM, NL
RS, RM, NL
RS, RM, NL
NL
NL
PR
4
1

53
53
47

21
21

Students
Private
Public school
school
students
students
133,000
133,000
105,000
7,200
3,600
5,000
386,800
90
51
7,550
10,800
10,800
7,200
3,600
2,700
35,100

TB
TB
TB
TB
TB
1
1

50
41
855
421,900
8,405

Page 4 of 23

Total

3,000 136,000
3,000 136,000
3,000 108,000
800
8,000
400
4,000
0
5,000
386,800
10,200 10,200
105
25
50
410
7,960
1,200 12,000
1,200 12,000
800
8,000
400
4,000
300
3,000
35,100
3,900
3,900
50
25
39
155
1,010
14,100 436,000
565
8,970

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E
Table 1.

February 26, 2014

Target sample sizes of assessed students, and expected number of participating
schools, for 2015 NAEP (Continued)
Spiral

Jurisdictions
States (incl.
DC, BIE,
Urban
Spiral Indic.
DoDEA)
districts
Grade 8
nat'l/state reading
nat'l/state math
nat'l/state science
reading - vocabulary
math KaSA
Puerto Rico KaSA
Total - alpha
Total- delta
Typical max. no. students/school
Average assessed students/school
Total schools - alpha, delta
Reading - TBA Start-up
Mathematics - TBA Start-up
Science - TBA Start-up
Science ICT Pilot
Science HOT Pilot
Total – beta
Total – epsilon
Typical max. no. students/school
Average assessed students/school
Total schools - beta, epsilon
Total number of students grade 8
Total number of schools grade 8

RS, RM, NL
RS, RM, NL
RS, RM, NL
NL
NL
PR
4
1

53
53
47

21
21

Students
Private
Public school
school
students
students
133,000
133,000
105,000
7,200
3,600
5,000
386,800
90
64
6,020
15,300
10,800
7,200
3,600
2,700
39,600

TB
TB
TB
TB
TB
1
1

50
44
900
426,400
6,920

Page 5 of 23

Total

3,000 136,000
3,000 136,000
3,000 108,000
800
8,000
400
4,000
0
5,000
386,800
10,200 10,200
105
26
62
400
6,420
1,700 17,000
1,200 12,000
800
8,000
400
4,000
300
3,000
39,600
4,400
4,400
50
25
41
175
1,075
14,600 441,000
575
7,495

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E

Table 1.

Target sample sizes of assessed students, and expected number of participating
schools, for 2015 NAEP (Continued)
Spiral

Jurisdictions
States (incl.
DC, BIE,
Urban
Spiral Indic.
DoDEA)
districts
Grade 12
nat'l/state reading
nat'l/state math
nat'l/state science
reading - vocabulary
Total – gamma
Total- delta
Typical max. no. students/school
Average assessed students/school
Total schools - gamma, delta
Reading - TBA Start-up
Mathematics - TBA Start-up
Science - TBA Start-up
Science ICT Pilot
Science HOT Pilot
Total – beta
Total – epsilon
Typical max. no. students/school
Average assessed students/school
Total schools - beta, epsilon
Total number of students grade 12
Total number of schools grade 12

NL
NL
NL
NL
1
1

Students
Private
Public school
school
students
students
11,700
12,600
9,900
5,400
39,600
100
76
520
13,500
10,800
8,100
3,600
2,700
38,700

TB
TB
TB
TB
TB
1
1

50
42
920
78,300
1,440

GRAND TOTAL STUDENTS
GRAND TOTAL SCHOOLS

II.

February 26, 2014

926,600
17,510

1,300
1,400
1,100
600
4,400
100
80
55
1,500
1,200
900
400
300
4,300
50
43
100
8,700
155

Total
13,000
14,000
11,000
6,000
39,600
4,400
77
575
15,000
12,000
9,000
4,000
3,000
38,700
4,300
42
1,020
87,000
1,595

37,400 964,000
1,990 19,500

Assessment Types

The assessment spiral types are shown in Table 2. Five different spirals will be used at each of the
grades 4 and 8, and two at grade 12. Session IDs contain six digits, traditionally. The first two digits
identify the assessment “type” (subjects and type of spiral in a general way). Grade is contained in
the second pair of digits, and the session sequential number (within schools) in the last two digits.
For example, session RS0401 denotes the first grade 4 reading, mathematics, and science operational
assessment in a given school.

Page 6 of 23

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E

Table 2.
ID

February 26, 2014

NAEP 2015 assessment types and IDs
Type

Subjects

Grades

Schools

Comments
Most public schools in states
doing science, plus TUDAs in
some of those states
Most public schools in states not
doing science, plus TUDAs in
those states and most others
All private schools; all public
schools at Grade 12; some
public schools in each
jurisdiction (except Puerto Rico
and DoD overseas) at grades 4
and 8

Operational
P&P

Reading, math, science
(1:1:1)

4, 8

Public

Operational
P&P

Reading, math, science
(10:10:1)

4, 8

Public

Operational
P&P

Reading, math, science,
Meaning Vocab, KaSA
(6:6:6:2:1, grades 4 & 8;
13:14:11:6:0, grade 12)

4, 8, 12

Public,
private

KaSA Mathematics

4, 8

Public

Puerto Rico only

Reading, math, science,
science ICT, science HOT

4, 8, 12

Public,
private

Booklets will not be used in any
other jurisdiction.

PR
TB

III.

Operational
P&P
TBA start-ups
& pilots

Sample Types and Sizes

In similar fashion to past years (but somewhat different), we will identify five different types of
school samples: Alpha, Beta, Gamma, Delta, and Epsilon. These distinguish sets of schools that will
be conducting distinct portions of the assessment.

Alpha Samples at Grades 4 and 8

These are public school samples for grades 4 and 8. They will be used for the operational state-bystate assessments in reading, math, and science and contribute to the national samples for these
subjects as well. They will also be used for the national meaning vocabulary assessment and for the
national KaSA special studies. There will be alpha samples for each state, DC, DoDEA, BIE, and
Puerto Rico. The alpha samples will not be used for any TBA start-up assessments or pilot
assessments.
The details of the target student sample sizes for the alpha samples are as follows:
A.

At each grade, the target student sample size for a state depends on whether or not it signed
on for science. For each state that signed on for science, the target student sample size is

Page 7 of 23

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E

February 26, 2014

7,600: roughly 2,200 assessed each for math, reading, and science. In these states the RS
session type will be used. For the four states not signed on for science and BIE, the target
student sample sizes are 5,300: 2,200 (assessed) each for math and reading and 220 for science.
In these jurisdictions the RM session type will be used.
B.

A national-level subsample will be applied to the sample schools at each grade with a target
student sample size of 86,600. Schools that are selected for this subsample will be given the
NL spiral.

There will be samples for twenty-one TUDA districts. For two of the five largest (New York
City and Chicago) where we require a sizeable science sample so as to ensure an adequate state
sample, the student target samples sizes are three-quarters the size of a science state (5,670).
For the remaining three large TUDA districts (Los Angeles, Miami-Dade, and Houston)
where only a smaller science sample is required, the student target sample sizes are threequarters the size of a non-science state sample (4,000). For Albuquerque, a small TUDA
where a sizeable science sample is required, the student target sample size is half the size of a
science state (3,780), whereas the remaining small TUDA districts where only a smaller science
sample is required, the student target sample sizes are one-half the size of a non-science state
sample (2,650).

Note, that above, there is a conflict between sample size requirements at the state level, and
the TUDA district level. This will be resolved as in previous years: the districts will have the
target samples indicated in B, and reflected in Table 4. For the states that contain one or more
of these districts, the target sample size indicated in A (and shown in Table 4) will be used to
determine a school sampling rate for the state, which will be applied to the balance of the state
outside the TUDA district(s). Thus the target student sample sizes, shown in Table 4, for
states that contain a TUDA district, are only ‘design targets’, and are smaller than the final
total sample size for the state, but larger than the sample for the balance of the state, exclusive
of its TUDA districts. In the case of the District of Columbia, the state sample size
requirement is that all schools and students be included. This renders moot any requirements
for the DC TUDA sample, which by default consists of all schools operated by the DCPS
district (but excludes charter schools in DC, even though those are all included in the state
sample, as these are not operated by DCPS).

In Puerto Rico, the target sample size is 5,750 per grade (grades 4 and 8), with the goal of
assessing 5,000 students. Only KaSA mathematics will be assessed in Puerto Rico (PR session
type).

As in past state-by-state assessments, schools with fewer than 20 students in the grade in question
will be sampled at a moderately lower rate than other schools (at least half, and often higher,
depending upon the size of the school). This is in implicit recognition of the greater cost and burden
associated with surveying these schools.
As mentioned above, the NAEP 2015 design includes an oversample of high proportion American
Indian schools in certain states (as part of the NIES design). These schools will be sampled at higher
rates than the other schools. The NIES oversample will take place in Arizona, Minnesota, North

Page 8 of 23

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E

February 26, 2014

Carolina, Oregon, Utah, and Washington. Schools with relatively large percentages of American
Indian students will be separately stratified, as explained below, and oversampled by factors ranging
from 2 to 6 based on state and grade. Table 3 below shows the thresholds used to define the NIES
oversampling strata along with their corresponding oversampling factors.

Table 3.

State
Arizona
Utah
Minnesota
North Carolina
Oregon
Washington

Percent American Indian thresholds and oversampling factors for the NIES school
oversample by state and grade
Grade 4
Percent American
Oversampling
Indian thresholds
factor
50
3
5
5
10
5
15
6
10
6
10
6

Grade 8
Percent American
Oversampling
Indian thresholds
factor
50
2
5
5
10
4
10
6
15
6
15
6

Table 4 shows the target student sample sizes, and the approximate counts of schools to be selected
in the alpha samples, along with the school and student frame counts, by state and TUDA districts
for grades 4 and 8. The table also identifies the jurisdictions where we take all schools and where we
take all students.

Page 9 of 23

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E
Table 4.

February 26, 2014

Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for
the 2013 state-by-state and TUDA district assessments (Alpha samples)
Grade 4

Jurisdiction
Alabama
Alaska
Arizona
Arkansas
Bureau of Indian Education
California
Colorado
Connecticut
Delaware
District of Columbia
DoDEA Schools
Florida
Georgia
Hawaii
Idaho
Illinois
Indiana
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri

Schools in
frame
719
361
1,184
488
136
5,907
1,041
575
118
122
108
2,183
1,248
203
373
2,310
1,059
647
704
734
754
326
893
955
1,731
934
422
1,160

Schools in
sample
110
158
111
113
136
103
97
114
94
122
108
96
92
112
127
121
110
138
145
113
115
169
110
111
119
117
106
130

Grade 8

Students
in frame
57,293
9,870
83,596
36,667
3,330
470,328
65,699
40,276
10,099
5,113
7,514
204,096
130,599
14,902
22,095
152,571
78,537
36,039
36,223
51,728
54,823
13,609
64,759
70,874
113,008
62,923
38,029
68,565

Overall
target
student
sample
size
7,600
5,300
7,600
7,600
3,330
7,600
5,300
7,600
7,600
5,113
7,514
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600

Page 10 of 23

**
**

Schools in
frame
462
276
784
300
112
2,876
554
315
61
73
63
1,173
552
83
212
1,610
496
374
395
407
479
199
379
477
1,071
702
286
691

Schools in
sample
96
118
104
99
112
96
93
96
61
73
63
94
87
62
101
110
90
108
115
99
102
116
93
90
102
114
95
110

Students in
frame
58,020
9,865
82,121
35,481
2,904
454,580
62,665
41,529
9,885
4,539
5,596
201,815
129,337
13,132
21,610
153,611
80,280
35,713
35,689
50,978
52,027
13,687
60,562
71,996
116,734
62,522
37,697
68,104

Overall
target
student
sample size
7,600
5,300
7,600
7,600
2,904
7,600
5,300
7,600
7,600
4,539
5,596
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600

*
**
**

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E
Table 4.

February 26, 2014

Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for
the 2013 state-by-state and TUDA district assessments (Alpha samples) (Continued)
Grade 4

Jurisdiction
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Puerto Rico
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virginia
Washington
West Virginia
Wisconsin
Wyoming

Schools in
frame
392
536
385
268
1,360
440
2,397
1,422
258
1,778
872
750
1,649
988
166
633
314
989
4,373
613
218
1,115
1,231
423
1,090
185

Schools in
sample
214
167
97
141
114
130
104
103
258
115
136
131
91
171
121
101
197
109
99
102
218
100
116
157
136
185

Grade 8

Students
in frame
11,146
22,939
33,934
13,975
99,003
25,977
199,215
116,094
7,760
129,775
50,101
42,558
129,728
33,865
10,751
55,665
10,084
76,003
386,022
48,529
6,247
95,536
79,005
20,728
61,306
7,185

Overall
target
student
sample
size
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,760
7,600
7,600
7,600
5,300
5,750
7,600
7,600
7,600
7,600
7,600
7,600
6,247
7,600
7,600
7,600
7,600
7,185

Page 11 of 23

Schools in
frame
273
303
161
141
750
225
1,443
703
183
1,118
584
422
900
401
59
307
245
581
2,224
244
122
384
610
196
642
87

Schools in
sample
157
126
84
90
97
102
94
91
183
100
120
108
87
125
59
89
160
97
95
91
122
86
99
98
109
87

Students in
frame
10,789
21,793
33,313
14,348
98,145
25,261
198,874
116,743
7,403
132,975
47,828
42,917
133,900
32,888
10,248
55,154
9,476
73,679
372,168
45,931
6,138
94,826
78,960
20,744
61,502
6,902

Overall
target
student
sample size
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,600
7,403
7,600
7,600
7,600
5,300
5,750
7,600
7,600
7,600
7,600
7,600
7,600
6,138
7,600
7,600
7,600
7,600
6,902

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E
Table 4.

February 26, 2014

Jurisdiction
Albuquerque
Atlanta
Austin
Baltimore City
Boston
Charlotte
Chicago
Cleveland
Dallas
Detroit
Duval County, FL
Fresno
Hillsborough County, FL
Houston
Jefferson County, KY
Los Angeles
Miami
New York City
Philadelphia
San Diego
District of Columbia PS
Total

Schools in
frame
95
55
79
127
72
103
501
71
147
66
115
71
171
176
100
556
271
768
162
145
80
52,240

Schools in
sample
52
42
44
56
52
43
102
71
43
48
44
44
44
64
44
66
67
74
48
47
80
7,831

Grade 8

Students
in frame
7,426
4,238
6,988
6,385
4,165
11,447
29,331
2,970
13,031
4,213
9,999
5,854
15,515
17,149
7,589
50,563
26,278
72,429
11,438
10,378
3,291
3,776,296

Overall
target
student
sample
size
3,780
2,650
2,650
2,650
2,650
2,650
5,670
2,970
2,650
2,650
2,650
2,650
2,650
4,000
2,650
4,000
4,000
5,670
2,650
2,650
3,291
443,343

Schools in
frame
40
24
24
102
41
45
482
66
42
50
48
23
85
64
46
191
162
527
126
62
32
28,770

Counts for states do not reflect the oversampling for their constituent TUDA districts.
Target student sample sizes reflect sample sizes prior to attrition due to exclusion, ineligibility, and nonresponse.
* identifies jurisdictions where all schools (but not all students) for the given grade are included in the NAEP sample.
** identifies jurisdictions where all students for the given grade are included in the NAEP sample.

Page 12 of 23

Schools in
sample
32
24
24
53
41
32
100
66
42
50
31
23
43
46
29
64
65
71
46
36
32
6,215

Students in
frame
6,721
3,509
5,542
5,654
3,867
10,717
28,794
2,755
10,799
3,224
9,098
5,086
15,159
13,400
7,345
45,215
27,130
68,327
9,573
9,231
2,289
3,725,584

Overall
target
student
sample size
3,780
3,509
2,650
2,650
2,650
2,650
5,670
2,755
2,650
3,224
2,650
2,650
2,650
4,000
2,650
4,000
4,000
5,670
2,650
2,650
2,289
441,811

**
*
*

**
*
**
*

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E

February 26, 2014

Table 5 consolidates the target student (and resulting school) sample size numbers, to show the total
target sample sizes in each state, combining the TUDA targets with those for the balance of the
state.

Page 13 of 23

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E
Table 5.

February 26, 2014

Total sample sizes, combining state and TUDA samples
Grade 4

Jurisdiction
Alabama
Alaska
Arizona
Arkansas
Bureau Of Indian Education
California
Colorado
Connecticut
Delaware
District Of Columbia
DoDEA Schools
Florida
Georgia
Hawaii
Idaho
Illinois
Indiana
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Montana

Schools
in frame
719
361
1,184
488
136
5,907
1,041
575
118
122
108
2,183
1,248
203
373
2,310
1,059
647
704
734
754
326
893
955
1,731
934
422
1,160
392

Schools
in sample
109
158
111
113
136
277
97
114
94
122
108
227
130
112
127
197
110
138
145
142
115
169
150
155
162
117
106
130
213

Grade 8
Overall
target
student
sample
size
7,600
5,300
7,600
7,600
3,330
15,818
5,300
7,600
7,600
5,113
7,514
14,969
10,004
7,600
7,600
11,805
7,600
7,600
7,600
9,137
7,600
7,600
9,501
9,803
9,966
7,600
7,600
7,600
7,600

Page 14 of 23

**
**

Schools in
frame
462
276
784
300
112
2,876
554
315
61
73
63
1,173
552
83
212
1,610
496
374
395
407
479
199
379
477
1,071
702
286
691
273

Schools in
sample
96
118
104
99
112
229
93
96
61
73
63
209
108
62
101
185
90
108
115
113
102
116
133
125
149
114
95
110
157

Overall
target
student
sample size
7,600
5,300
7,600
7,600
2,904
15,902
5,300
7,600
7,600
4,539
5,596
14,962
10,903
7,600
7,600
11,842
7,600
7,600
7,600
9,157
7,600
7,600
9,541
9,842
10,614
7,600
7,600
7,600
7,600

*
**
**

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E
Table 5.

February 26, 2014

Total sample sizes, combining state and TUDA samples (Continued)
Grade 4

Jurisdiction
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Puerto Rico
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virginia
Washington
West Virginia
Wisconsin
Wyoming
Total

Schools
in frame
536
385
268
1,360
440
2,397
1,422
258
1,778
872
750
1,649
988
166
633
314
989
4,373
613
218
1,115
1,231
423
1,090
185
52,240

Schools
in sample
167
97
141
114
151
142
137
258
182
136
131
131
171
121
101
197
109
250
102
218
100
116
157
136
185
7,831

Grade 8
Overall
target
student
sample
size
7,600
7,600
7,600
7,600
9,188
10,507
9,500
7,760
10,396
7,600
7,600
7,482
5,750
7,600
7,600
7,600
7,600
16,167
7,600
6,247
7,600
7,600
7,600
7,600
7,185
443,343

Students
in frame
22,939
33,934
13,975
99,003
25,977
199,215
116,094
7,760
129,775
50,101
42,558
129,728
33,865
10,751
55,665
10,084
76,003
386,022
48,529
6,247
95,536
79,005
20,728
61,306
7,185
3,776,296

Schools in
frame
303
161
141
750
225
1,443
703
183
1,118
584
422
900
401
59
307
245
581
2,224
244
122
384
610
196
642
87
28,770

Sample sizes for each state do reflect the samples in the TUDA districts within the state.
* identifies jurisdictions where all schools (but not all students) for the given grade are included in the NAEP sample.
** identifies jurisdictions where all students for the given grade are included in the NAEP sample.

Page 15 of 23

Schools in
sample
126
84
90
97
111
132
116
183
162
120
108
126
125
59
89
160
96
202
91
122
86
99
98
109
87
6,215

Students in
frame
21,793
33,313
14,348
98,145
25,261
198,874
116,743
7,403
132,975
47,828
42,917
133,900
32,888
10,248
55,154
9,476
73,679
372,168
45,931
6,138
94,826
78,960
20,744
61,502
6,902
3,725,584

Overall
target
student
sample size
7,600
7,600
7,600
7,600
9,347
10,659
9,551
7,403
10,198
7,600
7,600
7,571
5,750
7,600
7,600
7,600
7,600
16,291
7,600
6,138
7,600
7,600
7,600
7,600
6,902
441,811

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E

February 26, 2014

Stratification
Each state and grade will be stratified separately, but using a common approach in all cases. TUDA
districts will be separated from their state, and each part stratified separately. The first level of
stratification will be based on urban-centered type of location. This variable has 12 levels (some of
which may not be present in a given state or TUDA district), and these will be collapsed so that each
of the resulting location categories contains at least nine percent of the student population. Within
each of the resulting location categories, schools will be assigned a minority enrollment status. This
is based on the two race/ethnic groups that are the second and third most prevalent within the
location category. If these groups are both low in percentage terms, no minority classification will be
used. Otherwise three (or occasionally four) equal-sized groups (generally high, medium, and low
minority) will be formed based on the distribution across schools of the two minority groups.
Within the resulting location and minority group classes (of which there are likely to be from three
to fifteen, depending upon the jurisdiction), schools will be sorted by a measure derived from school
level results from the most recent available state achievement tests at the relevant grade. In general,
mathematics test results will be used, but where these are not available, reading results will be used.
In the few states that do not have math or reading tests at grades 4 and 8 (or where we are unable to
match the results to the NAEP school frame), instead of achievement data, schools will be sorted
using a measure of socio-economic status. This is the median household income of the 5-digit ZIP
Code area where the school is located, based on the 2012 ACS (5-year) data.
Once the schools are sorted by location class, minority enrollment class, and achievement data (or
household income), a systematic sample of schools will be selected using a random start. Schools
will be sampled with probability proportional to size. The exact details of this process are described
in the individual sampling specification memos.

Beta Sample

The beta sample comprises the national public school samples at grades 4, 8, and 12. This sample
will be used to conduct the TBA start-ups in reading, mathematics, and science and the pilot
assessments in science interactive computer tasks (ICTs) and hybrid hands-on tasks (HOTs) at all
three grades. Each of these samples will be nationally representative, with the exception that schools
that are included in the grades 4 and 8 alpha samples with certainty will not be represented. This is
because there is to be no overlap with the grade 4 and 8 alpha school samples.
Page 16 of 23

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E

February 26, 2014

The first stage of sampling for the beta sample is the selection of a sample of geographic primary
sampling units (PSUs). These will be selected using the same design as for the 2014 assessments, but
minimizing the overlap with PSUs in that assessment. A total 67 PSUs will be selected, representing
the U.S. (but not including Puerto Rico, or DoDEA schools that are located outside the 50 states
and D.C.). This PSU component is needed because of the operational complexities of administering
the computer-based assessment. A select group of staff will be trained to administer those
assessments. The school stratification of the beta sample within PSUs will be by type of location and
median household income. Schools with more than 15 percent black or Hispanic students will be
sampled at twice the rate of other schools, so as to increase the student sample sizes for these two
groups.
The number of students targeted per school will be 50 (either 25 start-up and 25 ICT/HOT or 50
start-up).

Gamma Sample

This is the public school sample at grade 12, used for the reading, mathematics, science, and
meaning vocabulary assessments. It will consist of a single sample of schools covering only nationallevel estimates. The number of students targeted per school will be 95 (28 reading, 30 for math, 24
science, and 13 meaning vocabulary).
As in past assessments, modest oversampling of Black and Hispanic students will be undertaken in
this sample. This will be carried out at the school level. Each school with more than 15 percent
Black and Hispanic students will be given twice the selection probability of other schools of
comparable size. This means that while about 53 percent of the student population (including over
90 percent of the Black and Hispanic students) are in the oversampled schools, about 70 percent of
the sample students will come from these schools.

Stratification
The Gamma sample will have an implicit stratification, using a hierarchy of stratifiers and a
serpentine sort. The highest level of the hierarchy is Census division (9 implicit strata). The next
stratifier in the hierarchy is type of location, which has twelve categories. Many of the type of
Page 17 of 23

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E

February 26, 2014

location strata nested within Census divisions will be collapsed with neighboring type of location
cells (this will occur if the expected school sample size within the cell is less than 4.0). These
geographic strata will be subdivided using a dichotomous high minority status category for
oversampling purposes. Schools with more than 10 Black or Hispanic students and greater than 15
percent Black or Hispanic students will be considered high minority and placed in an oversampling
stratum. All other schools will be considered low minority and placed in a regular sampling stratum.
If the expected sample size for a high or low minority stratum is less than 8.0, it will be left as is. If
the expected sample size is greater than 8.0, then the stratum will be subdivided into up to four
substrata (two for expected sample size up to 12.0, three for expected sample size up to 16.0, and
four for expected sample size greater than 16.0). For the regular sampling strata, the subdivision will
be by state or groups of contiguous states. For the oversampling strata, the subdivision will be by
percentage Black and Hispanic. Within these substrata, the schools are to be sorted by school type
(public, BIE, DoDEA) and median household income from the 2012 5-year ACS (using a serpentine
sort within the school type substrata).

Delta Samples

These are the private school samples at grades 4, 8, and 12 for conducting the operational
assessments in reading, mathematics, science, and meaning vocabulary, as well as the KaSA special
studies at grades 4 and 8 that are incorporated into the NL spiral. The number of students targeted
per schools will be 105 (30 reading, 30 math, 30 science, 10 meaning vocabulary, and 5 KaSA) at
grades 4 and 8, and 95 (28 reading, 30 for math, 24 science, and 13 meaning vocabulary) at grade 12.
The sample sizes are large enough to report results by Catholic and non-Catholic at grades 4 and 8,
but at grade 12 only for private schools as a whole. Approximately half the sample at each grade will
be from Catholic schools. This is very similar to the design used in 2013, at each grade.
Stratification
The private schools are to be explicitly stratified by private school type (Catholic/Other). Within
each private school type, stratification will be by Census region (4 categories), type of location (12
categories), and enrollment size. In general, where there are few or no schools in a given stratum,
categories will be collapsed together, always preserving the private school type.

Page 18 of 23

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E

February 26, 2014

Epsilon Sample

This sample is analogous to the beta sample, but for private schools, at grades 4, 8, and 12. The
same PSUs will be used as for the beta sample. There will be no oversampling of private schools for
this sample. The epsilon sample schools will not overlap with the delta sample schools.

IV.

New Schools

To compensate for the fact that files used to create the NAEP school sampling frames are at least
two years out of date at the time of frame construction, we will supplement the Alpha, Gamma, and
Delta sample with new school samples at each grade.
The new school samples will be drawn using a two-stage design. At the first stage, a minimum of ten
school districts (in states with at least ten districts) will be selected from each state for public
schools, and ten Catholic dioceses will be selected nationally for the private schools. The sampled
districts and dioceses will be asked to review lists of their respective schools and identify new
schools. Frames of new schools will be constructed from these updates, and new schools will be
drawn with probability proportional to size using the same sample rates as their corresponding
original school samples.
The school sample sizes in the above tables do not reflect new school samples.

Substitute Samples

Substitute samples will be selected for each of the Beta, Gamma, Delta, and Epsilon samples. The
substitute school for each original will be the next “available” school on the sorted sampling frame,
with the following exceptions:
A.

Schools selected for any NAEP samples will not be used as substitutes.

Private schools whose school affiliation is unknown will not be used as substitutes. Also,
unknown affiliated private schools in the original samples will not get substitutes.

A school can be a substitute for one and only one sample. (If a school is selected as a
substitute school for grade 12, for example, it cannot be used as a substitute for either grade 4
or grade 8.)
Page 19 of 23

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E
D.

VI.

February 26, 2014

A public school substitute will always be in the same state as its original school.

Contingency Samples

The districts that are taking part in the TUDA program are volunteers. Thus it is possible that at
some point over the next few months, a given district might choose to opt out of the TUDA
program for 2015. However, it is not acceptable for all schools in such a district to decline NAEP,
as then the state estimates will be adversely affected. Thus to deal with this possibility, in each
TUDA district, subsamples of the alpha sample schools will be identified as contingency samples. In
the event that the district withdraws from the TUDA program prior the selection of the student
sample, all alpha sampled schools from that district will be dropped from the sample, with the
exception of those selected in the contingency sample. The contingency sample will provide a
proportional representation of the district, within the aggregate state sample. Student sampling in
those schools will then proceed in the same way as for the other schools within the same state.

Page 20 of 23

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E

VII.

February 26, 2014

Student Sampling

Students within the sampled schools will be selected with equal probability. The student sampling
parameters vary by sample type (Alpha, Beta, Gamma, Delta, and Epsilon) and grade, as described
below.
Alpha Sample, Grades 4 and 8 Schools (Except Puerto Rico)
A.

The sample size for each school will depend upon which spiral the school was assigned, either
RS, RM or NL.

In schools assigned the RS spiral, all students will be selected, up to 100. If the school has
more than 100 students, 90 will be selected. In some schools, the school may be assigned
more than one ‘hit’ in sampling. In these schools we will select a sample of size 90 times the
number of hits, taking all students if this target is greater than or equal to 90/100 of the total
enrollment.

In schools assigned the RM spiral, all students, up to 70, will be selected. If the school has
more than 70 students, 63 will be selected. In some schools, the school may be assigned more
than one ‘hit’ in sampling. In these schools we will select a sample of size 63 times the number
of hits, taking all students if this target is greater than or equal to 63/70 of the total
enrollment.

In schools assigned the NL spiral, all students will be selected, up to 100. If the school has
more than 100 students, 90 will be selected.

The student sampling procedures for DC at grades 4 and 8 have yet to be determined.

Alpha Sample, Puerto Rico Grades 4 and 8
A.

All students, up to 55, will be selected.

If the school has more than 55 students, a systematic sample of 50 students will be selected
with no oversampling.

All students will be assigned to assessment type PR.

Page 21 of 23

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E

February 26, 2014

Delta Samples, Grades 4 and 8
A.

All students, up to 115, will be selected.

If the school has more than 115 students, a systematic sample of 105 students will be selected.

All students will be assigned to assessment type NL.

Beta and Epsilon Samples, Grade 8
A.

In each school, a sample for computer-based testing will be selected as follows: All students
up to 50 will be selected. If there are more than 50 students enrolled, a sample of 50 students
will be selected.

All students will be assigned to the TB session.

Gamma and Delta Samples, Grade 12
A.

All students, up to 105, will be selected.

If the school has more than 105 students, a systematic sample of 95 students will be selected.

All students will be assigned to assessment type NL.

VIII.

Weighting Requirements

Weighting activities for the 2015 NAEP assessments are not covered under the current NAEP
contracts. However, based on past experience, below is a description of the likely weighting
requirements.
The Operational Samples
These samples will have a single set of weights for each subject (reading, math, science, and meaning
vocabulary at grades 4, 8, and 12) applied to reflect probabilities of selection, school and student
nonresponse, any trimming, and the random assignment to the particular subject. There will be a
separate replication schemes by grade and public/private. Such weights will also be derived for the
Puerto Rico KaSA assessment and the national KaSA special studies at grades 4 and 8.

Page 22 of 23

Memorandum: 2015 - 1.1A/1.1B/1.1G/1.1D/1.1E

February 26, 2014

The Pilot Test Samples
Fully adjusted weights will not be provided for the students in the TBA start-up and science ICT
and HOT pilot test studies. However, preliminary weights will be available for these samples. These
will reflect the school and student selection probabilities, but with no adjustments for nonresponse.

The NIES Samples
The NIES survey samples consist of two grade-specific samples, comprising students selected for
each of the grade 4 and 8 operational samples. We will create one set of weights for each gradespecific sample. The NIES weights are designed for any aggregation of the NIES data, not involving
NAEP achievement data. NIES analyses involving NAEP achievement data should use the
appropriate NAEP operational weights.

Page 23 of 23

File Type	application/pdf
File Modified	2014-03-05
File Created	2014-02-26