Example of Sample Design

APPENDIX C -- Example of Sample Design Document (2007 Assessment)

TO:	John Burke William Wall Dwight Brock Sharon Hirabayashi Susan Fuss Nancy Caldwell Jennifer Kali Dianne Walsh Lou Rizzo Dward Moore Leslie Wallace Lauren Byrne Debby Vivari Kavemuii Murangi Rob Dymowski Chris Averett	DATE: June 6, 2006
FROM:	Lloyd Hicks Reviewer: Keith Rust	MEMO: 2007-1.2A/1.2G/1.2D
SUBJECT:	Sample Design for 2007 NAEP
CHANGES:	This version of the memorandum reflects the sample design in place at the time of school sampling. It reflects the reduction of the reading pilot test sample at grade 12; it no longer references the cancelled Participation and Engagement study at grade 12; and the frame data contained in the various tables reflect 2007 data instead of 2005.

I. Introduction

For 2007, the sample design involves several components:

• National assessments in reading and math at grades 4 and 8 and in writing at grades 8 and 12;

• State-by-state and Urban District assessments in reading and math for public schools at grades 4 and 8 and in writing for public schools at grade 8;

• Pilot tests in reading and math at grades 4 and 8 and in reading at grade 12 (public schools only);

Below is a summary list of the features of the 2007 sample design.

1. As in recent NAEP studies, Urban District Assessment (TUDA) samples will form part of the corresponding state samples, and the state samples will form part of the national sample. The same ten participants in 2005 will be involved. They include Los Angeles, San Diego, Atlanta, Chicago, Boston, New York City, Charlotte, Cleveland, Austin, and Houston.

2. The biggest change from 2005 is that there will be only one assessment type per school for a given grade. All subject area assessments, including pilot tests, will be in a single spiral and administered together.

3. As in 2005, there will be a large state oversample for reading, math, and writing assessments at grades 4 and 8 but at lower rates. California will receive a double sample instead of a triple; Texas will receive a one and a half sample, down from a double sample; Florida will receive a one and a half sample, same as 2005; and New York will not be oversampled (down from a one and a half sample in 2005). The oversample is in recognition that these large states have diverse populations, and increased sample sizes will permit meaningful breakdowns of the results to be made at finer levels than have been possible in the past. Applying increases in these large states will also significantly improve the precision of national estimates, both overall and by demographic subgroups.

4. There is to be no special study of charter schools. However, since charter school status is a NAEP reporting variable, NCES wants the 2007 charter school estimates to be as precise as they were in 2005. To maintain this level of precision, charter schools in California, Texas, and New York, but not in TUDA districts, will be further oversampled at grades 4 and 8 to compensate for the lower expected number of charter schools in sample due to the reduction in the large state oversampling rates.

5. At grade 8, all jurisdictions but Alaska, Maryland, Nebraska, Oregon, South Dakota, and the District of Columbia have signed on for writing. All jurisdictions doing writing will receive a 1:1:1 MRW spiral, and the six that did not will receive a 9:9:1 MRW spiral to ensure that these states are sufficiently represented in the national writing sample. A 1:1:1 MRW spiral means writing is spiraled in at the same rate as reading and math. The 9:9:1 MRW spiral means that for every 9 math and 9 reading booklets there will be 1 writing booklet.

6. At grade 8, all BIA schools will receive a 9:9:1 MRW spiral, and all private schools will receive a 1:1:1 MRW spiral.

7. At grade 8, the 1:1:1 MRW spiral will include pilot test booklets, and the 9:9:1 MRW spiral will not.

Consequently, the grade 8 pilot test samples will not include any students in non-participating writing states or in BIA schools; however, they will include private school students.

8. All ten TUDA districts will do writing in grade 8 since all their associated states have signed on for writing.

9. At grade 12, public and private schools will have different spirals. The public school spiral will include reading pilot test booklets along with operational writing booklets, whereas the private school spiral will only include operational writing booklets.

10. There will be no samples in territories other than Puerto Rico in math at grades 4 and 8.

11. There will be larger samples of BIA schools in the grades 4 and 8 operational assessments than in 2005. All BIA schools and students will be included in the sample. This is designed to enhance the reporting of results for American Indian students, at the state level (across school types), in those states with a sizable proportion of the nation’s American Indian students. It is also designed to provide detailed national results for American Indian and Alaskan Native (AIAN) students in reading and math as part of the National Indian Education Study (NIES).

Because writing will be spiraled with math and reading, the writing sample will be spread across almost all BIA schools at grade 8. As a result, the American Indian estimates for 2007 writing will be better than those for 2005 science, which only had 2 BIA schools in each grade-specific sample. All AIAN students sampled for NAEP at grade 4 will be included in the NIES survey. At grade 8, all BIA students will be included in NIES, plus all other AIAN students who are assigned to be assessed in operational reading and math. Non-BIA students selected for writing or the pilot tests at grade 8 will not be included in the NIES study.

12. In a handful of states (Washington, Oregon, Arizona, North Carolina, Minnesota, and Michigan) the public school sample at grades 4 and 8 will be increased somewhat, so as to give publishable results for AIAN students. This will affect school sampling only. There will be no special student sampling procedures for this purpose. This will be achieved by increasing, by an appropriate factor, the measures of size of schools that are in specially designated strata with a relatively high proportion of AIAN students.

13. As in 2005, the Department of Defense Schools will be reported as a single jurisdiction (DoDEA), instead of the two components of domestic (DDESS) and overseas (DoDDS). However, for design purposes, we will still sample and weight these as two separate entities.

14. Private schools will be oversampled in sufficient numbers so as to be able to report Catholic and non-Catholic schools separately. There will be no special oversampling as implemented in 2002, 2003, and 2005 for grades 4 and 8.

The sample sizes of assessed students for these various components are shown in Table 1 (which also shows the approximate numbers of participating schools). Note that the sample size for 8^th grade public schools for writing reflects the appropriate writing sample from each of the six “nonparticipating” states and BIA schools in order to ensure a nationally representative sample.

Table 1. Sample sizes of assessed students in participating schools for 2007 NAEP

	States, DC, PR, and including 2 states worth of oversampling (from 3 states)	Smaller jurisdictions (BIA and DoDEA)	Urban districts	Public school students	Private school students	Total students
Grade 4
Nat'l/state math	54	2	10	171,000	3,000	174,000
Nat'l/state read	53	2	10	168,200	3,000	171,200
Read pilot test	--	--	--	9,000	139	9,139
Math pilot test	--	--	--	1,500	23	1,523
Total				349,700	6,162	355,862
Schools				7,400	280	7,680
Grade 8
Nat'l/state math	54	2	10	167,300	3,000	170,300
Nat'l state read	53	2	10	164,500	3,000	167,500
Nat'l/state writing	47	1	10	149,000	3,000	152,000
Read pilot test	--	--	--	12,500	242	12,742
Math pilot test	--	--	--	1,500	29	1,529
Total				494,800	9,271	504,071
Schools				7,050	375	7,425

Table 1. Sample sizes of assessed students in participating schools for 2007 NAEP (continued)

	States, DC, PR, and including 2 states worth of oversampling (from 3 states)	Smaller jurisdictions (BIA and DoDEA)	Urban districts	Public school students	Private school students	Total students
Grade 12
Writing	--	--	--	16,000	4,000	20,000
Reading Pilot	--	--	--	2,000	0	2,000
Schools				490	135	625
Grand Total				876,900	21,533	898,433
Schools				14,940	790	15,730

II. Assessment Types

In 2007, schools will only be assigned to one assessment type at a given grade. Assessment types will vary by grade, and additionally at grade 8 public schools by whether or not a state signed on for the writing assessment and at grade 12 by whether the school is public or private. These different assessment types are summarized in Table 2. Session IDs contain 6 digits, traditionally. The first two digits identify the assessment “type” (subjects and type of spiral in a general way). Grade is contained in the second pair of digits, and the session sequential number (within schools) in the last 2 digits.

Table 2. NAEP 2007 assessment types and IDs

ID	Type	Subjects	Grades	Schools	Comments
RM	Operational/ Pilot	Reading, math, pilot test (reading and math)	4	Public, private, and BIA	Only assessment type at grade 4, except for Puerto Rico.
RS	Operational/ Pilot	Reading, math, writing, pilot test (reading and math)	8	Public & private	1:1:1 reading, math, writing spiral. For states signed on for writing, and private schools.
RN	Operational	Reading, math, writing	8	Public & BIA	9:9:1 reading, math, writing spiral: For states not signed on for writing and BIA schools.
RW	Operational/Pilot	Writing, pilot test (reading)	12	Public	National sample
WT	Operational	Writing	12	Private	National sample
PR	Puerto Rico	Math	4 & 8	Public	Booklets will not be used in any other jurisdiction.

III. Sample Types and Sizes

In similar fashion to 2003, we will identify three different types of school samples: Alpha, Gamma, and Delta. These distinguish sets of schools having different sample size and precision requirements.

1. Alpha Samples at Grades 4 and 8

These are public school samples for grades 4 and 8. They will be used for the operational state-by-state assessments in reading, math, and writing (grade 8 only) and contribute to the national samples for these subjects as well. There will be alpha samples for each state, DC, DoDEA, BIA, and Puerto Rico.

Table 3 shows the initial target student sample sizes, along with student frame counts, for each reporting jurisdiction and TUDA in the alpha samples at grades 4 and 8. These target student sample sizes were used to derive the initial school sampling rates for the jurisdictions and TUDAs. The details of the target student sample sizes for the alpha samples are as follows:

1. At grade 4, the target is to select a school sample for each state (other than California, Florida, and Texas) that will yield 6,500 sampled students – 3,150 each for math and reading and 200 for the pilot tests – before any allowance for exclusion, ineligibility, or nonresponse. In California the target will be two times as large (13,000), while in Florida and Texas it will be 50 percent larger (9,750).

2. At grade 8, the target student sample size for a state depends on whether or not it signed on for writing as well as whether it is oversampled or not. For each non-oversampled state that signed on for writing, the target student sample size is 9,750 – 3,150 each for math, reading, and writing and 300 for the pilot tests. For the oversampled states participating in writing, the target sample size is 19,500 for California and 14,625 for Florida and Texas. For the six states not signed on for writing, the target student sample sizes are 6,650 - 3,150 each for math and reading and 350 for writing.

3. In DC, DoDEA, BIA, and several small states, all students will be included (those states in which there are fewer students than are required in the above sample sizes, or just a few more than that).

4. There will be samples for ten TUDA districts, the same ten that participated in 2005. For the four largest (New York City, Los Angeles, Chicago, and Houston), the student target sample sizes are three-quarters the size of a non-oversampled state (4,875 at grade 4 and 7,313 at grade 8). For the other six districts (Atlanta, Austin, Boston, Charlotte, Cleveland, and San Diego), the student target sample sizes are one-half the size of a non-oversampled state (3,250 at grade 4 and 4,875 at grade 8).

5. Note that above there is a conflict between sample size requirements at the state level, and the TUDA district level. This will be resolved as in 2002, 2003, and 2005: The districts will have the target samples indicated in D. For the states that contain one or more of these districts, the target sample size indicated in A will be used to determine a school sampling rate for the state, which will be applied to the balance of the state outside the TUDA district(s). The target student sample sizes used in this approach are shown in Table 3.

6. In Puerto Rico, the target sample size is 3,150 per grade, as in 2003 and 2005.

Table 3. Grades 4 and 8 student frame counts and initial target student sample sizes for the NAEP 2007 state-by-state and TUDA district assessments (alpha samples)

	Grade 4			Grade 8
Jurisdiction	Students in the state/district	Target student sample size		Students in the state/district	Target student sample size
Alabama	57,310	6,500		58,712	9,750
Alaska	9,342	6,500		10,048	6,650
Arizona	78,289	6,500		77,213	9,750
Arkansas	34,947	6,500		37,069	9,750
Bureau of Indian Affairs	3,828	3,828		3,539	3,539
California	484,671	13,000		477,620	19,500
Colorado	56,963	6,500		58,964	9,750
Connecticut	43,500	6,500		43,617	9,750
Delaware	9,335	6,500		9,535	9,535
Department of Defense	8,696	8,696		6,620	6,620
District of Columbia	5,515	5,515		4,600	4,600
Florida	199,320	9,750		207,054	14,625
Georgia	118,326	6,500		122,087	9,750
Hawaii	14,332	6,500		13,654	9,750
Idaho	19,219	6,500		19,893	9,750
Illinois	157,501	6,500		160,902	9,750
Indiana	79,980	6,500		81,106	9,750
Iowa	34,311	6,500		37,200	9,750
Kansas	33,816	6,500		35,021	9,750
Kentucky	48,633	6,500		50,131	9,750
Louisiana	57,418	6,500		58,819	9,750
Maine	14,742	6,500		16,189	9,750
Maryland	63,546	6,500		67,388	6,650
Massachusetts	72,459	6,500		75,805	9,750
Michigan	125,706	6,500		134,965	9,750
Minnesota	59,729	6,500		64,506	9,750
Mississippi	38,841	6,500		39,619	9,750
Missouri	67,150	6,500		72,907	9,750
Montana	10,811	6,500		11,943	9,750
Nebraska	21,424	6,500		22,393	6,650
Nevada	30,858	6,500		30,761	9,750
New Hampshire	15,263	6,500		16,632	9,750
New Jersey	101,922	6,500		103,107	9,750
New Mexico	24,098	6,500		26,078	9,750
New York	203,679	6,500		211,477	9,750
North Carolina	106,777	6,500		108,558	9,750
North Dakota	7,323	6,500		8,022	8,022
Ohio	137,204	6,500		142,785	9,750
Oklahoma	46,020	6,500		47,283	9,750
Oregon	41,664	6,500		43,308	6,650
Pennsylvania	131,350	6,500		144,358	9,750

Table 3. Grades 4 and 8 student frame counts and initial target student sample sizes for the NAEP 2007 state-by-state and TUDA district assessments (alpha samples) (continued)

	Grade 4			Grade 8
Jurisdiction	Students in the state/district	Target student sample size		Students in the state/district	Target student sample size
Puerto Rico	45,801	3,150		42,017	3,150
Rhode Island	11,616	6,500		12,325	9,750
South Carolina	52,850	6,500		56,153	9,750
South Dakota	9,015	6,500		9,713	6,650
Tennessee	71,621	6,500		72,680	9,750
Texas	331,972	9,750		326,857	9,750
Utah	38,514	6,500		37,252	9,750
Vermont	6,947	6,500		7,114	7,114
Virginia	89,916	6,500		95,563	9,750
Washington	75,039	6,500		78,557	9,750
West Virginia	20,768	6,500		21,931	9,750
Wisconsin	59,763	6,500		64,558	9,750
Wyoming	6,029	6,029		6,829	6,829
Atlanta	4,270	3,250		3,515	3,515
Austin	6,171	3,250		5,241	5,241
Boston	4,143	3,250		4,729	4,729
Charlotte	9,426	3,250		9,332	4,875
Chicago	33,601	4,875		32,489	7,313
Cleveland	5,335	3,250		4,465	4,465
Houston	16,709	4,875		13,926	7,313
Los Angeles	59,934	4,875		52,661	7,313
New York City	71,567	4,875		68,641	7,313
San Diego	10,801	3,250		9,995	4,875

Notes: 1) State initial target sample sizes ignore the oversample for TUDA districts.

2) Target student sample sizes reflect sample sizes prior to attrition due to exclusion, ineligibility, or nonresponse.

As in past state-by-state assessments, schools with fewer than 20 students in the grade in question will be sampled at a moderately lower rate than other schools (at least half, and often higher, depending upon the size of the school). This is in implicit recognition of the greater cost and burden associated with surveying these schools.

As mentioned above, the NAEP 2007 design includes an oversample of charter schools and high proportion American Indian schools (as part of the NIES design). These schools will be sampled at higher rates, particularly those in the NIES oversample, than the other schools. The charter school oversample will occur in California, New York, and Texas to compensate for the reduction in the large-state oversample as explained in Items 3 and 4 in Section 1 above. Charter schools are oversampled by a factor of 1.5 in California and Texas and 1.33 in New York, but no oversampling takes place for charter schools that are part of TUDA districts.

The NIES oversample will take place in Arizona, Michigan, Minnesota, North Carolina, Oregon, and Washington. Schools with relatively large percentages of American Indian students will be separately stratified, as explained below, and oversampled by factors ranging from 2 to 6 based on state and grade. Table 4 below shows the thresholds used to define the NIES oversampling strata along with their corresponding oversampling factors. (See memorandum # 2007-7A for details of the analyses on which this approach was based.)

Table 4. Percent American Indian thresholds and oversampling factors for the NIES school oversample by state and grade

	Grade 4			Grade 8
State	Percent American Indian thresholds	Oversampling factor		Percent American Indian thresholds	Oversampling factor
Arizona	50%	3		50%	2
Michigan	10	6		10	6
Minnesota	10	5		10	4
North Carolina	15	6		10	6
Oregon	10	6		15	4
Washington	10	6		15	6

Table 5 shows the actual counts of schools selected in the alpha samples, along with the school frame counts, and final target student sample sizes by state and TUDA districts for grades 4 and 8. The school sample sizes reflect the undersampling of very small schools and oversampling of charter and American Indian schools where appropriate. The table also identifies the jurisdictions where we take all schools and where we take all students.

Table 5. Grade 4 and 8 school frame counts, school sample sizes, and final target student sample sizes for the NAEP 2007 state-by-state assessments (alpha samples) by state and TUDA districts

	Grade 4					Grade 8
Jurisdiction	Schools in frame	School Sample Size	Overall target student sample size			Schools in frame	School Sample Size	Overall target student sample size
Alabama	758	117	6,500			483	121	9,750
Alaska	356	201	6,500			273	137	6,650
Arizona	1,092	128	6,750			692	136	10,050
Arkansas	526	125	6,500			344	130	9,750
Bureau of Indian Affairs	143	143	3,828	**		122	122	3,539	**
California	5,601	329	19,200			2,506	317	29,650
Colorado	951	123	6,500			457	124	9,750
Connecticut	598	114	6,500			266	110	9,750
Delaware	100	100	6,500	*		54	54	9,535	**
Department of Defense	128	128	8,696	**		68	68	6,620	**
District of Columbia	122	122	5,515	**		46	46	4,600	**
Florida	1,972	166	9,750			1,083	178	14,625
Georgia	1,162	156	9,500			472	119	12,950

Table 5. Grade 4 and 8 school frame counts, school sample sizes, and final target student sample sizes for the NAEP 2007 state-by-state assessments (alpha samples) by state and TUDA districts (continued)

	Grade 4					Grade 8
Jurisdiction	Schools in frame	School sample size	Overall target student sample size			Schools in frame	School sample size	Overall target student sample size
Hawaii	197	117	6,500			70	70	9,750	*
Idaho	343	130	6,500			181	109	9,750
Illinois	2,305	183	9,800			1,535	210	15,050
Indiana	1,131	116	6,500			482	111	9,750
Iowa	701	141	6,500			404	137	9,750
Kansas	736	147	6,500			411	149	9,750
Kentucky	758	122	6,500			395	123	9,750
Louisiana	832	121	6,500			676	143	9,750
Maine	365	159	6,500			223	138	9,750
Maryland	860	113	6,500			314	108	6,650
Massachusetts	1,020	175	9,250			468	140	13,950
Michigan	2,032	132	7,050			1,071	130	9,950
Minnesota	945	147	7,350			652	163	10,950
Mississippi	447	114	6,500			287	116	9,750
Missouri	1,125	129	6,500			669	133	9,750
Montana	416	204	6,500			284	189	9,750
Nebraska	736	196	6,500			424	143	6,650
Nevada	324	111	6,500			122	77	9,750
New Hampshire	265	131	6,500			136	94	9,750
New Jersey	1,355	116	6,500			711	119	9,750
New Mexico	414	128	6,500			208	113	9,750
New York	2,343	154	9,250			1,171	158	13,850
North Carolina	1,323	168	10,100			664	155	15,000
North Dakota	285	221	6,500			200	200	8,022	**
Ohio	2,095	177	9,250			1,114	163	14,200
Oklahoma	900	140	6,500			598	153	9,750
Oregon	772	145	7,250			405	124	6,800
Pennsylvania	1,802	118	6,500			859	112	9,750
Puerto Rico	1,024	109	3,150			411	105	3,150
Rhode Island	189	119	6,500			58	58	9,750	*
South Carolina	588	111	6,500			273	108	9,750
South Dakota	332	203	6,500			262	154	6,650
Tennessee	971	117	6,500			555	121	9,750
Texas	3,921	297	17,400			2,059	234	25,800
Utah	514	112	6,500			215	110	9,750
Vermont	231	197	6,500			125	125	7,114	**
Virginia	1,120	111	6,500			380	106	9,750
Washington	1,191	140	7,250			611	143	10,700
West Virginia	443	148	6,500			195	118	9,750
Wisconsin	1,134	133	6,500			606	131	9,750
Wyoming	185	185	6,029	**		88	88	6,829	**
Total Public	52,179	7,989	393,868			27,438	7,043	552,534

Table 5. Grade 4 and 8 school frame counts, school sample sizes, and final target student sample sizes for the NAEP 2007 state-by-state assessments (alpha samples) by state and TUDA districts (continued)

	Grade 4							Grade 8
Jurisdiction	Schools in frame	School sample size			Overall target student sample size			Schools in frame	School sample size	Overall target student sample size
Atlanta	62		53	3,250				17	17	3,515	**
Austin	75		54	3,250				17	17	5,241	**
Boston	78		65	3,250				35	35	4,729	**
Charlotte	89		52	3,250				34	34	4,875	*
Chicago	464		87	4,875				433	108	7,313
Cleveland	88		60	3,250				47	47	4,465	**
Houston	194		84	4,875				56	45	7,313
Los Angeles	463		80	4,875				114	74	7,313
New York City	698		81	4,875				346	83	7,313
San Diego	130		55	3,250				42	33	4,875

Notes: 1) Counts for states include counts for their constituent TUDA districts.

2) Target student sample sizes reflect sample sizes prior to attrition due to exclusion, ineligibility, or nonresponse.

3) * identifies jurisdictions where all schools for the given grade are in the NAEP sample.

4) ** identifies jurisdictions where all students for the given grade are in the NAEP sample.

Stratification

In the six states where the NIES oversampling is taking place (Arizona, Michigan, Minnesota, North Carolina, Oregon, and Washington), we will first create a separate stratum of schools with relatively large percentages of American Indians (those above the thresholds given in table 4). These oversampling strata will not be further broken down. The remaining schools will be further stratified as described below.

Each state and grade will be stratified separately, but using a common approach in all cases. TUDA districts will be separated from their state, and each part stratified separately. The first level of stratification will be based on type of location. This variable has 8 levels (some of which may not be present in a given state or TUDA district), and these will be collapsed so that each of the resulting location categories contains at least 10 percent of the student population. Within each of the resulting location categories, schools will be assigned a minority enrollment status. This is based on the two race/ethnic groups that are the second and third most prevalent within the location category. If these groups are both low in percentage terms, no minority classification will be used. Otherwise three (or occasionally four) equal-sized groups (generally high, medium, and low minority) will be formed based on the distribution across schools of the two minority groups.

Finally, within the resulting location and minority group classes (of which there are likely to be from three to fifteen, depending upon the jurisdiction), schools will be sorted by a measure derived from school level results from the most recent available state achievement tests at the relevant grade. In general, mathematics test results will be used, but where these are not available, reading results will be used. In the few states that do not have math or reading tests at grades 4 and 8 (or where we are unable to match the results to the NAEP school frame), instead of achievement data, schools will be sorted using a measure of socio-economic status. This is the median household income of the 5-digit ZIP Code area where the school is located, based on 2000 Population Census data. Schools in the American Indian oversampling strata (in the six states having such strata) will be sorted by percentage of American Indian enrollment.

Once the schools are sorted by location class, minority enrollment class, and achievement data (or household income or enrollment data), a systematic sample of schools will be selected using a random start. Schools will be sampled with probability proportional to size. The exact details of this process are described in the individual sampling specification memos.

2. Gamma Sample

This is the public school sample at grade 12. It will consist of a single sample of schools all with the same assessment type.

As in past assessments, modest oversampling of Black and Hispanic students will be undertaken in this sample. This will be carried out at the school level. Each school with more than 15 percent Black and Hispanic students will be given twice the selection probability of a low minority school of comparable size. This means that while about 53 percent of the student population (including over 90 percent of the Black and Hispanic students) are in “high” minority schools, about 70 percent of the sample students will come from these schools.

Stratification

The Gamma sample will have an implicit stratification, using a hierarchy of stratifiers and a serpentine sort. The top of the hierarchy is Census division (9 implicit strata). The next stratifier in the hierarchy is type of location, which has eight categories. Of the 72 potential type of location strata nested within Census divisions, several will be collapsed with neighboring type of location cells, always within Census division, giving a total of 55 to 60 Census division-location type strata.

These geographic strata will be subdivided into 110 to 120 strata by a dichotomous high minority status category. Schools are in the high minority stratum if they had more than 10 minority students and greater than 15 percent minority students (minority defined as Black or Hispanic). Otherwise the school will be put in a low minority stratum. If the expected sample size within these strata is less than 8.0, they will be left as is. If the expected sample size is greater than 8.0, then the high-or-low minority strata will be subdivided into up to four substrata (two for expected sample size up to 12.0, three for expected sample size up to 16.0, and four for expected sample size greater than 16.0). For the low minority strata, the subdivision will be by state or groups of contiguous states. For the high minority strata, the subdivision will be by minority percentage. In total there will be between 160 and 180 implicit strata.

Within these substrata, the schools are to be sorted by estimated grade enrollment (using a serpentine sort within the school type substrata).

Note that the Gamma sample was actually designed and selected based on achieving a target of 16,000 assessed students, enough for the only the operational writing sample. This has now been revised to 18,000 students, as shown in Table 1, to accommodate a small reading pilot test sample. The procedures for obtaining the extra 2,000 students are discussed in Section VII.

3. Delta Samples

These are the private school samples at grades 4, 8, and 12. Schools in the three grade-specific samples will be assigned to one assessment type.

Stratification

The private schools are to be explicitly stratified by private school type (Catholic/non-Catholic). Within each private school type, stratification will be by Census region (4 categories), type of location (8 categories), and by proportion of minority enrollment. In general, where there are few or no schools in a given stratum, categories will be collapsed together always preserving the Catholic/non-Catholic stratification.

IV. New Schools

To compensate for the fact that files used to create the NAEP school sampling frames are at least two years out of date at the time of frame construction, we will supplement the Alpha, Gamma, and Delta samples with new school samples at each grade.

The new school samples will be drawn using a two-stage design. At the first stage, a minimum of ten school districts (in states with at least ten districts) will be selected from each state for public schools, and ten Catholic dioceses will be selected nationally for the private schools. The sampled districts and dioceses will be asked to review lists of their respective schools and identify new schools. Frames of new schools will be constructed from these updates, and new schools will be drawn with probability proportional to size using the same sample rates as their corresponding original school samples.

The school sample sizes in the above tables do not reflect new school samples.

V. Substitute Samples

Substitute samples will be selected for each of the delta and gamma samples. The substitute school for each original will be the next “available” school on the sorted sampling frame, with the following exceptions:

1. Schools selected for any NAEP samples will not be used as substitutes.

2. Schools selected for the TIMSS sample (originals or substitutes) will not be used as substitutes for any grades.

3. Private schools whose school affiliation is unknown will not be used as substitutes. Also, unknown affiliated private schools in the original samples will not get substitutes.

4. Schools can only be substitutes for one and only one sample. (If a school is selected as a substitute school for grade 12, for example, it can not be used as a substitute for either grade 4 or grade 8.)

5. A public school substitute will always be in the same state as its original school.

VI. Overlap Control with Other Samples

Two additional NCES studies (TIMSS and ECLSK) are being conducted at the same time as the NAEP 2007 assessments. When selecting the alpha and delta samples at grades 4 and 8, we will minimize overlap with schools in TIMSS. For ECLS-K for alpha and delta grade 8 samples, we will assign a flag at the school-level identifying whether a NAEP school contains at least one student from ECLS-K and at the student-level identifying whether or not an ECLS-K student was selected for NAEP.

VII. Student Sampling

Students within the sampled schools will be selected with equal probability. The student sampling parameters vary by sample type (Alpha, Gamma, and Delta), assessment type, and grade as described below.

Sample Alpha, Grade 4 Schools (Except Puerto Rico)

1. All students, up to 68, will be selected.

2. If the school has between 69 and 120 students (inclusive), the school will be asked if it wishes to have all students selected, or a sample of 62. If the school elects to have all students selected, all students will be selected. If the school asks for a sample of 62 students, a systematic sample of 62 students will be selected.

3. If the school has more than 120 students, a systematic sample of 62 students will be selected.

4. In some schools in small jurisdictions, the school may be assigned more than one ‘hit’ in sampling. In these schools we will select a sample of size 62 times the number of hits, taking all students if this target is greater than or equal to 62/68 of the total grade 4 enrollment.

5. All students will be assigned to assessment type RM.

Sample Alpha, Puerto Rico Grade 4

1. All students, up to 34, will be selected.

2. If the school has more than 34 students, a systematic sample of 30 students will be selected with no oversampling.

3. All students will be assigned to assessment type PR.

Sample Alpha, Grade 8 Schools (Except Puerto Rico)

For schools in states signed on for writing:

1. All students, up to 102, will be selected.

2. If the school has more than 102 students, a systematic sample of 93 students will be selected with no oversampling.

3. In some schools in small jurisdictions, the school may be assigned more than one ‘hit’ in sampling. In these schools we will select a sample of size 93 times the number of hits, taking all students if this target is greater than or equal to 93/102 of the total grade 8 enrollment.

4. All students will be assigned to assessment type RS.

For schools in states not signed on for writing:

1. All students, up to 68, will be selected.

2. If the school has more than 68 students, a systematic sample of 63 students will be selected with no oversampling.

3. In some schools in small jurisdictions, the school may be assigned more than one ‘hit’ in sampling. In these schools we will select a sample of size 63 times the number of hits, taking all students if this target is greater than or equal to 63/68 of the total grade 8 enrollment.

4. All students will be assigned to assessment type RN.

Sample Alpha, Puerto Rico Grade 8

1. All students, up to 34, will be selected.

2. If the school has more than 34 students, a systematic sample of 30 students will be selected with no oversampling.

3. All students will be assigned to assessment type PR.

Sample Gamma, Grade 12

As mentioned earlier, the gamma school sample was designed and selected based on an overall target sample size of 16,000 assessed students, enough for the operational writing sample only. The student sampling parameters for this design used a within-school student sample size of 62 and a take-all student threshold of 68. To accommodate the additional 2000 students for the reading pilot test sample, these parameters will have to be increased to 71 and 78, respectively, in the student sampling procedure. For documentation purposes, the parameters needed for student sampling are provided first followed by the corresponding parameters used in the sample design in parentheses.

1. All students, up to 78 (68), will be selected.

2. If the school has more than 78 (68) students, a systematic equal probability sample of 71 (62) students will be selected.

3. All students will be assigned to assessment type RW.

Sample Delta, Grade 4

1. All students, up to 68, will be selected.

2. If the school has between 69 and 120 students (inclusive), the school will be asked if it wishes to have all students selected, or a sample of 62. If the school elects to have all students selected, all students will be selected. If the school asks for a sample of 62 students, a systematic sample of 62 students will be selected.

3. If the school has more than 120 students, a systematic sample of 62 students will be selected.

4. All students will be assigned to assessment type RM.

Sample Delta, Grade 8

1. All students, up to 102, will be selected.

2. If the school has more than 102 students, a systematic sample of 93 students will be selected.

3. In some schools, the school may be assigned more than one ‘hit’ in sampling. In these schools we will select a sample of size 93 times the number of hits, taking all students if this target is greater than or equal to 93/102 of the total grade 8 enrollment.

4. All students will be assigned to assessment type RN.

Sample Delta, Grade 12

1. All students, up to 68, will be selected.

2. If the school has more than 68 students, a systematic equal probability sample of 62 students will be selected.

3. In some schools, the school may be assigned more than one ‘hit’ in sampling. In these schools we will select a sample of size 62 times the number of hits, taking all students if this target is greater than or equal to 62/68 of the total grade 12 enrollment.

4. All students will be assigned to assessment type WT.

VIII. Weighting Requirements

The Operational Samples

These samples will have a single set of weights for each subject (reading and math at grade 4; reading, math, and writing at grade 8; and writing at grade 12) applied to reflect probabilities of selection, school and student nonresponse, any trimming, and the random assignment to the particular subject. There will be a separate replication scheme by grade and public/private.

The Pilot Test Samples

We will not weight the students in the pilot test studies. However, preliminary weights will be available for pilot test samples, if needed.

The NIES Samples

The NIES assessment samples consist of two grade-specific samples comprising students selected for the grade 4 and 8 operational samples. We will create one set of weights for each grade-specific sample. The NIES weights are designed for any aggregation of the NIES data, not involving NAEP achievement data. NIES analyses involving NAEP achievement data should use the appropriate NAEP operational weights.

C-18