Potential respondent universe.
The possible universe of student respondents is estimated to be 12 million fourth-, eighth-, and twelfth-grade students attending approximately 154,000 public and private elementary and secondary schools. NAEP test booklets are administered in selected public and private schools to a sample of students in grades 4, 8, and 12.
Students are selected according to student sampling procedures with these possible exclusions:
The student is identified as an English Language Learner (ELL) if the student is classified by the school as ELL and has received language arts instruction primarily in English for less than three school years, including the current year.
The student is identified as having a disability which prevents participation in NAEP, even with accommodations as allowed in NAEP, and has an Individualized Education Plan (IEP) or equivalent classification, such as those identified as part of the 504 plan.
Based on experience from previous assessments, we estimate about three to five percent of grade-eligible students will be excluded because of disabilities or language barriers that preclude their assessment. NAEP relies upon the professional judgment of school administrators as to how students or schools should be classified.
2. Procedures for collection of information.
Survey Design and Sampling:
The sampling information in this system clearance package is an overview of the sampling techniques and criteria used by the current sampling contractor for the sampling in NAEP assessments. Each specific assessment will involve different selected samples based on the volumes and subjects in that particular assessment. Planned sample sizes are based on the need to obtain representative samples on which to report proficiency information.
Design Features
As in the past, NAEP samples are based on multistage designs. The state assessment designs consist of stratified samples of public schools; where the stratification is derived from type of location (urban/suburban/large town/small town/rural), proportion minority enrollment, school level achievement on statewide testing programs, and a measure of household income in the zip code area of the school. The second stage of sampling is the selection of the students from within each selected school. This is an equal probability systematic sample from among all students in the appropriate grade. For the national samples, a three-stage design is used. The first stage is the selection of primary sampling units (PSU's), which are individual counties or groups of contiguous counties. The second stage is the selection of schools within PSUs, and the third stage is the selection of students within schools. The following are characteristic features of NAEP sampling designs:
Approximately equal sample sizes for each state's public schools
About 2500-3000 assessed students for each subject, at grades 4 & 8 in each state
National sample sizes of approximately 10-12K for operational subjects
In each school some students are to be assessed in each subject
Lists of schools are obtained from the NCES Common Core of Data (CCD)
Schools are grouped into strata
Schools are assigned a measure of size
Sample is selected with probability proportional to the measure of size
School stratification is based on characteristics such as: Type of Location, Minority Enrollment, School Achievement
(Refer to Appendix C for an example of the sampling procedures contained in the 2007 assessment).
Students - Students in fourth, eighth and twelfth grades complete assessment booklets that commonly contain two 25-minute cognitive blocks, followed by two background question sections, which require 15 minutes to complete. The first background question block contains core questions related to demographic information. The second background block contains subject-specific questions.
Teachers - The teachers of fourth- and eighth-grade students participating in NAEP will be asked to complete questionnaires about their teaching background, education, training, and classroom organization.
Principals/Administrators - The school administrators in the sample schools will be asked to complete a questionnaire. As with the teacher questionnaires, the core questions are designed to measure school characteristics and policies that research has shown are highly correlated with student achievement.
SD/ELL questionnaires will be completed for students identified with disabilities or as English language learners.
E-Filing - Survey sample information is collected from schools in the form of lists of potential students who may participate in NAEP. This sample information can be gathered manually or electronically at the school, district, or state level. If done at the school or district level some burden will be incurred by school personnel.
High School Transcript Study (HSTS) information is provided by a sample of the public and private school. The sample of schools is nationally representative of all schools in the United States, and the sample of students is representative of graduating seniors from each school. The transcript study includes only those students whose transcripts indicate that they graduated the year that the study was conducted. Most of the students sampled in the transcript study are in schools that participated in NAEP. The data collected from those students that participated in NAEP make it possible to link course-taking patterns to academic performance, as measured by NAEP.
3. Methods to maximize response rates and deal with issues of nonresponse.
NAEP attempts to minimize nonresponse of both students and schools. Chief State School Officers and LEA superintendents are provided with lists of schools in the sample in their jurisdiction and their cooperation is requested. For the assessments, schools within each state will be selected and the chief state school officer and the state coordinator will be asked to solicit their cooperation. Plans also call for requesting NCES to provide letters to states and districts in support of the operational and filed tests. Since states and school districts receiving Title I funds are required to participate in the NAEP reading and mathematics (grades 4 and 8) assessments under No Child Left Behind, NAEP response rates have improved for these assessments.
In previous NAEP administrations 95 percent or more of students have responded, between 85 percent and 90 percent of school administrators are respondents, and among teachers, 85 percent provide background specific information and 75 percent provide class-period specific information.
Not all of the students in the sample will respond. Some will be unavailable during the sample time period because of absenteeism or other reasons. If a student decides not to complete an exercise, the action will be recorded, but no steps will be taken to obtain an answer.
4. Tests of procedures or methods to be undertaken.
The 2008-2010 operational, pre-calibration, and pilot tests will be administered in the January-March time window of each year. The long-term trend administration is conducted in the fall (October-December) of 2007 for age 13 students, the winter (January-March) of 2008 for age 9 students, and the spring (March-May) 2008 for age 17 students. Refer to information in item 16 for specific schedules. Each student in a session will receive one booklet from a spiral of booklets, in which all booklets are spiraled. In general, the operational, pre-calibration, and pilot test materials will be administered in the same sessions. For some subjects (Arts, U.S. history) separate sessions will be required because of different time requirements or book layout. The 2008-2010 administration procedures will be similar those of previous NAEP operational assessments.
5. Consultants on statistical aspects of the design.
AIR, ETS, and Westat staffs have collaborated on the statistical aspects of the design. The primary persons responsible are:
Nancy Caldwell
Vice-President, Westat
Executive Director, NAEP Project Director, ETS
Catherine McClellan
Director of NAEP Psychometrics, ETS
Keith F. Rust
Vice-President, Westat
Leslie Scott
Principal Research Analyst
American Institutes for Research
In addition, the NAEP Design and Analysis Committee (DAC) and the NAEP Validity Study (NVS) panel members (see Appendix B) have also contributed to NAEP designs on an on-going basis.
| File Type | application/msword |
| Author | joconnell |
| Last Modified By | DoED |
| File Modified | 2007-05-22 |
| File Created | 2007-05-22 |