Part B TIMSS2023 Field Test Data Collection Main Study Sampling and Recruitment

There are four target populations for the TIMSS 2023 field test: two school populations and two student populations. The two school target populations include schools offering fourth-grade instruction and those offering eighth-grade instruction in the winter/spring of 2022 in the 50 states and the District of Columbia. These schools include:

Public schools, including charter and magnet schools;
Private schools;
Bureau of Indian Education schools; and
Domestic Department of Defense schools.

Although the following schools are eligible for TIMSS 2023 and could be considered for the field test, we plan to exclude them from the field test in line with our proposal to exclude them from the main study. However, we note that changes in the number of virtual schools, the number of students attending virtual schools, and/or school classifications in light of increased remote learning during the coronavirus (COVID-19) pandemic may require adjustment of the virtual schools’ exclusion. Currently, our plan excludes:

Special education schools,
Virtual schools, and
Schools that include temporary housing like correctional facilities and hospitals.

There are two student target populations for the field test of TIMSS 2023: fourth- and eighth-grade students enrolled in winter/spring of 2022 in the United States (50 states and the District of Columbia) in a school in one of the two school target populations.

There are four target populations for the TIMSS 2023 main study: two school populations and two student populations. The two school target populations include schools offering fourth-grade instruction and those offering eighth-grade instruction in the winter/spring of 2023 in the 50 states and the District of Columbia.

There are two student target populations for the main study of TIMSS 2023: fourth- and eighth-grade students enrolled in winter/spring of 2023 in the United States (50 states and the District of Columbia) in a school in one of the two school target populations.

B.2 Statistical Methodology

Field Test Sampling Plan and Sample.

The purpose of the field test is to evaluate survey instruments and procedures, including classroom sampling procedures. In selecting a school sample for this purpose, it is important to minimize the burden on schools, districts, and states, to minimize impact on these entities while also ensuring that the field test data are collected effectively. In accordance with the TIMSS International Study Center, the field test participation goal will be 35 schools offering fourth grade and 35 schools offering eighth grade. To maximize the efficiency of data collection, only schools that report at least 45 fourth-grade or 45 eighth-grade students will be included in the sample. The field test sample is not required to be nationally representative but should include a diverse sample of schools, covering such features as public (including charter schools), private, large, small, urban, and rural schools, and schools from a variety of different states.

The 2018–2019 Common Core of Data and 2018–2019 Private School Universe Survey (or later, if available) will be used to identify the set of schools eligible for TIMSS 2023 field test, and the reported fourth- and eighth-grade enrollment counts will be used to create the student population distributions. The proposed sampling design is based on quasi-nationally representative samples of schools from six purposively selected geographically diverse metropolitan statistical areas (MSAs). Within each selected MSA, eligible schools will be stratified by school type, poverty status (high/low), and locale. Participation goals for each stratum will be determined by distributing the 35 schools for each sample approximately proportional to the student population in each stratum eligible for the 2023 TIMSS main study. A stratified random sample of 105 schools will selected and from these, 35 schools will be selected for initial recruitment. The remaining 70 schools will be available as a pool of replacement schools, and schools will be selected as needed to achieve the stratum-specific school participation goals.

Participating schools will be asked to provide a list of students and classes. Students will be selected for participation by drawing a random sample of two classes in each school using software provided by the international TIMSS study coordinator that will also be used in the main study. At grade 8, students will be selected by drawing a sample of two intact mathematics classrooms (in which grade 8 students are enrolled) in each sampled school. Smaller classes may be combined to form ‘pseudoclasses’ for sampling. All selected students will be asked to participate in a combined TIMSS mathematics and science assessment. Only students in intact classrooms will be assessed at each grade. We estimate that on average 45 students will be selected from each school, and up to 10 percent of those students will be ineligible or excluded¹ for the field test, yielding approximately 1,400 fourth grade and 1,400 eighth grade students.

TIMSS samples are typically designed to minimize overlap with NAEP data collections happening in the same time frame. There are currently no NAEP data collections scheduled for 2022; however, if NAEP or other NCES data collections are determined to be scheduled in a similar time frame and we are provided with a list of participating schools, then steps will be taken to ensure the same schools are not selected for the TIMSS 2023 field test.

Class and student lists will be gathered from participating schools electronically using a secure electronic filing process (as explained in Part A). Electronic filing provides advantageous features such as efficiency and data quality checks. Schools will access the electronic filing system through a web site.

Main Study Sampling Plan and Sample.

The school sample design for the main study uses probability sampling to select a sample of schools for grade 4 and a sample of schools for grade 8. The school samples will represent all schools, with the requisite grades, in the entire United States. The main study for TIMSS will take place in the spring of 2023.

The sample size for the TIMSS main study will be 300 schools at each of grades 4 and 8. The sampling frames of grade 4 and 8 schools will be obtained from NCES’ 2019-2020 Common Core of Data (CCD) and preliminary 2019-2020 Private School Universe Survey (PSS) files, restricted to schools having grade 4 or 8, respectively, and limited to schools in the 50 United States and the District of Columbia.

The school samples will be explicitly and implicitly stratified according to school characteristics. Explicit strata will be created by cross-classifying school type (public/private), Census region (Northeast, Midwest, South, and West), and poverty status (High/Low, as measured by the percentage of students in the school receiving free or reduced-price lunch in the National School Lunch Program (NSLP)). This will ensure an appropriate representation of each type of school in the selected sample of schools. Within each explicit stratum, schools will be hierarchically sorted by locale (city, suburb, town, rural), percent non-white (less than 15 percent, greater than or equal to 15 percent), State, and enrollment² in the respective grade. Hierarchical sorting will result in samples within explicit strata being distributed approximately proportionately across the groups defined by the implicit stratification variables.

Schools will be selected with probability proportional to enrollment in the appropriate grade (4 or 8) using systematic probability proportional to size sampling. Because the number of classes in a given grade at a school is approximately proportional to the number of students enrolled in that grade, and because one or two classrooms will be sampled from participating schools, the use of a probability proportional to enrollment sample design ensures that all students have an approximately equal chance of selection, since two classes will be selected from each school regardless of the size of the school.

Student sampling will be accomplished by selecting one or two classes per school. Participating schools will be asked to provide a list of students and classes. Students will be selected for participation by drawing a random sample of up to two classes in each school using software provided by the international TIMSS study coordinator. At grade 4, students will be selected by drawing a sample of two intact homeroom classrooms. At grade 8, students will be selected by drawing a sample of two intact mathematics classrooms (in which grade 8 students are enrolled) in each sampled school. Smaller classes may be combined to form ‘pseudoclasses’ for sampling. All selected students will be asked to participate in a combined TIMSS mathematics and science assessment. Only students in intact classrooms will be assessed at each grade. We estimate that on average 45 students will be selected from each school, and up to 10 percent of those students will be ineligible or excluded³ for the main study, yielding approximately 10,000 fourth grade and 10,000 eighth grade students.

TIMSS samples are typically designed to minimize overlap with NAEP data collections happening in the same time frame; however, as the TIMSS sample will be selected before any NAEP samples for the year 2023, the TIMSS samples may be provided for NAEP to consider when selecting its’ sample.

Nonresponse Bias Analysis, Weighting, and Sampling Errors

We will conduct a nonresponse bias analysis pursuant to NCES statistical standards in order to determine where and how respondents and nonrespondents differ on available characteristics. Sampling errors, that account for unequal weighting, stratification, and probability sampling will be calculated for a selection of key estimates.

B.3 Maximizing Response Rates

Gaining cooperation from school districts and schools is paramount to the success of TIMSS 2023, and also the most significant challenge of the study. Given that classrooms are selected, student participation is not as great of a challenge. Historically, student participation rates in TIMSS have never fallen below 90 percent (see table 1). That said, it is important to U.S. TIMSS that students are engaged and try to do their best on the assessment.

Table 1. Historical TIMSS school and student participation rates

Year	Grade	School Participation Rate		Overall Student Participation Rate
Year	Grade	Before Replacement	After Replacement	Overall Student Participation Rate
2019	4	76	88	96
2019	8	72	85	94
2015	4	79	85	96
2015	8	78	84	94
2011	4	79	84	95
2011	8	87	87	94
2007	4	70	89	95
2007	8	68	83	93
2003	4	70	82	95
2003	8	71	78	94
1999	8	83	90	94
1995	4	86	NA	94

Our approach to maximizing school recruitment is to:

Begin recruitment activities as early as possible;
Engage stakeholders at all levels throughout each cycle of TIMSS;
Obtain endorsements about the value of TIMSS from relevant national organizations, state and regional agencies;
Engage with the broad school community (i.e., not sampled schools) through involvement at small-scale school leader conferences and events, to provide an opportunity to test-drive our anticipated recruitment messages, build relationships with influential school-level leaders and highlight anticipated benefits of participation to build hope and excitement among schools that they may be selected into the study;
Inform Chief State Officers and state assessment directors about the sample of schools in their state;
Provide a framework alignment guide for state assessment and accountability offices, which provides information on how to compare and contrast the TIMSS framework with individual state frameworks;
Use the assistance of NAEP State Coordinators to recruit districts and schools, providing key state agency involvement in recruitment;
Send letters and informational materials to schools and districts. These letters will be customized by type of school. In addition, alternative versions are provided in Appendix A to account for school composition during recruitment, as different letters may be used for schools conducting learning virtually;
Train experienced NAEP State Coordinators about TIMSS;
Follow-up mailings with telephone calls and emails to explain the study and school involvement, including placing the TIMSS assessment date on school calendars;
Maintain continued contact until schools have built a relationship with the recruiter and fully understand TIMSS;
Use monetary school incentives of at least $200 and up to $800 for select schools (all school incentives will be $200 for the field test, and graduated incentives will be introduced in the main study for target schools, per description in Part A.9);
Offer salient, interactive, topical webinars to participating district and school staff;
Provide school reports augmented with survey response data;

Produce and distribute a newsletter compiling relevant elementary and middle-school math and science research reports and practice guides from federal Regional Education Laboratories, Comprehensive Centers, and the What Works Clearinghouse for all levels;

Provide study updates for all levels on key study milestones and releases; and
Make in-person visits to some districts and schools, as necessary.

Our approach to maximizing student recruitment is to:

Encourage schools to use implied permission forms or notification letters. Written permission will be collected if required by the school district or school;
Send parental permission forms home to parents 4-6 weeks ahead of the TIMSS sessions to allow ample time for parents to grant permission;
Offer participating students a small token of appreciation valued at approximately $4, for example ear buds, a digital watch, or a pair of sunglasses;
Provide students with a certificate with their name thanking them for participating and representing the United States in TIMSS;
If permitted by the school, certificate will also serve as a certificate of service (4^th and 8^th grades) and/or community service hours (8^th grade), as permitted by the school;
Encourage teachers to encourage student participation;
Ask schools to hang colorful, engaging posters in the school announcing participation;
When feasible, have the test administrator (TA) speak to student prior to the scheduled session day to encourage participation.

Our approach to maximizing teacher recruitment is to:

Send letters and materials reinforcing the importance of participation to teachers;
Follow up with prompting phone calls from RTI’s recruitment experts, who have already established relationships with the school during the school recruitment phase;
Offer a $25 incentive for participation;
Have the TA speak to teachers in person on the day of the student session, or, if teachers are not available, have TA leave personalized postcard reminders in teacher mailboxes.

B.4 Purpose of the Field Test and Main Study and Data Uses

The central goals for the field test are to evaluate new assessment items and background questions, and to ensure that classroom and student sampling procedures proposed for the main study are successful. Information learned from the field test will be integrated into main study processes and protocols, where appropriate.

The goals of the TIMSS main study are to (1) provide trend information about student achievement in mathematics and science to inform education policy discussions and identify existing educational inequalities; (2) provide comparative indicators on student performance and school practices across countries in order to benchmark U.S. student performance.

Data compiled and collected from TIMSS 2023 will allow for evidence-based decisions to be made for educational improvement. These high-quality, internationally comparative trend data are key to informing education policy discussions.

B.5 Individuals Consulted on Study Design

Overall direction for TIMSS is provided by Lydia Malley, National Research Coordinator, National Center for Education Statistics (NCES), U.S. Department of Education, in consultation with a number of NCES statistical staff.

The following persons are responsible for the statistical design of TIMSS:

Pierre Foy, TIMSS International Study Center, Boston College (617-552-6253); and
Peter Siegel, RTI International (919-541-6348).

RTI International is the contractor responsible for sampling, data collection, and data analysis:

Debbie Herget, Project Director, RTI International (919-485-7793);
Ben Dalton, Associate Project Director, RTI International (919-541-7228); and
David Wilson, Senior Statistician, RTI International (919-541-6990).

Analysis and reporting will be performed by:

National Center for Education Statistics, U.S. Department of Education;
TIMSS International Study Center, Boston College;
RTI International; and
AnLar, under contract to RTI International.

1 Students are considered ineligible if they are enrolled in the classroom but not in the target grade (grade 4 or grade 8) or if a student transferred from the school or class between the time that the student roster was prepared and the scheduled session at the school. Students are excluded if they meet the internationally defined criteria as having a functional disability, an intellectual disability, or if they are a non-native language speaker and unable to read or speak the language of the assessment. Students with functional or intellectual disabilities or have at least one year of English instruction or are able to overcome the language barrier would be included in the testing.

2 Enrollment less than 5 will be set to 5 for purposes of implicit stratification and sampling.

3 Students are considered ineligible if they are enrolled in the classroom but not in the target grade (grade 4 or grade 8) or if a student transferred from the school or class between the time that the student roster was prepared and the scheduled session at the school. Students are excluded if they meet the internationally defined criteria as having a functional disability, an intellectual disability, or if they are a non-native language speaker and unable to read or speak the language of the assessment. Students with functional or intellectual disabilities or have at least one year of English instruction or are able to overcome the language barrier would be included in the testing.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Millar, Kimrey
File Modified	0000-00-00
File Created	2021-09-10

Part B TIMSS2023 Field Test Data Collection Main Study Sampling and Recruitment