Part B TIMSS 2019 Main Study

Trends in International mathematics and science study (TIMSS) 2019 main Study

REQUEST FOR OMB Clearance

OMB# 1850-0695 v.14

Supporting Statement Part B

Submitted by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

September 2018

2. COLLECTION OF INFORMATION EMPLOYING STATISTICAL INFORMATION

B.1 Respondent Universe

The respondent universe for the TIMSS 2019 main study is all students enrolled in grade 4 and grade 8 during the 2018-19 school year. A sample of 335 schools will be selected for the main study at each grade level. Within sampled schools, students will be selected for participation by drawing a random sample of two classes. At grade 8, students will be selected by drawing a sample of two intact mathematics classrooms (in which grade 8 students are enrolled) in each sampled school. Only students in intact classrooms will be assessed at each grade. Most selected students will be asked to participate in a combined electronically administered TIMSS mathematics and science assessment using the eTIMSS assessment platform, along with a paper questionnaire. A sample of 1,500 students at each grade will take TIMSS on paper, in order to allow for a bridge analysis with paper-based trend lines.

The respondent universe for the TIMSS 2019 field test was all students enrolled in grade 4 and grade 8 during the 2017-18 school year. A sample of 40 schools was selected for the field test, with the goal of obtaining participation from a minimum of 35 schools at each grade. Within sampled schools, students were selected for participation by drawing a random sample of two classes. At grade 8, students were selected by drawing a sample of two intact mathematics classrooms (in which grade 8 students are enrolled) in each sampled school. All selected students were asked to participate in a combined TIMSS mathematics and science assessment. Only students in intact classrooms were assessed at each grade.

B.2 Statistical Methodology

Main Study Sampling Plan and Sample

The school sample design for the main study must be a probability sample of schools (one sample for each of grades 4 and 8) that fully represents the entire United States. At the same time, to ensure maximum participation it must be designed so as to minimize overlap with other NCES studies involving student assessment that will be conducted around the same time.

The main study for TIMSS will take place in the spring of 2019, about two months after a very large NAEP assessment. NAEP will assess several thousand schools nationally, at grades 4 and 8. The NAEP sample will be relatively heavy in smaller states, and in a number of these states all eligible schools will be included in NAEP, especially at grade 8. Thus to be fully representative, the TIMSS sample may include some schools that will have participated in NAEP at the same grade. However, this number can be kept to minimum using the overlap control procedures outlined below.

Overlap control procedures in studies such as this, where stratified probability proportional to size samples of schools are selected, can be implemented via a procedure that applies Bayes Theorem to modify the conditional probability of selection of a given school for one study, depending upon its selection probability for a second study, and whether or not it was selected for that study. This approach was first documented in a survey sampling application by Keyfitz (1951)¹. The principles involved can be extended to more than two studies simultaneously, and a procedure for doing this is described by Chowdhury et al. (2000)².

The sample size for the TIMSS main study will be about 335 schools at each of grades 4 and 8. The sampling frames of grade 4 and 8 schools will be obtained from the most current versions of NCES’s Common Core of Data (CCD) and Private School Survey (PSS) files, restricted to schools having grade 4 or 8 respectively, and eliminating schools in Puerto Rico, U.S. territories, and Department of Defense overseas schools.

The sample will be stratified according to school characteristics such as public/private, Census region, poverty status (as measured by the percentage of students in the school receiving free or reduced-price lunch in the National School Lunch Program (NSLP)). This will ensure an appropriate representation of each type of school in the selected sample of schools.

Schools will be selected with probability proportional to the number of estimated classes at the appropriate grade (4 or 8), with schools expected to have either one or two classes being given the same selection probability. The use of a probability proportional to sample design ensures that all students have an approximately equal chance of selection, since two classes will be selected from each school regardless of the size of the school.

Student sampling will be accomplished by selecting two classes per school. Each grade 4 school will be asked to prepare a list of classes that is comprehensive and includes each grade 4 student in the school in one of the listed classes. As described above, schools will submit these class and student lists via secure E-filing. Grade 8 schools will be asked to prepare a similar list but with students listed in their mathematics classes. At either grade, any class with fewer than ten students will be combined with another class to form a ‘pseudoclass’ with at least ten students in it. We will then select two classes (regular classes or pseudoclasses) from each school, with equal probability, and all students in those classes/pseudoclasses will be included in the sample. If a school has only one or two classes, then all students in the grade will be included in the sample. At grade 8, mathematics classes are used for three reasons. First, this minimizes the burden on mathematics teachers, as only two mathematics teachers need to fill out a teacher questionnaire (but typically more than two science teachers are required to do so because the students in the two selected math classes often attend more than two different science classes). Second, it makes for sound data for conducting analyses of the extent to which classroom factors moderate the relationship of student factors to achievement (e.g., “Does having a well-qualified math teacher reduce the correlation between math achievement and parental education?”). Third, at grade 8, most students take one and only one mathematics class, and thus mathematics classes make for a foolproof partitioning of the eligible students. A subset of approximately 70 classes sampled at each grade level will take paper-based TIMSS, while the remainder will be assessed using the digitally based eTIMSS assessment platform.

Nonresponse Bias Analysis, Weighting, and Sampling Errors

It is possible that nonresponse will occur at both levels: school and student. If so, we will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards. After the calculation of weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification (see Appendix D for more details).

B.3 Maximizing Response Rates

With the recent exception of TIMSS 2011, the most significant challenge in recruitment for TIMSS has been engaging the schools and gaining their cooperation. The circumstances that aided our success in 2011—the NAEP-TIMSS Linking Study and the involvement of NAEP State Coordinators—did not recur in 2015. However, there are important lessons to be learned from our TIMSS 2011 experience that were used in TIMSS 2015 and will be used in the TIMSS 2019 main study. Given that classrooms are selected, student participation is not as great of a challenge. Historically, student participation rates have never fallen below 90 percent (see table 1). That said, it is important to U.S. TIMSS that students are engaged and try to do their best on the assessment.

Table 1. Historical TIMSS school and student participation rates

Year	Grade	School Participation Rate		Overall Student Participation Rate
		Before Replacement	After Replacement
2015	4	79	85	96
	8	78	84	94
2011	4	79	84	95
	8	87	87	94
2007	4	70	89	95
	8	68	83	93
2003	4	70	82	95
	8	71	78	94
1999	8	83	90	94
1995	4	86	NA	94

Our approach to school recruitment is to:

• Obtain endorsements about the value of TIMSS from relevant organizations;

• Inform Chief State Officers and test directors about the sample of schools in their state.

• Use the assistance of NAEP State Coordinators to recruit districts and schools, providing key state agency involvement in recruitment;

• Send letters and informational materials to schools and districts. These letters will be customized by type of school;

• Train experienced NAEP State Coordinators about TIMSS;

• Implement strategies from NAEP’s Private School Recruiting Toolkit. This toolkit, developed for NAEP, includes well-honed techniques used to recruit a very challenging type of schools;

• Follow-up mailings with telephone calls to explain the study and schools involvement, including placing the TIMSS assessment date on school calendars;

• Maintain continued contact until schools have built a relationship with the recruiter and fully understand TIMSS;

• Use second-tier school incentives of up to $800 for select schools; and

• Make in-person visits to some schools, as necessary.

B.4 Purpose of Pilot, Field Test, Pre-Testing and Data Uses

The U.S. participated in eTIMSS pilot in spring 2017 to test out items that were transferred from paper to digital format, assess the performance of new items developed to take advantage of technology, and evaluate the mode effect of transitioning from paper to digital format.

The TIMSS field test was conducted in spring 2018 across most participating education systems to evaluate sampling and data collection procedures, new and legacy TIMSS items in the eTIMSS electronic platform, and student, teacher, and school questionnaires in preparation for the main study in 2019. In the U.S., 35 schools at grade 4 and 34 schools at grade 8 participated in the field test, including approximately 1,500 fourth-graders, 1,400 eighth-graders, and 200 teachers. Gaining participation of schools continues to be a challenge, and we will most likely need to implement the second-tier incentives in the main study to achieve sufficient participation. The assessment window for the main study is a few weeks longer than in the field test, allowing us to be more flexible in rescheduling schools as necessary to accommodate school conflicts.

In addition, during the field test we evaluated a new eTIMSS delivery system that uses routers provided by us that connected to each student tablet. This method was successful and streamlined the data collection and data upload process, so we will use the router method again in the main study. We were able to determine that data upload for teacher and school questionnaires and for student cognitive data was successful, with no information losses. Pearson, under subcontract to Westat, was also able to score open-ended questions using the international scoring system which was new in the field test, and will be used again in the main study.

Because there continues to be updates and revisions to the eTIMSS assessment platform, in addition to fully testing it internally, prior to the main study, we will conduct pre-testing sessions between late December 2018 and early January 2019 with fourth and eighth-graders (OMB# 1850-0803 v. 238).

B.5 Individuals Consulted on Study Design

Overall direction for TIMSS is provided by Lydia Malley, National Research Coordinator, National Center for Education Statistics (NCES), U.S. Department of Education, in consultation with a number of NCES statistical staff.

The following persons are responsible for the statistical design of TIMSS:

• Pierre Foy. TIMSS International Study Center, Boston College (617-552-6253); and

• Sylvie LaRoche, Statistics Canada (613-863-9480).

Westat is the contractor responsible for sampling and data analysis:

• Chris Averett, Project Director, Westat (301-314-2492); and

• David Ferraro, Senior Statistician, Westat (301-251-4261).

Analysis and reporting will be performed by:

• National Center for Education Statistics, U.S. Department of Education;

• TIMSS International Study Center, Boston College

• Westat; and

• American Institutes for Research, under contract to Westat.

1 Keyfitz, N. (1951). Sampling with Probabilities Proportional to Size: Adjustment for Changes in Probabilities. Journal of the American Statistical Association, 46, 105-109.

2 Chowdhury, S., Chu, A., & Kaufman, S. (2000). Minimizing overlap in NCES surveys. Proceedings of the Survey Methods Research Section, American Statistical Association, 174-179. Retrieved from http://www.amstat.org/sections/srms/ proceedings/papers/2000_025.pdf.