Part A TIMSS2023 Field Test Data Collection Main Study Sampling and Recruitment

The Trends in International Mathematics and Science Study (TIMSS), conducted by the National Center for Education Statistics (NCES), within the U.S. Department of Education (ED), is an international assessment of fourth and eighth grade students’ achievement in mathematics and science. Since its inception in 1995, TIMSS has continued to assess students every 4 years (1995, 1999, 2003, 2007, 2011, 2015, and 2019), with the next TIMSS assessment, TIMSS 2023, being the eighth iteration of the study. In TIMSS 2023, approximately 65 countries or education systems will participate. The United States will participate in TIMSS 2023 to continue to monitor the progress of its students compared to that of other nations and to provide data on factors that may influence student achievement.

TIMSS is led by the International Association for the Evaluation of Educational Achievement (IEA), an international collective of research organizations and government agencies that create the frameworks used to develop the assessment, the survey instruments, and the study timeline. IEA decides and agrees upon a common set of standards, procedures, and timelines for collecting and reporting data, all of which must be followed by all participating countries. As a result, TIMSS is able to provide a reliable and comparable measure of student skills in participating countries. In the U.S., NCES conducts this study in collaboration with the IEA and a number of contractors (RTI International, AnLar, Measurement Incorportated, and Strategic Communications) to ensure proper implementation of the study and adoption of practices in adherence to the IEA’s standards. Participation in TIMSS is consistent with NCES’s mandate of acquiring and disseminating data on educational activities and student achievement in the United States compared with foreign nations [The Educational Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543)].

TIMSS 2023 will be a computer-based assessment (referred to as “eTIMSS”), administered using the eTIMSS player via Chromebook tablets with attached keyboards. TIMSS 2023 builds on the work of TIMSS 2019, which primarily used an electronic assessment format but included a bridge study to examine the effect of administering the assessment on tablet versus paper and establish the link to maintain trends. TIMSS 2023 will be the second eTIMSS assessment in the United States.

Among the key features of TIMSS 2023:

TIMSS 2023 will be an innovative, group adaptive design to allow for a better alignment of the assessment with the student populations in the participating countries;
Problem solving and inquiry tasks integrated into the assessment design so students can experience real-world laboratory situations and apply process skills and content knowledge;
Enhanced international reporting to include reporting of process data to allow TIMSS 2023 to better understand student approaches to problem solving and scientific inquiry;
Innovative item types that engage students; students can create a range of data displays, move and rotate objects on the screen, and show their work with typed text and drawings; and
Policy-relevant informaiton on contexts for learning mathematics and science.¹

Because TIMSS is a collaborative effort among many parties, the United States must adhere to the international schedule set forth by the IEA, including the availability of final field test and main study plans as well as draft and final questionnaires. In order to meet the international data collection schedule, to align with recruitment for other NCES studies (e.g., the National Assessment of Education Progress, NAEP), and for schools to put the TIMSS 2023 field test assessment on their Spring 2022 calendars, recruitment activities for the field test were approved in May 2021 (OMB# 1850-0695 v16) and will begin in June of 2021. The current package requests approval to conduct the TIMSS 2023 field test data collection and the TIMSS 2023 main study sampling, recruitment, and data collection. The field test data collection will be conducted in March and April 2022. Recruitment for the main study will begin in March 2022 with data collection occurring in March through June 2023. The final internationally approved questionnaires for the field test and main study will be submitted via change memos in January 2022 and January 2023, respectively.

A. Justification

A.1 Importance of Information

TIMSS is a cornerstone assessment for measuring U.S. student achievement internationally. TIMSS provides a unique opportunity to compare U.S. students‘ mathematics and science knowledge and skills at fourth and eighth grade with that of their peers in countries around the world. Science, technology, engineering, and mathematics (STEM) preparedness is key to economic improvement.

The continuation of U.S. participation allows for the study of past and current education policies that have shaped science and mathematics achievement over the past 28 years. Furthermore, participating countries are not only able to obtain information about students' knowledge and abilities in the specified subjects, but also about the cultural environments, teaching practices, curriculum goals, and institutional arrangements that are associated with student achievement in the respective subject areas.

Along with other international assessments conducted by the National Center for Education Statistics, TIMSS benchmarks U.S. student achievement against peer countries and other jurisdictions, tracks trends in student performance over time, and illuminates relationships between achievement and the contextual factors that influence it.

TIMSS 2023 will represent a significant milestone in the assessment of the long-term education impacts of one of the largest disruptions in the history of education from the coronavirus (COVID-19) pandemic. Given these profound disruptions, TIMSS data are paramount for our educational system and will provide a greater insight to our nation’s strengths and weaknesses relative to a wide range of existing educational practice.

Based on earlier TIMSS data releases and wide interest in measuring the impact from coronavirus (COVID-19) pandemic disruptions, it is likely that the results of TIMSS 2023 will draw great attention in the United States and elsewhere. It is therefore expected that TIMSS will contribute to ongoing national and international debates and efforts to improve mathematics and science learning and achievement.

A.2 Purposes and Uses of Data

TIMSS assesses mathematics and science knowledge and skills at grades 4 and 8. TIMSS is designed to align broadly with curricula in the participating countries. The results, therefore, suggest the degree to which students have learned concepts and skills likely to have been taught in school. TIMSS also collects background information on students, teachers, schools, curricula, and official education policies in order to allow cross-national comparison of educational contexts that may be related to student achievement.

Data compiled and collected from TIMSS 2023 allows for evidence-based decisions to be made for the purposes of educational improvement. Each successive participation in TIMSS provides trend information about student achievement in mathematics and science relative to other countries, as well as indicators that show how this achievement relates to demographic and curricular, school, teacher, and student factors that provide the educational context for achievement. This high quality, internationally comparative trend data provide key information to inform education policy discussions while identifying existing educational inequalities across socioeconomic, racial/ethnic, and gender lines.

Through participation in TIMSS and other international assessment programs, NCES is able to provide comparative indicators on student performance and school practices across countries in order to benchmark U.S. student performance, and to suggest hypotheses about the relationship between student performance and factors that may influence performance as well as areas in which students have strengths or weaknesses. The international studies identify differences among countries over time in instructional practices, school policies, and opportunity-to-learn that informs discussions about how to organize instruction.

NCES’s mandate [Section 406 of the General Education Provisions Act, as amended (20 U.S.C. §1221e-1)] specifies that "The purpose of the Center [NCES] shall be to collect and analyze and disseminate statistics and other information related to education in the United States and in other nations." and the Educational Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543) specifies that NCES shall collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations, including acquiring and disseminating data on educational activities and student achievement in the United States compared with foreign nations. TIMSS is essential for any international perspective on students’ mathematics and science knowledge and skills and is aligned with both the national and international aspects of NCES' mission.

TIMSS 2023 Components

The mathematics and science assessments at grades 4 and 8 are organized around a content dimension that specifies the subject matter to be assessed and a cognitive dimension that specifies the thinking processes to be assessed. The cognitive domains are the same in mathematics and science: knowing, applying, and reasoning. The TIMSS 2023 frameworks will be published mid-2021 and are similar to the TIMSS 2019 frameworks, but slightly revised to provide more specificity for item writers, and to better reflect current curricula in participating countries. It is not anticipated that there will be any revisions to the content domains or cognitive domains, nor to the target percentages for the content domains or cognitive domains at either subject at either grade.

In fourth-grade mathematics the cognitive content domains include: number, geometric shapes and measures, and data display. More advanced content in these three domains are assessed in eighth grade, supplemented by a data and chance domain. TIMSS assesses a range of problem-solving situations within mathematics, with about two-thirds of the questions requiring students to use applying and reasoning skills.

In science at fourth grade, the content domains include: life science, physical science, and earth science. At eighth grade the content domains transition to a more discipline-based approach, reflecting the differences in instruction from elementary school. The content domains at eighth grade are: biology, chemistry, physics, and earth science. TIMSS 2023 will also measure science practices and science inquiry, reflecting recent emphasis on these skills in many countries’ curricula and content standards.

Assessment Mode

TIMSS 2023 is mainly a digital assessment, however countries unable to transition to digital assessment will be offered a paper-based option comprising trend items only. The United States will participate in digitally-based TIMSS (or eTIMSS). Studies have been conducted in 2017, 2018, and 2019 to operationalize the new digital mode of assessment, as well as to analyze mode effects so that the important TIMSS trend reporting can be maintained. eTIMSS was first administered in the TIMSS main study in 2019. Similar to 2019, the assessments in 2023 will be conducted on Chromebook tablets, with touchscreen capability, if required by the IEA, and an external keyboard and mouse.

Assessment Instruments

In order to minimize burden and to ensure broad subject-matter coverage, TIMSS will use a matrix sampling approach where the mathematics and science items at each grade level are organized into a set of item blocks arranged into digital block combinations, with each student taking a subset of the blocks. Test items are either multiple-choice or constructed response items.

Questionnaires

The background questionnaires for TIMSS 2023 are being developed to address a background questions framework developed internationally. The United States will adapt the questions to fit the U.S. education context, including adding a few questions, such as about the race/ethnicity of students. Teacher and school questionnaires will be offered online via a web-based survey. All students will answer their questionnaires and cognitive assessment questions on Chromebook tablets.

School Questionnaire. A representative from each participating school will be asked to provide information on mathematics and science resources, teacher availability and retention, principal leadership, school emphasis on academic success, school climate, and parental involvement in school activities. The TIMSS school questionnaire is expected to take 30 minutes to complete and will be offered online via a web-based survey.

Teacher Questionnaire. At grades 4 and 8, mathematics and science teachers of students in selected classes will be asked to complete a teacher questionnaire. Teacher questionnaires will include questions about teacher preparation and experience, mathematics and science topics taught, instructional resources and technology, instructional time, instructional engagement, classroom assessment, and technology resources and instruction in their classes. The teacher questionnaire is expected to take 35 minutes to complete and will be offered online via a web-based survey.

Student Questionnaire. Student information will be collected about home resources, student motivation, self-concept, self-efficacy, and student characteristics such as gender and race/ethnicity. The student questionnaire is expected to take 30 minutes to complete and is administered after the cognitive assessment.

A.3 Improved Information Technology (Reduction of Burden)

The TIMSS 2023 design and data collection procedures are prescribed internationally including student assessments and questionnaires being administered on tablets. Each participating nation is expected to adhere to the internationally prescribed design. In the United States, the school and teacher questionnaires will be made available to school administrators and teachers online as the main mode of administration; paper-and-pencil assessments were phased out during TIMSS 2019.

A communication website, MyTIMSS USA, was used during the previous iterations of TIMSS which is familiar to many states, districts, and schools. A similar portal will be used in the 2023 field test and main study in order to provide continuity as a simple, single source of information to engage sample schools and maintain high levels of their involvement. This portal will be used throughout the assessment cycle to inform schools of their tasks and to provide them with easy access to information tailored for their anticipated needs. We plan to gather class and student lists from participating schools electronically using RTI’s secure roster upload process. The study portal includes an electronic system for submitting lists of student information, including student background information in school records. Electronic rosters have been used successfully in NCES studies for more than 10 years and schools will access the electronic system through RTI TIMSS website.

A mobile computer lab will be set up for the student session at each participating school. The student session will be completed on tablet-like Chromebook computers with touchscreen capability and an attached keyboard and mouse. The computerized assessment is made possible by connecting the Chromebooks to an independent local area network (LAN) housed on a laptop computer set up at the school by study field staff. The laptop computer is loaded with the eTIMSS player that collects the data students enter on each Chromebook. All equipment is provided by the study, and neither the school’s internet access nor any internet access in general is required for the computerized administration of the student session.

A.4 Efforts to Identify Duplication

In the United States, mathematics and science achievement is systematically assessed at (1) the Federal level, where trend data have been collected on a fairly regular basis since 1971 through the National Assessment of Educational Progress (NAEP); (2) the state level, where data are routinely collected as part of state testing programs, though they vary across the states in terms of the frequency of testing, age/grades tested, and types of cognitive items administered; and (3) the district level, where data are collected through the use of commercially or locally developed standardized tests as well as tests developed in conjunction with the instructional programs used in schools. TIMSS 2023 does not duplicate these assessments.

TIMSS 2023 is part of a program of international cooperative studies of educational achievement supported and funded, in part, by the U.S. Department of Education. These studies represent the U.S. participation in international studies involving approximately 65 countries. As part of international cooperative studies, the United States must collect the same information at the same time as the other participating nations for purposes of making both valid international comparisons with other countries and with the previous TIMSS data. While some studies in the United States collect similar, though not identical, types of information (e.g., NAEP), the data from those studies cannot be substituted for the information collected in TIMSS in that they do not allow for comparisons outside the United States. Furthermore, the data collected through TIMSS is based on unique frameworks that are not shared by any other state, national, or international data collection effort. In order to participate in these international studies, the United States must agree to administer the same core instruments that are administered in all other participating countries. Because the items measuring mathematics and science achievement have been developed with intensive international coordination, any changes to the instruments require international coordination and approval.

A.5 Minimizing Burden for Small Entities

No small entities are part of this sample. The school samples for TIMSS contain small-, medium- and large-size schools, including private schools, selected based on probability proportionate to their size. All school sizes are needed to ensure an appropriate representation of each type of school in the selected sample of schools. Burden will be minimized wherever possible. For example, schools will be selected so as to avoid as much as possible overlap with other NCES assessments such as NAEP. In addition, contractor staff will conduct all test administrations, provide all equipment, and will assist with parental notification, sampling, and other tasks as much as possible within each school.

A.6 Frequency of Data Collection

The TIMSS 2023 field test will occur March through April 2022. The main study data collection is scheduled for March through June 2023. This schedule is prescribed by the international collective for TIMSS, and adherence to this schedule is necessary to establish consistency in survey operations among participating countries as well as to maintain trend lines.

A.7 Special Circumstances

None of the special circumstances identified in the Instructions for Supporting Statement apply to the TIMSS study.

A.8 Consultations outside NCES

Consultations outside NCES have been extensive and will continue throughout the life of the project. The IEA studies are developed as a cooperative enterprise involving all participating countries. An International Steering Committee has general oversight of the study and each National Research Coordinator participates in extensive discussions concerning the projects, usually with advice from national subject matter and testing experts. In addition, the IEA convened separate panels of mathematics and science experts from around the world to develop cognitive items.

The majority of the consultations (outside NCES) have involved the IEA-Amsterdam, in the Netherlands, the IEA-Data Processing Center (DPC) in Hamburg, Germany, and the TIMSS International Study Center (ISC) at Boston College in the United States. Key to these ongoing consultations are: Dirk Hastedt (executive director of the IEA); Oliver Neuschmidt (head of the IEA Data Processing and Research Center); Michael Martin, Ina V.S. Mullis, Dana Kelly, and Victoria Centurino (directors of the ISC TIMSS teams), all of whom have extensive experience in developing and operating international education surveys (especially related to TIMSS).

A.9 Provisions of Payments or Gifts to Respondents

In order to achieve acceptable response rates in international studies, incentives are typically offered at the school, staff, and student level to thank them for their participation – the time they invest in and the space they make available for the international assessments. High response rates are required by both IEA and NCES and are difficult to achieve in voluntary school-based studies. TIMSS standards for participating countries require minimum participation rates for schools, classrooms, and students. Without sufficient participation levels, U.S. results may be flagged as unreliable in international reports or excluded from international reporting altogether. To be included in cross-national comparisons, each country must meet the following standards:

A minimum school participation rate of 85%, based on sampled schools;
A minimum classroom participation rate of 95%, computed across sampled schools and substitute schools;
A minimum student participation rate of 85%, computed across sampled schools and substitute schools

A minimum combined school, classroom, and student participation rate of 75%, based on sampled schools (although classroom and student participation rates include substitute schools).

The U.S. has historically had difficulties in achieving sufficient participation levels. To maximize likelihood of participation, we propose a set of monetary and nonmonetary incentives. The list of incentives for TIMSS 2023 are introduced in Table 1, followed by a description and justification for each.

Table 1. TIMSS 2023 Field Test and Main Study Incentive Amounts

Participant	Incentive
School	$200 or equivalent gift certificate for school supplies Interactive webinar (for up to 3 staff) for schools to select one topic of either social and emotional learning, project-based learning, or STEM In the main study only, $800 or equivalent gift certificate for targeted schools who are the most difficult to recruit (see details in section below)
School coordinator	$100 or equivalent gift certificate for school supplies
Teacher	$25 or equivalent in gift card for school supplies
Student	Token incentive, ~$4 value Certificates of service for students (in schools where permitted) Community service hours for 8^th grade students (in schools where permitted)

School incentives. As with the field test and past administrations of TIMSS, all schools contacted to participate in TIMSS 2023 will be offered $200 as a token of our appreciation for participating. Each participating school will also be invited to send up to three staff to a 90-minute interactive webinar delivered online as a virtual workshop. Staff will be able to choose from one of three timely topics:

STEM and Inquiry-Based Learning. This inquiry-based learning session will embed a common language and instructional strategy across disciplines to foster blended learning and other processes that model curiosity, design thinking, and reflective learning. This option will provide a system that maximizes the staff’s opportunity to grow and succeed in teaching STEM through inquiry.

Social and Emotional Learning (SEL). Schools will learn how to create an environment where SEL can take place and build competence in self-awareness and social awareness in adults.
Project-Based Learning. The session is designed to support inquiry-driven project-based learning (PBL) for educators who are early or intermediate in their exploration of the concept. This workshop will engage participants for designing, assessing, and managing standards-focused exploration as well as using performance assessment to judge the relevant work generated by 21^st-century learners.

For the TIMSS 2023 main study, participating schools will also receive a school-level report; those with sufficient participation will receive indicators of performance. These reports are a requirement for participation of many schools in large districts who expect timely, relevant, and actionable data in exchange for participation.

While we hope that these incentives will be sufficient to achieve the IEA participation requirements, school recruitment has become increasingly difficult in the U.S. and will likely be even more difficult given loss of instruction and learning during COVID. To ensure that we are able to achieve the required participation targets for TIMSS 2023, we plan to offer a second-tier incentive of up to $800 for the most challenging schools to recruit. This second-tier incentive would be used judiciously and strategically to increase participation in the following target groups:

Schools that have declined to participate but are in a sampling strata that is falling short of participation targets, or we may go back and offer it to original schools where the original school and both of its substitutes decline to participate;

Private schools in the original or substitute sample. In ICILS 2018, only 40% of original private public schools participated, with the overall rate of about 57% after substitutes were added. Participation of private schools is particularly important as there are only about 20 private schools selected per grade for TIMSS 2023; and;
Substitute schools for whom recruitment does not begin prior to January 2022. During previous TIMSS cycles, original schools typically refused in the mid to late fall prior to the spring assessment, at which point substitute schools are recruited. Recruitment of substitute schools is a challenge (less than 25 percent participate) because the school year is already underway, and calendars are set.

This second-tier incentive has been repeatedly and successfully used across the NCES international studies—and has been recently approved for two international studies currently in the field (PIRLS 2021 and PISA 2022)—to help ensure that the studies meet international participation requirements to be included in cross-national comparisons. Table 2 shows the history and impact of this second-tier incentive across U.S. participation in international studies.

Table 2. History and Impact of the Second-Tier Incentive on NCES International Studies

Study	Approved Incentive	Timing of Initiation	Impact
PISA 2012 Field Test	Tier 1: $200 Tier 2: Up to $800 for a subset of schools.	Conducted as an experiment	Considered successful and used for main study
PISA 2012 Main Study	Tier 1: $200 Tier 2: Up to $800 for a subset of schools.	At the end of the prior academic year after all original schools were contacted	Met the minimum participation requirement after including substitute schools
PISA 2015	Tier 1: $200 Tier 2: Up to $800 for a subset of schools.	At the end of the prior academic year after all original schools were contacted	Met the minimum participation requirement after including substitute schools
PIRLS 2016	Tier 1: $200 Tier 2: Up to $800 for a subset of schools.	Mid-way through recruitment	Accepted by eight schools (or about 20 percent of the schools offered the $800) which enabled the study to achieve the minimum participation target after including substitute schools.
ICILS 2018	Tier 1: $200 Tier 2: Up to $800 for a subset of schools.	Offered in the middle of data collection through an OMB change request after it became clear participation rates were not going to be met with the Tier 1 $200 incentives.	Able to successfully recruit about 20 percent of these schools even with the late start. Though ICILS fell short of its target recruitment percentage, an earlier implementation may have helped achieve required participation.
TIMSS 2019	Tier 1: $200 Tier 2: Up to $800 for a subset of schools.	Throughout recruitment for private and NAEP overlap schools; beginning Fall 2018 for substitute and refusal schools	Accepted by 159 schools (or about 46 percent of the schools offered the $800), which enabled the study to achieve the minimum participation target after including substitute schools.
PIRLS 2021	Tier 1: $200 Tier 2: Up to $800 for a subset of schools.	Recruitment in progress	Recruitment in progress
PISA 2022	Tier 1: $200 Tier 2: Up to $800 for a subset of schools.	Recruitment in progress	Recruitment in progress

School Coordinators. School coordinators will be offered a $100 monetary incentive or equivalent gift certificate for school supplies. They play an especially important role in the study and are critical to its success. The coordinator in each participating school will coordinate logistics with RTI; provide class and student list information for sampling; communicate with teachers, students, and parents about the study to encourage their participation; distribute and collect parental consent forms; and assist the session facilitator in ensuring that the sampled students attend the testing sessions.

Teachers. The incentive proposed for students’ teachers is $25 per teacher survey as either a check or gift card for school supplies. This amount is aligned with the amounts used on TIMSS 2019 as well as other NCES studies, such as PISA and PIRLS.

Students. Consistent with prior administrations of TIMSS, as a token of appreciation for their participation, students will receive a small gift valued at approximately $4. Each participating student will be offered a choice of ear buds, a digital watch, or a pair of sunglasses. Students will also receive a certificate with their name thanking them for participating and representing the United States in TIMSS. In schools where it is permitted, eighth-grade students participating in TIMSS may also receive a certificate from the U.S. Department of Education for four hours of volunteer service. Additionally, some schools also offer recognition parties with pizza or other treats for students who participate; however, these are not reimbursed by NCES or the contractor. In districts or schools that require active parental consent, which historically has been difficult to collect, we will offer a small party with refreshments for students who bring in their parental consent forms.

A.10 Assurance of Confidentiality

Data security and confidentiality protection procedures have been put in place for TIMSS to ensure that the TIMSS contractor for the U.S. and its subcontractors comply with all privacy requirements, including:

The statement of work of this contract;
Privacy Act of 1974 (5 U.S.C. §552a);
Family Educational and Privacy Act (FERPA) of 1974 (20 U.S.C. §1232(g));
Privacy Act Regulations (34 CFR Part 5b);
Computer Security Act of 1987;
U.S.A. Patriot Act of 2001 (P.L. 107-56);
Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9573);
Confidential Information Protect and Statistical Efficiency Act of 2002;
E-Government Act of 2002, Title V, Subtitle A;
Cybersecurity Enhancement Act of 2015 (6 U.S.C. §151);
The U.S. Department of Education General Handbook for Information Technology Security General Support Systems and Major Applications Inventory Procedures (March 2005);
The U.S. Department of Education Incident Handling Procedures (February 2009);
The U.S. Department of Education, ACS Directive OM: 5-101, Contractor Employee Personnel Security Screenings;
NCES Statistical Standards; and
All new legislation that impacts the data collected through the inter-agency agreement for this study.

Furthermore, the contractor will comply with the Department’s IT security policy requirements as set forth in the Handbook for Information Assurance Security Policy and related procedures and guidance, as well as IT security requirements in the Federal Information Security Management Act (FISMA), Federal Information Processing Standards (FIPS) publications, Office of Management and Budget (OMB) Circulars, and the National Institute of Standards and Technology (NIST) standards and guidance. All data products and publications will also adhere to the revised NCES Statistical Standards, as described at the website: http://nces.ed.gov/statprog/2012/.

The laws pertaining to the use of personally identifiable information are clearly communicated in correspondence with states, districts, schools, teachers, students, and parents. Letters and information materials will be sent to parents and school administrators describing the study, its voluntary nature, and the extent to which respondents and their responses will be kept confidential (see copies in appendix A-B).

Letters to teachers, school coordinators, and supporting materials will read (Appendix A-1 and A-2):

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Letters to states, districts, and schools and parent notification letters and supporting materials will read (Appendix A-1, A-2, B-1, and B-2):

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information provided by school staff and students may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

The following statement will appear on the login page for eTIMSS and on the TIMSS portal:

The National Center for Education Statistics (NCES), within the U.S. Department of Education, conducts TIMSS in the United States as authorized by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0695. The time required to complete this information collection and participate in study activities is estimated to average 30 minutes per school administrator, 35 minutes per teacher, and 240 minutes per school coordinator, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments or concerns regarding the accuracy of the time estimate(s), suggestions for improving the form, or questions about the status of your individual submission of this form, write directly to: Trends in International Mathematics and Science Study (TIMSS), National Center for Education Statistics, Potomac Center Plaza, 550 12^th Street, SW, 4^th floor, Washington, DC 20202.

OMB No. 1850-0695, Approval Expires xx/xx/20xx.

The TIMSS 2023 confidentiality plan includes signing confidentiality agreements and notarized nondisclosure affidavits by all contractor and subcontractor personnel and field workers who will have access to individual identifiers. By law (20 U.S.C. §9573), a violation of the confidentiality restrictions is a felony, punishable by imprisonment of up to 5 years and/or a fine of up to $250,000. Also included in the plan is personnel training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; controlled and protected access to computer files under the control of a single data base manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and operator-manned in-house computing facility. Data files, accompanying software, and documentation will be delivered to NCES at the end of the project. Neither names nor addresses will be included on any data file. In eTIMSS, students log into the automated assessment using non-identifying ID’s and thus the resulting data are collected and stored with only the non-identifying TIMSS assigned ID. The data are collected from the tablets using a process that encrypts the data, and uploads the data onto an RTI secure FTP site.

NCES understands the legal and ethical need to protect the privacy of the TIMSS respondents and has extensive experience in developing data files for release that meet the government’s requirements to protect individually identifiable data from disclosure. The contractor will conduct a thorough disclosure analysis of the TIMSS 2019 data when preparing the data files for use by researchers, in compliance with ESRA 2002 (20 U.S.C. §9573). Schools with high disclosure risk will be identified and, to ensure that individuals may not be identified from the data files, a variety of masking strategies will be used, including swapping data and omitting key identification variables (i.e., school name and address) from both the public- and restricted-use files (though the restricted-use file will include an NCES school ID that can be linked to other NCES databases to identify a school); omitting key identification variables such as state or ZIP Code from the public-use file; and collapsing or developing categories for continuous variables to retain information for analytic purposes while preserving confidentiality in public-use files.

A.11 Sensitive Questions

The questionnaires do not have items considered to be of a sensitive nature.

A.12 Estimates of Burden

This request is to conduct sampling, recruitment activities and data collection for the TIMSS 2023 field test and main study. Therefore, the burden estimates include burden for

contacting states, districts, schools, and parents in order to recruit for the TIMSS 2023 field test and main study, including
1. sending recruitment letters to districts and schools selected,
2. contacting and seeking research approvals from special handling districts, where applicable, and
3. notifying parents of sampled students about their participation in the field test and main study.
field test and main study data collection.

The estimated burden for these efforts is shown in Table 3.

The district and school contact letters for TIMSS are assumed to impose a small burden on all contacted parties, both those that refuse and those that agree to participate in the TIMSS studies. The burden hours have been separated for those districts and schools which elect to participate and those that may refuse.

The special handling districts are those known to require completion of a research application before they will allow schools under their jurisdiction to participate in a study. Based on an initial assessment of previous TIMSS data collections, we estimate that there may be between 10-20 special handling districts in the field test sample and approximately 80 special handling districts in the main study sample. Contacting special handling districts begins with updating district information based on what can be gleaned from online sources. Calls are then placed to verify the information about where to send the completed required research application forms, and, if necessary, to collect contact information for this process. During the call, inquiry is also made about the amount of time the districts spend reviewing similar research applications. The estimated number of such districts represents those with particularly detailed application forms and lengthy processes for approval. This operation should begin in the spring of the year preceding the start of the data collection to allow sufficient time for special handling districts’ review processes. We will continue to work with these districts until we receive a final response from each district (approval or denial of request) up until April 1, 2022 for the field test and up until May 1, 2023 for the main study.

The total district and school response burden estimate for recruitment is based on 10 minutes for districts to read materials and respond, and 20 minutes for schools to read materials and respond. The total response burden estimate for IRB approvals is based on 120 minutes for staff approval and 60 minutes for panel approval.

Based on the estimated hourly rates for principals/administrators, school coordinators, and parents of $49.52, $31.39, and $27.07, respectively, and based on the estimated total of 1,048 burden hours for TIMSS 2023 field test recruitment, the estimated respondent burden time cost is $36,203. Based on the estimated total of 1,592 burden hours for TIMSS 2023 field test data collection, the estimated respondent burden time cost is $16,746. Based on the estimated total of 6,336 burden hours for TIMSS 2023 main study recruitment, the estimated respondent burden time cost is $201,729. Based on the estimated total of 11,360 burden hours for TIMSS 2023 main study data collection, the estimated respondent burden time cost is $119,192.

Table 3. Burden estimates for TIMSS 2023 field test recruitment, data collection, and main study recruitment and data collection

Activity	Sample size	Expected response rate	Number of respondents	Number of responses	Average Burden Time per Response (minutes)	Total burden (hours)	Estimated Respondent Average¹	Estimated Respondent Burden Time Cost
Field Test Recruitment
Nonparticipating Districts	97²	30%	29	29	10	5	$49.52	$248
Participating Districts	97²	70%	68	68	10	12	$49.52	$594
District IRB Staff Study Approval	20	100%	20	20	120	40	$49.52	$1,981
District IRB Panel Study Approval³	100	100%	100	100	120	200	$49.52	$9,904
Original Nonparticipating Eligible Schools⁴	70	30%	21	21	20	7	$49.52	$347
Original Participating Eligible Schools	70	70%	49	49	20	17	$49.52	$842
Replacement Nonparticipating Eligible Schools	140⁵	50%	21	21	20	7	$49.52	$347
Replacement Participating Eligible Schools	140⁵	50%	21	21	20	7	$49.52	$347
School Coordinators	70	100%	70	70	240	280	$31.39	$8,789
Students’ Parents (permission)	3,150⁶	90%	2,835	2,835	10	473	$27.07	$12,804
Total Field Test Recruitment	3,647		3,234	3,234		1,048		$36,203
Field Test Data Collection
Student Survey	3,150	89%	2804	2804	30	1402	$7.25	$10,165
Student Assessment	3,150	89%	2804	2804	72	3365	--	--
Teacher Survey	280⁷	95%	266	266	35	156	$31.39	$4,897
School Survey	70	95%	67	67	30	34	$49.52	$1,684
Total Field Test Data Collection	3,500		3,137	3,137		1,592		$16,746
Field Test Total	7,147		6,371	6,371		2,640		$52,949
Main Study Recruitment
Nonparticipating Districts	396	30%	119	119	10	20	$49.52	$990
Participating Districts	396	70%	277	277	10	47	$49.52	$2,327
District IRB Staff Study Approval	80	100%	80	80	120	160	$49.52	$7,923
District IRB Panel Study Approval³	240	100%	240	240	120	480	$49.52	$23,770

Original Nonparticipating Eligible Schools⁴	600	30%	180	180	20	60	$49.52	$2,971
Original Participating Eligible Schools	600	70%	420	420	20	140	$49.52	$6,933
Replacement Nonparticipating Eligible Schools	1,000⁵	50%	80	80	20	27	$49.52	$1,337
Replacement Participating Eligible Schools	1,000⁵	50%	80	80	20	27	$49.52	$1,337
School Coordinators	500	100%	500	500	240	2,000	$31.39	$62,780
Students’ Parents (permission)	22,500	90%	20,250	20,250	10	3,375	$27.07	$91,361
Total Main Study Recruitment	25,316		22,226	22,226		6,336		$201,729
Main Study Data Collection
Student Survey	22,500	89%	20,025	20,025	30	10,013	$7.25	$72,594
Student Assessment	22,500	89%	20,025	20,025	72	24,030	--	--
Teacher Survey	2,000	95%	1,900	1,900	35	1,109	$31.39	$34,812
School Survey	500	95%	475	475	30	238	$49.52	$11,786
Total Main Study Data Collection	25,000		22,400	22,400		11,360		$119,192
Main Study Total	50,316		44,626	44,626		17,696		$320,921
Total Requested	57,463		50,996	50,996		20,336		$373,870

Italicized rows represent assessment activities, which are excluded from calculations of burden.

The average hourly earnings of parents derived from May 2020 Bureau of Labor Statistics (BLS) Occupation Employment Statistics is $27.07, teachers is $31.39, of education administrators is $49.52. If mean hourly wage was not provided, it was computed assuming 2,080 hours per year. The exception is the student wage, which is based on the federal minimum wage. Source: BLS Occupation Employment Statistics, http://data.bls.gov/oes/ datatype: Occupation codes: All employees (00-0000); Teachers (25-2020); Education Administrators (11-9032); accessed on May 4, 2021.

² Based on a simulated sampling of schools.

³ Based on the estimate that, on average, there will five individuals per panel.

⁴ Satisfactory sampling participation rate includes a final unweighted school response rate of at least 50% of original schools and at least 85% of original plus replacement schools, with original school sample as the denominator.

⁵Two replacements schools will be sampled for each original school but a replacement school will only be contacted if the original school declines to participate. Based on estimated response rates of original participating schools, 42 replacement schools will be contacted for the field test and 160 replacement schools will be contacted for the main study, and it is estimated that 50% will participate.

⁶Sample size assumes that approximately one percent of students will be ineligible, and that 90 percent of eligible students’ parents will review permission materials.

⁷Based on estimate average of 4 teachers per school.

A.13 Total Annual Cost Burden

There are no additional costs to respond beyond the time to respond.

A.14 Annualized Cost to Federal Government

The cost to the federal government for conducting all of the phases of TIMSS 2023, including preparations, field test and main study data collection, and scoring is estimated to be $9,809,064 over a 5-year period (see table breakdown below). Figures include all direct and indirect costs.

TIMSS 2023
Contract costs
Instrumentation and materials	$3,428,073
Data Collection	$4,619,157
Systems and data processing	$1,761,834
Total	$9,809,064

A.15 Program Changes or Adjustments

This submission is for the TIMSS 2023 field test data collection and the main study sampling, recruitment, and data collection activities. Burden has increased because the prior package included only field test sampling and recruitment activities while this package includes sampling, recruitment, and data collection for both the field test and main study. TIMSS 2023 differs from prior administrations in that the data collection will be entirely electronic and no pilot or bridge study will be conducted.

A.16 Publication Plans and Time Schedule

The TIMSS field test is designed to provide a statistical review of the performance of items on the cognitive assessment and questionnaires in preparation for the main study data collection. They will provide valuable experience informing TIMSS main study assessment on Chromebook tablets and the performance of TIMSS items in a digitally-based environment. No field test data will be published.

Based on the data collected in the main study, the TIMSS International Study Center will prepare separate reports for mathematics and science at grades 4 and 8. These reports will be released by December 2024. As has been customary, NCES will also release a report at the same time as the international reports are released, interpreting the results for the U.S. audience. NCES reports on initial data releases are generally limited to simple bivariate statistics. There are currently no plans to conduct complex statistical analyses of either dataset. Examples of past reports on TIMSS can be found at https://nces.ed.gov/timss/results19/index.asp#/math/intlcompare. In the spring of 2025, the International Study Center will also prepare technical reports for TIMSS 2023, describing the design and development of the assessments as well as the scaling procedures, weighting procedures, missing value imputation, and analyses. After the release of the international data, NCES plans to release the national data and an accompanying User’s Guide for the study.

Electronic versions of each publication are made available on the NCES website. Schedules for tabulation and publication of TIMSS 2023 results in the United States are dependent upon receiving data files from the international sponsoring organization. With this in mind, the expected data collection dates and a tentative reporting schedule are shown in Table 4.

Table 4. Operational schedule for TIMSS 2023 field test and main study

	Activity	Start Date	End Date
	Field Test
Prepare data collection manuals, forms, assessment materials, and questionnaires		January 2021		March 2022
Select school sample		January 2021		June 2021
Recruitment of states, districts, schools		August 2021		April 2022
Collect field test data		March 2022		April 2022
Deliver raw data to international sponsoring organization		April 2022		May 2022
Review field test results		April 2022		July 2022
	Main Study
	Prepare data collection manuals, forms, assessment materials, and questionnaires	May 2022	March 2023
	Select school sample	March 2021	July 2021
	Recruitment of states, districts, schools for main study	January 2022	May 2023
	Collect main study data	March 2023	June 2023
	Deliver raw data to international sponsoring organization	July 2023	July 2023
	Receive final data files from international sponsors	June 2024	June 2024
	Produce reports	June 2024	December 2024

A.17 Display OMB Expiration Date

The OMB expiration date will be displayed on all data collection materials.

A.18 Exceptions to Certification Statement

No exceptions to the certifications are requested.

1 Source: https://timssandpirls.bc.edu/timss2023/downloads/T2023_TIMSS_Brochure.pdf

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Calvin Choi
File Modified	0000-00-00
File Created	2021-11-25

Part A TIMSS2023 Field Test Data Collection Main Study Sampling and Recruitment