Justification

Trends in International Mathematics and Science Study (TIMSS) 2019 Pretest

OMB# 1850-0803 v.238

Volume 1

Submitted by:

National Center for Education Statistics (NCES)

Institute of Education Sciences (IES)

U.S. Department of Education

Washington, DC

September 2018

Table of Contents

Table of Contents

1) Submittal-Related Information 3

2) Background and Study Rationale 3

3) Recruitment and Data Collection 4

4) Consultations outside the agency 5

5) Justification for Sensitive Questions 6

6) Paying Respondents 6

7) Assurance of Confidentiality 6

8) Estimate of Hourly Burden 7

9) Cost to federal government 8

10) Project Schedule 8

1. Submittal-Related Information

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB# 1850-0803), which provides for NCES to conduct various procedures (such as pilot tests, cognitive interviews, and usability studies) to test new methodologies, question types, or delivery methods to improve survey and assessment instruments and procedures.

2. Background and Study Rationale

The Trends in International Mathematics and Science Study (TIMSS) is an international assessment of fourth and eighth grade students’ achievement in mathematics and science. Since its inception in 1995, TIMSS has continued to assess students every 4 years (1995, 1999, 2003, 2007, 2011, and 2015), with the next TIMSS assessment, TIMSS 2019, being the seventh iteration of the study. In TIMSS 2019, 59 countries and 6 other education systems plan to participate at grade 4, and 40 countries and 5 other education systems plan to participate at grade 8. The United States will participate in TIMSS 2019 in grades 4 and 8 to continue to monitor the progress of its students compared to that of other nations and to provide data on factors that may influence student achievement.

TIMSS is conducted by the International Association for the Evaluation of Educational Achievement (IEA), an international collective of research organizations and government agencies that create the frameworks used to develop the assessment, the survey instruments, and the study timeline. IEA decides and agrees upon a common set of standards, procedures, and timelines for collecting and reporting data, all of which must be followed by all participating countries. As a result, TIMSS is able to provide a reliable and comparable measure of student skills in participating countries. In the U.S., the National Center for Education Statistics (NCES) conducts this study in collaboration with the IEA and a number of contractors (Westat, Avar Consulting (Avar), AIR, Pearson, and Hager Sharp) to ensure proper implementation of the study and adoption of practices in adherence to the IEA’s standards. Participation in TIMSS aligns with NCES’s mandate of acquiring and disseminating data on educational activities and student achievement in the United States compared with foreign nations [The Educational Sciences Reform Act of 2002 (ESRA 2002), 20 U.S.C. §9543].

New in 2019, TIMSS will be a technology-based assessment conducted in an electronic format (referred to as “eTIMSS”) and administered using the IEA eTIMSS Assessment System, an assessment delivery system developed by the IEA. There are two primary goals for the transition to a technology-based assessment: (1) maintaining continuity to measure trends with the past paper-and-pencil TIMSS assessments, and (2) developing relevant assessment innovations that take advantage of new technologies.

This request is to conduct a real-world test of the IEA eTIMSS Assessment System with students, allowing the system to be tested prior to the main study data collection in spring 2019 in the manner that will be used in the main study to help identify system issues, including any router performance issues. The rationale for this study is based on lessons learned and issues encountered by students related to the electronic administration of TIMSS during the field test in spring 2018. It is believed that students use and interact with the systems differently than adult QC testers. Conducting pretesting and QC process with students will allow us to identify existing issues and to address them prior to the operational use of the system in the main study in spring 2019.

The pretesting will be conducted with students from grades 4 and 8 in a real school classroom environment as well as in a simulated classroom using the preliminary final version of the eTIMSS Assessment System. Up to five pretesting events will take place during winter 2018-2019, to include one grade 4 and two grade 8 in-school pretest sessions and one grade 4 and one grade 8 session in the simulated classroom environment. Each grade 4 event session will last approximately 105 minutes and each grade 8 event session will last approximately 125 minutes.

3. Recruitment and Data Collection

Recruitment and Sample Characteristics

In-School Pretest:

For the in-school pretest, a state will be asked to participate in the study on a voluntary basis. The NAEP State Coordinator (“Coordinator”) in the volunteer state will recruit in their state schools that are not part of the main TIMSS 2019 sample (see Appendix A for a sample recruitment letters). The Coordinator will leverage relationships within the state to contact schools and identify those willing to participate in the study. The NAEP State Coordinator will forward the contact information for participating schools to Westat.

A total of three schools will participate in the study. While the study will seek participation from schools with various demographic characteristics, including students with a mix of race/ethnicity and socioeconomic backgrounds, detailed sampling requirements will not be targeted because of the small number of schools that will be asked to participate in the study. Additionally, schools that are within a relatively close proximity to each other within a state will be selected to ensure that one field administration staff member is able to conduct the study for all schools in the state. Private schools will not be recruited for this study.

In each school, an intact class of 4^th or 8^th graders will participate in one pretrial session (for a total of one 4^th grade class and two 8^th grade classes). Schools may select a class to participate or work with Westat to randomly sample one of their classes in the participating grade. TIMSS test administrators will communicate with the school coordinator to determine accommodation needs to discuss whether any students should be excluded from the pretest session. Schools will be asked to provide the number of participating students in the selected class but will not be required to submit student lists or any information about students. The session will be held in the school, on a date between November 2018 and January 2019 that is selected in coordination with the school.

Simulated Classroom Pretest:

For the simulated classroom environment, EurekaFacts, under contract with Westat, will recruit up to 30 students for each session of TIMSS pretesting. There will be a maximum of two sessions. One TIMSS session will be exclusively composed of 4^th graders, and the other session will be exclusively composed of 8^th graders. Each session will be held at the EurekaFacts facility in Rockville, Maryland, most likely on a Saturday.

EurekaFacts will recruit participants for the pretesting study from the District of Columbia, Maryland, Virginia, and West Virginia. Although the sample will include a mix of student characteristics, the results will not explicitly measure differences by those characteristics. Students will be recruited to obtain the following criteria:

• A mix of race/ethnicity (Black, Asian, White, Hispanic);

• A mix of socioeconomic background; and

• A mix of urban/suburban/rural areas.

While EurekaFacts will use various outreach methods (see Appendix B) to recruit students to participate, the bulk of the recruitment will be conducted by telephone and be based on EurekaFacts’ acquisition of targeted mailing lists containing residential addresses and landline telephone listings. EurekaFacts will also use a participant recruitment strategy that integrates multiple outreach methods and resources such as newspaper and internet ads, community organizations (e.g., Boys and Girls Clubs, Parent-Teacher Associations), and mass media recruitment (e.g., postings on the EurekaFacts website).

Interested students will be screened (see Appendix B) to ensure that they meet the criteria for participation in the pretesting study (i.e., the students are from the targeted demographic groups outlined above and their parents/legal guardians have given consent). When recruiting participants, EurekaFacts staff will speak to the parent/legal guardian of the interested minor before starting the screening process. During this communication, the parent/legal guardian will be informed about the objectives, purpose, and participation requirements of the data collection effort as well as the activities that it entails. After confirming that a participant is qualified, willing, and available to participate in this study, he or she will receive a confirmation email/letter and phone call. Written informed parental consent (see Appendix H) will be obtained for all respondents who are interested in participating in the data collection efforts. Shortly after the grade 4 and grade 8 preliminary IEA TIMSS assessment versions are released, the two simulated classroom pretesting events with students will be held, as well as the three pretesting events in schools.

In both the school and simulated classroom environments, all events will have one session lasting approximately 105 or 125 minutes each, depending on the grade level, and it will be structured as follows:

• During each session, each student will be asked to take the assessment under standard TIMSS assessment conditions (approximately 95 minutes at grade 4 and 115 minutes at grade 8, which includes the directions/tutorial, the assessment, and a brief questionnaire about students’ experience with the eTIMSS platform). Students will not be asked to complete the TIMSS student questionnaire.

• A group debrief (up to 10 minutes) will be conducted in each session to solicit feedback from the students. See Volume 2 for the debriefing script.

Data Collection Process

For both the simulated classroom and in-school pretests, normal data collection will be enabled by the TIMSS systems, and any errors generated will be collected automatically by the system. Note that student responses will not be scored. In addition to the TIMSS systems recording information, administrators and observers from NCES, Westat, and/or EurekaFacts will monitor the assessments and record notes detailing any issues encountered by the students, as well as what the students were doing at the time each issue occurred. In addition, observers may ask individual students for clarification of the actions he or she took prior to an issue or error occurring. For example, observers may ask questions such as, “What is the error?”; “What was the last thing you saw before the error?”; “What were you expecting to happen?”; or “What did you do right before the error happened?”. Understanding and documenting what caused the system error is necessary in order to obtain enough information for staff to replicate the error and develop a fix for it.

The sessions in the simulated classroom environment will be audio and/or video recorded to capture information regarding any student actions that resulted in system errors or issues. Sessions that occur in schools will not be recorded.

4. Consultations outside the agency

Westat is the contractor for TIMSS and will provide the tablets for students’ use and administer the pretesting study.

Consultations outside NCES have been extensive and will continue throughout the life of the project. The IEA studies are developed as a cooperative enterprise involving all participating countries. An International Steering Committee has general oversight of the study and each National Research Coordinator participates in extensive discussions concerning the projects, usually with advice from national subject matter and testing experts. In addition, the IEA convened separate panels of mathematics and science experts from around the world to develop cognitive items.

The majority of the consultations (outside NCES) have involved the IEA and, in the United States, the TIMSS International Study Center at Boston College. Key to these ongoing consultations are: Dirk Hastedt (executive director of the IEA), Michael Martin, Ina V.S. Mullis, Victoria Centurino, and Kerry Cotter, all of whom have extensive experience in developing and operating international education surveys (especially related to TIMSS).

EurekaFacts is located in Rockville, Maryland. It is an established for-profit research and consulting firm, offering facilities, tools, and staff to collect and analyze both qualitative and quantitative data. EurekaFacts is working as a subcontractor for Westat to recruit participants and provide the facilities to be used for the study. In addition, EurekaFacts staff may assist in administering and/or observing some of the study sessions.

5. Justification for Sensitive Questions

Throughout the item and debriefing question development processes, effort has been made to avoid asking for information that might be considered sensitive or offensive.

6. Paying Respondents

Incentives used for the in-school sessions will reflect those planned for the TIMSS 2019 main study. Consistent with prior administrations of TIMSS, schools will receive a check for $200 after the session is completed and students will receive a small gift valued at approximately $4 as a token of appreciation for their participation. In the 2018 TIMSS field test, each participating student received a string backpack with a globe printed on it.

For the EurekaFacts simulated classroom environment sessions, to encourage participation and thank students for their time and effort, a $35 gift card from a major credit card will be offered to each participating student. If a parent or legal guardian brings their student to and from the testing site, they will also receive a $35 gift card to thank them for their time and effort in transporting their child. Similar NAEP studies (e.g. OMB# 1850-0803 v.199) have offered $25 to the student participant and $25 to the parent/guardian for sessions that last approximately 1.5 hours. In addition, in the ICILS pretest, student participants and parent/guardians were offered $50 for a 3-hour session. Given that this research study requires more time than NAEP and less time than ICILS (1.75 to 2.08 hours per session, depending on the grade), $35 will be offered to both the student and parent/guardian to aid in recruitment and gain their cooperation in the study.

7. Assurance of Confidentiality

The study will not retain any personally identifiable information. Prior to the start of the study, students will be notified that their participation is voluntary. As part of the study, students will be notified that the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

In the in-school environment, parents will be notified using the same procedures as in the TIMSS 2018 Field Test. A notification letter (see Appendix A) will be sent to the parent/legal guardian of each sampled student prior to the pretesting session and parents/legal guardians will have the opportunity to opt the student out of the session. Before each pretesting session is administered in the simulated classroom environment, a written consent will be obtained from the parent/legal guardian of each participating student (see Appendix B).

All participants will be assigned a unique student identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files, secured for the duration of the study, and will be destroyed after the final report is released. Pretesting activities may be recorded using audio and/or screen capture technology. The only identification included on the files will be the participant ID. The recorded files will be secured for the duration of the study and will be destroyed after the final report is completed.

8. Estimate of Hourly Burden

Tables 1, 2, and 3 detail the estimated burden of the various TIMSS pretesting activities.

Estimated Burden for In-School Environment Pretesting

The estimates assume recruitment of 3 schools – 1 grade 4 and 2 grade 8 schools. Within each school 25 students from a single classroom will be recruited to participate in the study, and all parents will be notified. For estimates of hourly burden in the in-school environment, see Table 1 below.

Table 1. Estimate of Hourly Burden – School Environment

Activity	Sample size	Expected response rate	Number of respondents	Number of responses	Time per respondent (minutes)	Total burden (hours)
In-School Pretest Recruitment
Contacting Districts	3	1.00	3	3	10	1
Contacting Schools	3	1.00	3	3	20	1
Parental notification	90	1.00	90	90	10	15
Pretest Recruitment Burden	-	-	96	96	-	17
In-School Pretest Data Collection
Students Grade 4
Directions	30	0.93	28	28	20	9
Assessment	30	0.93	28	28	72	34
eTIMSS questionnaire	30	0.93	28	28	3	1
Debrief	30	0.93	28	28	10	5
Students Grade 8
Directions	60	0.93	56	56	22	21
Assessment	60	0.93	56	56	90	84
eTIMSS questionnaire	60	0.93	56	56	3	3
Debrief	60	0.93	56	56	10	9
Pretest Data Collection Burden	-	-	84	252	-	48
Total burden – In-School Pretest			180	348		65

Note: Numbers have been rounded and therefore may affect totals. The cognitive item portions of the study, shaded in grey in the table, are not included in burden totals because they are not subject to the Paperwork Reduction Act (PRA).

Estimated Burden for the Simulated Classroom Pretesting

The estimated burden for recruitment assumes attrition throughout the process. In all events, each student will participate in one session for a total of 105 minutes at grade 4 and 125 minutes at grade 8. Table 2 details the estimated burden for the simulated classroom pretesting.

Table 2. Estimate of Hourly Burden – Simulated Classroom Environment

Respondent	Number of respondents	Number of responses	Hours per respondent	Total hours
Parent or Legal Guardian for Student Recruitment
Initial contact	268	268	0.05	14
Follow-up via phone	134*	134	0.15	20
Consent & confirmation	67*	67	0.15	10
Recruitment Totals	268	469	-	44
Student Participation (Pretesting)
Students in 4th Grade TIMSS Pretest	30	30	1.75	53
Students in 8th Grade TIMSS Pretest	30	30	2.08	63
Participation Totals	60	60	-	116
Total burden – Simulated Classroom Pretest	328	529	-	160

* Subset of initial contact group; assumptions for approximate attrition rates are 50 percent from initial contact to follow-up, 50 percent from follow-up to confirmation, and 90 percent from confirmation to participation.

Note: Numbers have been rounded and therefore may affect totals

Table 3 details the total estimated burden for the In-School Environment Pretesting and the Simulated Classroom Pretesting sessions.

Table 3. Estimate of Total Hourly Burden

Respondent	Number of Respondents	Number of Responses	Total Hours
Field Trial Pretesting Sub Total	180	348	65
Simulated Classroom Pretesting Sub Total	328	529	160
Total Burden	508	877	225

Note: numbers have been rounded and therefore may affect totals

9. Cost to federal government

The total cost to federal government for this study is $148,369 as detailed in Table 4.

Table 4: Estimate of Costs

	Activity	Provider	Estimated Cost
School Environment
Recruiting states, schools, and students		Westat	$6,123
Administering the study		Westat	$52,011
Simulated Classroom Environment
Recruiting students and providing facilities for the study		EurekaFacts	$49,285
Administering the study		Westat	$35,882
Total			$143,301

10. Project Schedule

The schedule for this study, including all activities (recruitment, pretesting, data collection, results), will begin in September 2018, as soon as OMB approval is granted, and the study will conclude by January 2019 .