Volume I - PISA Pretest

Program for International Student Assessment (PISA) 2021 Field Test Pretest

OMB# 1850-0803 v.258

Volume 1

Submitted by:

National Center for Education Statistics (NCES)

Institute of Education Sciences (IES)

U.S. Department of Education

Washington, DC

November 2019

Table of Contents

1) Submittal-Related Information 3

2) Background and Study Rationale 3

3) Recruitment and Data Collection 3

4) Consultations outside the agency 5

5) Justification for Sensitive Questions 6

6) Paying Respondents 6

7) Assurance of Confidentiality 6

8) Estimate of Hourly Burden 6

9) Cost to federal government 8

10) Project Schedule 8

1. Submittal-Related Information

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB# 1850-0803), which provides for NCES to conduct various procedures (such as pilot tests, cognitive interviews, and usability studies) to test new methodologies, question types, or delivery methods to improve survey and assessment instruments and procedures.

2. Background and Study Rationale

The Program for International Student Assessment (PISA) is an international assessment conducted every 3 years that focuses on 15-year-olds’ capabilities in reading, mathematics, and science literacy. In 2021, PISA will assess these subjects as well as financial literacy. PISA emphasizes functional skills that students have acquired, both in and out of school, as they near the end of mandatory schooling. PISA is coordinated by the Organization for Economic Cooperation and Development (OECD), an intergovernmental organization of industrialized countries, and is conducted in the United States by the National Center for Education Statistics (NCES). The 30-day public comment period for the PISA 2021 field test and school recruitment is currently in progress. The purpose of this request is to conduct a real-world test of the PISA Student Delivery System (SDS) with students, allowing the system to be tested prior to the field test data collection in spring 2020 in the manner that will be used in the field test to help identify system issues, including any router performance issues. The rationale for this study is based on lessons learned and issues encountered by students related to the electronic administration of PISA during the main study in 2018. It is believed that students use and interact with the systems differently than adult Quality Control (QC) testers. Conducting pretesting and QC process with students will allow us to identify existing issues and to address them prior to the operational use of the system in the field test in spring 2020.

The pretesting will use the preliminary final version of the PISA SDS, and will be conducted in a real school environment as well as in a simulated school environment. Students born in 2004 and in grade 7 or higher will be eligible to participate. Up to two pretesting events will take place during winter 2019-2020, to include one in-school pretest session and one session in the simulated school environment. Each session will last approximately three and a half hours.

3. Recruitment and Data Collection

Recruitment and Sample Characteristics

In-School Pretest:

For the in-school pretest, a state will be asked to participate in the study on a voluntary basis. The NAEP State Coordinator (“Coordinator”) in the volunteer state will recruit public schools in their state that are not part of the PISA 2021 field test sample (see Appendix A for a sample recruitment letters). The Coordinator will leverage relationships within the state to contact schools and identify those willing to participate in the study. The NAEP State Coordinator will forward the contact information for participating schools to Westat.

One public school will participate in the study. While the study will seek participation from a school with various demographic characteristics, including students with a mix of race/ethnicity and socioeconomic backgrounds, detailed sampling requirements will not be targeted.

In the school, up to 60 PISA-eligible students will participate in one pretest session. PISA test administrators will communicate with the school coordinator to determine accommodation needs to discuss whether any students should be excluded from the pretest session. The school will be asked to submit a student list so that students can be randomly selected for the session. The session will be held in the school, on a date between December 2019 and January 2020 that is selected in coordination with the school.

Simulated School Pretest:

For the simulated school environment, EurekaFacts, under contract with Westat, will recruit up to 60 students for the session of PISA pretesting. The session will be held at the EurekaFacts facility in Rockville, Maryland, most likely on a Saturday. EurekaFacts has successfully conducted focus groups for a number of studies conducted by the US Department of Education including TIMSS 2019 and is currently conducting work for PIRLS 2021.

EurekaFacts will recruit participants for the pretesting study from the District of Columbia, Maryland, Virginia, and West Virginia. Although the sample will include a mix of student characteristics, the results will not explicitly measure differences by those characteristics. Students will be recruited to obtain the following criteria:

A mix of race/ethnicity (Black, Asian, White, Hispanic; Multiracial);

A mix of socioeconomic background; and

A mix of students living in or attending schools in urban/suburban/rural areas.

While EurekaFacts will use various outreach methods (see Appendix B) to recruit students to participate, the bulk of the recruitment will be conducted by telephone and be based on EurekaFacts’ acquisition of targeted mailing lists containing residential addresses and landline telephone listings. EurekaFacts will also use a participant recruitment strategy that integrates multiple outreach methods and resources such as newspaper and internet ads, community organizations (e.g., Boys and Girls Clubs, Parent-Teacher Associations), and mass media recruitment (e.g., postings on the EurekaFacts website).

Interested students will be screened (see Appendix B) to ensure that they meet the criteria for participation in the pretesting study (i.e., the students are from the targeted demographic groups outlined above and their parents/legal guardians have given consent). When recruiting participants, EurekaFacts staff will speak to the parent/legal guardian of the interested minor before starting the screening process. During this communication, the parent/legal guardian will be informed about the objectives, purpose, and participation requirements of the data collection effort as well as the activities that it entails. After confirming that a participant is qualified, willing, and available to participate in this study, he or she will receive a confirmation email/letter and phone call. Written informed parental consent (see Appendix B) will be obtained for all respondents who are interested in participating in the data collection efforts. Shortly after the PISA SDS assessment versions are received by Westat, the simulated school pretesting event with students will be held as well as the pretesting event in the school.

In both the school and simulated school environments, all events will have one session lasting approximately three and a half hours each and structured as follows:

During each session, each student will be asked to take the assessment under standard PISA assessment conditions (3 hours), which includes the directions/tutorial, the assessment, and a brief questionnaire about students’ experiences in and out of school).

A group debrief (up to 10 minutes) will be conducted in each session to solicit feedback from the students. See Volume 2 for the debriefing script.

For both the simulated school and in-school pretests, normal data collection will be enabled by the PISA systems, and any errors generated will be collected automatically by the system. Note that student responses will not be kept or scored, as the intent of these focus groups is not on students’ academic ability but on their facility with the PISA student assessment delivery system and its functionality and to test the functioning of the system. In addition to the PISA systems recording information, administrators and observers from NCES, Westat, and/or EurekaFacts will monitor the assessments and record notes detailing any issues encountered by the students, as well as what the students were doing at the time each issue occurred. In addition, observers may ask individual students for clarification of the actions he or she took prior to an issue or error occurring. For example, observers may ask questions such as, “What is the error?”; “What was the last thing you saw before the error?”; “What were you expecting to happen?”; or “What did you do right before the error happened?” Understanding and documenting what caused the system error is necessary in order to obtain enough information for staff to replicate the error and develop a fix for it.

The sessions in the simulated school environment will be audio and/or video recorded to capture information regarding any student actions that resulted in system errors or issues. Sessions that occur in the public school will not be recorded.

4. Consultations outside the agency

Westat is the contractor for PISA and will provide the tablets for students’ use and will administer the pretesting study.

Consultations outside NCES have been extensive and will continue throughout the life of the project. The nature of the study requires this because international studies typically are developed as a cooperative enterprise involving all participating countries. PISA 2021 is being developed and operated under the auspices of the OECD by a consortium of organizations. Key persons from these organizations who are involved in the design, development and operation of PISA 2021 are listed below.

Organization for Economic Cooperation and Development

Andreas Schleicher, Indicators and Analysis Division

2, rue André Pascal, 75775 Paris Cedex16, FRANCE, Tel: +33 (1)-4524-9366, Fax: +33 (1)-4524-9098

Educational Testing Service

Irwin Kirsch, Project Director, ETS Corporate Headquarters

660 Rosedale Road, Princeton, NJ 08541 USA, Tel: 1-609-921-9000, Fax: 1-609-734-5410

Westat

Keith Rust, Director of Sampling

1600 Research Boulevard, Rockville, Maryland 20850-3129 USA, Tel: 301-251-8278, Fax: 301-294-2034

EurekaFacts is located in Rockville, Maryland. It is an established for-profit research and consulting firm, offering facilities, tools, and staff to collect and analyze both qualitative and quantitative data. EurekaFacts is working as a subcontractor for Westat to recruit participants and provide the facilities to be used for the study. In addition, EurekaFacts staff may assist in administering and/or observing some of the study sessions.

5. Justification for Sensitive Questions

Throughout the item and debriefing question development processes, an effort has been made to avoid asking for information that might be considered sensitive or offensive.

6. Paying Respondents

Incentives used for the in-school sessions will reflect those planned for the PISA 2021 Field Test. Consistent with prior administrations of PISA, the school will receive a check for $250 after the session is completed, the school coordinator will also receive a check for $200 after the session, and students will receive $25 and volunteer service certificate of 4 hours from the U.S. Department of Education.

For the simulated school environment sessions, to encourage participation and thank students for their time and effort, students will receive a $60 gift card from a major credit card for completing the session. Parents that transport the student to the session will also receive a $60 gift card. Similar NAEP studies (e.g. OMB# 1850-0803 v.199) have offered $25 to the student participant and $25 to the parent/guardian for sessions that last approximately 1.5 hours. Given that this study requires an unusually long session, 3.5 hours, $60 will be offered to both the student and parent/guardian to aid in recruitment and gain their cooperation in the study.

7. Assurance of Confidentiality

The study will not retain any personally identifiable information. Prior to the start of the study, students will be notified that their participation is voluntary. As part of the study, students will be notified that the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

In the in-school environment, parents will be notified using the same procedures as proposed in the PISA 2021 Field Test. A notification letter (see Appendix A) will be sent to the parent/legal guardian of each sampled student prior to the pretesting session and parents/legal guardians will have the opportunity to opt the student out of the session. In the simulated school environment, a written consent will be obtained from the parent/legal guardian of each participating student before each pretesting session is administered (see Appendix B).

All participants will be assigned a unique student identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files, secured for the duration of the study, and will be destroyed after the final focus group report is released. Pretesting activities may be recorded using audio and/or screen capture technology. The only identification included on the files will be the participant ID. The recorded files will be secured for the duration of the study and will be destroyed after the final focus group report is completed.

8. Estimate of Hourly Burden

Tables 1, 2, and 3 detail the estimated burden of the various PISA pretesting activities.

Estimated Burden for In-School Environment Pretesting

The estimates assume recruitment of one high school with PISA-eligible students. Within each school 60 students will be recruited to participate in the study, and all parents will be notified. For estimates of hourly burden in the in-school environment, see Table 1 below.

Table 1. Estimate of Hourly Burden – School Environment

Activity	Sample size	Expected response rate	Number of respondents	Number of responses	Time per respondent (minutes)	Total burden (hours)
In-School Pretest Recruitment
Contacting Districts	1	1	1	1	10	1
Contacting Schools	1	1	1	1	20	1
Parental notification	60	1	60	60	10	10
Pretest Recruitment Burden	-	-	62	62	-	12
In-School Pretest Data Collection
Students
Directions	60	1	60	60	20	20
Assessment	60	1	60	60	120	120
PISA questionnaire	60	1	60	60	60	60
Debrief	60	1	60	60	10	10
Pretest Data Collection Burden	-	-	60	180	-	90
Total burden – In-School Pretest			122	242		102

Note: Numbers have been rounded and therefore may affect totals. The cognitive item portions of the study, shaded in grey in the table, are not included in burden totals because they are not subject to the Paperwork Reduction Act (PRA).

Estimated Burden for the Simulated School Pretesting

The estimated burden for recruitment assumes attrition throughout the process. Each student will participate in one session for a total of 3.5 hours. Table 2 details the estimated burden for the simulated classroom pretesting.

Table 2. Estimate of Hourly Burden – Simulated School Environment

Respondent	Number of respondents	Number of responses	Time per respondent (minutes)	Total hours
Parent or Legal Guardian for Student Recruitment
Initial contact	200	200	3	10
Follow-up via phone	100*	100	9	15
Consent & confirmation	50*	50	9	8
Recruitment Totals	200	350	-	33
Student Participation (Pretesting)
Directions	45	45	20	15
Assessment	45	45	120	90
Student questionnaire	45	45	60	45
Debrief	45	45	10	8
Participation Totals	45	135	-	68
Total burden – Simulated School Pretest	245	485	-	101

* Subset of initial contact group; assumptions for approximate attrition rates are 50 percent from initial contact to follow-up, 50 percent from follow-up to confirmation, and 90 percent from confirmation to participation. The cognitive item portions of the study, shaded in grey in the table, are not included in burden totals because they are not subject to the Paperwork Reduction Act (PRA).

Note: Numbers have been rounded and therefore may affect totals

Table 3 details the total estimated burden for the In-School Environment Pretesting and the Simulated School Pretesting sessions.

Table 3. Estimate of Total Hourly Burden

Respondent	Number of Respondents	Number of Responses	Total Hours
Field Trial Pretesting Sub Total	122	242	102
Simulated School Pretesting Sub Total	245	485	101
Total Burden	367	727	203

Note: numbers have been rounded and therefore may affect totals

9. Cost to federal government

The total cost to federal government for this study is $74,819 as detailed in Table 4.

Table 4: Estimate of Costs

Activity	Provider		Estimated Cost
School Environment
Recruiting states, schools, and students		Westat		$2,029
Administering the study		Westat		$9,222
Simulated School Environment
Recruiting students and providing facilities for the study		EurekaFacts		$43,870
Administering the study		Westat		$19,698
Total				$74,819

10. Project Schedule

The schedule for this study, including all activities (recruitment, pretesting, data collection, results), will begin in November 2019, as soon as OMB approval is granted, and the study will conclude by January 2020.