Program for International Student Assessment 2022 (PISA 2022) Main Study Recruitment and Field Test
Supporting Statement Part B
OMB# 1850-0755 v.28
National Center for Education Statistics (NCES)
U.S. Department of Education
Institute of Education Sciences
Washington, DC
April 2023
revised July 2023
TABLE OF CONTENTS
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
B.2 Procedures for the Collection of Information 1
B.2a Statistical Methodology 1
B.3 Maximizing Response Rates 9
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
PISA 2025 assesses students nearing the “end of their compulsory school experience” and, as all prior administrations of PISA, is conducted in the United States by the National Center for Education Statistics (NCES) within the U.S. Department of Education. For international comparability, the target population is defined as students who are 15 years old, in grades 7 or higher. A range of exact birthdates is specified by the international coordinating committee based on the months in which the data will be collected. However, students must be between the ages of 15 years and 3 completed months and 16 years and 2 completed months at the beginning of the testing period. In the U.S., the universe for the selection of schools is all types of schools in the 50 states and the District of Columbia. Within sampled schools, students will be selected for participation by drawing a random sample among the 15-year-old students.
This section presents information on the PISA international standards and description for school and student sampling, recruitment, and data collection procedures for the PISA 2025 main study. Gaining schools’ and students’ cooperation in voluntary research is increasingly challenging and employing effective strategies for gaining the cooperation of schools is central to the data collection effort. PISA 2025 main study states, districts, and schools will be recruited beginning in October 2024 and data collection will be conducted from October-November 2025.
The Technical Standards for PISA 2025 main study established by the international governing board include the following:
Standard 6.10 The student sample size for the computer-based mode is a minimum of 6,300 assessed students, and 2,100 for additional adjudicated entities, or the entire PISA Defined Target Population where the PISA Defined Target Population is below 6,300 and 2,100 respectively. The student sample size of assessed students for the paper-based mode is a minimum of 5,250.
Standard 6.11 The school sample size needs to result in a minimum of 150 participating schools, and 50 participating schools for additional adjudicated entities, or all schools that have students in the PISA Defined Target Population where the number of schools with students in the PISA Defined Target Population is below 150 and 50 respectively. Countries not having at least 150 schools, but which have more students than the required minimum student sample size, can be permitted, if agreed upon, to take a smaller sample of schools while still ensuring enough sampled PISA students overall.
Standard 6.12 The minimum acceptable sample size in each school is 25 students per school (all students in the case of school with fewer than 25 eligible students enrolled).
Standard 6.13 The final weighted school response rate is at least 85 percent of sampled eligible and non-excluded schools. If a response rate is below 85 percent, then an acceptable response rate can still be achieved through agreed upon use of replacement schools.
Standard 6.14 The final weighted student response rate is at least 80 percent of all sampled students across responding schools.
In addition, NCES has a standard in which student response rate should be at least 85 percent, and the sampling design described below is based on that rate.
The design for this study will be self-weighting, stratified, consist of two stages, and will use probability proportional to size (PPS). There will be no oversampling of schools or students. Schools will be selected in the first stage with PPS and students will be sampled in the second stage yielding overall equal probabilities of selection.
The national PISA target population is 15-year-old students attending education institutions located within the U.S. in grades 7 and higher. The target population for any participating state is the same. The plan is to implement the main survey in the fall of 2025, with a field test in the spring of 2024. The specific definition of age eligibility that will be used in the survey is “…between 15 years and 3 (completed) months to 16 years and 2 (completed) months at the beginning of the testing window.”
The population of schools for PISA 2025 is defined as all schools containing any 15-year-olds in grades 7 through 12. As in previous PISA cycles, the school sampling frame for the PISA 2025 main study sample will be developed from the most up to date NCES Common Core of Data (CCD) and Private Schools Survey (PSS) datasets. For the PISA 2025 field test, we will select a purposive sample using the sampling frame prepared for the National Assessment of Educational Progress (NAEP) 2024, which uses the 2020-2021 CCD and the 2019-2020 PSS school data. We do not plan to minimize overlap for the PISA 2025 school sample
The grade structure of the school is a key stratification variable designed to reduce sampling error, but this is especially so in PISA because data analyses have shown that achievement is highly related to grade. Other stratification variables may include public/private, region of the country, location (urban/suburban/town/rural, etc.), enrollment by race/ethnicity, and state.
International standards do not require a formal probability sample of schools for the PISA field test. It is sufficient that the samples of schools be representative of a broad range of schools from across the U.S.. The national field test requires a minimum student sample of 2,000 students. In 2023, the U.S. will select a sample of 50 schools each with two substitute schools, with the expectation that 50 schools will ultimately participate, to provide for an adequate participating student sample. Among the 50 schools, 45 will be public schools and 5 will be private schools. This allows for school and student non-response and also for school level and within-school level exclusions.
To obtain a school sample that is broadly representative of schools across the U.S., we will target a convenience sample of schools with grade 9 and above and enrollment of at least 50 students in grades 9 and 10 (where most 15-year-olds are found) excluding schools with grades 7 and 8 only, small public schools, and schools sampled by NCES for other educational studies in 2024. We will use the sample stratification characteristics used in previous PISA cycles including census region, locality (city/urban fringe/town/rural MSA), school type (public/private), grade span, minority enrollment, and state. The sample will be a stratified systematic sample, with sampling probabilities proportional to measures of size, where the measure of size is the estimated number of 15-year-olds.
The Maple sampling software provided by the PISA International Consortium will be used to select the student sample in each school. For the field test, the target cluster size of students per school will be 52 students with the goal of assessing at least 45 students per school (after refusals and student ineligibility). For the PISA 2025 main study, we expect to keep the target cluster size of 52, the same as it was in PISA 2022.
Both in the field test and the main study, each school will have a school coordinator to provide assistance with arranging for the study and prepare a list of all eligible students in the school using a standardized Student Listing Form. Each completed list will be submitted to Westat and the information entered into the Maple sampling software. The collected data will be used only to select the student sample in each school. Once no further follow-up with sampled students is necessary, all student listing data will be destroyed.
The PISA field test has the following goals:
evaluation of the invariance of item parameters compared to previous PISA cycles for the 2025 cycle (CBA and PBA);
evaluation of the invariance of item characteristics (in preparation for adaptive testing) when delivered in fixed unit order vs. variable unit order;
estimation of initial item parameters to evaluate the quality of the new items in preparation for the selection of main study items and for the implementation of the adaptive design for science in the main study;
evaluation of sampling and survey operations aspects; and,
assessment of how well the computer platform functions within and across participants.
Cognitive Design. The PISA 2025 design is planned to follow closely that of PISA 2022, in that the field test will contribute information to construct a multistage adaptive testing (MSAT) design for science, as was done for reading in 2018 and mathematics in 2022. The reading and mathematics MSAT will be carried forward in PISA 2025.
For the PISA 2025 field test, there are a total of 69 forms in the field test containing a combination of clusters in trend science and mathematics MSAT, trend science and reading MSAT, trend science and new science, new science and Learning in a Digital World (LDW), or only new science. For the assessment, which will be administered in a 2-hour session, students will receive one form with a combination of clusters depending on the form. The forms are combined and organized in three distinct groups in order to ensure adequate coverage of newly developed items and to examine the psychometric properties of the items. There will be MSAT in 2025 for reading and mathematics as a minor domains. MSAT was successfully developed and administered for reading as a major domain in PISA 2018 and for mathematics in PISA 2022. As stated above, one of the goals to be met by the field test is collecting information in preparation of the planned introduction of MSAT for the major domain of science literacy, the use of a reduced reading and mathematics MSATs, and the use of previously used nonadaptive designs for the other minor and innovative domains. The field trial design will thus include variable unit positioning within clusters and will investigate the effects of variable unit positioning versus fixed positions in preparation for the main study, the hypothesis being that item parameter invariance is only supported when using intact clusters.
Cognitive items to be administered in the field trial consist of the following subjects and number of clusters (groups of items/units):
Science = 6 intact trend clusters and 12 new clusters;
Mathematics = a reduced version of the 2022 mathematics MSAT;
Reading = a reduced version of the 2018 reading MSAT;
LDW = 8 clusters of new items.
The field test assessment design utilizes 6 trend clusters and 12 new clusters of science items, a reduced version of the reading MSAT, a reduced version of the mathematics MSAT, and 8 new clusters for LDW. These clusters are organized in a rotation within three groups of students. Within a school, sampled students will be assigned to each of the three groups.
Group 1 will receive 4 trend clusters of combinations of science and mathematics, mathematics and reading, mathematics. These clusters will be fixed unit order. Group 1 is expected to yield 137 responses per science item and 80 responses per reading and mathematics item per participant.
The approach to reduce the distinction between major and minor domains has been supported by analytic approaches that utilize data from multiple cycles. Data collected during the major domain cycle provides a basis for the analyses of two subsequent minor domain cycles. In terms of the field test data analysis, Group 1 forms can be directly linked to the existing data from prior cycles, The variability in the psychometric characteristics of the PISA2025 field test data collected relative to the 2022 analysis results gives a baseline for the magnitude of error expected across data collections when moving to mathematics adaptive testing. The variability in Group 1, given fixed unit order within a cluster with full construct coverage is the lower bound that can be used to evaluate the variability of psychometric characteristics observed for Group 2 (variable unit order) and Group 3 (fixed unit order) of the new mathematics clusters. Also, the same variability will be referred to for comparisons of mathematics trend clusters of Group 1 (fixed unit order) and Group 2 (variable unit order).
Group 2 will receive new and trend items in science. These items will apply variable unit ordering within clusters. The design will provide variations in unit ordering within clusters and can be examined relative to Group 1. Each of the 24 Group 2 forms contains a combination of one to six trend science clusters and 3 of the 12 new science clusters. Every trend cluster will be paired with a new cluster once and appears once in each position. The design is expected to yield, per participant, 103 responses per trend item and 154 responses per new item.
Group 3 is based on a fixed order of units to provide a basis for comparisons with variable unit order forms administered in Group 2. There are two sets of forms in this group. The first set of 12 forms (Forms 43-54) is assembled from new Science clusters and each cluster is presented once in every position and paired with other clusters at least once. The second set of 10 forms (Forms 55-64) includes two new Science clusters and two new LDW clusters. Cluster positions are balanced for both domains. The third set of 5 forms (Forms 65-69) includes four new LDW clusters. Every form will be administered to 28 students; thus these two sets will be administered to a total of 757 students. This will yield an average of 159 responses per new Science item and 224 responses per new LDW item.
Learning in a Digital World Design. The LDW is a new innovative domain for PISA 2025. The assessment design will include LDW in the core PISA assessment and will assign students LDW items as part of the rotation of domains and items for the core assessment. The LDW will contain 2 clusters of interactive items. The PISA 2025 assessment will measure how well students can solve open problems and further their knowledge and understanding by engaging with technology in an autonomous learning process. In each 30-minute unit, students will work towards a learning objective in an open-ended, scaffolded, and interactive learning environment that provides them with computational tools and learning resources. The assessment is grounded in a social constructivist approach to learning that emphasizes the active and iterative process of discovery by interacting with external tools and resources – in this context, computational tools. Students will progress through increasingly complex tasks in which they will demonstrate the extent to which they can effectively combine what they already know and can do, with the learning opportunities afforded to them by technology.
Background Questionnaire Instruments. The questionnaires have been developed to address the questionnaire framework developed for PISA 2025. The framework defines 20 modules across the school and student questionnaires comprising student background characteristics, teaching and learning practices, school governance, and non-cognitive/metacognitive constructs dealing with science-related outcomes, attitudes, and motivational strategies. In addition, the questionnaires include items that have been included in multiple cycles of PISA, allowing the investigation of patterns and trends over time.
The school and student questionnaires, including these newly developed items, are presented in appendix C.
School questionnaire. The principal or designate from each participating school will be asked to provide information on basic demographics of the school population and more in-depth information on one or more specific issues (generally related to the content of the assessment in the major domain, which is science in 2025). Basic information to be collected includes data on school location; measures of socio-economic context of the schools’ student population, including location, school resources, facilities, and community resources; school size; staffing patterns; instructional practices; and school organization. The in-depth information is designed to address a very limited selection of issues that are of particular interest and that focus primarily on the major content domain, science. As in 2022, the Global Crisis items will also be included with revisions that aim at making the questions relevant beyond the COVID-19 pandemic and relevant to the experiences of the students in the year previous to test administration. They should help assess the readiness and resiliency of students, schools and education systems in responding to educational disruptions (e.g., school closures). For the field test, it is anticipated that the school questionnaire will take approximately 45 minutes. For the main study the questionnaire will remain at 45 minutes. The school questionnaire will be available to respondents online, as it has been since PISA 2015. The intent is that the school principal will respond to the questionnaire, but they may designate someone to complete the questionnaire. Principals, or their designate, access the questionnaire through a secure website with a username and password provided to them in their invitation (see Appendix A). Responding to the instrument is flexible. Respondents may break off and return to continue responding to the questionnaire by logging in again with their credentials. In addition to completing questions, respondents may review and change their responses to previously answered questions.
Student questionnaires. In the 2025 cycle, the U.S. will administer two student questionnaires that will be completed in a single student questionnaire session following the assessment session: the core student questionnaire and the information and communication technology (ICT) familiarity questionnaire. The content of these instruments is described below. Students access the questionnaires through the PISA assessment platform in the same way they access the assessment, approximately 15 minutes after the assessment session has ended. Unlike the school questionnaire, the student questionnaires are only administered from the student assessment platform and students only access the questionnaires in the questionnaire session. That is, student access to the questionnaires can only be accomplished during the in-school session; there is no access to the student questionnaire outside of school hours.
Student core questionnaire. Participating students will be asked to provide information pertaining primarily to the major assessment domain in 2025, science. Information to be collected includes demographics (e.g., age, gender, language, race, and ethnicity); socio-economic background of the student (e.g., parental education, economic background); student's education career; and access to educational resources and their use at home and at school, which have been standard questions in PISA since the earliest rounds. Domain-specific information will include instructional experiences and time spent in school, as perceived by the students, and student attitudes towards science. Attitudes toward science may be viewed as science identity, defined as a set of dispositions, agency, attitudes towards science and personal capital indicated by an interest in science and technology; valuing of scientific approaches to enquiry where appropriate, and a perception and awareness of environmental issues. Thus, a major new development included in this PISA 2025 Questionnaire Framework is a new module on science identity that includes new constructs such as science capital and environmental agency. As in the school questionnaire, the questions included in the Global Crisis Module are again included at the end of the core questionnaire
The goal is for the student questionnaire to take approximately 35 minutes to complete in the field test and the main study. As in PISA 2022, the PISA 2025 field test will implement a matrix sampling design where different respondents will receive different sets of items to reduce student burden while extending content coverage across relevant areas. This approach is viable for PISA 2025 due to the limited time available for the questionnaire and the large student sample size in large-scale assessments. This design is expected to be carried to the main study.
The approach being proposed for PISA 2025 will utilize an alternative matrix sampling design that rotates questions within constructs instead of across constructs. In the PISA 2025 proposed within-construct matrix sampling design, every student will receive questions on all constructs but only answer a subset of all questions for each construct, thus resulting in a complete database in terms of construct-level indices. This approach will be implemented for a select number of scales in the field test. A decision on the use of this design for the main study will be made based on an empirical evaluation of the PISA 2025 field test results. Following the Field Trial, a decision will be made about the feasibility of multiple partly overlapping forms for the main study, including consideration of a within-construct matrix sampling design.
Learning in a Digital World questionnaire which will collect information about students’ use of digital tools for learning – both inside and outside of the classroom – as well as their attitudes towards ICT and self-regulated learning in general. Modules in the school questionnaire will supplement this information, providing a description of environmental enablers of learning in the digital world, such as the school’s pedagogical culture, availability of supporting technology.
Information and Communication Technology (ICT) Familiarity questionnaire. The ICT questionnaire aims to examine students’ ICT activities and domain-specific attitudes including access to and use of ICT at home and at school, students’ attitudes towards and self-confidence in using computers, self-confidence in doing ICT tasks and activities; and navigation indices extracted from log-file data (number of pages visited, number of relevant pages visited). The ICT questionnaire for students is expected to take approximately 15 minutes to complete. The U.S. successfully administered the ICT questionnaire in the 2018 and 2022 cycles of PISA.
For the core computer-based assessment in mathematics, reading, and science, the international required minimum number of completed assessments is 6,300 students in 150 schools. In past PISA rounds up until 2018, the U.S. typically assessed between 5,600 and 5,900 students in 165 schools and sampled 42 students per school. However, as was already done in PISA 2018 and PISA 2022 main study, in PISA 2025, in order to achieve a larger number of students assessed required in the total assessed yield of 6,300 students, and to account for anticipated nonparticipation and student ineligibility, wherever possible, we will sample 52 students per school. Assuming a higher response level than in PISA 2022, the initial target is a total sample of about 288 schools, with estimated 256 schools eligible, to yield about 218 participating schools (assuming a total participation rate among schools of 85 percent, after replacement). As allowed under the international sampling standards, to achieve the target final school response rate, we will use replacement schools to complete the sample.
The student-per-school target for the core assessment is at least 42 completed student assessments per school. Assuming a within-school student assessment response rate of 85 percent (rates were 85 percent in 2000, 82 percent in 2003, 91 percent in 2006, 86 percent in 2009, 89 percent in 2012, 89 percent in 2015, 85 percent in 2018, and 80 percent in 2022) the original sample size of students within schools will be 52.1 In schools that do not have 52 PISA-eligible students, all eligible students will be sampled. Should any states participate in the 2025 assessment, each state would have a sample of 54 schools and 2,808 students to yield approximately 2,330 assessed students. As the main study plans for states and subnational jurisdictions are finalized, this information will be updated in the respondent burden table in the Supporting Statement Part A.
It is inevitable that nonresponse will occur at the school and student level. We will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, as required by NCES statistical standards. After the international contractor calculates weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification.
Based on recruitment knowledge gained in PISA 2018, PISA 2022, and other NCES studies, states and districts will be notified about the study before the sampled schools in their jurisdictions are contacted. School staff often wants to be assured that their school district knows about the study before they agree to participate. The planned PISA 2025 approach to state, district, and school recruitment is described in this section, and all of the respondent recruitment materials for the field test are provided in Appendix A-1 and B-1.
State Recruitment. For the field test in spring 2024, in October 2023, state education agencies (SEAs) in states that contain schools sampled for the field test will be mailed a package that includes the state letter and PISA 2025 field test advanced materials. We are working with the NAEP State Coordinators (NSCs) in this state recruitment effort. Some NSCs will send the state letter and personally follow-up to answer any questions; otherwise, Westat school recruiters will mail the package and follow-up with the districts and schools being recruited. The main study contact will begin in October 2025 and follow the same procedures.
District Recruitment. Also, in October 2023, shortly after the state mailing, advance packages to district superintendents will be mailed. Each package contains an introductory letter, including a list of sampled schools in the district’s jurisdiction, and the PISA 2025 field test advanced materials. The district mailings will come from the NSC or Westat, depending on each NSC’s preference. Shortly after the mailing, the district superintendent will be contacted by phone to inform him/her of the study, ensure they received the PISA 2025 field test package, and answer any questions they may have. Any issues with approaching schools in the district are also discussed at the time. The PISA 2025 main study will follow the same procedures with district contact beginning in January 2025, shortly after the state contact.
The PISA 2025 study staff responds to districts’ requirements such as research applications or meetings to provide more information about the study. If a district chooses not to participate, the recruiter documents all concerns listed by the district so that a refusal conversion strategy can be formulated. As in PISA 2018, PISA 2022, and previous NCES studies, some NSCs talk to district staff themselves, others mail the package but do not further contact districts, while still others do not want to be involved in district recruitment at all. In cases where the NSCs do not wish to follow-up, Westat’s recruiters will work directly with the districts.
Based on prior recruitment experience on a variety of NCES studies, some districts are designated as “special handling districts.” Contacting special handling districts begins with updating district information based on researching online sources, followed by calls to verify the information about where to send the completed required research application forms, and, if necessary, to collect contact information for this process. During the call, inquiries are also made about the amount of time the district spends reviewing similar research applications.
School Recruitment. After each district of sampled public schools has been informed of the study and has confirmed the receipt of the PISA 2025 field test package, the school mailings will be triggered. All of the school mailings, taking place from April through August of 2023, contain an introductory letter offering each sampled school a $250 check as a thank you for participation (to be mailed with a thank you letter after the end of the data collection); the PISA brochure and timeline; and school administrator and student Frequently Asked Questions (FAQs) sheets. Shortly after the mailing, the NSCs or Westat recruiters contact the school administrator to discuss the study, gain cooperation, and assign a school staff person to serve as the PISA 2025 field test school coordinator who will work with the PISA staff to manage the data collection in the school. The school coordinator will act as the liaison between study staff and their school.
In cases where recruitment proves more difficult, school recruiters consult with the PISA home office to evaluate a conversion plan for each school. Typically, the types of general issues are principals who are difficult to reach, school staff who are considering participation but not providing a final decision, principals and/or staff who express a concern and need follow-up, and principals who may require additional appreciation for agreeing to participate. The PISA 2025 main study contact of schools will follow the same procedures, with contact anticipated to begin in late February 2025 and continue through the fall of 2025.
School Coordinator Contact. Shortly after permission has been granted from the school administrator, Westat emails the school coordinator the MyPISA website registration information. The MyPISA website (described in more detail below) contains the e-filing instructions and templates for within-school student sampling, and other logistical information for the assessment. Each school will receive a unique ID that will support multiple school users. Each user must then provide their own contact information and set up their own unique account. This registration process has been used across all of NCES’ international studies and in NAEP.
For the field test, beginning in January 2024, school coordinators of participating schools will receive a handbook detailing the procedures for administering the PISA 2025 field test in the school, and providing timelines and instructions for submission of the list of students via MyPISA. Westat PISA staff will also call the school coordinator to discuss the PISA activities at the school, including when to begin constructing and e-filing the student list. School coordinators of participating schools in the main study will receive these materials beginning in August of 2025.
Student sampling. Beginning in January 2024 for the field test, student lists will be collected to draw a student sample in each school. The student lists are designed to collect students’ names, grade, gender, and month and year of birth. Edit checks will be run during sampling to check column mappings and completeness of data to ensure that all student listing forms are constructed properly for sampling (see Appendix A-1).
After student sampling has been completed, student tracking forms will be generated in the Maple software system and distributed to the school coordinator via MyPISA. These forms will also be used by the PISA field staff for administering the assessment and recording student participation status.
After students are sampled and school coordinators are notified of the sampled students, each school coordinator will notify and provide consent materials to parents of sampled students using the materials provided in Appendix B-1. Prior to the assessment, the test administrator assigned to each school will collect a copy of the notification and consent materials the school sent to parents as proof that the school took care of parental notification and consent. The process for the main study is the same as described for the field test. The main study student sampling will begin in August 2025, closer to start of the main study data collection in October-November 2025. The materials for the main study are presented in Appendices A-2 and B-2.
Role of the MyPISA Website. The central purpose of MyPISA is to provide a way for the schools to securely upload a list of students and to provide the tracking forms with the sampled students to the school coordinator and PISA field staff. MyPISA is also used as a source for disseminating information about PISA, such as providing copies of advanced materials and descriptions of survey activities, school actions, and instructions and templates for submitting lists. School registrants are encouraged to update their contact information and access information about the study in MyPISA.
PISA 2025 field test data collection will occur in March – April 2024. Prior to data collection, Westat PISA staff will complete various pre-survey activities.
Pre-survey Activities
In January 2024, the school coordinators will receive a handbook and instructions for assembling a student list. The lists will be submitted to Westat via MyPISA and the samples will be drawn during August and September 2021. Beginning in January 2024, school coordinators will be asked to do the following activities:
Create and submit a list of eligible students via MyPISA;
Receive and distribute to sampled study materials from the PISA Home Office;
Encourage the principal to complete their questionnaire; and
Meet with the PISA test administrator to review assessment logistics and hold a brief meeting with the sampled students to show a PISA video presentation explaining the study and the students’ role and contribution and to answer questions about PISA. Depending on the situation with the coronavirus pandemic, this meeting may be virtual facilitated by the school coordinator via Google Meet or another electronic meeting platform used by the school. In the event that Westat field staff cannot hold a student meeting, the school coordinator may hold the meeting. This has sometimes occurred in past PISA rounds due to scheduling issues and travel.
Data Collection
For both the field test and main study, the school questionnaire will be available electronically. For the main study, a hard copy available upon request. The principal will be given links to their questionnaires as part of invitation to PISA and will also be emailed their personalized link to the questionnaire. The student questionnaire will be administered as part of the assessment platform provided to countries by the International Consortium. The assessment platform includes the PISA core assessment and the student questionnaires. The field test PISA questionnaires are provided in Appendix C.
The PISA assessments are administered to students by trained PISA test administrators hired and trained by Westat. The PISA test administrators will bring all assessment equipment to the school including student tablets and peripheral equipment (power cords, additional extension cords, routers). They are responsible for set-up and breakdown of equipment and administration of the assessment to the students. All that is required from the school is an adequate space to set up the equipment and hold the assessment.
Students begin the data collection activities by entering a room containing desks and PISA tablets. Upon entering, each student is directed to their assigned tablet by a PISA test administrator. On the tablet is a paper log-in form (see Appendix A-1, A-2, p. 32) with that student’s unique log-in information. This form also has the OMB statement printed below the log-in information2. The first screen that students see is the assessment login screen where the student enters the student assessment platform to begin PISA student data collection activities. The PISA test administrator gives the students the verbal instruction to enter their log-in information to begin. Students complete the cognitive assessment, take a short break (about 15 minutes), and return to complete the student questionnaires which end the student data collection.
Throughout the data collection period, PISA staff and the school coordinator will monitor the return of school questionnaires and, working in conjunction, will follow-up with non-responders as needed. A ‘window-is-closing’ non-response follow-up effort will be utilized, gently reminding principals to complete their questionnaires. Near the end of data collection, the two reminders before the last are designed to establish a deadline effect and will be followed with an extension email. This campaign-style approach is designed to provide soft reminders across the data collection window, while creating a sense of urgency to respond towards the end.
School coordinator incentive checks will be distributed after Westat receives the student list and completes within-school student sampling. School incentive checks will be distributed after the assessment is completed. Incentives will be mailed to schools on a weekly basis throughout the data collection period. Student incentives will be distributed to the students at the end of the questionnaire session.
Our approach to maximizing school and student response rates in the main study includes the following:
Use of a fall test administration in the main study, to avoid major conflicts with state testing;
Selecting and notifying schools several months in advance;
Communicating with state and district officials early in the process and applying a more proactive approach with states by coordinating with NAEP State Coordinators to gain assistance with sampled schools;
Assigning personal recruiters for specific schools and use of personal visits to districts and/or schools early in the contact process;
Monetary incentives for schools, school coordinators, and students (see Section A.9) and below;
School report incentive for schools;
Volunteer service certificate of 4 hours from the U.S. Department of Education incentives for students;
Contact with schools and school coordinators at set intervals throughout the year preceding the assessment;
Use of an informational video about PISA 2025 to motivate student participation and full effort during the assessments; and
Use of individually tailored refusal conversion strategies to encourage participation (as detailed above in the section entitled “Respondent Recruitment”).
Our approach to gaining cooperation from respondents is multipronged and takes into consideration feedback and advice that NCES has received over the years. Beyond monetary incentives, we offer feedback to the school (in the form of the school report) and service credit for student participants (in the form of volunteer service time). In addition, we try to engage districts and schools in the study by informing them of findings and results from previous rounds in the form of short videos and focused findings offered by OECD and NCES.
PISA 2025 main study schools that meet the criteria for receiving a school report (see section A.9 of Supporting Statement Part A), will be provided school-level PISA 2025 results. While individual-level scores cannot be produced from PISA data, a school level report showing comparative data for the school can be produced when the school has a participation rate of 85 percent or better and at least 10 assessed students. The results in the school-level report will be comparative results that do not provide actual school scores, but rather indicate how the school performed compared to country averages and to other US schools with similar demographic characteristics. For PISA 2025, we are also attempting to design a second, alternate report to provide information from the contextual questionnaires for schools that do not meet the requirements for receiving the standard school report. This alternate report version will need to be reviewed and approved by the NCES chief statistician prior to implementation.
These approaches are based on recommendations from an NCES panel, experience with previous PISA administrations, as well as extensive discussions with NCES’ chief statistician.
Participation in the field test is an international requirement for participating in the PISA 2025 main study. The main focus of the field test is to collect enough assessment data to perform reliable tests of the items, to evaluate newly developed assessment items and new or revised multi-stage adaptive test designs, and to test the survey operations. The field test is also used to evaluate recruitment, data collection, and data management procedures in preparation for the main study, including recruitment methods for obtaining school and student participation. The results of the field test are analyzed by OECD. In the U.S., NCES uses the field test results to: (a) determine the final main study design and decide in which international options the U.S. will participate, (b) improve recruiting strategies and materials for the main study in the U.S., and (c) finalize all study procedures.
Many people at OECD, ACER, and other organizations around the world have been involved in the design of PISA. Some of the lead people are listed in section A8. Overall direction for PISA conducted in the U.S. is provided by Dr. Samantha Burg, the PISA National Project Manager, and other staff at NCES.
1 For the national main study sample, we expect to draw an initial sample of 298 schools. Taking into account closed, merged, and ineligible schools (historically, around 14% of sampled schools), as well as the historical school-level response rate, we anticipate interacting with/recruiting about 256 of these schools, of which, we estimate, 218 will participate in the PISA 2025 main study. To obtain the required number of students, we will ask to sample up to 52 students in each school. However, some of the smaller schools will not have 52 students available. We estimate: 218 schools x 52 students sampled x 0.945 (a factor used to account for variations in student population sizes across the schools) = 10,713 starting student sample size that we will work to recruit. Based on historical student assessment rates (~80%), we estimate that, in the end, we will assess about 8,570 students (10,713 x 0.80), which will assure that we meet the minimum required 6,300 assessed students.
2 Originally, we planned to include the OMB statement, including assurances that the data collection was voluntary, the OMB number and expiration date, and the paperwork burden statement, on the assessment platform log-in screens. We have since learned that those screens are uneditable, making this the best way to make sure that students are fully informed.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | PREFACE |
| Author | Janice Bratcher |
| File Modified | 0000-00-00 |
| File Created | 2023-08-01 |