PROGRAM FOR INTERNATIONAL STUDENT ASSESSMENT 2018
(PISA 2018) Main Study
OMB# 1850-0755 v.21
SUPPORTING STATEMENT PART A
Submitted by:
National Center for Education Statistics (NCES)
U.S. Department of Education
Institute of Education Sciences
Washington, DC
February 2018
revised March 2018
TABLE OF CONTENTS
A.1 Importance of Information 3
A.2 Purposes and Uses of Data 3
A.3 Improved Information Technology (Reduction of Burden) 5
A.4 Efforts to Identify Duplication 6
A.5 Minimizing Burden for Small Institutions 7
A.6 Frequency of Data Collection 7
A.8 Consultations Outside NCES 7
A.9 Payments or Gifts to Respondents 8
A.10 Assurance of Confidentiality 10
A.13 Total Annual Cost Burden 13
A.14 Annualized Cost to Federal Government 14
A.15 Program Changes or Adjustments 14
A.16 Plans for Tabulation and Publication 15
A.17 Display OMB Expiration Date 16
A.18 Exceptions to Certification Statement 16
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
B.1 Respondent Universe
B.2 Procedures for the Collection of Information
B.3 Maximizing Response Rates
B.4 Purpose of Main Study and Data Uses
B.5 Individuals Consulted on Study Design
Appendix A: Recruitment Materials
Appendix B: Parental Contact Materials
Appendix C: Questionnaires
The Program for International Student Assessments (PISA) is an international assessment of 15-year-olds which focuses on assessing students’ reading, mathematics, and science literacy. PISA was first administered in 2000 and is conducted every three years. The seventh cycle of the study, PISA 2018, is being administered at a time when interest is increasing, both worldwide and in the United States (U.S.), in how well schools are preparing students to meet the challenges of the future, and how the students perform compared with their peers in other education systems of the world. Approximately 80 education systems, including the U.S., are expected to participate in 2018. The U.S. has participated in all previous cycles and is participating in 2018 in order to track trends and to compare the performance of U.S. students with that in other education systems.
PISA 2018 is sponsored by the Organization for Economic Cooperation and Development (OECD). In the U.S., PISA 2018 is conducted by the National Center for Education Statistics (NCES) of the Institute of Education Sciences (IES), U.S. Department of Education. PISA is a collaboration among the participating countries, the OECD, and a group of international organizations each under contract to the OECD (hereafter referred to as the PISA International Consortium), including the Educational Testing Service (ETS), Westat, the German Institute for Educational Research (DIPF), and Pearson.
In each administration of PISA, one of the subject areas (reading, mathematics, or science literacy) is the major domain and has the broadest content coverage, while the other two subjects are the minor domains. Reading literacy will be the major domain in PISA 2018. Other areas may also be assessed, such as, in the case of PISA 2018, financial literacy. PISA assesses students’ knowledge and skills gained both in and out of school environments. The focus on the “yield” of education in and out of school makes it different from other international assessments such as the Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PIRLS), which are closely tied to school curriculum frameworks and assess younger and grade-based populations.
Like PISA 2015, in PISA 2018 the entire assessment and the questionnaires will be administered on computer. While it is possible for countries to continue using paper-based instruments, and some countries are choosing to do so, those instruments will not include new items. The U.S. will administer PISA 2018 on computer. In addition to the cognitive assessments, PISA 2018 will include questionnaires administered to assessed students, school principals, and teachers. The school and teacher questionnaires will be delivered online. The school and student questionnaires are core components of PISA and as such are required for countries, while the teacher questionnaire is optional for countries. The U.S. will implement all three questionnaires in 2018.
To prepare for the main study in 2018, PISA countries conducted a field test in the spring of 2017, primarily to evaluate newly developed assessment and questionnaire items but also to test the assessment operations. The U.S. PISA field test data collection occurred from April-May 2017 and the main study data collection will occur October-November 2018. In order to meet the international data collection schedule for the fall 2018 main study, questionnaires must be finalized by September 2018.
Previously approved recruitment materials, including letters to state and district officials and school principals, and text for a PISA brochure, summary of activities, and “Frequently Asked Questions” are included in Appendix A. New contact materials for teachers, principals, school coordinators, and students have been added to Appendix A. Previously approved parental consent letters and related materials for the main study are included in Appendix B. The final international versions of the questionnaire items with adaptations to these items for use in the U.S. are provided in Appendix C.
OMB approved the initial phase of this collection in September 2016 and, most recently, in September 2017, approved subsequent revisions to planned main study recruitment materials (OMB# 1850-0755 v.18-20). With those submissions, NCES adequately justified the need for and overall practical utility of the main study as proposed and an overarching plan for the phases of the data collection over the next 3 years, and provided as much detail on the measures to be used as was available at the time of the submission. Thus OMB approved all aspects of the initial phase of this collection, and now NCES published a notice in the Federal Register allowing a 30-day public comment period on the details of the subsequent phase of the PISA 2018 main study collection described in this submission. This request is to conduct the PISA 2018 main study.
As part of a continuing cycle of international studies, the U.S., through NCES, participates in several international education assessments and surveys. PISA, sponsored by OECD, is one of these studies. In light of the growing concerns related to international economic competitiveness, the changing face of our workplace, and the expanding international marketplace in which we trade, knowing how our students and adults compare with their peers around the world has become an even more prominent issue than ever before. Nationwide, interest in understanding what other nations are doing to further the educational achievement of their populations has increased beyond simple comparisons. Data at critical points during the education career of U.S. students, such as that collected through PISA, have been used by policymakers in efforts to guide and examine the American education system. Consequently, generating comparative data about students in school, at the end of schooling, and about adults in workplace and in community has become an important focus for NCES.
PISA measures students' knowledge, skills, and competencies primarily in three subject areas – reading, mathematics, and science literacy. The overall strategy is to collect in-depth information on student capabilities in one of these three domains every 3 years so that detailed information on each becomes available every nine years. During each 3-year survey cycle, the major focus is on one content domain, with a minor focus on the other two content domains. The major focus for the data collection in 2018 is on reading literacy, with a minor focus on mathematics and science. As in the 2012 and 2105 rounds of PISA, the U.S. will also administer the optional financial literacy assessment.
The results from PISA assessments, published every 3 years along with other related indicators, allow national policymakers to compare the performance of their education systems with those of other countries and provide a basis for monitoring the effectiveness of education systems at the national level. Without these kinds of data, U.S. policymakers will be limited in their ability to gain insight into the educational performance and practices of other nations as they compare to the United States. NCES provides extensive information to the public on PISA through its publications and its website (http://nces.ed.gov/surveys/pisa).
Governments and the general public want solid evidence of education outcomes. In the late 1990s, the OECD launched an extensive program for producing policy-oriented and internationally comparable indicators of student achievement on a regular basis and in a timely manner. PISA is at the heart of this program. How well are schools preparing students to meet the challenges of the future? Parents, students, the public, and those who run education systems need to know whether children are acquiring the necessary skills and knowledge, whether they are prepared to become tomorrow's workers, to continue learning throughout life, to analyze, to reason, and to communicate ideas effectively.
The results of OECD’s PISA, published every 3 years (with more detailed measures of each of the three major subject domains every 9 years) along with related indicators, allow national policymakers to compare the performance of their education systems with those of other countries, and to analyze the relationship between constructs measured through the PISA questionnaires with assessment results at national and international levels. Through PISA, the OECD and NCES produce three types of indicators:
Basic indicators that provide a baseline profile of the knowledge, skills, and competencies of students;
Contextual indicators that show how such skills relate to important demographic, social, economic, and education variables; and
Trend indicators that emerge from the ongoing, cyclical nature of the data collection.
PISA 2018 Components
The primary focus for the assessment and questionnaires for PISA 2018 will be on reading literacy. The PISA reading framework defines reading literacy as an individual’s:
“understanding, using, evaluating, reflecting on and engaging with texts in order to achieve one’s goals, to develop one’s knowledge and potential and to participate in society”
As in all administrations of PISA, mathematics and science literacy also will be assessed, although they will be “minor domains” in 2018. The instruments to be administered in 2018 are as follows:
Assessment Instruments: the cognitive assessment is computer-based and each student receives four clusters of assessment items (approximately 30 minutes each) in 2 or 3 of the domains. Every participating student will take at least one cluster of reading because reading literacy is the primary domain for 2018. Students will be presented with a randomly assigned form of the PISA assessment with a combination of clusters depending on the form. Based on the results of the field test, the main study will implement a multi-stage adaptive testing (MAT) design, albeit in a conservative fashion. The MAT model will be applied only to the reading domain. Reading clusters will use a variable unit order design based on favorable results from the field test, which examined fixed unit order versus the variable unit ordering of clusters.
Financial literacy (FL), an international option in which the U.S. has elected to participate, will be administered to a separate sample of approximately 12 students per school, selected during the within-school sampling stage, similar to the PISA 2012 design, in which a core sample was chosen followed by a FL sample. Students sampled for FL will receive 2 clusters of FL items plus a cluster each of mathematics and reading items. Students sampled for core PISA and those sampled for FL will be assessed in the same session as only the assessment content differs in terms of the clusters and domain-specific material they receive.
The field test evaluated the cognitive and non-cognitive items for bias and interpretation issues, following standard protocols. For the main study, the pool of items was reduced to only include the items that demonstrated validity across the participating education systems and met the goals of content coverage to adequately measure the framework and provide the desired distribution of item types.
Background Questionnaire Instruments: Every participating country must implement two core background questionnaires for PISA 2018: school and student. Several optional questionnaires are also available, of which the U.S. will implement two: a teacher questionnaire and an additional student questionnaire on Information and Communication Technology (ICT) familiarity. These instruments have been developed to address the PISA 2018 questionnaire framework, which defines 14 modules across the school, student, and teacher questionnaires comprising student background characteristics, teaching and learning practices, professional development of teachers, school governance, and non-cognitive/metacognitive constructs dealing with reading-related outcomes, attitudes, and motivational strategies. In addition, the questionnaires include items that have been administered in multiple cycles of PISA, allowing the investigation of patterns and trends over time. Countries adapt the questions to fit their national context and the questionnaires are reviewed and verified to ensure they remain comparable across countries.
The principal (or his or her designee) from each participating school will be asked to provide information on basic demographics of the school population and more in-depth information on one or more specific issues (generally related to the content of the assessment in the major domain – reading). Basic information to be collected includes data on school location; measures of socio-economic context of the school, including location, school resources, facilities, and community resources; school size; staffing patterns; instructional practices; and school organization. The school questionnaire will take 45 minutes on average to complete, and will be available to respondents online.
The teacher questionnaire, which was also implemented in 2015, gathers school-level contextual information about the structural and process characteristics of schools (e.g. teaching practices and learning opportunities in classrooms, leadership and school policies for professional development, vertical and horizontal differentiation of the school system) and will be analyzed alongside data received through the school questionnaire to provide a context for the student achievement scores.
Participating students will be asked to provide information pertaining primarily to the major assessment domain, reading, and about their demographics (e.g., age, gender, language, race, and ethnicity); socio-economic background of the student (e.g., parental education, economic background); student's education career; and access to educational resources and their use at home and at school. Domain-specific information will include instructional experiences and time spent in school, as perceived by the students, and student attitudes towards reading. The student questionnaire is designed to take approximately 30 minutes to complete in the main study.
The FL questionnaire will be administered to all students, regardless of whether they are sampled for the FL assessment or not. The FL questionnaire examines students’ experiences with money matters, such as having savings accounts, debit or prepaid cards, as well as whether they have experienced financial-related lessons in their school careers. Many of the items in the FL questionnaire were previously administered in 2012 and 2015, with a small number of new items being piloted in the field trial. The FL questionnaire for students is designed to take no more than 15 minutes to complete.
The ICT questionnaire, in turn, aims to examine students’ ICT activities and domain-specific attitudes including access to and use of ICT at home and at school, students’ attitudes towards and self-confidence in using computers, self-confidence in doing ICT tasks and activities; and navigation indices extracted from log-file data (number of pages visited, number of relevant pages visited). The U.S. elected to administer the ICT questionnaire to better understand whether students’ familiarity with and use of ICT may be related to their performance on PISA, especially now that PISA is delivered via computer. The ICT questionnaire for students is designed to take no more than 15 minutes to complete.
The core student questionnaire, FL, and ICT questionnaire will be computer-based and delivered to students via a thumb-drive. The school and teacher questionnaires will be administered online, though hard copy versions will also be made available to those who make the request.
The PISA 2018 design and procedures are prescribed internationally. The school and teacher questionnaires will be available on-line as the main mode of administration. Student data collection will consist of online questionnaires and computer-based responses for reading, mathematics, science, and financial literacy, thus responses to the assessments and questionnaires will be captured electronically. In the U.S., the assessment and student questionnaire will be implemented using laptops carried into schools by the data collection staff. This greatly reduces the burden on schools and staff by eliminating the need to use school-based equipment and computer labs. Online questionnaires and carry-in laptop data collection was successfully used in the 2015 cycle.
A communication website, MyPISA.us, will be used during the 2018 main study in order to provide a simple, single source of information to engage sampled schools and maintain high levels of their involvement. This secure portal will be used throughout the assessment by school coordinators to perform their tasks and to provide them with easy access to information tailored for their anticipated needs. We plan to gather student and teacher lists from participating schools electronically using an adaptation of Westat’s secure E-filing process through the MyPISA.us portal. E-filing is an electronic system for submitting lists of student and teacher information, including limited background information in school records. Instructions to school coordinators on how to submit student and teacher lists are included in Appendix A. E-filing has been used successfully in the National Assessment of Educational Progress (NAEP) for more than 10 years, and was used in TIMSS 2015 and the PISA 2012 and 2015 assessments. The E-filing system provides advantageous features such as efficiency and data quality checks.
A number of international comparative studies already exist to measure achievement in science, mathematics, and reading, including TIMSS and PIRLS. The Program for the International Assessment of Adult Competencies (PIAAC), administered in 2012, measures the reading literacy, numeracy, and problem-solving skills of adults. In addition, the U.S. has been conducting its own national surveys of student achievement for more than 40 years through the NAEP program. PISA differs from these studies in several important ways:
Content. PISA is designed to measure “literacy” broadly, while other studies, such as TIMSS and NAEP, have a strong link to curriculum frameworks and seek to measure students’ mastery of specific knowledge, skills, and concepts taught in schools. The content of PISA is drawn from broad content areas, such as understanding, using, and reflecting on written information for reading, in contrast to more specific curriculum-based content such as decoding and literal comprehension. Moreover, PISA differs from other assessments in the tasks that students are asked to do. PISA focuses on assessing students’ knowledge and skills in science, reading, and mathematics literacy in the context of everyday situations. That is, PISA emphasizes the application of knowledge to everyday situations by asking students to perform tasks that involve interpretation of real-world materials as much as possible. A study based on expert panelists’ reviews of mathematics and science items from PISA, TIMSS, and NAEP reported that PISA items required multi-step reasoning more often than either TIMSS or NAEP.1 The study also showed that PISA mathematics and science literacy items often involved the interpretation of charts and graphs or other “real world” material. These tasks reflect the underlying assumption of PISA: as 15-year-olds begin to make the transition to adult life, they need to know not only how to read, or know particular mathematical formulas or scientific concepts, but also how to apply this knowledge and these skills in the many different situations they will encounter in their lives. The computer-based assessments add additional “real world” tasks, given the predominance of technology in young adults’ lives and workplaces.
Age-based sample. The goal of PISA is to represent outcomes of learning rather than outcomes of schooling. By placing the emphasis on age, PISA intends to show not only what 15-year-olds have learned in school, but also outside of school and over the years, not just in a particular grade. In contrast, NAEP, TIMSS, and PIRLS are all grade-based samples: NAEP assesses students in grade 4, 8, and 12; TIMSS assesses students in grades 4 and 8 (and, occasionally, grade 12); and PIRLS assesses students in grade 4. PISA thus seeks to show the overall yield of an education system and the cumulative effects of all learning experiences. Focusing on students at age 15 provides an opportunity to measure broad learning outcomes while all students are still required to be in school across the many participating nations. Finally, because years of education vary among countries, choosing an age-based sample makes comparisons across countries somewhat easier than a grade-based sample.
Information collected. The kind of information PISA collects also reflects a policy purpose slightly different from the other assessments. PISA collects only background information related to general school context and student demographics. This differs from other international studies such as TIMSS, which collects background information related to how teachers in different countries approach the task of teaching and how the approved curriculum is implemented in the classroom. The results of PISA will certainly inform education policy and spur further investigation into differences within and between countries, but PISA is not intended to provide direct information about improving instructional practice in the classroom. The purpose of PISA is to generate useful indicators to benchmark performance more broadly and inform education policy.
Alternate sources for these data do not exist. This study represents the U.S. participation in an international study involving approximately 80 countries and jurisdictions in the PISA 2018 field test in spring of 2017 and the main study in fall of 2018. The U.S. must collect the same information, using the same instruments and procedures, at the same time as the other nations for purposes of making valid and meaningful international comparisons. No other study in the U.S. will be using the instruments developed by the OECD, and thus no alternative sources of comparable data are available.
No small entities are part of this sample. The school sample for PISA was drawn in July 2017 and contains small-, medium-, and large-size schools from a wide range of school types, including private schools, and burden will be minimized wherever possible for all institutions participating in the data collection. For example, the selection of schools to be assessed in the PISA 2018 main study avoids overlap with the selection of schools for the Teaching and Learning International Survey (TALIS – an international survey of teachers and their professional lives), which will be in the field in the spring of 2018. Schools included in the PISA 2018 field test in the U.S. had a low likelihood of being included in the PISA 2018 main study. Student burden will be reduced through the use of multiple forms of the assessment, allowing PISA to administer a large number of assessment items while not requiring students to take all items, and allowing good coverage of the domains and efficient and reasonable time for the assessment. In addition, contractor staff will undertake all test administration and assist with parental notification, sampling, and all other PISA 2018 related tasks as much as possible within each school.
PISA is conducted on a 3-year cycle as prescribed by the OECD, and adherence to this schedule is necessary to establish consistency in survey operations among the many participating countries.
The special circumstances identified in the Instructions for Supporting Statement do not apply to this study.
Consultations outside NCES have been extensive and will continue throughout the life of the project. The nature of the study requires this, because international studies typically are developed as a cooperative enterprise involving all participating countries. PISA 2018 is being developed and operated under the auspices of the OECD by a consortium of organizations. Key persons from these organizations who are involved in the design, development and operation of PISA 2018 are listed below.
Organization for Economic Cooperation and Development
Andreas Schleicher, Indicators and Analysis Division
2, rue André Pascal, 75775 Paris Cedex16, FRANCE, Tel: +33 (1) 4524 9366, Fax: +33 (1) 4524 9098
Educational Testing Service
Irwin Kirsch, Project Director, ETS Corporate Headquarters
660 Rosedale Road, Princeton, NJ 08541 USA, Tel: 1-609-921-9000, Fax: 1-609-734-5410
Westat
Keith Rust, Director of Sampling
1600 Research Boulevard, Rockville, Maryland 20850-3129 USA, Tel: 301 251 8278, Fax: 301 294 2034
Currently, the minimum response rate targets required by the OECD are 65 percent of original schools and 80 percent of students, while the NCES statistical standards require a minimum response rate target of 85 percent at the student level. Historically, these high response rates have been difficult to achieve in school-based studies. The U.S. failed to reach the school response rate targets for the study in all previous PISA administrations (2000, 2003, 2006, 2009, 2012, and 2015) and had to adjust incentives upwards in the middle of the recruitment and data collection period in order to meet minimum response rate requirements. Gaining sufficient student cooperation is also challenging. The U.S. has historically met the NCES target rate of 85 percent of students responding; however, this takes a great deal of effort. Student response rates exceeded the NCES requirement in PISA 2006 by 6 percent, in PISA 2009 by 2 percent, and in PISA 2012 by 4 percent. Unweighted results from 2015—the most recent round of data collection--suggest that the U.S. student response rate is 89 percent, as it was in 2012. The monetary incentives, particularly for school coordinators, had an impact in maintaining the student response rates. School coordinators indicated that the incentives were meaningful to them as well as to the students. Field staff reiterated this as well, reporting what they heard from school coordinators and students.
We are using a multi-pronged approach to address the challenge of gaining school and student cooperation and learn as much as possible during the main study about how to achieve acceptable participation rates. First, our PISA contractor reviewed the most recent PISA 2015 and PISA 2018 field test experience to understand where possible improvements could be made in materials and communication with schools. Staff with experience working on NAEP, PISA, other international assessments, other large-scale data collections, and with expertise in effective approaches to school recruitment have provided input towards identifying strategies for achieving high response rates and continue to serve as an ongoing source of ideas and feedback. Additionally, PISA 2018 continues the use of effective incentives. The proposed amounts are described below and are based on the amounts used in PISA 2018 field test (please also see Part B, section B.3).
In the rare cases where state or school district laws or labor contracts do not allow school staff to receive incentives for participating in PISA, the school or school district will be offered the total amount of incentives that would have otherwise been distributed to the individual respondents. NCES and its contractor will work with schools to determine when this option will need to be implemented.
Schools. Schools participating in PISA will receive $250. In order to meet the minimum school response rates mandated by the PISA international governing board, and to thank each school for accommodating the disruption, we believe it is necessary to offer schools this incentive to encourage participation.
Although the field test was successful, we anticipate difficulty in reaching the required school response rates in the main study, as has been the case in all past administrations of PISA. Although the field test had more schools participate than required, and the disposition of the schools was positive toward PISA, the recruitment of these schools did not follow the model that will be used in the main study because we were not required to build a response rate for original schools. In the field test, we had to obtain participation from an adequate number of schools to get the required number of student responses for the test items. Therefore, we pooled our schools (both originally sampled schools and their replacement schools), approached the original schools first, and moved to the replacement schools quickly with very little, if any, conversion effort. This will not be the case in the main study where we must pursue the original schools until we obtain a satisfactory rate of participation. The historical experience for the U.S. is that obtaining a sufficient response rate of 65 percent of originally sampled, eligible schools is difficult and has required additional efforts. Moreover, in the field test we did not go to schools in states where we have traditionally had difficulty gaining school cooperation, but must do so in the main study. Finally, we are learning from our recruitment efforts for other NCES studies that the current climate regarding voluntary assessments is even more challenging this year than in the past years and we are concerned that we may face even greater challenges than in the past administrations of PISA.
Given these anticipated difficulties in securing sufficient school participation, we propose a second-tier of incentive, also approved and used in the PISA 2012 and 2015 main studies, which will allow us to offer to schools, when necessary, an $800 participation incentive, instead of the standard $250 (this amount was tested in the incentive experiment conducted as part of the PISA 2012 field test and then used in the PISA 2012 and 2015 main studies). This second tier will not be initiated until near the end of the current academic year, in June 2018, after we have approached all original schools and had an opportunity to try different conversion efforts, such as addressing the specific concerns of refusing schools and making personal visits to schools to discuss the study face to face. If, by that time, we have not reached a participation rate of at least 68 percent of the original schools, we will implement the higher incentive rate (to meet the 65 percent of original schools minimum requirement we need to recruit at least 68 percent of original schools factoring in a 3 percent attrition of schools over the summer months before data collection begins in fall 2018). We will approach refusing schools with the second-tier incentive only if necessary and at the point of our last chance to convert them.
In addition to a monetary incentive, participating schools will be offered a school-level report with basic comparisons of the performance means of students in the school with overall means for the U.S., OECD countries, and other similar schools as measured in PISA 2018. Working with the NCES Chief Statistician, we have devised a set of sample and response rate requirements for a participating school to receive a report (schools will be grouped into 6 categories based on sample size and response rates). Failure to achieve the designated sample and response rate requirements will mean that a school will not be eligible to receive a school report with comparative achievement results. We are also designing an alternate version that would report the non-cognitive questionnaire data, showing profiles of the school across a variety of variables, compared to other similar schools (for additional details see Part B, section B.3). This would allow us to provide a source of feedback to potentially all schools, if they request it. We will also directly ask school principals during recruitment if there is any reason they do not want to receive a report. In these rare cases, a school report will not be produced. School-level reports will be shared only with the principal of the school, and will not be shared or distributed to anyone else. Prior to distribution, the school reports will be reviewed by the NCES Chief Statistician for accuracy and compliance with the sample and response rate requirements designated for PISA 2018.
Additionally, for the last several cycles of PISA, we have offered each sampled school the opportunity to send one delegate to a summer training workshop, held in Washington, DC, where details on the PISA study, including the procedures around submitting needed school-, teacher-, and student-level information are discussed. The school principal may send anyone s/he chooses, but we strongly encourage them to send the school coordinator who benefits most from the training workshop (please also see Part B, section B.3). The summer training for schools is offered in the PISA 2018 main study recruitment materials (Appendix A).
School coordinators. The school coordinator will be offered $200. The role of the school coordinator is critical for the success of the study. The coordinator is expected to coordinate logistics with the data collection contractor; supply a list of eligible students and teachers for sampling to the data collection contractor; communicate with teachers, students, and parents about the study to encourage participation; assist the test administrator in ensuring that the sampled students attend the testing sessions; and assist the test administrator in arranging for make-up sessions as needed.
Teachers: Up to 25 teachers will be selected from each school to take an approximately 30 minute online teacher questionnaire. As in TALIS, selected teachers will be offered $25 for completing the questionnaire.
Students. The student burden in PISA 2018 will be the same as in previous rounds of PISA, in 2012 and 2015 and, as in previous data collection cycles, all participating students will be offered $25. Unlike PISA 2012 and 2015, where students were subsampled for additional, optional assessments and assessed in an additional session after the core assessment, the assessment design for PISA 2018 will facilitate the assessment of all students in a single, core assessment session. This means that the added incentive offered to students attending the second session in 2012 and 2015 will not be necessary in 2018.
Students participating in the assessment during non-school hours (after school or on a Saturday), which is an accommodation offered in the main study when it is not possible to find a suitable time within school hours, and one that is exercised rarely, will be offered $35. The increased incentive amount is designed to thank students for travelling to the assessment site and potentially missing outside of school activities (e.g., work, sports) in order to participate in the assessment outside of school hours. Incentives for students will only be provided with the explicit permission of the school principal.
All student incentives will be offered directly to the students. Parents will be informed of the amount of the payment the students will receive in the consent form/letter in advance of the assessment. The payments will be provided as a personal check, as has been successfully done since PISA 2009.
All contractor and subcontractor PISA 2018 staff also undergo personnel training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; controlled and protected access to computer files under the control of a single data base manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and operator-manned in-house computing facility.
The laws pertaining to the use of personally identifiable information are clearly communicated in correspondence with states, districts, school administrators, teachers, and parents. Letters and information materials describe the study, its voluntary nature, and the extent to which respondents and their responses will be kept confidential (see Appendix A). Recruitment letters, supporting materials, and login pages and front cover of each data collection instrument, including questionnaires, indicate:
The National Center for Education Statistics (NCES), within the U.S. Department of Education, conducts the Program for International Student Assessment (PISA) in the United States as authorized by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).
Login pages and front cover of each data collection instrument, including questionnaires, also include the following statement (the phrase “gather the data needed, and complete and review the information collection” will not be included on the student questionnaire):
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0755. The time required to complete this information collection is estimated to average [240/45/30/120/60] minutes per [school coordinator/school administrator/teacher/student], including the time to review instructions, gather the data needed, and complete and review the information collection. If you have any comments or concerns regarding the accuracy of the time estimate(s), suggestions for improving the form, or questions about the status of your individual submission of this form, write directly to: Program for International Student Assessment (PISA), National Center for Education Statistics (NCES), Potomac Center Plaza, 550 12th Street, SW, 4th floor, Washington, DC 20202.
OMB No. 1850-0755, Approval Expires xx/xx/2021
NCES understands the legal and ethical need to protect the privacy of the PISA respondents and has extensive experience in developing data files for release that meet the government’s requirements to protect individually identifiable data from disclosure. The contractor will conduct a thorough disclosure risk analysis of the PISA 2018 data when preparing the data files for use by researchers, in compliance with ESRA (20 U.S.C. §9573). Schools with high disclosure risk will be identified and, to ensure that individuals may not be identified from the data files, a variety of masking strategies will be used, including swapping data and omitting key identification variables (e.g., school name and address) from both the public- and restricted-use files (though the restricted-use file will include an NCES school ID that can be linked to other NCES databases to identify a school); omitting key identification variables such as state or zip code from the public-use file; and collapsing or developing categories for continuous variables to retain information for analytic purposes while preserving confidentiality in public-use files. IES’s Disclosure Review Board (DRB) carefully reviews all datasets prior to release to ensure that disclosure risks have been properly addressed. The PISA 2018 data will be reviewed and approved by the DRB prior to any public release, as has been the protocol for all previous rounds of PISA.
PISA 2018 does not include questions usually considered to be of a sensitive nature, such as items concerning religion, substance abuse, or sexual activity. Several items in the background questionnaires may be considered sensitive by some of the respondents, such as the socioeconomic context of the school, parents’ education and occupation, family possessions, and students’ belongings. Research indicates that the constructs these items represent are strongly correlated to academic achievement, and they have been used in the five previous cycles of PISA (2000, 2003, 2006, 2009, 2012, and 2015) as well as a number of other national and international studies. These items are considered essential for the anticipated analyses and to retain consistency in planned comparisons with the international data.
This package requests approval for the PISA 2018 main study. Burden estimates are shown in table A.1. The time required for students to respond to the assessment (cognitive items) portion of the study, and associated directions, are shown in gray font and are not included in the totals because they are not subject to the PRA. Student, administrator, and teacher questionnaires are included in the requested burden totals. Recruitment and pre-assessment activities (approved under OMB# 1950-0755 v.18-20) include the time involved in: (1) special handling school district staff reviewing and processing NCES’s research application requests to conduct PISA 2018 in schools under these districts’ jurisdiction, and (2) schools deciding to participate, completing teacher and student listing forms, distributing parent consent materials, and arranging assessment space.
The total main study burden requested here includes staff and students participating in PISA national and state samples. Although NCES has yet to receive a firm commitment from any state interested in PISA, the main study burden estimates reflect burden for the inclusion of up to 3 states (TBD).2
The total response burden for PISA 2018 main study is based on the following:
We estimate that there may be up to 30 special handling districts in the main study sample – those known to require completion of a research application before they will allow schools under their jurisdiction to participate in a study. Estimated burden hours for special handling districts are included in table A.1 under “Special Handling Districts IRB Staff Approval” (about 2 hours per IRB special handling district staff to process and review PISA application to conduct study in their schools) and “Special Handling Districts IRB Panel Approval” (about 1 hour per IRB special handling district panel member, assuming the average panel size of 6 members, to discuss and respond to the PISA application). Contacting special districts begins with updating district information based on what can be gleaned from online sources, followed by calls to verify the information about where to send the completed required research application forms, and, if necessary, to collect contact information for this process. During the call, inquiry is also made about the amount of time the districts spend reviewing similar research applications. The estimated number of such districts represents those with particularly detailed application forms and lengthy processes for approval. To allow sufficient time for special districts’ review processes, this operation will begin upon receiving OMB’s approval, and continue until we receive final approval or denial of our request from each contacted district, up until October 1, 2018.
The PISA 2018 main study burden estimate also includes the time to complete:
recruitment activities (approximately 90 minutes per school administrator) in 229 eligible schools;
pre-assessment activities (an average of 4 hours per school coordinator) in 190 participating schools to: (a) coordinate logistics with the data collection contractor, (b) supply a list of eligible students and teachers for sampling to the data collection contractor, (c) communicate with teachers, students, and parents about the study to encourage participation, (d) assist the test administrator in ensuring the sampled students attend the testing sessions, and (e) assist the test administrator in arranging make-up sessions as needed; and
data collection activities in 190 participating schools, including: (a) a 45-minute school administrator questionnaire, (b) a 30-minute teacher questionnaire, (c) a 3-minute review of consent forms by parents, (d) a 30-minute computer-based core background student questionnaire, (e) a 15-minute ICT student questionnaire, and (f) a 15-minute set of FL student questionnaire items.
Table A-1. Burden estimates for PISA 2018 main study
|
Sample |
Expected response rate |
Number of respondents |
Number of responses |
Burden per respondent (minutes) |
Total burden (hours) |
US national sample |
|
|
|
|
|
|
Recruitment and Pre-Assessment Activity |
|
|
|
|
|
|
Special Handling Districts IRB Staff Approval (US sample) |
30 |
1 |
30 |
30 |
120 |
60 |
Special Handling Districts IRB Panel Approval (US sample) |
180 |
1 |
180 |
180 |
60 |
180 |
School Administrator* |
257 |
0.89 |
229 |
229 |
90 |
344 |
School Coordinator |
190 |
1 |
190 |
190 |
240 |
760 |
School Administrator |
|
|
|
|
|
|
Questionnaire |
190 |
1 |
190 |
190 |
45 |
143 |
Teacher |
|
|
|
|
|
|
Questionnaire |
4,750 |
0.85 |
4,038 |
4,038 |
30 |
2,019 |
Parent |
|
|
|
|
|
|
Student Participation Consent |
9,337 |
1 |
9,337 |
9,337 |
3 |
467 |
Student |
|
|
|
|
|
|
Directions |
9,337 |
0.83 |
7,750 |
7,750 |
10 |
1,292 |
Assessment |
9,337 |
0.83 |
7,750 |
7,750 |
120 |
15,500 |
Student questionnaire (main questionnaire) |
9,337 |
0.83 |
7,750 |
7,750 |
30 |
3,875 |
Student questionnaire (FL questionnaire) |
9,337 |
0.83 |
7,750 |
7,750 |
15 |
1,938 |
Student questionnaire (ICT questionnaire) |
9,337 |
0.83 |
7,750 |
7,750 |
15 |
1,938 |
State samples (up to 3 states) |
||||||
Recruitment and Pre-Assessment Activity |
|
|
|
|
|
|
School Administrator (US states) |
162 |
1.00 |
162 |
162 |
90 |
243 |
School Coordinator (US states) |
162 |
1.00 |
162 |
162 |
240 |
648 |
School Administrator |
|
|
|
|
|
|
Questionnaire (US states) |
162 |
1.00 |
162 |
162 |
45 |
122 |
Teacher |
|
|
|
|
|
|
Questionnaire (US states) |
4,050 |
0.85 |
3,443 |
3,443 |
30 |
1,722 |
Parent |
|
|
|
|
|
|
Student Participation Consent |
8,845 |
1 |
8,845 |
8,845 |
3 |
442 |
Student |
|
|
|
|
|
|
US states (includes up to 3) |
|
|
|
|
|
|
Directions |
8,845 |
0.83 |
7,342 |
7,342 |
10 |
1,224 |
Assessment |
8,845 |
0.83 |
7,342 |
7,342 |
120 |
14,684 |
Student questionnaire (main questionnaire) |
8,845 |
0.83 |
7,342 |
7,342 |
30 |
3,671 |
Student questionnaire (FL questionnaire) |
8,845 |
0.83 |
7,342 |
7,342 |
15 |
1,836 |
Student questionnaire (ICT questionnaire) |
8,845 |
0.83 |
7,342 |
7,342 |
15 |
1,836 |
Total Burden Requested in this Submission |
|
41,708 |
72,244 |
|
22,244 |
|
NOTE 1: The estimated burden is conservatively high because PISA 2018 may include up to 3 states, however the burden is held consistent with national sample schools because of potential variability between states. Total student burden does not include time for cognitive assessment and its associated instructions. NOTE 2: Cells highlighted in green reflect the cells that contribute to the totals (the same respondent groups are not double counted in the total number of respondents). *Special note: For the national sample, we drew an initial sample of 257 schools. Taking into account closed, merged, and ineligible schools (historically, around 11% of sampled schools), as well as the historical school-level response rate, we anticipate interacting with/recruiting 229 of these schools. Of the 229 schools, we estimate that 190 will participate in PISA 2018. To obtain the required number of students, we will ask to sample up to 52 students in each school. However, some of the smaller schools will not have 52 students available. We estimate: 190 schools x 52 students sampled x 0.945 (a factor used to account for variations in student population sizes across the schools) = 9,337 starting student sample size that we will work to recruit. Based on historical student assessment rates (~83%), we estimate that, in the end, we will assess 7,750 students (9,337 x 0.83), which will assure that we meet the minimum required 6,900 assessed students. |
||||||
The estimated hourly rates for secondary school teachers/instructional staff, non-instructional staff/coordinators, principals, and parents ($29.46, $22.58, $45.86, $23.86 respectively) are based on Bureau of Labor Statistics (BLS) May 2016 National Occupational and Employment Wage Estimates.3 The federal minimum wage of $7.25 is used as the hourly rate for students. For the PISA main study, for the national and state samples, a total of 22,244 burden hours are anticipated, resulting in an estimated burden time cost to respondents of approximately $323,202.
Other than the burden associated with completing the PISA questionnaires and assessments (estimated above in Section A.12), the field test and main study impose no additional cost to respondents.
The cost to the Federal Government for conducting the PISA 2018 main study is estimated to be $3,806,890 over a 3-year period. The total cost to the Federal Government for conducting the PISA 2018 main study is estimated to be of $5,283,399 over a 4-year period. This is based on the national data collection contract, valued at $6,215,965 from January 2017 to January 2021, and includes costs for creating the national sample. These estimates also include all estimated direct and indirect costs of the project.
Components with breakdown |
Estimated costs |
FIELD TEST (2017) |
|
Recruitment |
244,757 |
Preparations (e.g., adapting instruments, sampling) |
202,540 |
Data collection, scoring, and coding |
1,029,212 |
MAIN STUDY (2018) |
|
Recruitment |
605,241 |
Preparations (e.g., adapting instruments, sampling) |
127,459 |
Data collection, scoring, and coding |
2,440,751 |
Reporting and dissemination |
633,439 |
Current package components |
$3,806,890 |
Grand total |
$5,283,399 |
The apparent increase in burden from last approval is due to the fact that the last request was to conduct the PISA 2018 recruitment and field test, while this request is to conduct the PISA 2018 main study.
With regards to content, there are some changes to PISA 2018 from the previous rounds of data collection. The main change is that the assessment will focus on reading literacy during this cycle, so the bulk of the items will be reading items with mathematics and science being the secondary components. PISA 2018 involves minor changes in wording to some of the questionnaire items, and questions that focused on student attitudes toward mathematics or science now focus on attitudes toward reading. PISA 2018 includes a new global competence assessment, in which the U.S. will not participate4, and a new ICT questionnaire which is being fielded in the U.S., with each sample student taking the 15-minute ICT questionnaire. The FL questionnaire has been lengthened to 15 minutes for PISA 2018. Each student will take the standard student 30-minute background questionnaire plus the ICT and FL questionnaires.
The PISA 2018 field test was designed to provide a statistical review of the performance of items on the assessments and questionnaires in preparation for the main study. The international contractor, ETS, provided the international instruments to be used in the field test and reported to the participating countries on the results of the field test. Based on the field test results, ETS, with input and agreement from the participating countries, made final revisions in the survey instruments, materials, and documents in preparation for the main study.
For PISA 2018 main study, an analysis of the U.S. and international data will be undertaken to provide an understanding of the U.S. national results in relation to the international results. Based on proposed analyses of the international data set by ETS, and the need for NCES to report results from the perspective of an American constituency, a plan is being prepared for the statistical analysis of the U.S. national data set as compared to the international data set. Analysis of data will include examinations of the science, reading, and mathematics literacy and financial literacy of U.S. students in relation to their international counterparts; and the relationships between student performance and student and school background variables.
All reports and publications will be coordinated with the release of information from the international organizing body. Planned publications and reports for the PISA 2018 main study include the following:
General Audience Report. Approximately one year after data collection, in December 2019, to correspond with the international data release, NCES will publish the U.S. National First Look Report with information on the status of reading, mathematics, and science education among students in the U.S. in comparison to their international peers, written for a non-specialist, general U.S. audience. This report will present the results of analyses in a clear and non-technical way, conveying how U.S. students compare to their international peers, and what factors, if any, may be associated with the U.S. results.
The results for financial literacy will be released internationally in 2020, with NCES releasing a Data Point report to present the results of financial literacy for the U.S. Accompanying both the First Look report and the Data Point, will be a set of tables and figures released on the NCES website to provide additional details.
Following the release of the national report, additional data will be made available to secondary users in the form of the International Data Explorer (IDE), an online tool on the NCES website, and a U.S. public-release dataset.
Survey Operations/Technical Report. This document will detail the procedures used in the main study (e.g., sampling, recruitment, data collection, scoring, weighting, and imputation) and describe any problems encountered and the contractor’s response to them. The primary purpose of the main study survey operations/technical report is to document the steps undertaken by the U.S. in conducting and completing the study. This report will include an analysis of non-response bias, which will assess the presence and extent of bias due to nonresponse. Selected characteristics of respondent students and schools will be compared with those of non-respondent schools and students to provide information about whether and how they differ from respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards.
Electronic versions of each publication are made available on the NCES website. Schedules for tabulation and publication of PISA 2018 results in the U.S. are dependent upon receiving data files from the international sponsoring organization. With this in mind, the expected data collection dates and a tentative reporting schedule are as follows:
Timeframe |
Activity |
October 2016-February 2017 |
Contact and gain cooperation of states, districts, and schools for field test |
March – May 2017 |
Select student samples and collect field test data |
July 2017 |
Deliver raw data to international sponsoring organization |
August – September 2017 |
Receive Field test Report from international sponsors |
October 2017–September 2018 |
Prepare for the main study phase and recruit schools |
June/July 2018 |
Summer training workshop for sampled schools |
October 2018–November 2018 |
Collect main study data |
March - April 2019 |
Receive final data files from international sponsors |
August - December 2019 |
Produce General Audience Report and Survey Operations/Technical Report for the U.S. |
The OMB expiration date will be displayed on all data collection materials.
No exceptions to the certifications are requested.
1 Neidorf, T.S., Binkley, M., Gattis, K., and Nohara, D. (2006). Comparing Mathematics Content in the National Assessment of Educational Progress (NAEP), Trends in International Mathematics and Science Study (TIMSS), and Program for International Student Assessment (PISA) 2003 Assessments (NCES 2006-029). U.S. Department of Education. Washington, DC: National Center for Education Statistics.
2 Puerto Rico indicated its intent to participate in PISA 2018, as it did in 2015 and began work on preparation for the field test sample, then ended their participation in PISA 2018 citing change in government administration and cost factors.
3 The average hourly earnings of secondary school teachers/instructional staff in the May 2016 National Occupational and Employment Wage Estimates sponsored by the Bureau of Labor Statistics (BLS) is $29.46, of noninstructional staff is $22.58, of principals/education administrators is $45.86, and of all occupations to estimate parent wages is $23.86. Source: BLS Occupation Employment Statistics, https://www.bls.gov/oes/current/oes_nat.htm#(4) data type: Occupation codes: Secondary School Teachers (25-2030); Education, Training, and Library Workers, All Other (Elementary and Secondary Schools) (25-9099); Education Administrators, Elementary and Secondary Schools (11-9032); and All employees (00-0000); accessed on February 27, 2017. If mean hourly wage was not provided it was computed assuming 2,080 hours per year.
4 Update: In March 2017, NCES determined that the U.S. will not administer the global competence component of PISA 2018. Items related to global competence will be eliminated from the student questionnaire and the global competence assessment will not be administered. No items related to global competence were part of the teacher or school questionnaire.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | PREFACE |
| Author | Gonzales, Patrick |
| File Modified | 0000-00-00 |
| File Created | 2021-01-21 |