OMB83C Change Memo - PISA 2018 Main Study Recruitment

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: August 7, 2017

TO: Robert Sivinski, OMB

THROUGH: Kashka Kubzdela, OMB Liaison, NCES

FROM: Patrick Gonzales, NCES

RE: Program for International Student Assessment (PISA 2018) Main Study Recruitment Change Request (OMB# 1850-0755 v.20)

The Program for International Student Assessment (PISA) is an international assessment of 15-year-olds which focuses on assessing students’ reading, mathematics, and science literacy. PISA was first administered in 2000 and is conducted every three years. The United States has participated in all of the previous cycles, and will participate in 2018 in order to track trends and to compare the performance of U.S. students with that of students in other education systems. PISA 2018 is sponsored by the Organization for Economic Cooperation and Development (OECD). In the United States, PISA is conducted by the National Center for Education Statistics (NCES), within the U.S. Department of Education. In each administration of PISA, one of the subject areas (reading, mathematics, or science literacy) is the major domain and has the broadest content coverage, while the other two subjects are the minor domains. PISA emphasizes functional skills that students have acquired as they near the end of mandatory schooling (aged 15 years), and students’ knowledge and skills gained both in and out of school environments. PISA 2018 will focus on reading literacy as the major domain. Mathematics and science literacy will also be assessed as minor domains, with additional assessments of global competence and financial literacy. In addition to the cognitive assessments described above, PISA 2018 will include questionnaires administered to assessed students, school principals, and teachers. The PISA 2018 Recruitment and Field Test were originally approved in September 2016 with a change request approved in October 2016 (1850-0755 v.18-19). The PISA 2018 field test was conducted in March-April 2017. The PISA 2018 main study data collection is scheduled for September-November 2018. Recruitment for the PISA 2018 main study is scheduled to begin by October 2017. This request provides the final plans, burden, and materials for the PISA 2018 main study recruitment.

Based on feedback and experience from the PISA 2018 field trial, NCES is requesting OMB approval for the changes to aspects of the PISA 2018 main study descriptions in the Supporting Statements Parts A and B and to the main study recruitment materials in Appendix A2 & B2 as detailed below.

The described here changes to the PISA 2018 main study assessment components, sampling, procedures, and recruitment materials do not result in changes to the cost to the federal government. Updates to the burden estimates of the main study are reflected below, and if any further changes will be made to the main study data collection burden, these will be reflected in a January 2018 request for approval of the main study data collection effort once negotiations over sample size have been finalized with the Organization for Economic Cooperation and Development (OECD), the international sponsor of PISA.

The following revisions have been made to the Supporting Statement Part A:

• The start of main study school recruitment was changed in the preface and in the schedule in Section A.16 to begin main study recruitment by October 2017 (rather than in September 2017).

• The United States will not field the global competence assessment. The assessment development for global competence required a serious redesign prior to the field test due to a consensus agreement among countries that the global competence framework and item sets needed considerable improvement to go forward. As a result, the development schedule and timing for implementation of the assessment became too compressed to comfortably pursue for the main study in 2018 and the United States has elected to forego including Global Competence in PISA 2018.

  • In Section A.2, under “Assessment Instruments”, a footnote was added stating: “Update: In March 2017, NCES determined that the United States will not administer the global competence component of PISA 2018. Both cognitive and non-cognitive items related to the global competence measure will not be included in the main study in the United States.”

  • In Section A.2, under “Background Questionnaire Instruments”, a footnote was added stating: “Update: In March 2017, NCES determined that the United States will not administer the global competence component of PISA 2018. For the non-cognitive portions of PISA, items related to global competence will be eliminated from the student questionnaire. No items related to global competence were part of the teacher or school questionnaire.”

• In Section A.9, under “Schools”, the following text was added:

“In addition to a monetary incentive, participating schools will be offered a school-level report on findings based on students’ knowledge and skills measured in PISA 2018. Working with the NCES Chief Statistician, we have devised a set of sample and response rate requirements for a school to receive a report (schools will be grouped into 6 categories based on sample size and response rates). Failure to achieve the designated sample and response rate requirements will mean that a school will not be eligible to receive a school report. We will also directly ask school principals during recruitment if there is any reason they do not want to receive a report. In these rare cases, a school report will not be produced. School-level reports will be shared only with the principal of the school, and will not be shared or distributed to anyone else. Prior to distribution, the school reports will be reviewed by the Chief Statistician for accuracy and compliance with the sample and response rate requirements designated for PISA.

Additionally, for the last several cycles of PISA, we have offered each sampled school the opportunity to send one delegate to a summer training workshop, held in Washington, DC, where details on the PISA study, including the procedures around submitting needed school-, teacher-, and student-level information are discussed. The school principal may send anyone s/he chooses, but we strongly encourage them to send the school coordinator who benefits most from the training workshop (please also see Part B, section B.3). The summer workshop is offered in the PISA 2018 main study recruitment materials (Appendix A). To reduce costs, we are considering reducing or eliminating this recruitment tool in 2018. Should we decide to eliminate the summer training workshop, all references to it in the main study recruitment materials will be deleted and the revised recruitment materials and Part A will be submitted to OMB as a change request prior to us beginning to contact schools.”

• The following text has been appended with the footnote that is copied here, right below it:

“The following statement will appear on the front cover of the questionnaires (the phrase “gather the data needed, and complete and review the information collection” will not be included on the student questionnaire):

Your answers will be combined with answers from other [respondent type] to calculate totals and averages. All information you provide may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Science Reform Act of 2002 (ESRA 2002), 20 U.S. Code, Section 9573].”

…the added footnote:

“Please note that starting with the Main Study recruitment, to account for the Cybersecurity Enhancement Act of 2015 (6 U.S.C. §151), this portion of the statement will read: The National Center for Education Statistics (NCES), within the U.S. Department of Education, conducts the Program for International Student Assessment (PISA) in the United States as authorized by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information provided by school staff and students may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). The U.S. Office of Management and Budget has approved this data collection under OMB #1850-0755.”

• In Section A.12, the following note was added to update the status of Puerto Rico in PISA 2018: “Puerto Rico began work on preparation for the field test sample and then ended their participation in PISA 2018 citing change in government administration and cost factors.”

• Section A.12 includes updated burden estimates to account for changes in the sampling design for the main study that are now known. The initial Table A-1 that was submitted with the April 2016 clearance package is provided below, followed by the updated Table A-1 that is now presented in Section A.12. Changes are based on the revised sampling assumptions for the main study which are included in the updated Section B.2. The revised rows in Table A-1 are under the Main Study heading and include minor adjustments to the sample sizes, number of respondents and responses, total burden hours, and the deletion of Puerto Rico. The total burden requested in this submission has decreased from 8,775 to 8,437 because of the deletion of Puerto Rico from the main study burden.

Previous Table A-1 (April 2016):

Table A-1. Burden estimates for PISA 2018 field test and main study

	Sample	Expected response rate	Number of respondents	Number of responses	Burden per respondent (minutes)	Total burden (hours)
FIELD TRIAL—Based on core + international options
Recruitment and Pre-Assessment Activity (includes Puerto Rico)
School Administrator (US sample)	70	1.00	70	70	90	105
Special Handling Districts IRB Staff Approval (US sample)	9	1	9	9	120	18
Special Handling Districts IRB Panel Approval (US sample)	54	1	54	54	60	54
School Administrator (Puerto Rico sample)	60	1.00	60	60	90	90
School Coordinator (US sample)	70	1.00	70	70	240	280
School Coordinator (Puerto Rico sample)	60	1.00	60	60	240	240
School Administrator
Questionnaire (US sample)	70	1.00	70	70	45	53
Questionnaire (Puerto Rico sample)	60	1.00	60	60	45	45
Teachers
Questionnaire (US sample)	1,750	0.80	1,400	1,400	45	1,050
Total School Burden Field Trial			1,723	1,853		1,935
Student
US national sample
Directions	3,500	0.85	2,975	2,975	10	496
Assessment	3,500	0.85	2,975	2,975	120	5,950
Student questionnaire (Main questionnaire)	3,500	0.85	2,975	2,975	30	1,488
Student questionnaire (ICT questionnaire)	3,500	0.85	2,975	2,975	15	744
Student questionnaire (Financial Literacy questionnaire)	3,500	0.85	2,975	2,975	15	744
Puerto Rico sample
Directions	3,000	0.85	2,550	2,550	10	425
Assessment	3,000	0.85	2,550	2,550	120	5,100
Student questionnaire (main questionnaire)	3,000	0.85	2,550	2,550	30	1,275
Total Student Burden Field Trial			5,525	11,475		4,251
Total Burden Field Trial			7,248	13,328		6,186
MAIN STUDY —Based on core + international options
US national sample
Recruitment and Pre-Assessment Activity
School Administrator	205	1.00	205	205	90	308
Special Handling Districts IRB Staff Approval (US sample)	30	1	30	30	120	60
Special Handling Districts IRB Panel Approval (US sample)	180	1	180	180	60	180
School Coordinator	205	1.00	205	205	240	820
School Administrator
Questionnaire	205	1.00	205	205	45	154
Teacher
Questionnaire	5,125	0.85	4,356	4,356	30	2,178
Student
Directions	10,250	0.90	9,225	9,225	10	1,538
Assessment	10,250	0.90	9,225	9,225	120	18,450
Student questionnaire (main questionnaire)	10,250	0.90	9,225	9,225	30	4,613
Student questionnaire (FL questionnaire)	10,250	0.90	9,225	9,225	15	2,306
Student questionnaire (ICT questionnaire)	10,250	0.90	9,225	9,225	15	2,306
State samples (up to 3 states & Puerto Rico)
Recruitment and Pre-Assessment Activity
School Administrator (US states)	162	1.00	162	162	90	243
School Administrator (Puerto Rico)	60	1.00	60	60	90	90
School Coordinator (US states)	162	1.00	162	162	240	648
School Coordinator (Puerto Rico)	60	1.00	60	60	240	240
School Administrator
Questionnaire (US states)	162	1.00	162	162	45	122
Questionnaire (Puerto Rico)	60	1.00	60	60	45	45
Teacher
Questionnaire (US states)	3,750	0.85	3,188	3,188	30	1,594
Questionnaire (Puerto Rico)	-	0.85	-	-	30	-
Student
US states (includes up to 3)
Directions	8,100	0.90	7,290	7,290	10	1,215
Assessment	8,100	0.90	7,290	7,290	120	14,580
Student questionnaire (main questionnaire)	8,100	0.90	7,290	7,290	30	3,645
Student questionnaire (FL questionnaire)	8,100	0.90	7,290	7,290	5	608
Student questionnaire (ICT questionnaire)	8,100	0.90	7,290	7,290	15	1,823
Puerto Rico
Directions	3,000	0.90	2,700	2,700	10	450
Assessment	3,000	0.90	2,700	2,700	120	5,400
Student questionnaire (main questionnaire)	3,000	0.90	2,700	2,700	30	1,350
Total School Recruitment Burden - Main Study			1,064	1,064		2,589
Total Burden Requested in this Submission			8,312	14,392		8,775
NOTE: OMB Clearance Requested: Total Burden includes all burden associated with conducting the PISA 2018 Field Test and the recruitment and preassessment activities for the PISA 2018 Main Study. The PISA 2018 Main Study burden is conservatively high because the PISA 2018 Main Study may include up to 3 states and Puerto Rico, however the burden is held consistent with national sample schools because of potential variability between states. Total student burden does not include time for cognitive assessment and its associated instructions. Puerto Rico has expressed intent to participate in PISA 2018 and would be expected to conduct a field trial as it moves from paper-based assessment to computer-based assessment. Any participating states are not required to conduct field trial as the national field trial is sufficient for this purpose. The estimates for Puerto Rico schools and students have been added to the burden estimates for the field trial and the main study. The Teacher Questionnaire will not be administered in Puerto Rico.

Updated Table A-1 (August 2017):

Table A-1. Burden estimates for PISA 2018 field test and main study

	Sample	Expected response rate	Number of respondents	Number of responses	Burden per respondent (minutes)	Total burden (hours)
FIELD TRIAL—Based on core + international options
Recruitment and Pre-Assessment Activity (includes Puerto Rico)
School Administrator (US sample)	70	1.00	70	70	90	105
Special Handling Districts IRB Staff Approval (US sample)	9	1	9	9	120	18
Special Handling Districts IRB Panel Approval (US sample)	54	1	54	54	60	54
School Administrator (Puerto Rico sample)	60	1.00	60	60	90	90
School Coordinator (US sample)	70	1.00	70	70	240	280
School Coordinator (Puerto Rico sample)	60	1.00	60	60	240	240
School Administrator
Questionnaire (US sample)	70	1.00	70	70	45	53
Questionnaire (Puerto Rico sample)	60	1.00	60	60	45	45
Teachers
Questionnaire (US sample)	1,750	0.80	1,400	1,400	45	1,050
Total School Burden Field Trial			1,723	1,853		1,935
Student
US national sample
Directions	3,500	0.85	2,975	2,975	10	496
Assessment	3,500	0.85	2,975	2,975	120	5,950
Student questionnaire (Main questionnaire)	3,500	0.85	2,975	2,975	30	1,488
Student questionnaire (ICT questionnaire)	3,500	0.85	2,975	2,975	15	744
Student questionnaire (Financial Literacy questionnaire)	3,500	0.85	2,975	2,975	15	744
Puerto Rico sample
Directions	3,000	0.85	2,550	2,550	10	425
Assessment	3,000	0.85	2,550	2,550	120	5,100
Student questionnaire (main questionnaire)	3,000	0.85	2,550	2,550	30	1,275
Total Student Burden Field Trial			5,525	11,475		4,251
Total Burden Field Trial			7,248	13,328		6,186
MAIN STUDY —Based on core + international options
US national sample
Recruitment and Pre-Assessment Activity
School Administrator	257	.89	229	229	90	344
Special Handling Districts IRB Staff Approval (US sample)	30	1	30	30	120	60
Special Handling Districts IRB Panel Approval (US sample)	180	1	180	180	60	180
School Coordinator	228	.85	194	194	240	776
School Administrator
Questionnaire	193	.85	164	164	45	123
Teacher
Questionnaire	4,825	0.85	4,101	4,101	30	2,051
Student
Directions	10,036	0.90	9,032	9,032	10	1,505
Assessment	10,036	0.90	9,032	9,032	120	18,064
Student questionnaire (main questionnaire)	10,036	0.90	9,032	9,032	30	4,516
Student questionnaire (FL questionnaire)	10,036	0.90	9,032	9,032	15	2,258
Student questionnaire (ICT questionnaire)	10,036	0.90	9,032	9,032	15	2,258
State samples (up to 3 states)
Recruitment and Pre-Assessment Activity
School Administrator (US states)	162	1.00	162	162	90	243
School Coordinator (US states)	162	1.00	162	162	240	648
School Administrator
Questionnaire (US states)	162	1.00	162	162	45	122
Teacher
Questionnaire (US states)	4,050	0.85	3,443	3,443	30	1,722
Student
US states (includes up to 3)
Directions	8,424	0.90	7,582	7,582	10	1,264
Assessment	8,424	0.90	7,582	7,582	120	15,164
Student questionnaire (main questionnaire)	8,424	0.90	7,582	7,582	30	3,791
Student questionnaire (FL questionnaire)	8,424	0.90	7,582	7,582	5	632
Student questionnaire (ICT questionnaire)	8,424	0.90	7,582	7,582	15	1,896
Total School Recruitment Burden - Main Study			957	957		2,251
Total Burden Requested in this Submission			8,205	14,285		8,437
NOTE: OMB Clearance Requested: Total Burden includes all burden associated with conducting the PISA 2018 Field Test and the recruitment and preassessment activities for the PISA 2018 Main Study. The PISA 2018 Main Study burden is conservatively high because the PISA 2018 Main Study may include up to 3 states, however the burden is held consistent with national sample schools because of potential variability between states. Total student burden does not include time for cognitive assessment and its associated instructions. Puerto Rico has expressed intent to participate in PISA 2018 and would be expected to conduct a field trial as it moves from paper-based assessment to computer-based assessment. Puerto Rico began work on preparation for the field test sample and then ended their participation in PISA 2018 citing change in government administration and cost factors. Any participating states are not required to conduct field trial as the national field trial is sufficient for this purpose.

The following revisions have been made to the Supporting Statement Part B:

• In Section B.2, the sampling assumptions for the main study have been updated.

“The international minimum number of completed assessments—for the core computer-based assessments in reading, mathematics, and science,—is 6,300 students in 150 schools. In PISA, the United States typically assesses between 5,600 and 5,900 students in 165 schools when sampling 42 students per school. To achieve a larger number of students assessed in 2018 as well as to account for anticipated nonparticipation and student ineligibility, the number of students sampled within schools will be increased to 52 students because of the added sample for financial literacy (42 students for the core assessment + 10 students for financial literacy). Assuming the same response level as PISA 2015, the initial target is a total sample of about 257 schools to yield about 193 participating schools (assuming a 75 percent participation rate among schools). To achieve the target final school response rate, we will use replacement schools to complete the sample, as allowed under the international sampling standards.

The student-per-school target for the core assessment is at least 42 completed student assessments per school. Assuming a within-school response rate of 90 percent (rates were 85 percent in 2000, 82 percent in 2003, 91 percent in 2006, 86 percent in 2009, 89 percent in 2012 and 89 percent in 2015), the original sample size of students within schools will be 52. Should any states participate in the 2018 assessment, each state would have a sample of between 50 and 60 schools and 2,700 students to yield 2,430 assessed students. Like the states, Puerto Rico will also have a sample size of between 50 and 60 schools and 2,700 students. As the main study plans for states and subnational jurisdictions are finalized, this information will be updated in the burden table.”

• In section B.3, the bullet point describing the training workshop was made into a stand-alone paragraph with added detail about the training attendees, goals and outcomes of the training event.

“In addition to these recruitment actions, and pending availability of funding, we plan to hold a summer training workshop for representatives of sampled schools in June of 2018 to inform them about PISA and keep them engaged in the study (see Section A.9). The Summer Training for PISA 2018 Schools provides an important channel of communication between NCES and Westat and the schools participating in PISA. This one-and-a-half-day training was held in each of the four previous cycles of PISA in 2006, 2009, 2012, and 2015. In each instance the training was valuable for answering questions from schools about PISA, conveying the usefulness of PISA data both nationally and internationally, and working with school staff to help them understand the logistical requirements of the study in their schools. The summer training workshop for PISA 2018 will be held in June 2018 in Washington D.C. The school coordinator from each participating school will be invited to attend. Airfare, hotel accommodation, and per diem will be provided for school participants who attend the training. Should we decide to eliminate the summer training workshop due to funding limitations, a change request will be submitted to OMB documenting these changes prior to us beginning to contact schools.”

• In section B.3, the bullet describing the school-level report was made into a stand-alone paragraph with added detail about the content of the report.

“Finally, we will provide school-level results on PISA to schools that meet the criteria for receiving a report (see section A.9 of Supporting Statement Part A). While individual-level scores cannot be produced from PISA data, a school level report can be produced when the school has a participation rate of 85 percent or better and at least 10 assessed students. The results in the school-level report will be comparative results that do not provide actual school means, but rather indicate how the school performed compared to country averages and to other US schools with similar demographic characteristics.”

A new appendix has been created and is included in this submission to provide the PISA 2018 main study recruitment materials (Appendix A-2 & B-2). These materials are largely based on the previously approved field test recruitment materials that have been revised to ready them for use in the main study. The revisions primarily consist of:

• Edits have been made throughout the recruitment materials to update references to the main study, remove references to the field test, reflect the main study components and the timing of main study activities, and generally improve the content and flow of the text.

• The following text has been added to the public and private school study brochures and FAQs for school administrators, and to the school letter for private schools: “In addition to receiving a U.S. national report with PISA 2018 results, schools may also receive a school-level report (pending sample size and response rate requirements). The school report presents comparisons of your school’s average scores with the average scores of participating education systems including the United States. The report also provides comparisons of your school with other similar U.S. schools based on specific school-level characteristics.”

• Global competence was removed from the list of PISA 2018 assessment components in all main study recruitment materials.

• As in previous cycles of PISA, the main study includes an enhanced mix of incentives for schools, teachers, students, and school coordinators that are not offered in the field trial. The incentives for the PISA 2018 main study were approved in September 2016 (OMB# 1850-0755 v.18-19) and are the same as in PISA 2015 main study and are detailed in Part A. The added text reads: “As a token of appreciation for participating, schools will receive $250, the school coordinator will receive $200, students who participate will each receive $25 as well as 4 hours of volunteer service time, and teachers who complete an online questionnaire will each receive $25. NCES also sponsors one delegate from each school to attend a summer training workshop in Washington, D.C. to learn about PISA.”

• The authorization language and confidentiality pledge cited throughout the recruitment materials have been updated, including to account for the Cybersecurity Enhancement Act of 2015 (6 U.S.C. §151), and now read: “The National Center for Education Statistics (NCES), within the U.S. Department of Education, conducts the Program for International Student Assessment (PISA) in the United States as authorized by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information [you provide / provided by school staff and students / provided by your teenager] may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). The U.S. Office of Management and Budget has approved this data collection under OMB #1850-0755.”

• Much of the School Administrator FAQ has been revised for the main study. Also, portions of the Student FAQ and Teacher FAQ have been updated for the main study.

• Much of the School Coordinator Email announcing E-Filing for Main Study and Instructions for School Coordinators to E-file student and teacher lists have been revised to reflect the main study and system changes.

• A new set of parallel materials for private schools was developed and is included in these appendices. The school letter; brochure; school administrator, teacher, student, and parent FAQs; teacher email; and PISA for Private Schools Information Sheet provide expanded information designed to address specific issues private schools have about participation in a study like PISA.

8