Change Memo

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: March 12, 2021

TO: Robert Sivinski, OMB

THROUGH: Carrie Clarady, OMB Liaison, NCES

FROM: Holly Xie, Program Officer, PIAAC, NCES

SUBJECT: Program for the International Assessment of Adult Competencies (PIAAC) Cycle II Operational Field Test 30D Review (OMB# 1850-0870 v.10)

The Program for the International Assessment of Adult Competencies (PIAAC) is a cyclical, large-scale study of adult skills and life experiences focusing on education and employment. PIAAC is an international study designed to assess adults in different countries over a broad range of abilities, from simple reading to complex problem-solving skills, and to collect information on individuals’ skill use and background. PIAAC is coordinated by the Organization for Economic Cooperation and Development (OECD) and developed by participating countries with the support of the OECD. In the United States, the National Center for Education Statistics (NCES), within the U.S. Department of Education (ED) conducts PIAAC. The U.S. participated in the PIAAC Main Study data collection in 2012 and conducted national supplement data collections in 2014 and 2017. All three of these collections are part of PIAAC Cycle I.

A new PIAAC cycle is to be conducted every 10 years, and PIAAC Cycle II Main Study data collection will be conducted from August 2022 through March 2023. In preparation for the main study collection, PIAAC Cycle II was due to begin with a Field Test in Spring of 2021, in which 33 countries participate with the goal of evaluating newly developed assessment and questionnaire items and to test the operations for the PIAAC 2023 Main Study (2022-2023). PIAAC 2022 defines four core competency domains of adult cognitive skills deemed key to facilitating the social and economic participation of adults in advanced economies: (1) literacy, (2) numeracy, (3) reading and numeracy components, and (4) adaptive problem solving. The U.S. will administer all four domains of the PIAAC 2022 assessment to a nationally representative sample of adults, along with a background questionnaire with questions about their education background, work history, the skills they use on the job and at home, their civic engagement, financial literacy and sense of their health and well-being. The results are used to compare the skills capacities of the workforce-aged adults in participating countries, and to learn more about relationships between educational background, employment, and other outcomes.

The request to conduct the PIAAC Cycle II Field Test in April-June 2020 was approved by OMB in December 2019 (OMB# 1850-0870 v.7-8). As described in the previously approved amendment in September 2020 (OMB# 1850-0870 v.9), the 2020 PIAAC Cycle II Field Test, scheduled to begin in April 2020, was postponed due to the 2020 novel coronavirus (COVID-19) global pandemic. The OECD delayed both the Field Test and Main Study by 12 months, meaning that the Field Test was due to be carried out in American homes beginning in April 2021.

In recognition of the continued constraints that countries face in meeting the current Field Test goals and timeline during the continuing global pandemic, the OECD has relaxed the current field test standards to allow for a smaller Operational Field Test of survey implementation. To determine the safety of in-person interviewing, the NCES data collection contractor, Westat, uses the average of new daily cases of COVID-19 per 100,000 people over the last 7 days to classify counties in green (less than one case), yellow (between 1 and 9 cases), orange (between 10 and 24 cases), and red (more than 24 cases) risk levels. Westat ONLY considers counties with a green or yellow risk safe to conduct in-person interviewing. That is, counties with an average of new daily cases of COVID-19 per 100,000 less than 10. Currently, very few counties in the U.S. are in the green or yellow zone and none of the 13 PSUs in the PIAAC field test sample is considered safe for field interviewers. After several months of careful monitoring of new daily case rates and considering OECD guidance, NCES has elected to conduct an Operational Field Test.

Therefore, this request updates Part A and Part B of the package to reflect the change to an Operational Field Test and relevant changes to materials and procedures resulting from this change. Additionally, NCES has imposed new requirements on study nomenclature that require changes to the U.S. outreach and recruitment materials and program nomenclature in the instruments. The revisions made to the last approved clearance documents are described below.

1. Changes Related to COVID-19 Delay and shift to the Operational Field Test

Cycle II Field Test Design Changes

Table 1 compares the main features of the operational field test to the previously approved PIAAC Cycle II Field Test (December 2019, OMB# 1850-0870 v.7-8). Two options are proposed to accommodate the possibility of continued concerns about in-person data collection. Proposed Option A will allow NCES to deliver to the Consortium the analysis of 250 cases of newly developed (previously approved by OMB) assessment and questionnaire items. Westat will continue to closely monitor the average of new daily cases of COVID-19 cases in the local area. If local rates exceed the average of 10 per 100,000, the Operational Field Test will delay until August and NCES will submit a non-substantive change request updating the timeline for data collection. We anticipate that all other materials will remain unchanged.

Table 1. PIAAC Cycle II Field Test Options

Data Collection	Approved PIAAC Cycle II Field Test	Proposed Option A	Proposed Option B
Training Start Date	March 15	June 14	August 16
Field Period	April 1-June 30	June 28-August 7	August 30-Sept. 30
Data Delivery	Consortium Batch 1 August 15	Consortium Batch 1 August 16-30	Consortium Batch 2 October 29
Completed Cases (BQ & Assessment)	1500	250	250
Sample Design	Household nonprobability sample	Convenience sample, recruited volunteers	Convenience sample, recruited volunteers
Sample Source	Electronic Address List, Cycle 1 PIAAC	Local, volunteer recruitment	Local, volunteer recruitment
Response Rate Estimate	50%	n/a	n/a

The Approved Cycle II Field test featured a stratified purposive subsample of 13 PSUs selected from the PIAAC Cycle I 2017 sample of 80 PSUs. The expected sample size of 1500 was required to estimate item parameters for all newly developed Assessment items and to test the stability of the trend item parameters for each tested language in a participating country. The Operational Field Test will include tests of survey implementation as well as the functionalities of the tablet device and ease of use of the device and interface for adults with low levels of familiarity with ICT devices. The field test will use a convenience sample of volunteers in the greater Washington DC metropolitan area for a total of 250 completed cases. Volunteers will be screened on relevant demographic characteristics and quotas will be set to achieve a diverse set of study participants. Screening characteristics include: age, gender, level of education, English language proficiency and computer/tablet familiarity. Quotas will be set in each category, and once filled, volunteers with those characteristics will no longer be eligible to participate. Under this data collection, interviews will be conducted in a location with established COVID mitigation procedures.

Part A: In addition to the changes described above and throughout this memo, specific changes to burden, cost to the government, and the schedule of the project are detailed below.

A.12 Estimates Burden

Table 2. Estimates of burden for PIAAC Cycle II Field Test

Data collection instrument	Sample size	Expected response rate	Number of respondents	Number of responses	Burden per respondent (minutes)	Total burden hours
Households Volunteers
PIAAC Volunteer Recruitment Screener	3,433 500	71.4% 100%	2,451 500	2,451 500	5	205 42
PIAAC Screener	250	100%	250	250	5	21
Background Questionnaire	2,213 250	71.4% 100%	1,580* 250	1,580 250	45	790 188
Locator and Orientation Module	1,580 250	100.00%	1,580* 250	1,580 250	10	263 42
Assessment**	1,580 250	98.10%	1,550* 245	1,550 245	60	1,550 245
Total	NA	NA	2,451 500*	5,611 1250	NA	1,258 293

* Duplicate counts of individuals are not included in the total number of respondents estimate.

** Assessments are exempt from Paperwork Reduction Act reporting and thus are not included in the burden total.

Assuming an average hourly cost of $24.98 25.72¹ for respondents, the 1,258 293 total burden hours are estimated to translate to $31,425 7,536 total burden time cost to PIAAC Cycle II Field Test respondents.

A.14 Annualized Cost to Federal Government

The total cost to the federal government, including all direct and indirect costs of preparing for and conducting the PIAAC Cycle II Field Test is estimated to be $3,113,759 2,648,499 (see Table 3 for cost detail).

Table 3. Cost for conducting the PIAAC Cycle II Field Test

Item	Cost
Labor	1,102,506 665,810
Other Direct Costs	489,252
Respondent Incentives	113,124 84,560
Overhead, G&A, and Fee	1,273,577
Salaries of Federal Employees	135,300
TOTAL PRICE	$3,113,759 $2,648,499

A.16 Plans for Tabulation and Publication

Because the Field Test is intended for testing sampling and operational procedures in the field only, the data from the PIAAC Cycle II Field Test will not be used to produce national estimates.

Technical Report

NCES will not produce a report for the Field Test and there are currently no plans to conduct statistical analyses of the Field Test dataset. The schedule reflects estimates of the ability to safely conduct face-to-face interviewing in light of the novel coronavirus (COVID-19). We will continue to closely monitor the average of new daily cases of COVID-19 cases in the local area. If local rates exceed the average of 10 per 100,000, the Operational Field Test will delay start until August 2021.

Table 4. PIAAC 2022 Cycle II Schedule

Dates	Activity
November 2019—February 2020 May-August 2021	Select sample Recruit volunteers for field test the Field Test
September 2019—March 2020—May 2021	Prepare field test data collection manuals, forms, training materials, assessment materials, questionnaires, and train data collection staff
April - June 2020 2021-August 2021*	Collect field test data – includes recruitment and assessment activities
August 2020 September 2021*	Deliver field test raw data results to international consortium
January—June 2021 2022	Select sample for main study
August 2020 2021—July 2021 2022	Prepare main study data collection manuals, training materials, forms, assessment materials, questionnaires, and train data collection staff
August 2021 2022—March 2022 2023	Collect main study data – includes recruitment and assessment activities
May 2022 2023	Deliver main study raw data to international consortium
January 2023 2024	Receive data files from international consortium
January 2023 2024 —August 2023 2024	Produce reports

* Assumes an average of new daily cases of COVID-19 per 100,000 less than 10 in the greater Washington DC metropolitan area.

Part B: The field test no longer employs statistical methods for sampling, therefore Part B has been updated significantly and provides a description of the volunteer recruitment and screening approach. The population will consist of 250 volunteers, aged 16-65 years. Since older Adults are at increased risk of serious illness if they become infected with the coronavirus (CDC, 2020 Older Adults and COVID-19 | CDC), persons age 66 and older will not be selected to participate in the 2021 Operational Field Test. To achieve a diverse sample, volunteers will be screened on age, gender, education, and race/ethnicity. Ensuring variation in education, English language proficiency and computer/tablet skills will help test functionality across volunteers with varying skill and education levels.

In addition, an experiment with incentives for completing the Screener was originally planned for completion during the Field Test. Due to the changes in the structure of the PIAAC Cycle II Field Test, that experiment has been cancelled.

Other Relevant Changes

This update also includes the following updates:

• Date References All date references in the submission have been advanced one year and can be found in the study descriptions (Part A, Part B). Also, data collection instruments are updated to reflect the revised date-related edits required from this delay. 

• Changes to Respondent Incentives. The original approved design featured field workers visiting sampled respondents’ households and collecting data in their homes. The revised design requires respondents to volunteer for inclusion, and then travel to another site to participate in the 2-hour data collection. Because we need to incentivize respondents to contact us to participate, and because the data collection will now be done outside the home, we have changed the promise incentive amount from $50 to $100.

• Additional Materials. Volunteer recruitment materials appear in Appendix A1, as does an interview introduction guide for the Operational Field Test and some respondent communication materials from the Main Study that will be used in the Operational Field Test. Appendices A2-D are the PIAAC instruments, unchanged since previous submissions. Appendix E describes detailed COVID mitigation strategies that Westat management and field staff must follow. This document also provides a description of the criteria Westat follows to determine when field deployment is allowable within a specific PSU.

II. PIAAC Study Name and Branding Revisions

In Cycle I, a user-friendly name for PIAAC Cycle II was created – the International Study of Adult Skills and Learning (ISASL) – to represent the program to the public and for use on all public-facing materials and reports. In the initial submissions for Cycle II (OMB# 1850-0870 v.7-9), this U.S.-specific user-friendly branding was carried forward. The International Consortium has since prohibited country-specific logos or branding. In this submission, all recruitment and communication materials have been updated to refer to the study as PIAAC. Version 7 of this submission called attention to the ISASL study name in the public facing materials. All references to ISASL have been removed from Part A and Part B. Public-facing documents with PIAAC study name logos and branding appear within Appendix E.

1 The average hourly earnings of adults derived from May 2018 2019 Bureau of Labor Statistics (BLS) Occupation Employment Statistics is $24.98 25.72. If mean hourly wage was not provided, it was computed assuming 2,080 hours per year. Source: BLS Occupation Employment Statistics, http://data.bls.gov/oes/ datatype: Occupation code: All employees (00-0000); accessed on August 30, 2019 December 10, 2020.