HPOG 2.0 Long-Term Follow-up Survey_Supporting Statement A_REV_June 2022_V4

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

National and Tribal Evaluation of the 2nd Generation of the Health Profession Opportunity Grants

OMB Information Collection Request

0970 - 0462

Supporting Statement

Part A

June 2022

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Lisa Zingman

Nicole Constance

Part A

Executive Summary

1. Type of Request: This Information Collection Request is for a revision to OMB #0970-0462. We are requesting a 3-year approval.

2. Progress to Date: In 2015, the U.S. Department of Health and Human Services (HHS), Administration for Children and Families (ACF) awarded grants to 32 organizations to administer the second generation of the Health Profession Opportunity Grants (HPOG 2.0) Program. ACF contracted with Abt Associates and partners to conduct the HPOG 2.0 National and Tribal Evaluations. In August 2015, ACF received approval to collect baseline data from HPOG 2.0 National and Tribal Evaluation study participants under OMB #0970-0462. Since that initial approval, OMB has approved

   • The participant contact update forms;

   • Three follow-up surveys and supporting materials and a phone-based assessment protocol;

   • Several descriptive evaluation protocols; and

   • A program cost survey.

Grantees began enrolling participants in 2016 and continued through the end of the grant period in September 2021. Data collection in support of the Tribal Evaluation is complete. Data collection in support of the National Evaluation is ongoing, including Participant Contact Update Forms for the in-progress COVID-19 Cohort Short-Term Follow-up Survey (COVID-STS) data collection and the upcoming HPOG 2.0 Long-Term Survey (LTS), part of the Long-Term Follow-Up Study. Overall, the 27 non-tribal grantees—the focus of this information collection—randomly assigned more than 52,000 study participants and enrolled an additional 3,000 participants who were not subject to random assignment.¹

3. Previous Terms of Clearance: There were no terms of clearance for the most recent approval.

4. Summary of changes requested: This is the fifth request for revisions to OMB #0970-0462. ACF is now preparing to conduct the HPOG 2.0 Long-Term Follow-Up Study, which will use the LTS and administrative data to estimate longer-term program impacts. This revised request for clearance seeks approval for:

1. HPOG 2.0 LTS (Instrument 21 and 21a): This new survey extends data collection for HPOG 2.0 participants to approximately 66 months (five and a half years) after random assignment. The HPOG 2.0 LTS will provide insights into the long-term impacts of HPOG 2.0 for outcomes that are not captured in administrative records, such as details about educational experiences, characteristics of employment (including employment in healthcare—which is a special focus of the program and the evaluation—and self-employment), earnings from such jobs, receipt of public assistance, physical and mental well-being, and child outcomes.

2. New supporting materials to support the LTS including an advance letter, flyer, and email script.

We do not intend for this information to be used as the principal basis for public policy decisions.

5. Time Sensitivity: The goal is to begin survey data collection for the LTS in May 2023. We are seeking OMB approval by October 2022 to ensure ample time to program and test the approved instrument.

A1. Necessity for Collection

The Health Profession Opportunity Grants (HPOG) Program was authorized by the Affordable Care Act (ACA), Public Law 111-148, 124 Stat. 119, March 23, 2010, sect. 5507(a), “Demonstration Projects to Provide Low-Income Individuals with Opportunities for Education, Training, and Career Advancement to Address Health Professions Workforce Needs,” adding sect. 2008(a) to the Social Security Act, 42 U.S.C. 1397g(a). In 2015, the U.S. Department of Health and Human Services (HHS), Administration for Children and Families (ACF) awarded grants to 32 organizations—27 non-tribal and 5 tribal—to administer the second generation of the Health Profession Opportunity Grants (HPOG 2.0) Program. HPOG’s authorizing legislation calls for a comprehensive evaluation of the funded demonstration projects. Accordingly, ACF is using a multi-pronged and comprehensive evaluation strategy to understand the effectiveness of HPOG 2.0 programs. To support this effort, the ACF Office of Planning, Research, and Evaluation launched the HPOG 2.0 National and Tribal Evaluations (OMB #0970-0462) in 2015. Data collection instruments approved for the evaluations include:

• collection of uniform performance measures from HPOG programs;

• the National Evaluation, which includes a descriptive evaluation, impact evaluation, and cost-benefit analysis of 27 non-tribal programs; and

• the Tribal Evaluation, which included implementation and outcomes studies of the five tribal HPOG programs.

Data collection activities began in 2016 when grantees began enrolling participants. Enrollment continued through the end of the grant period in September 2021. Data collection in support of the Tribal Evaluation is complete. The collection of uniform program measures from HPOG programs concluded in December 2021. Data collection in support of the National Evaluation is ongoing.

ACF is now preparing to conduct the HPOG 2.0 Long-Term Follow-Up Study, which will use the HPOG 2.0 Long-Term Survey (LTS) and administrative data to estimate longer-term (approximately five and a half years after random assignment) program impacts at the local and national level and to explore characteristics of local programs that are associated with more favorable outcomes. By extending data collection to include a long-term survey, ACF can address important unanswered questions for policymakers and practitioners. This submission seeks approval for the LTS instrument and supporting materials.

A2. Purpose

Purpose and Use

HPOG provided healthcare occupational training for Temporary Assistance for Needy Families (TANF) recipients and other adults with low incomes to train for jobs that pay well and were expected to experience labor shortages or be in high demand. The evaluation of non-tribal programs will assess the implementation and impacts of HPOG in non-tribal HPOG programs and will include a cost-benefit analysis. Key participant outcomes of interest include (but are not limited to) educational progress, employment, and earnings.

This information collection request seeks approval for the LTS instrument (Instrument 21); the LTS Critical Items Instrument (LTS-CII, Instrument 21a) which, as discussed below, is a shorter version of the LTS; and supporting materials (HPOG 2.0 LTS advance letter, flyer, and email script, Attachments AB, AD, and AE, respectively). The HPOG 2.0 LTS will provide insights into the long-term impacts of HPOG 2.0 for outcomes that are not captured in administrative records, such as details about educational experiences, characteristics of employment, self-employment, and earnings from jobs not covered in administrative data, receipt of public assistance, physical and mental well-being, and child outcomes. Survey-based outcomes have been critical to understanding the evolution of outcomes and impacts in other Career Pathways evaluations, specifically, the evaluations of Pathways for Advancing Careers and Education (PACE) and the first round of HPOG (HPOG 1.0) (OMB #0970-0394 and #0970-0397, respectively). Findings from the evaluation of HPOG 2.0 support ACF in understanding the extent to which HPOG met its statutory goals. Should HPOG be reauthorized, findings from the National Evaluation will be used to make adjustments to grant requirements outlined in the Notice of Funding Opportunity. The HPOG 2.0 Short-Term Impact Report (STIR) showed that HPOG 2.0 does help interested potential healthcare workers make educational progress and start work in the healthcare field. Findings from the Intermediate-Term Impact Report are not yet available. ACF expects that the LTS will be an important data source for the Long-term Impact Report. The LTS is critical for measuring the occupation and industry of workers, welfare participation, and financial stability. In addition, new modules will allow ACF to examine potential impacts on multi-generational outcomes, under-the-table earnings, and perceived barriers to long-term education and training.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests

Exhibit A-1 summarizes the research questions for this information collection request. These research questions largely mirror those from the Intermediate Term Survey (ITS) (Instrument 18 and 18a, previously approved under OMB #0970-0462). This information collection will provide information on the longer-term sustainability of short- and intermediate-term impacts on education, employment, and overall adult well-being. New to this information collection request is the addition of the child outcomes module, which offers an opportunity to measure child outcomes about five and a half years (66 months) after random assignment (see more on child sample selection in Supporting Statement B).

Exhibit A-1: HPOG 2.0 Long-Term Follow-Up Study—Research Questions Addressed by the Long-Term Survey

Evaluation Component	Data Source2	Research Questions
Impact Evaluation	HPOG 2.0 Long-Term Survey (Instrument 21) HPOG 2.0 Long-Term Survey, Critical Items Instrument (Instrument 21a)	Research questions on the impact of being offered an HPOG 2.0 slot five and a half years (66 months) after random assignment: What is the impact of an offer of an HPOG 2.0 slot on participant outcomes across a broad set of domains? How does the impact on key outcomes—training completion, earnings from healthcare occupations, and earnings—vary with baseline (i.e., pre-randomization) characteristics of individuals, including gender, education, race/ethnicity, age, and receipt of public assistance? For specific programs, what is the impact of an offer of an HPOG 2.0 slot on key outcomes of training completion, earnings, and earnings from healthcare occupations? How does the impact an offer of an HPOG 2.0 slot has on key outcomes—educational progress, labor market, and well-being—vary with HPOG 2.0 local program characteristics, including speed of implementation to get people quickly trained and quality of counseling services? What is the impact of an offer of an HPOG 2.0 slot on outcomes of participants’ families, including outcomes of their children by child age cohort as defined in Supporting Statement B (based on parental report for selected focal child)?

Study Design

The HPOG 2.0 National Evaluation is guided by the career pathways framework, as shown in the HPOG logic model (Attachment H). The framework puts into practice the assertion that “post-secondary training should be organized as a series of manageable and well-articulated steps accompanied by strong supports and connections to employment” (Fein et al., 2012).³ These articulated steps provide opportunities for students to advance through successively higher levels of education and training, exiting into employment at multiple possible points. The framework also incorporates customization, supports, and employer connections. The design for the HPOG 2.0 National Evaluation features a

• descriptive evaluation (including an implementation study, a systems study, and an outcome study);

• an impact evaluation (using a classic experimental design to measure and analyze key participant outcomes including completion of education and training, receipt of certificates and/or degrees, earnings, and employment in a healthcare career); and

• a cost-benefit analysis.

Exhibit A-2 provides a visual description of the major components of the HPOG 2.0 National Evaluation. The short- and intermediate-term follow-up surveys are described in more detail in Supporting Statement A for the second and third revisions to OMB #0970-0462, approved in June 2018⁴ and July 2019⁵, respectively. The COVID-STS was approved in July 2021, with nonsubstantive changes approved in October 2021. This information collection request seeks approval for the LTS instrument (both the full version and a shorter critical items only version) and supporting materials.

Exhibit A-2: Components of the HPOG 2.0 National Evaluation

In August 2015, ACF received approval to collect a list of baseline data items from HPOG 2.0 National and Tribal Evaluation study participants. The continued use of that list was approved with each subsequent revision to OMB #0970-0462. Subsequently OMB has approved:

• Contact update forms for survey respondents (Instruments 2-4, 5a and 5b approved in June 2017);

• The STS and supporting materials (Instrument 12 approved in July 2018);

• Descriptive evaluation protocols (Instruments 13-17 approved in July 2019);

• The ITS (Instruments 18 and 18a originally approved in July 2019, with nonsubstantive changes approved in June 2020);

• Phone-based Skills Assessment Pilot and Program Cost Survey (Instruments 19 and 20 also approved in July 2019); and

• The COVID-STS (Instrument 12a approved in July 2021 and 12b approved in October 2021).

This revised submission focuses only on the ongoing and new information collections for the HPOG 2.0 National Evaluation⁶.This includes:

New Instruments:

• Instrument 21: HPOG 2.0 Long-Term Survey (LTS)

• Instrument 21a: HPOG 2.0 Long-Term Survey Critical Items Instrument (LTS-CII)

Previously Approved Instruments Still in Use:

• Instrument 5b: HPOG 2.0 National Evaluation Participant Contact Update Letter and Form

• Instrument 5b: HPOG 2.0 National Evaluation Contact Update Form Phone Version

• Instrument 12a: COVID-19 Cohort Short-Term Follow-up Survey

• Instrument 12b: COVID 19 Cohort Short-Term Follow-up Survey Critical Items Only

Exhibit A-3: Overview of the HPOG 2.0 Long-Term Survey

Data Collection Activity	Instrument(s)	Respondent, Content, Purpose of Collection	Mode and Duration
Follow-up survey with HPOG 2.0 impact study participants approximately five and a half years (66 months) after random assignment	HPOG 2.0 Long-Term Survey (Instrument 21) HPOG 2.0 Long-Term Survey, Critical Items Instrument (Instrument 21a) Supporting Materials	Respondents: 4,938 participants (those randomized between September 2017-January 2018), target 73 percent response rate Content: Current education, credential attainment, and education/career goal Barriers to long-term health care training Current/most recent job conditions, job quality, benefits, on the job training Informal employment Income and economic well-being, student debt, financial resilience Adult well-being including physical health, housing conditions Household composition, family formation, and marital stability 21st century skills/cognitive skills Time child spends out of home/child supervision Child education-related goals and support Child outcomes Transition to adulthood Contact Information Purpose: The LTS will collect information on events that have occurred since the date of last interview through roughly five and a half years (66 months) after random assignment in most areas—particularly earnings, employment in healthcare, and receipt of credentials. This survey information will provide outcomes for the impact analysis. These data are not available through any current sources. Many of the questions to be asked in this survey were approved for the STS and ITS for the HPOG 2.0 National Evaluation (OMB #0970-0462), along with other studies in ACF’s Career Pathways portfolio, specifically the Pathways for Advancing Careers and Education 15-, 36-, and 72-month follow-up surveys (OMB #0970-0397); and the impact study of the first round of HPOG 15-, 36-, and 72-month surveys (OMB #0970-0394). Though not previously approved under this OMB control number, a nearly identical child outcomes module was previously approved under the other Career Pathways projects (OMB #0970-0394 and #0970-0397). A summary of the LTS item sources is provided in Attachment AC. The shorter version, Instrument 21a, will collect data on just the critical items of interest. Instrument 21a would be used as a tool to increase response rates and reduce the response rate differential between treatment and control group members. An equivalent shorter version was used to great success as part of the HPOG 2.0 ITS (Instruments 18 and 18a).	Mode: Phone with in-person follow-up if feasible Duration: 60 minutes

Other Data Sources and Uses of Information

The research team will continue to match participant data collected through the impact evaluation for both the treatment and control groups to employment and earnings data from the National Directory of New Hires (NDNH) and school enrollment and credential data from the National Student Clearinghouse (NSC). These other data sources are complementary to the survey data. There is little explicit overlap. In particular, the administrative data provide high quality data for a limited set of outcomes.

NDNH data only has information on quarterly earnings (from which the evaluation infers employment) and receipt of UI benefits. A survey is needed to collect information on hours of work, hourly wage, benefits, working conditions, and sector of employment. The latter is crucial because increasing the healthcare workforce is an explicit statutory goal.

NSC data only include information on education in colleges (more precisely, degree granting institutions). Information on whether the training was in healthcare—a crucial detail for HPOG—is not available from the NSC. Further, the NSC only covers college enrollment and outcomes. The STS and ITS found that about one-third of the trainings in which HPOG 2.0 participants enroll do not take place at colleges. A survey is therefore crucial to capture a full sense of training received.

A3. Use of Information Technology to Reduce Burden

The HPOG 2.0 National Evaluation has generated and will continue to generate a substantial amount of data using a combination of data collection methods. For each data collection activity, the evaluation team has selected the form of technology that enables the collection of valid and reliable information in an efficient way while minimizing burden. As described in the originally approved supporting statement (approved in August 2015, with previously noted revisions), participant- and grantee-level data were collected through PAGES, a cloud-based data system (PAGES data collection ended December 2021).

The HPOG 2.0 National Evaluation’s impact evaluation offers study participants the option to update their contact information online, by mail, or by telephone. The follow-up surveys are administered using computer assisted personal interviewing (CAPI) technology. CAPI technology allows interviewers to navigate through the survey faster, ensures that the skip patterns are properly implemented, and builds in checks to minimize data entry errors.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The evaluation team uses the quantitative data collected through PAGES, NDNH, and NSC to reduce the amount of data collected from participants during the follow-up survey. See Section A2, Other Data Sources and Uses of Information for additional information.

A5. Impact on Small Businesses

There is no impact on small businesses.

A6. Consequences of Less Frequent Collection

The HPOG 2.0 LTS will provide insights into the long-term impacts of HPOG 2.0 for outcomes that are not captured in administrative records, such as details about education and training experiences, characteristics of employment (including whether in healthcare), self-employment, and earnings from jobs not covered in administrative data, receipt of public assistance, physical and mental well-being, and child outcomes. The HPOG 2.0 Long-Term Follow-Up Study will use these survey-based outcomes to inform other planned analyses, including subgroup analyses, estimates of local program impacts, and explorations of the characteristics of local programs with favorable outcomes. Survey-based outcomes have been critical to understanding the evolution of outcomes and impacts in the evaluations of PACE and HPOG 1.0. In the absence of a survey, the evaluation team would only be able to estimate the program’s impact on outcomes measured in NDNH and NSC administrative data.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on April 8, 2022 (87 FR 20869) and provided a sixty-day period for public comment. No public comments were received during the notice and comment period.

Consultation with Experts Outside of the Study

Please see previously approved revised submissions from June 2017, June 2018, July 2019, and July 2021 for more detail on people consulted outside of the evaluation team for previously approved information collection requests under OMB #0970-0462. This external feedback informed the previous studies as well as this information collection request. In developing this submission, we also consulted with two colleagues—Dr. Laura Peck and Dr. Dan Litwok—who are leading the HPOG 1.0 Six-Year Follow-up efforts under OMB #0970-0394. Their comments (and their ongoing analyses) have informed this submission.

A9. Tokens of Appreciation

The HPOG 2.0 Long-Term Follow-Up Study will survey selected impact evaluation participants five and a half years (66 months) after random assignment. Panel retention during the follow-up period is critical to minimizing the risk of nonresponse bias and to achieving sufficient sample size to detect meaningful effects in the analysis. The evaluation team proposes to continue to use tokens of appreciation consistent with the design and amounts offered for other aspects of the impact study. (Please refer to the previously approved supporting statements for more details about approved tokens of appreciation for other HPOG 2.0 data collection activities and the associated justifications).

For the HPOG 2.0 LTS, the team hopes to achieve a response rate that exceeds that achieved on the recently completed ITS (65.8 percent).⁷ Respondents for the two previous follow-up surveys (STS and ITS) received an electronic gift card valued at $40 and $45, respectively. The evaluation team believes that a $50 token of appreciation is the appropriate amount for the LTS to help ensure that the evaluation ultimately meets the quality targets defined by OMB. Like the ITS, the LTS sample is smaller than that for the STS, so minimizing attrition is critical, especially given the longer follow-up period. This proposed amount represents a modest $5 increase— from $45 at the ITS to $50 at the LTS, the same increase as previously approved between the STS and the ITS. As previously approved in July 2021, participants who respond to the mailed contact update requests, using Instrument 5b, will continue to receive a $5 electronic gift card. For one round of contact update requests, participants in the LTS sample will be invited to complete the request by phone instead of by mail, allowing the interviewers to reinforce the study participant’s connection to the study, preview the upcoming LTS, and increase the quality of contact information for participants in the survey sample.⁸ As described in the previously approved contact update forms, respondents to the phone contact updates will receive a $10 electronic gift card. This amount is the same as the tokens of appreciation provided for similar efforts on other career pathways studies (HPOG 1.0 Six-Year Follow-up OMB #0970-0394 and PACE Six-Year Follow-up OMB #0970-0397). The higher amount for the phone contact updates is to help offset any expenses incurred, such as cell phone minutes.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

The HPOG 2.0 National Evaluation collected the name, date of birth, Social Security Number (SSN), address, phone number, and email information for all study participants through the PAGES system (previously approved under this OMB number). Name and contact information are necessary to aid in the contact update and survey data collection procedures. SSNs are used to aid in matching to the NDNH and NSC administrative databases.

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individual’s personal identifier. For all analyses, study members are linked by one of two randomly generated IDs. One of these, the “AbtID” is used to link surveys across time, to the baseline administrative database, and to retrieve data from NSC. There is a “key” file that links every AbtID to the SSN. This key file is stored in a separate folder on Abt’s secure server. Only personnel with a strict need to know are allowed access to this folder. The other randomly generated ID is a pseudo-SSN created by the Office of Child Support Enforcement (OCSE) at ACF. They use it to allow Abt personnel to link survey data with NDNH information. Abt personnel do not have access to NDNH data with names or SSNs attached. Only OCSE personnel and their contractors process such files.

Assurances of Privacy

Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information. All data collection protocols will receive IRB approval before data collection begins.

Data Security and Monitoring

As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ PII. The Contractor will ensure that all its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.

As specified in the contract, the Contractor will use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor will securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor will: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor will submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or PII that ensures secure storage and limits on access.

A11. Sensitive Information ⁹

Please refer to the information collections for the HPOG 2.0 National Evaluation previously approved in August 2015, June 2017, June 2018, July 2019, and July 2021 for a description of the sensitive items previously collected. Exhibit A-4 below provides an overview of the sensitive questions contained in the HPOG 2.0 LTS.

All previously approved information collections were approved by the Abt Associates Institutional Review Board (IRB). The HPOG 2.0 LTS (Instrument 21) and the HPOG 2.0 LTS-CII (Instrument 21a) and supporting materials Attachments AB, AD, and AE are in review with the Abt IRB.

Exhibit A-4: Sensitive Question Summary for HPOG 2.0 Long-Term Survey

	Potentially Sensitive Questions and Topic Areas	Justification
HPOG 2.0 Long-Term Survey, (Instrument 21) and Long-Term Survey-Critical Items Instrument (Instrument 21a)	Questions about income, employment status, receipt of government benefits, fertility, and household composition. Module on child outcomes.	As it is hoped that HPOG 2.0 will have favorable impacts in all these areas—failure to ask any of these questions would limit the findings of the evaluation. Questions about respondents’ children’s experiences during the follow-up could be viewed as sensitive. Interviewers will remind study members during the interview that they may refuse to answer individual items. Interviewers will also remind study members that their responses will be kept private to encourage their candid responses. Interviewers are also trained to recognize when questions are upsetting to participants and how to deescalate the situation swiftly.

A12. Burden

Explanation of Burden Estimates

The burden estimates presented here first update the remaining burden on previously approved instruments still in use, followed by estimated burden for new instruments.

Burden Remaining from Previously Approved Instruments

At the time of this request (May 2022), only Instrument 5b (Participant Contact Update Form, supporting both the LTS and the COVID-STS) and Instruments 12a and 12b (COVID-STS and COVID-STS-CII) are still active. The remaining burden estimates for those instruments are summarized below in Exhibit A-5.

Exhibit A-5: Remaining Burden on Previously Approved Instrument

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total Burden (in hours)	Annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Instrument 5b: HPOG 2.0 National Evaluation Letter and Participant Contact Update Form/Instrument 5b: HPOG 2.0 National Evaluation Contact Update Form Phone Version	2,244	7	.1	1,571	524	$10.15	$5,318.60
Instrument 12a: COVID-19 Cohort Short-Term Follow-up Survey	5,107	1	1	5,107	1,702.3	$10.15	$17,278.68
Instrument 12b: COVID-19 Cohort Short-Term Follow-up Survey – Critical Items Only	518	1	.33	171	57.0	$10.15	$578.35
Total	7,869			6,849	2,283.3		$23,175.63

Burden Estimates for This Information Collection Request

Exhibit A-6 summarizes the new burden estimates for Instrument 21 (HPOG 2.0 LTS) and Instrument 21a (HPOG LTS CII), the subjects for this information collection request. The total number of respondents is based on a total sample of 4,938 and a projected response rate of 73 percent. We estimate a 62 percent response rate for the full survey (3,064 respondents), with another 11 percent completing the CII (541 respondents).

Exhibit A-6: Burden Estimate for New Information Collection Request

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total Burden (in hours)	Annual Burden (in hours)	Avg. Hourly Wage Rate	Total Annual Respondent Cost
Instrument 21: HPOG 2.0 Long-Term Survey	3,064	1	1	3,064	1,021	$10.15	$10,363.15
Instrument 21a: HPOG 2.0 Long-Term Survey Critical Items Instrument	541	1	0.33	179	60	$10.15	$609.00
Total	3,605			3,243	1,081		$10,972.15

Estimated Annualized Cost to Respondents

We used the same wage rates for calculating annualized cost to respondents that we have used in all previously approved revisions to OMB #0970-0462. The wage rate for respondents, $10.15 per hour, is based on the Federal minimum wage of $7.25 with a 40 percent fringe and overhead loading. Based on these wage rates, the evaluation team estimates that the annual costs for the remaining previously approved data collection and this new information collection request is $34,147.78, annualized over the three-year request for clearance. The total remaining costs are $102,443.34.

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

The costs in Exhibit A-7 below are estimates to proceed with the HPOG 2.0 LTS based on the evaluation team’s experiences conducting the prior STS and ITS data collection efforts conducted under OMB #0970-0462. The exhibit also includes the remaining costs to complete the previously approved information collections (ongoing contact updates and the COVID-STS).

Exhibit A-7: Preliminary Cost Estimates for this Information Collection Request

Activity	Detail	Estimated Cost
Survey administration and Participant Contact Update Activities	FTE time Operational expenses (such as equipment, overhead, printing, and staff support) Other expenses which would not have been incurred without this collection of information	$10,392,850
Analysis and Initial Dissemination	FTE time Operational expenses (such as equipment, overhead, printing, and staff support) Other expenses which would not have been incurred without this collection of information	$2,900,000
Total costs over the request period		$13,292,850
Annual costs		$4,430,950

A15. Reasons for changes in burden

This request for clearance includes new surveys to continue the HPOG 2.0 National Evaluation. As such, those new instruments (Instruments 21 and 21a) and the associated burden have been added, as shown in Exhibit A-6.

We also plan to complete data collection with previously approved instruments (Instruments 5b, 12a, and 12b). This request does not include any changes to content of those instruments or to the estimated time to complete. We have, however, updated the number of respondents to reflect the current status of data collection. Exhibit A-5 shows the remaining burden for the previously approved information collections that are still active.¹⁰

Since the most recent approval, the PAGES data collection has been completed, so that has been removed from the burden section.

A16. Timeline

ACF awarded grants to 32 grantee organizations in 2015. Participant enrollment began in February 2016 and ended in September 2021. OMB approved several information collection requests under OMB #0970-0462 in support of the HPOG 2.0 National and Tribal Evaluations. The timeline for OMB approval, data collection, reporting, and archiving for each component of the HPOG 2.0 National and Tribal Evaluations are summarized by study component in Exhibit A-8 below.

Exhibit A-8: Project Schedule for Data Collection, Analysis, Publication, and Archiving¹¹

Task	Timing	Data to be Archived (yes/no)
HPOG 2.0 National Evaluation: Descriptive Evaluation (Instruments 2-4 and 13-17 approved in June 2017 and July 2019)
Data Collection
Descriptive evaluation data collection for implementation and systems studies, and for in-depth participant interviews	June 2017-February 2020	Grantee and Participant Interview Data will be archived; Systems Interview Data will not be archived because it would be difficult to anonymize given small number of sites, detailed information on partners, and small number of respondents.
National evaluation descriptive study site visits	Fall 2018	No, the data would be difficult to anonymize due to small number of sites and respondents.
Reporting
Systems Study Report	Mid 2022
Implementation Study Report	Mid 2022
Participant Interview Briefs	Ongoing beginning early 2022
Outcomes Report	Late 2022
HPOG 2.0 National Evaluation: Impact Evaluation (Instrument 5a and 5b, 12, 18 and 18a (approved in June 2017, June 2018, and July 2019)
Data Collection
Contact update mailing for survey sample (Instrument 5a and 5b)	Quarterly beginning 3 months after random assignment up to 3 months before the survey fielding period (November 2017-Winter 2023)	No, this only collects participant PII, there is no way to anonymize it.
Short-Term (15-Month) Follow-up Survey Data Collection	October 2018-November 2019	Yes, after publication of the Short-Term Impact Report
Intermediate (36-Month) Follow-up Survey Data Collection (Instrument 18 and 18a)	September 2020-June 2021	Yes, after publication of the Intermediate-Term Impact Report
COVID-19 Cohort Short-term (15-Month) Follow-up Survey Data Collection (Instrument 12a)	December 2021-April 2023	Yes, after publication of the COVID-19 Cohort Impact Report
Long-Term Survey	Planned May-October 2023	Yes, after publication of the Long-Term Impact Report
Reporting
Short-Term Impact Report	March 2022	Survey data only. Not data from NDNH or NSC.
Draft Intermediate-Term Impact Report to ACF	June 2022	Survey data only. Not data from NDNH or NSC.
Draft COVID-19 Cohort Impact Report	Spring 2024	Survey data only. Not data from NDNH or NSC.
Draft Longer-Term Impact Report to ACF	Fall 2024	Survey data only. Not data from NDNH or NSC.
HPOG 2.0 National Evaluation: Cost-benefit analysis (Instrument 20, Approved in July 2019)
Program cost data collection	Fall 2019 – Spring 2020	Yes, after publication of the Long-Term Impact Report
Draft Cost-Benefit report to ACF	Spring 2024
HPOG 2.0 National and Tribal Evaluation Participant Accomplishment and Grantee Evaluation System (PAGES) (Instrument 1, Approved in August 2015)
PAGES grantee-level and participant-level data collection	September 2015 – December 2021	Yes, for the 27 non-Tribal grantees only
Reporting
6 Semi-annual Performance Progress Reports	September 2015 – December 2021
Annual reports	September 2015 – September 2022

A17. Exceptions

No exceptions are necessary for this information collection.

Instruments and Attachments – Current Request

New Instruments:

• Instrument 21: HPOG 2.0 Long-Term Survey (LTS)

• Instrument 21a: HPOG 2.0 Long-Term Survey Critical Items Instrument (LTS-CII)

Previously Approved Instruments Still in Use:

• Instrument 5b: HPOG 2.0 National Evaluation Participant Contact Update Letter and Form

• Instrument 5b: HPOG 2.0 National Evaluation Contact Update Form Phone Version

• Instrument 12a: COVID-19 Cohort Short-Term Follow-up Survey

• Instrument 12b: COVID 19 Cohort Short-Term Follow-up Survey Critical Items Only

New Attachments

• Attachment AB: HPOG 2.0 Long-Term Survey Advance Letter

• Attachment AC: HPOG 2.0 Long-Term Survey Sources

• Attachment AD: HPOG 2.0 Long-Term Survey Trying to Reach You Flyer

• Attachment AE: HPOG 2.0 Long-Term Survey Email Reminder Text

• Attachment AF: HPOG 2.0 Previously Approved Sample Selection for the STS and ITS

• Attachment AG: HPOG 2.0 Previously Approved Methods to Maximize Response Rates

Previously Approved Attachments Still in Use

• Attachment H: HPOG Logic Model

• Attachment K-Revised: COVID-19 Cohort Short-Term Survey Advance Letter

• Attachment L-Revised: COVID-19 Short-Term Survey Sources

• Attachment M-Revised: COVID-19 Cohort Short-Term Survey Trying to Reach You Flyer

• Attachment N-Revised: COVID-19 Cohort Short-Term Survey Email Reminder Text

• Attachment AA: HPOG 2.0 Participant Newsletter

Instruments and Attachments – Previously Approved, No Longer in Use¹²

Previously Approved Instruments No Longer in Use:

• Instrument 1: PAGES Grantee- and Participant-Level Data Items List

• Instrument 2: HPOG 2.0 National Evaluation Screening Interview

• Instrument 3: HPOG 2.0 National Evaluation first-round telephone interview protocol

• Instrument 4: HPOG 2.0 National Evaluation in-person implementation interviews

  • Instrument 4A HPOG 2.0 National Evaluation In-Person Implementation Interview

  • Instrument 4B HPOG 2.0 National Evaluation In-Person Implementation Interviews Basic Skills Training

  • Instrument 4C HPOG 2.0 National Evaluation In-Person Implementation Interviews Career Pathways

  • Instrument 4D HPOG 2.0 National Evaluation In-Person Implementation Interviews Work-Readiness

  • Instrument 4E HPOG 2.0 National Evaluation In-Person Implementation Interviews Sustainability

• Instrument 5: HPOG 2.0 National Evaluation welcome packet and participant contact update forms

  • Instrument 5a: HPOG 2.0 National Evaluation welcome packet and contact update form_REV

• Instrument 6: HPOG 2.0 Tribal Evaluation grantee and partner administrative staff interviews

• Instrument 7: HPOG 2.0 Tribal Evaluation program implementation staff interviews

• Instrument 8: HPOG 2.0 Tribal Evaluation employer interviews

• Instrument 9: HPOG 2.0 Tribal Evaluation program participant focus groups

• Instrument 10: HPOG 2.0 Tribal Evaluation program participant completer interviews

• Instrument 11: HPOG 2.0 Tribal Evaluation program participant non-completer interviews

• Instrument 12: HPOG 2.0 National Evaluation Short-term Follow-up Survey

• Instrument 13: HPOG 2.0 Screening Interview Second Round

• Instrument 14: HPOG 2.0 Second Round Telephone Interview Guide

• Instrument 15: HPOG 2.0 Program Operator Interview Guide for Systems Study

• Instrument 16: HPOG 2.0 Partner Interview Guide for Systems Study

• Instrument 17: HPOG 2.0 Participant In-depth Interview Guide

• Instrument 18: HPOG 2.0 Intermediate Term Follow-up Survey_ REV_June2020

  • Instrument 18a: HPOG 2.0 Intermediate Term Follow-up Survey_Critical Items Only

• Instrument 19: HPOG 2.0 Phone-based Skills Assessment Pilot Study Instrument

• Instrument 20: HPOG 2.0 Program Cost Survey

Previously Approved Attachments No Longer in Use

• Attachment A: References

• Attachment B: Previously Approved Informed Consent Forms

  • Attachment B: National Evaluation informed consent form A (Lottery Required)

  • Attachment B: National Evaluation informed consent form B (Lottery Not Required)

  • Attachment B: National Evaluation Informed Consent Form C (Lottery Required) _Verbal

  • Attachment B: National Evaluation Informed Consent Form D (Lottery Not Required) _Verbal

• Attachment B: Previously Approved Informed Consent Forms, Updated Time Period

  • Attachment B: National Evaluation Informed Consent Form A (Lottery Required) _REV

  • Attachment B: National Evaluation Informed Consent Form C (Lottery Required) _Verbal_REV

  • Attachment B2: Tribal Evaluation informed consent form A (SSNs)

  • Attachment B3: Tribal Evaluation informed consent form B (Unique identifiers)

  • Attachment B2: Tribal Evaluation Informed Consent Form C (SSNs)_Verbal

  • Attachment B3: Tribal Evaluation Informed Consent Form D (Unique identifiers) _Verbal

• Attachment C: 60-Day Federal Register Notice

• Attachment D: Previously Approved Sources and Justification for PAGES Grantee- and Participant-Level Data Items

• Attachment E: Previously Approved Final Updated Attachment E PPR Data List and Mockup

• Attachment F: First Round of HPOG Grantees Research Portfolio

• Attachment G: Previously Approved Participant Contact Information Update Letter and Form (Obsolete, replaced by Instrument 5a and 5b)

• Attachment I: Previously Approved Focus Group Participant Consent Form

• Attachment I: New Focus Group Participant Consent Form_Remote

• Attachment J: Previously Approved Interview Verbal Informed Consent Form

• Attachment J: New Interview Verbal Informed Consent Form_Remote

• Attachment K: HPOG 2.0 National Evaluation Short-term Follow-up Survey Advance Letter

• Attachment L: HPOG 2.0 National Evaluation Short-term Follow-up Survey Sources

• Attachment M: HPOG 2.0 National Evaluation Short-term Follow-up Survey Trying to Reach You Flyer

• Attachment N: HPOG 2.0 National Evaluation Short-term Follow-up Survey Email Reminder

• Attachment O: Research Questions for Previously Approved Data Collection Efforts (National Evaluation and Tribal Evaluation)

• Attachment P: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Advance Letter

• Attachment P: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Advance Letter_REV

• Attachment Q: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Sources

• Attachment Q: Intermediated Follow-up Survey Sources_REV

• Attachment R: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Trying to Reach You Flyer

• Attachment R: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Trying to Reach You Flyer_REV

• Attachment S: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Email Reminder

• Attachment S: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Email Reminder_REV

• Attachment T: HPOG 2.0 National Evaluation phone-based Skills Assessment Pilot flyer

• Attachment U: HPOG 2.0 National Evaluation phone-based Skills Assessment Pilot grantee letter

• Attachment V: HPOG 2.0 National Evaluation phone-based Skills Assessment Pilot participant letter

• Attachment W: HPOG 2.0 National Evaluation phone-based Skills Assessment Pilot recruitment script

• Attachment X: Complete list of previously approved data collection instruments

• Attachment Y: 60-day Federal Register Notice

• Attachment Z: Participant Interview Recruitment Materials

1

2 Additional data sources will be used to answer the research questions, beyond those listed here. Only the data sources relevant to this submission are listed.

3 Fein, David J. 2012. Career Pathways as a Framework for Program Design and Evaluation: A Working Paper from the Pathways for Advancing Careers and Education (PACE) Project. OPRE Report 2012-30. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

4 ICR Ref. No: 201802-0970-001 

5 ICR Ref. No: 201904-0970-006 

6 Please refer to OMB #0970-0462, revised submission #1 approved in June 2017, for more detail on the HPOG 2.0 Tribal Evaluation, for which data collection is complete.

7 Generally speaking, response rates decline between waves on longitudinal studies. However, the ITS was done strictly by telephone due to COVID-19 social distancing requirements. The evaluation contractor expects to be able to conduct in-person follow-up for the LTS, which should yield a higher response rate.

8 Phone contact updates will be done using Instrument 5b- HPOG 2.0 Contact Update Form Phone Version, previously approved in July 2021.

9 Examples of sensitive topics include (but are not limited to): SSN; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships ( e.g., family, pupil-teacher, employee-supervisor); mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment, WIC, or SNAP); and immigration/citizenship status.

10 All data collection for instruments 1, 2-4, 6-17, 18 and 18a, and 19-20 are now complete. Since they are complete and have no remaining burden, they are no longer reflected in Exhibit A-5.

11 Tribal Evaluation data collection using Instruments 6-11, approved in June 2017, and renewed in subsequent revisions under this OMB # took place between Summer 2017 and Fall 2020. Reports were published annually after each data collection. The Final Report for the Tribal Evaluation was published in December 2021. Tribal data will not be archived.

12 Please refer to the information collections for the HPOG 2.0 National Evaluation previously approved in August 2015, June 2017, June 2018, July 2019, and July 2021

18