HPOG2 Supporting Statement_Part A_10 18 16_REVISED 4-21-17

Instrument 4A HPOG 2.0 National Evaluation In-Person Implementation Interview
Instrument 4B HPOG 2.0 National Evaluation In-Person Implementation Interviews Basic Skills Training
Instrument 4C HPOG 2.0 National Evaluation In-Person Implementation Interviews Career Pathways
Instrument 4D HPOG 2.0 National Evaluation In-Person Implementation Interviews Work-Readiness
Instrument 4E HPOG 2.0 National Evaluation In-Person Implementation Interviews Sustainability

Instrument 5: HPOG 2.0 National Evaluation welcome packet and participant contact update forms

Instrument 5a: HPOG 2.0 National Evaluation welcome packet and contact update form
Instrument 5b: HPOG 2.0 National Evaluation participant contact update letter and form

Instrument 6: HPOG 2.0 Tribal Evaluation grantee and partner administrative staff interviews

Instrument 7: HPOG 2.0 Tribal Evaluation program implementation staff interviews

Instrument 8: HPOG 2.0 Tribal Evaluation employer interviews

Instrument 9: HPOG 2.0 Tribal Evaluation program participant focus groups

Instrument 10: HPOG 2.0 Tribal Evaluation program participant completer interviews

Instrument 11: HPOG 2.0 Tribal Evaluation program participant non-completer interviews

Attachment A: References

Attachment B: Informed Consent Forms

Attachment B2: Tribal Evaluation informed consent form A (SSNs)
Attachment B3: Tribal Evaluation informed consent form B (Unique identifiers)

Attachment C: 60 Day Federal Register Notice

Attachment D: Sources and Justification for PAGES Grantee- and Participant-Level Data Items

Attachment E: Final Updated Attachment E PPR Data List and Mockup

Attachment F: First Round of HPOG Grantees Research Portfolio

Attachment G: Participant Contact Information Update Letter and Form

Attachment H: HPOG Logic Model

Attachment I: Focus group participant consent form

Attachment J: Interview Verbal Informed Consent Form

Part A: Justification

This document provides supporting statements for the collection of information for the National and Tribal Evaluations of the Health Profession Opportunity Grants (HPOG) program, funded by the U.S. Department of Health and Human Services (HHS), Administration for Children and Families (ACF). The HPOG grants fund programs that provide education and training to Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals for occupations in the health care field that pay well and are expected to either experience labor shortages or be in high demand. ACF awarded the first set of HPOG grants in September 2010, and the second set of HPOG grants in September 2015 (referred to as HPOG 2.0). Under HPOG 2.0, ACF funded 32 grants—five to tribal-affiliated organizations and 27 to non-tribal entities.

The ACF Office of Planning, Research and Evaluation (OPRE) has developed a multi-pronged research and evaluation portfolio for the HPOG 2.0 Program to better understand and assess the activities conducted and their results. This submission is in support of two components of the evaluation portfolio, the HPOG 2.0 National Evaluation and HPOG 2.0 Tribal Evaluation. Abt Associates and their partners, MEF Policy Associates, the Urban Institute, Insight Policy Research, and NORC at the University of Chicago, are leading the evaluation of HPOG 2.0. These studies will use data collected from the HPOG management information system, the HPOG Participant Accomplishment and Grantee Evaluation System (PAGES), designed under The Evaluation and System Design for Career Pathways Programs: 2nd Generation of Health Profession Opportunity Grants (HPOG Next Gen Design). OMB previously approved baseline data collection in the PAGES system and informed consent forms for the HPOG 2.0 evaluation under OMB Control Number 0970-0462. PAGES is internet-based and gathers data from the HPOG 2.0 grantees on: (1) grantee program designs and offerings; (2) intake information on eligible applicants (both treatment and control) through baseline data collection; and (3) individual enrolled program participants’ activities and outcomes. The original OMB submission was approved in August 2015. A nonsubstantive change request was approved in January 2016 for changes to the informed consent forms for non-tribal grantees. A second nonsubstantive change request was approved in July 2016 for changes to the informed consent forms for Tribal grantees. This submission seeks OMB approval for additional data collection efforts for the HPOG 2.0 National Evaluation and the HPOG 2.0 Tribal Evaluation.

A1: Necessity for the Data Collection

ACF at HHS seeks approval for the data collection activities described in this request to support the HPOG 2.0 National Evaluation and HPOG 2.0 Tribal Evaluation.

A1.1 Study Background

The HPOG Program, established by the Patient Protection and Affordable Care Act of 2010 (ACA), funds training in high-demand healthcare professions, targeted to TANF recipients and other low-income individuals. The HPOG program is administered by the ACF Office of Family Assistance (OFA). The first round of HPOG grants was awarded in 2010. In September 2015, OFA awarded a second round of HPOG grants—approximately $72 million was awarded to 32 organizations located across 21 states. Grantees include six community based organizations, four state government entities, seven local workforce development agencies, ten institutions of higher education, two tribal colleges, one tribal human service agency, one tribe, and one Indian Health Board. Those 32 grantees oversee 43 individual HPOG programs.

HPOG programs: (1) target skills and competencies demanded by the healthcare industry; (2) support career pathways, such as an articulated career ladder; (3) result in an employer- or industry-recognized credential (which can include a license, third-party certification, postsecondary educational certificate or degree, as well as a Registered Apprenticeship certificate); and (4) combine supportive services with education and training services to help participants overcome barriers to employment, as necessary.

HPOG’s authorizing legislation calls for a comprehensive evaluation of the funded demonstration projects. Accordingly, ACF plans to evaluate rigorously the effectiveness of funded HPOG 2.0 programs. The federal evaluation activities are intended to expand the career pathways evidence base and to build on what has been learned to date about how to design and implement successful career pathways programs for low-income and low-skilled individuals, and improve the outcomes of individuals who participate in these programs. All grantees will participate in a federal evaluation. The federal evaluation for the non-Tribal HPOG 2.0 grantees involves random assignment of individual participants. The national evaluation may expand to include the randomization of selected program design components, or “enhancements,” in a select number of programs. Participants would be assigned to a treatment, treatment enhanced, or control group, if these enhancements are implemented. Tribal grantees are participating in a coordinated evaluation that does not involve random assignment.

The OMB-approved HPOG PAGES data system (OMB Control Number 0970-0462) is collecting and storing uniform data needed for performance management and the federal evaluations, incorporating the required semi-annual grantee performance reports to ACF (Attachment E). These reports include a quantitative section with metrics automatically generated from data in the PAGES system and a narrative section that must be filled out by grantees. The system also provides necessary data for other research and evaluation efforts, including the HPOG 2.0 National Evaluation and HPOG 2.0 Tribal Evaluation.

Abt Associates is the prime contractor for the HPOG 2.0 Evaluation. Abt and the Urban Institute led the design of the PAGES data system and both organizations are overseeing PAGES data collection. Abt is leading the HPOG 2.0 National Evaluation. Partners MEF Associates, Insight Policy Research and the Urban Institute are assisting with the site monitoring, descriptive evaluation and cost benefit analysis. NORC at the University of Chicago is leading the HPOG 2.0 Tribal Evaluation under subcontract to Abt Associates. Abt and their partners are also conducting several other evaluations on behalf of ACF as part of the HPOG research portfolio on the first round of HPOG grantees, for which there are numerous data collections already approved by OMB (see Attachment F for further details). ACF and its contractors are engaged in many efforts to coordinate these evaluation activities and avoid duplication of work. The HPOG 2.0 Evaluation team has used the extensive knowledge generated to date from the research activities on the first round of HPOG and current Pathways for Advancing Careers and Education (PACE) programs to inform the proposed new data collection efforts for the second round of HPOG grantees.

A1.2 Legal or Administrative Requirements that Necessitate the Collection

H.R. 3590, the ACA requires an evaluation of the HPOG demonstration projects (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(B)). The Act further indicates that the evaluation will be used to inform the final report to Congress (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(C)). The Act calls for evaluation activities to assess the success of HPOG in “creating opportunities for developing and sustaining, particularly with respect to low-income individuals and other entry-level workers, a health professions workforce that has accessible entry points, that meets high standards for education, training, certification, and professional development, and that provides increased wages and affordable benefits, including healthcare coverage, that are responsive to the workforce’s needs” (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(B)).

There were two Funding Opportunity Announcements (FOAs) for the second round of HPOG grants—one for non-Tribal grantees (HHS-2015-ACF-OFA-FX-0951) and one for Tribal grantees (HHS-2015-ACF-OFA-FY-0952). Both FOAs required all HPOG 2.0 grantees to participate in a federal evaluation and to follow all evaluation protocols established by ACF or its designated contractors. Participating in the federal evaluations includes, but is not limited to, the use of the PAGES data system to collect uniform data elements and, for non-Tribal grantees participating in the National Evaluation, the facilitation of random assignment.

Data collected under PAGES will be used to automatically generate the federally required semi-annual reports, to inform ACF reports to Congress, to monitor and manage the performance of the grant-funded projects, to inform the HPOG 2.0 National Evaluation impact, outcomes and implementation studies and HPOG 2.0 Tribal Evaluation, and to inform other future research and evaluation efforts.

A2: Purpose of Survey and Data Collection Procedures

A2.1 Overview of Purpose and Approach

ACF is rigorously evaluating the effectiveness of the second round of HPOG grants. OPRE oversees the federal evaluation activities, which include an implementation and impact evaluation (with long-term follow-up) of the non-Tribal HPOG grantees under the National Evaluation, and a separate but coordinated Tribal Evaluation of the Tribal HPOG grantees. The federal evaluation activities are intended to expand the career pathways evidence base and to build on what has been learned to date about how to design and implement successful career pathways programs for low-income individuals, and improve the outcomes of individuals who participate in these programs. Under this information collect request, ACF seeks approval for data collection protocols for both the National and Tribal evaluations.

HPOG 2.0 National Evaluation

The National Evaluation involves random assignment of individual participants. As stated in the FOA, the non-Tribal HPOG 2.0 grantees are required to abide by random assignment procedures and facilitate the random assignment process for individuals, which involves entering eligible HPOG program applicants into a lottery to determine if they will be invited to participate in the program.

Applicants who are not invited to participate will serve as a control group in the evaluation. The control group members will not receive HPOG program services, but may enroll in any other program or service for which they are eligible. Individuals must complete the application process prior to random assignment; only individuals who have been deemed both eligible and suitable for program participation may be entered into the lottery.

For the National Evaluation, the request covers the descriptive evaluation protocols (Instruments 2-4) and the participant contact update activities for the impact evaluation (Instruments 5a and 5b).

HPOG 2.0 Tribal Evaluation

The purpose of the HPOG 2.0 Tribal Evaluation is to conduct a comprehensive implementation and outcome evaluation of the five Tribal HPOG 2.0 grantee programs. The HPOG 2.0 Tribal Evaluation will employ sound scientific methods and be grounded in sound cultural methods. At the start and throughout the process, the evaluators will engage with tribal leadership or authorized designee(s) to ensure that the evaluation is firmly anchored in questions that are meaningful to local stakeholders and that assist local service providers in better serving their communities. The evaluation of the Tribal HPOG 2.0 grantees will not involve random assignment. The evaluation will assess the HPOG 2.0 programs administered by tribes, tribal organizations, and tribal colleges to identify and assess how programmatic health profession training operations are working; determine differences in approaches used when programs are serving different sub-populations, including participants with different characteristics and skill levels; and identify programs and practices that seem to be successful in supporting the target population to achieve portable industry-recognized certificates or degrees as well as employment-related outcomes.

For the Tribal Evaluation, the request covers all of the data collection protocols to be used in the evaluation (Instruments 6-11).

A2.2 Research Questions

There is alignment between the National and Tribal evaluations; key questions applicable to both are in bold.

The National Evaluation will address several research questions through the descriptive and impact evaluations. We present here the research questions specifically related to the descriptive evaluation. The impact evaluation research questions will be summarized in a future request for clearance, along with the impact evaluation data collection protocols. The descriptive evaluation will address the following key research questions:

What is the nature of the labor market in which HPOG 2.0 programs operate?
What groups are targeted for HPOG participants?
What are eligibility criteria and processes?
What occupational training opportunities are available to HPOG participants? What is the nature of pre-training, supportive services, job placement and retention services?
To what degree do HPOG 2.0 programs conform to the career pathways framework? What are the pathways?
What are the roles of grantee and partner organizations in delivering services?
What changes to the service delivery system are associated with HPOG 2.0 implementation?
How are health profession training programs being implemented across the grantee sites?
What occupational training opportunities are available to HPOG participants? What is the nature of pre-training, supportive services, job placement and retention services?
What are the individual-level outputs and outcomes for participants in HPOG programs?
What were the component and unit costs of HPOG 2.0?

Overall, the Tribal Evaluation will address the following key research questions:

In what ways was the program designed or modified for Tribal organizations?
To what degree do the HPOG programs conform to the career pathways framework? What are the pathways?
What changes to the service delivery system are associated with program implementation?
How are health profession training programs being implemented across the grantee sites?
What occupational training opportunities are available to HPOG participants? What is the nature of pre-training, supportive services, job placement and retention services?
What key components or approaches seem to be successful or contribute to the success of these programs?
What are the individual-level outputs and outcomes for participants in the Tribal HPOG programs?
Do some programs or program components appear to be associated with positive outputs and outcomes for tribal populations? If so, what are the hypothesized reasons for differences between outcomes?
Do different program models, strategies, or components appear to lead to different outcomes for participants?
Is there evidence that participation in the program is positively associated with successful employment and work force capacity building outcomes?

A2.3 Study Design

HPOG 2.0 National Evaluation Study Design

The HPOG 2.0 National Evaluation is guided by the career pathways framework, as shown in the HPOG logic model (Attachment H). The framework puts into practice the assertion that “post-secondary training should be organized as a series of manageable and well-articulated steps accompanied by strong supports and connections to employment” (Fein et al., 2012). These articulated steps provide opportunities for students to advance through successively higher levels of education and training, exiting into employment at multiple possible points. The framework also incorporates customization, supports and employer connections.

The design for the HPOG 2.0 National Evaluation features a descriptive evaluation (including implementation, systems, and outcome studies) and a cost benefit study. In addition, the National Evaluation will conduct an impact evaluation, using a classic experimental design to measure and analyze key participant outcomes including completion of education and training, receipt of certificates and/or degrees, earnings, and employment in a healthcare career.

Guided by the framework, the goal of the descriptive evaluation is to describe and assess the implementation, outcomes, and systems related to the 27 non-Tribal HPOG grantees. The descriptive evaluation will also inform the impact evaluation; the research team will use results from the implementation study to evaluate the effectiveness of the approaches HPOG grantees use to improve HPOG participants’ attainment of education, training, employment, and advancement within the healthcare field.

The impact evaluation design includes randomizing program-eligible participants to treatment and control status in all non-Tribal sites. Treatment group members will participate in the HPOG program and the research study. Control group members will not have access to HPOG but will have access to other programs and services as available in the local community. The impact evaluation will also evaluate variation in participant impact that may be attributable to different HPOG program components and models. Additionally, the research team will match participant data collected through the impact evaluation for both the treatment and control groups to long-term employment and earnings data from ACF’s National Directory of New Hires (NDNH). An agreement with the Office of Child Support Enforcement (OCSE) is in progress.

The impact evaluation will conduct two types of participant data collection: 1) quarterly contact update requests; and 2) two follow-up surveys roughly 15 and 36 months after random assignment. The follow-up survey data collections will be submitted for OMB review and approval at a later date. The descriptive evaluation will involve document reviews and telephone and in-person data collection from grantees and their partners and stakeholders.

Exhibit 1 provides a visual description of the major components and sub-components of the HPOG 2.0 National Evaluation.

Exhibit 1: Components of the HPOG 2.0 National Evaluation

Shape1

HPOG 2.0 Tribal Evaluation Study Design

The HPOG 2.0 Tribal Evaluation is designed as a comprehensive implementation and outcome evaluation. The approach for the evaluation is guided by the seven values outlined in the Roadmap for Collaborative and Effective Evaluation in Tribal Communities, developed by the Child Welfare Research and Evaluation Tribal Workgroup.¹ The values provide guidance for partnering with tribal communities and are grounded in community-based participatory research. All five tribal grantees will participate in the HPOG 2.0 Tribal Evaluation. The evaluation will use a mixed-methods approach, including collection of qualitative data through interviews and focus groups and analysis of program documentation and program data. Qualitative data will be collected during annual site visits to each of the five Tribal HPOG 2.0 grantees.

HPOG Participant Accomplishment and Grantee Evaluation System (PAGES)

The previously approved PAGES system is designed to meet the performance data needs of the grantees and of OFA to monitor the grantee performance and prepare the report to Congress on the grants. PAGES will support the National and Tribal Evaluations, as well as other future research and evaluation efforts sponsored by ACF. The information collection gathers data on (1) grantee program designs and offerings; (2) intake information on eligible applicants (at both the tribal and non-Tribal grantees) through baseline data collection and (3) individual enrolled program participants’ activities and outcomes.

Grantees will use the data collected through the system to generate the required Performance Progress Reports (PPRs) for OFA. The PPR includes two sections—a narrative section and a quantitative section. In the narrative section, grantees are required to provide OFA updates on their programs including major activities and accomplishments, problems, significant findings and events, dissemination activities, other activities, and activities planned for the next reporting period. This section is taken from the ACF Performance Progress Report, a standard form approved by OMB under number 0970-0406. The narrative portion of this form is embedded in PAGES so that grantees can type the narrative directly into the PAGES system (note the burden for this narrative portion is included in the burden estimates so that the narrative section assumed the OMB number and expiration date of this information request). In the quantitative section, grantees enter their projections for the grant year on enrollment, basic skills training enrollment and completions, healthcare training enrollment and completions, and employment. The quantitative section also includes metrics that are automatically calculated from data entered in other parts of PAGES—i.e., actual levels of enrollment, completions, and employment, as well as information on participant status, other skill development activity participation, and support service receipt. This part of the PPR also includes narrative fields for grantees to provide information on why projected values differ from actual values. Both the narrative and quantitative sections of the PPR are able to be saved and printed as one file for submission to OFA by grantees. See the full list of PPR items and a mockup of the PPR in Attachment E.

Data collected in PAGES will also be used in other components of the HPOG 2.0 National Evaluation and HPOG 2.0 Tribal Evaluation. Program level data will help analyze each grantee’s and program’s inputs and outputs and place analytic results into the appropriate context. Participant-level data will be used in the impact evaluation to identify balance between the treatment and control groups, to increase the precision of estimates regarding the impact of program components, and to identify subgroups for subgroup impact analysis at follow-up. PAGES will support the National Evaluation descriptive evaluation by providing information on grantee program characteristics and program performance to date. Participant-level data will also enable the HPOG 2.0 National Evaluation and HPOG 2.0 Tribal Evaluation teams to track the participant educational and employment outcomes.

A2.4 Universe of Data Collection Efforts

To address these research questions, the HPOG 2.0 National and Tribal Evaluation will use a number of data collection instruments. Instruments in this clearance request include the following:

Screening Interview to identify respondents for the HPOG 2.0 National Evaluation first-round telephone grantee interviews. (Instrument 2)
HPOG 2.0 National Evaluation first-round telephone interviews. . These interviews, conducted with management staff, partners and stakeholders, will collect information about the HPOG program context and about program administration, activities and services, partner and stakeholder roles and networks, and respondent perceptions of the program’s strengths. (Instrument 3)
HPOG 2.0 National Evaluation in-person implementation interviews will collect information from five to ten HPOG 2.0 programs with promising approaches to the topic areas of specific interest to ACF, including employer engagement; basic skills instruction; career pathways training opportunities; work-readiness training; and program sustainability after the end of the HPOG 2.0 grant period. In consultation with ACF, the programs selected for this limited data collection will be identified through the first-round telephone interviews as those that show the most promising or innovative approaches in each topic area. (Instrument 4)
HPOG 2.0 National Evaluation contact update forms. This form will collect updated participant contact information for impact evaluation participants (treatment and control) during the follow-up period. This form is included as part of the welcome packet (Instrument 5a) and then sent every three months accompanied with the contact update letter and form. Attachment G is replaced by this contact update letter and form. (Instrument 5b)
HPOG 2.0 Tribal Evaluation grantee and partner administrative staff interviews will collect information on high-level program strategies, partnerships in place to implement the Tribal HPOG 2.0 program, program development and lessons learned. (Instrument 6)
HPOG 2.0 Tribal Evaluation program implementation staff interviews will collect information from instructors, trainers, recruitment and orientation staff, and providers of program or supportive services on Tribal HPOG 2.0 program processes including recruitment, screening, orientation, provision of supportive services, and program implementation. (Instrument 7)
HPOG 2.0 Tribal Evaluation employer interviews will collect information from local or regional employers that are partnering with Tribal HPOG 2.0 programs or have employed program participants and collect information on employers’ impressions of the Tribal HPOG 2.0 program and program graduates. (Instrument 8)
HPOG 2.0 Tribal Evaluation program participant focus groups will collect information on participants’ perceptions, experience, outcomes and satisfaction with the Tribal HPOG 2.0 program. (Instrument 9)
HPOG 2.0 Tribal Evaluation program participant completer interviews will collect information on the current employment status of the participants who completed a training program and their perceptions of and satisfaction with the Tribal HPOG 2.0 program. (Instrument 10)
HPOG 2.0 Tribal Evaluation program participant non-completer interviews will collect information on reasons participants left the program, short-term outcomes, how they feel the program could be improved, and any plans for future academic training. (Instrument 11)

These data are not available through any current sources.

Study instruments approved by OMB in prior information collection requests include the following:

PAGES Grantee- and Participant-Level Data Collection. This includes grantee-level data collection on program components (e.g. training courses offered, types of supports offered) and participant-level data on participation, services provided, and program outputs. (Instrument 1)
PAGES Participant-Level Baseline Data Collection (participants at non-tribal grantees participating in the impact evaluation). This includes data on characteristics of eligible individual participants at intake (e.g., demographics, household characteristics, employment and education experiences, a child roster, and baseline data on expectations for the program) at the non-tribal grantees. (Instrument 1)
PAGES Participant-Level Baseline Data Collection (participants at tribal grantees). This includes data on characteristics of eligible individual participants at intake (e.g., demographics, household characteristics, employment and education experiences, and tribal specific data items) at the tribal grantees. (Instrument 1)
Informed Consent Forms (Form A: National Evaluation lottery required; Form B: National Evaluation, lottery not required; Tribal Informed Consent Form A -SSNs Included; Tribal Informed Consent Form B-Unique Identifiers Included). The informed consent forms provide information to participants to ensure they understand the nature of the research and evaluation activities being conducted. (Attachment B, B2 and B3)

As part of the HPOG 2.0 data collection, we anticipate submitting a future additional OMB clearance request for the following for the HPOG 2.0 National Evaluation impact evaluation protocols:

HPOG 2.0 National Evaluation Participant Follow-up Surveys to measure outcomes at 15-months post random assignment and to measure the outcomes at 36-months post random assignment.
HPOG 2.0 Cost Forms which will collect data on costs associated with the implementation of the HPOG program to support a cost benefit analysis.

The future OMB clearance request may also include the following protocols for use in the HPOG 2.0 National Evaluation descriptive evaluation:

HPOG 2.0 National Evaluation second-round telephone interviews with management and staff. These interviews may collect information about notable implementation and performance issues as well as changes to the HPOG network and systems.

Other extant data sources will be used for the HPOG 2.0 National and Tribal Evaluation. These include the following:

National Directory of New Hires (NDNH). These data will provide information about employment and earnings of HPOG participants.
National Student Clearinghouse (NSC). These data will provide information on student enrollment in credit-bearing courses (and some enrollment in non-credit bearing courses) and receipt of post-secondary degrees.
HPOG program management information, including initial applications and ongoing management reports, which will provide supplemental information in tracking the evaluation of the grant, and information on the local healthcare labor market and needs for occupational training.
Government sources of labor market data, from the U.S. Census Bureau and Bureau of Labor Statistics (BLS), such as County Business Patterns, Local Area Unemployment Statistics (LAUS), Quarterly Workforce Indicators QWI), which will provide a picture of the local labor market.

A3: Improved Information Technology to Reduce Burden

The HPOG 2.0 National Evaluation and HPOG 2.0 Tribal Evaluations are designed to limit the reporting burden for respondents. As described in the originally approved supporting statement May 2015, with revisions in January and July 2016, participant and grantee level data will be collected through PAGES, a cloud-based data system. The evaluation teams will use the quantitative data collected through PAGES to reduce respondent burden wherever possible. The team will rely on administrative data—such as NDNH—to capture employment and wage data. This removes the burden from collecting this information from participants during the follow-up survey. Any requests for program documentation will be collected electronically as well.

The HPOG 2.0 National Evaluation impact evaluation will offer study participants the option to update their contact information online, by mail, or by telephone.

A4: Efforts to Identify Duplication

The purposes of the HPOG 2.0 National Evaluation telephone interviews and site visits are to obtain information about HPOG grantee programs that is not available through any other source. The research team will use existing sources of information including PAGES, available information from the site-specific evaluation design and other existing site-specific materials developed earlier by the National Evaluation team to guide implementation of random assignment procedures and site monitoring notes to collect key information prior to conducting the telephone interviews and site visits. The team will not ask for this information in the data collection efforts described here. These data sources and information available in each include the following:

Evaluation Design and Other Site-Specific Materials
1. Target population and recruitment strategies
2. Eligibility criteria and application processes
3. Control group services and conditions
PAGES and Program Performance Progress Reports (PPRs)
1. Available training courses and support services
2. Participant characteristics
3. Participant take-up of training courses and support services
4. Participant education and employment outcomes
Site Team Monitoring Reports
1. Documented program changes
Grantee applications
1. Organization type

The purposes of the HPOG 2.0 Tribal Evaluation interviews and focus groups are to obtain information about Tribal HPOG grantee programs that is not available through any other source. The qualitative data collected will complement other data sources, not duplicate or replace them. The research team will use existing sources of information, including the following:

PAGES and Program PPRs
1. Available training courses and support services
2. Participant characteristics
3. Participant take-up of training courses and support services
4. Participant education and employment outcomes
Grantee applications
1. Organization type
Program documentation including program brochures and materials, organizational policies, and curricula.
1. Available training courses
2. Recruitment strategies

A5: Involvement of Small Organizations

The National Evaluation and Tribal Evaluation will have minimal impact on small organizations. The primary organizations involved in this study will be tribal and community colleges, workforce development agencies, tribal organizations, and community-based organizations that operate occupational training programs. The funding announcement informed all grantees of the federal evaluation and reporting requirements, and adequate resources have been provided to coordinate the data collection and reporting. There should be no adverse impact for any grantees participating in the study.

Small business professionals will only be interviewed if they are employers of National or Tribal HPOG 2.0 program graduates or grantee administrative partners. In an effort to reduce burden, the duration of each employer interview will be no more than 45 minutes.

A6: Consequences of Less Frequent Data Collection

For the HPOG 2.0 National Evaluation descriptive evaluation and the HPOG 2.0 Tribal Evaluation, the consequences of less frequent data collection would result in a lack of essential and timely information. Collecting data less frequently would preclude the Tribal Evaluation team from collecting essential information about program progression from year to year. Since this grant is administered over five years, four years of annual data collection at each site will provide a rich set of data that will clearly demonstrate program successes or struggles over time, and allow for the evaluation of Tribal HPOG 2.0 grantee program. ACF will benefit from having annual information available on the processes and outcomes of these grant programs.

For the National Evaluation descriptive evaluation, less frequent data collection would prohibit timely collection of data about program implementation, costs, and systems over time. The participant contact updates are critical to maintaining up to date contact information on impact evaluation participants during the three-year follow-up period. Prior career pathways studies (the impact study of the first round of HPOG grants, and Pathways for Advancing Careers and Education (PACE), OMB control numbers 0970-0394 and 0970-0397 respectively), enrolled participants with similar characteristics to those we expect will enroll in the HPOG 2.0 National Evaluation. Those prior studies have shown that this sample is likely to be very difficult to track and locate over time. Thus, less frequent collection of participant contact updates could result in out-of-date contact information, and ultimately lower response rates to the follow-up survey efforts.

A7: Special Circumstances

There are no special circumstances for the proposed data collection.

A8: Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on August 15, 2016, Volume 81, Number 157, page 54096-7, and provided a sixty-day period for public comment. A copy of this notice is attached as Attachment C. During the notice and comment period, the government received no requests for information or substantive comments.

Consultation with Experts Outside of the Study

The HPOG 2.0 Tribal Evaluation team consulted with outside experts about the proposed data collection and evaluation plan. Experts in the fields of health professions training and research in tribal communities reviewed the HPOG 2.0 Tribal Evaluation design and all comments, questions and suggestions were resolved during consultation. Additionally, the HPOG 2.0 Tribal Evaluation team worked closely with each tribal grantee for their input on the evaluation design and draft protocols. Consultants are listed below. This consultation took place in 2016.

Name	Title/ Organization	Contact Information
Mark Doescher, MD, MSPH	Stephenson Cancer Center, University of Oklahoma	Mark-Doescher@ouhsc.edu
Loretta Heuer, PhD, RN, FAAN	School of Nursing, North Dakota State University	loretta.heuer@ndsu.edu 701.231.8205
Joan LaFrance, Ed.D	Mekinak Consulting	lafrancejl@gmail.com
Myra Parker, JD, MPH, PhD	Center for the Study of Health and Risk Behaviors, University of Washington	myrap@uw.edu (206) 616-5887
Rick Haverkate	Deputy Director, Indian Health Service	Richard.Haverkate@ihs.gov 301-945-3224

The majority of PAGES grantee- and participant-level data items are adapted from previously approved data collection instruments for PACE (clearance number 0970-0343) and HPOG ISO and HPOG-Impact (both under clearance number 0970-0394), as described in Attachment D.

PAGES data items were also developed in consultation with senior methodological and substantive experts, including: Karen Staha, Department of Labor; Yvette Chocolaad, National Association of State Workforce Agencies; Burt Barnow, George Washington University; Tim Harmon, Workforce Enterprise Services, Sung-Woo Cho, Matthew Zeidenberg, and David Fein, Abt Associates; Keith Watson, Lauren Eyster, Alan Dodkowitz, Urban Institute.

A9: Incentives for Respondents

There are no incentives provided to respondents for the data collection via PAGES, as that information is necessary for program participation, not simply for evaluation purposes. The evaluators plan to offer incentives to respondents for both the National Evaluation impact evaluation and the Tribal Evaluation. The justification and incentive plans for each study are provided below.

Tokens of Appreciation—National Evaluation

In a longitudinal study such as the HPOG 2.0 National Evaluation, panel retention during the follow-up period is critical to minimizing the risk of nonresponse bias and to achieving sufficient sample for analysis. Although low response rates do not necessarily lead to nonresponse bias and it is at least theoretically possible to increase nonresponse bias by employing some techniques to boost response rates (Groves, 2006), most statisticians and econometricians involved in the design and analysis of randomized field trials of social programs agree that it is generally desirable to obtain a response rate close to 80 percent in all arms of the trial (Deke and Chiang, 2016). The work of Deke and Chiang underlies the influential guidelines of the What Works Clearinghouse (WWC). Under those guidelines, the evidential quality rating of an evaluation is sharply downgraded if the differential exceeds a certain tolerance (like 4 percentage points at an overall response rate of 80 percent). Based on the research team’s experience with differential response rates in the PACE and HPOG 1.0 data collection efforts, the team believes that there is some risk that the HPOG 2.0 study might be in the penalized range if the team continues to use the follow-up protocols employed in the prior studies. PACE had a differential response rate of 5.1 percentage points. In the HPOG 1.0 three-armed experiment, the response rate differential for the standard treatment vs. the control group was 7.17 percentage points, and 6.57 percentage points for the enhanced treatment vs. the control group.

The team has taken several steps to help maximize retention rates in both arms and thereby ensure a high rating for the HPOG 2.0 National Evaluation. To achieve this goal, the team focused on steps that would help address two challenges:

Participant mobility—over a long follow-up period, study participants may relocate, making it difficult to find them and update their contact information or complete a follow-up interview; and
Participant engagement—the ability to keep participants engaged in the research study.

In an effort to strengthen participants’ engagement, the study team will send each study participant a welcome packet the month after enrollment. (See Instrument 5a for the welcome packet and Instrument 5 b for the contact update form.) We propose including a non-monetary incentive, such as a portable cell phone charger or comparably valued item, to all participants as part of the welcome packet. This item will be branded with the HPOG study logo. This token of appreciation is a tool to remind the participant about the study (rather than the program). The chargers are a practical item that can be used in a computer’s USB port or in a charging block for a cell phone or other portable device. The fact that the charger is practical will encourage participants to use it, and thus to see a regular reminder of their participation in the HPOG study. The team anticipates that this will be particularly important for the control group since most members of the treatment group will have many reminders of HPOG as a result of the training they will be receiving.

The welcome packets also include the participant contact update form to deal with the mobility challenge. These contact update forms will be sent to participants every three months during the follow-up period. The participant contact update form does not collect any data for analytic use, but these updates are crucial to ensuring that the contact information in the sample database is as up to date as possible during the follow-up period. Although the team is not aware of any true experiments on the effects of requests for contact updates, the team does have strong circumstantial evidence for their effectiveness in raising response rates in the follow-up survey. This evidence arises in particular from the first follow-up survey for the PACE study, where no requests were sent to early cohorts, but requests were sent to subsequent cohorts. While the final response rate differential did not differ substantially between the earlier and later cohorts, the earlier cohorts had to be worked much longer to achieve completion targets. This increased cost to the government due to the higher level of effort needed by the survey team. This also meant that study participants in the early cohorts were actually interviewed as long as 24 months after random assignment, instead of closer to 15 months post-random assignment, as intended. Such variation in lags undercuts the utility of the survey to explore factors influencing progression for outcomes such as cumulative credits earned and attainment of credentials, which are very sensitive to the lag between randomization and interview.

Furthermore, the study team proposes to offer a token of appreciation valued at $5 for each round of quarterly participant contact updates (see Instrument 5b for the contact update form). The proposed amount is comparable to what OMB approved for the participant contact update efforts leading up to the 15-month follow-up survey efforts for both the PACE study and the first round of HPOG grants (OMB Nos. 0970-0397 and 0970-0394 respectively). Participants will receive their token of appreciation after they provide updated contact information.

Incentives show study participants that the study team appreciates their continued support and cooperation with the study requirements and their ongoing participation in study information collection activities. The team theorizes that these tokens of appreciation will be a particularly powerful tool for maintaining a high response rate in the control group given that these sample members do not receive any (other) program benefits or services. Although evidence of the effectiveness of incentives in reducing nonresponse bias appears to be nonexistent, it is well established that incentives strongly reduce attrition in panel studies.²

Tokens of Appreciation—Tribal Evaluation

The Tribal Evaluation will use incentives to encourage participation in focus groups and individual follow-up interviews. Offering incentives to gain cooperation and solicit participation is a well-established practice in social science research and program evaluation for both small-scale studies and sample surveys. Participants are provided incentives as a gesture of appreciation for voluntary participation in data collection activities.

We have worked closely with the tribal grantees to design and implement a culturally responsive evaluation. Based on our previous experience with the Tribal Evaluation of the first round of HPOG, we learned that there is the potential for non-response bias due to circumstances experienced by Tribal HPOG participants. HPOG participants in tribal programs very often have substantial family commitments, including caregiving for both children and elderly family members, and may live considerable distances from grantee organizations (where focus groups and interviews typically are conducted). These commitments and travel time required pose additional burdens to participating in research compared to other populations. In addition, the expenses associated with participation, including childcare and transportation, place additional burden on potential respondents.

Additionally, based on our experience working with Tribal grantees and HPOG participants during the first round of HPOG, tribal members can be reluctant to participate in research activities. Monetary incentives are used regularly when conducting research in tribal communities (Sobeck, 2003). Researchers have found financial incentives to be a motivator for tribal participation in research. Use of incentives also increases the likelihood that recruited participants will participate in the data collection activities.

Given these circumstances, there is the potential for non-response bias in the data collection as participants who have family commitments, longer travel time, expenses associated with research participation, or reluctance to participate in research may not participate in data collection activities. Our prior work with this population showed that participants were more likely to be single mothers, and many participants traveled significant distance to participate in evaluation activities. An insufficient incentive is likely to reduce participation among those with family commitments or longer travel requirements such that they would be underrepresented in data collection activities, thereby resulting in non-response bias. Offering an incentive to participate in the study will therefore help to offset the potential of non-response bias.

Given the travel time required for an in-person focus group or interview, incentives for participation in the in-person 90 minute focus group or in-person 60 minute completer or non-completer interview will be a non-cash honorarium valued at $50. The HPOG 2.0 Tribal Evaluation team will consult with each grantee to determine the most appropriate non-cash honorarium (e.g., gift certificate to a local grocery store) to send to the participant.

A10: Privacy of Respondents

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. Participants will be allowed to receive services under the Tribal Evaluation if they do not provide a social security number. For the National Evaluation, study participants must provide an SSN in order to enroll in the program. Informed Consent Form A—Lottery Required and Form B—Lottery Not Required explain this information for participants in the HPOG 2.0 National Evaluation. Comparably, Tribal Informed Consent Form A-SSN Required and Tribal Informed Consent Form B-Unique Identifier Required provide this information to HPOG 2.0 Tribal Evaluation participants. Consent forms are found in Attachment B. For national evaluation participants and tribal evaluation participants who provide SSNs, the consent forms (Informed Consent Form A, Informed Consent Form B, and Tribal Informed Consent Form A) clearly state how SSNs will be used in the evaluation.

As specified in the evaluator’s contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Security and Monitoring Plan that assesses all protections of respondents’ personally identifiable information. The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. All project and grantee staff with access to PAGES sign a New User Data Security Agreement and they undergo training on data privacy and security. Grantees participating in the National Evaluation that do not have their own Institutional Review Board (IRB) or Federalwide Assurance Number (FWA) sign individual investigator agreements, which allows the protection under Abt’s FWA. Grantees participating in the Tribal Evaluation that do not have their own IRB or FWA will sign individual investigator agreements, which will allow them protections under NORC’s FWA.

As specified in the evaluator’s contract, the Contractor shall use Federal Information Processing Standard (currently, FIPS 140-2) compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor has submitted a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.

For the HPOG 2.0 National Evaluation descriptive evaluation and the Tribal Evaluation, none of the respondents that participate in interviews or focus groups will be identified in any report or publication of this study or its results; their participation will be voluntary; and their information will be kept private. This information will be provided verbally to interview respondents in both studies, and verbal consent will be requested.

The Tribal Evaluation focus group participants will receive a written consent form. The Program Participant Focus Group Informed Consent (Attachment I) will provide all of the assurances of privacy and data protection information to focus group participants. The Tribal Evaluation interview respondents will provide verbal consent (Attachment J). These consent forms are for the tribal evaluation interviews and focus groups, separate from the consent forms that study participants sign at the time of enrollment.

PAGES data collection includes personally identifiable information (PII). Respondent privacy will be protected to the extent allowed by law. ACF recognizes that HPOG 2.0 grantees serve vulnerable populations (per the authorizing legislation), and that grantees must protect those populations from any risks of harm from the research and evaluation activities. Accordingly, the HPOG 2.0 Evaluation team will obtain informed consent forms from all study participants. This informed consent will ensure that participants understand the nature of the research and evaluation activities being conducted (previously approved consent forms can be found in Attachment B).

As a part of informed consent, the following rationale for data collection and privacy assurances will be provided to HPOG 2.0 participants by grantees:

Research is being conducted to see if and how HPOG 2.0 makes a difference in people’s lives by helping them complete training and get healthcare jobs. This program and research are funded by HHS, and HHS may fund other research on this program in the future.
In this program, grantees will collect some personal information from individuals, such as their name, date of birth, Social Security number, and involvement in other programs.³ The researchers studying the program for the government also need this information. Researchers will use data security procedures to keep all of the study data private and to protect individuals’ personal information. All of the information collected for the program or for the research studies will be kept completely private to the extent allowed by law, and no one’s name will ever appear in any report or discussion of the evaluation results.
Researchers may contact applicants at grantees participating in the impact evaluation in the future. Individuals may refuse to answer any of their specific questions at any time.

OPRE published a Privacy Impact Assessment (PIA) to ensure that information handling conforms with applicable legal, regulatory, and policy requirements regarding privacy; determine the risks of collecting and maintaining PII; assists in identifying protections and alternative processes for handling PII to mitigate potential privacy risks; and communicates an information system’s privacy practices to the public. This PIA, titled Participant Accomplishment and Grant Evaluation System (PAGES), was approved on October 9, 2015 and is available online through HHS at http://www.hhs.gov/pia/#System.

PAGES was developed using the highest standards of technology and data security. Data for grantee-level and individual-level records will be stored securely in an SQL server database. The web interface for data entry and reporting is built on the industry leading Microsoft Dynamics customer relationship management (CRM) platform. The system is hosted and maintained on Microsoft Dynamics CRM Online Government, a highly secure Federal Information Security Management Act (FISMA) Moderate compliant cloud-based Software as a Service (SaaS) solution.

Accounts on the web server will be protected with dual factor authentication, to include a password and an additional means of authentication. Dynamics CRM uses HTTPS with the SSL/TLS protocol providing encrypted communication and secure identification of the network web server. The platform is heavily utilized in other Federal Government organizations with externally facing instances and has undergone and passed all Authority to Operate (ATO) and security protocols within these organizations. All data is filtered using the security model so records and fields containing Personally Identifiable Information (PII) data are removed for users that do not have authorization. Any logging or output files will not contain private data and will be limited to generic system and error data.

PAGES will support field-level security so users without authorization to specific data do not see the data on forms, views or reports. Thus, private participant data such as Social Security numbers will be entered into the system and encrypted at the field level, but will not be visually displayed or downloadable by system users. User-identifiable participant-specific data will be stored separately from grantee-level data and will be available for updating only by the grantee representative who originally entered the data. Grantee-specific data will be available to the project team in specific extracts and reports once the information has been entered and submitted. Information will not be maintained in a paper or electronic system from which they are actually or directly retrieved by an individuals’ personal identifier.

Grantees have received detailed guidelines and training on data entry and security procedures. Clearly defined variables and labeled fields specify how to enter each data element. Training and supporting guidance documents have been provided to grantees and technical assistance on the system is available to grantees throughout their grant period of performance.

A11: Sensitive Questions

None of the HPOG 2.0 National Evaluation instruments include questions of a sensitive nature. Several questions in the HPOG 2.0 Tribal Evaluation program participant focus groups (Instrument 9), HPOG 2.0 Tribal Evaluation program participant completer interviews (Instrument 10), and HPOG 2.0 Tribal Evaluation program participant non-completer interviews (Instrument 11) may be considered sensitive by some program participants. These questions ask about participant and family needs and what types of supportive services received, including academic, social, and employment related. These questions are necessary because supportive services are a key component of the HPOG Program. Data collected will be used to identify how HPOG programs assess student needs and what types of services they are offering as part of their program. Participants will be informed that their participation is voluntary, that they may decline to answer any question that they wish, and that their information will be kept private and they will not be identified in any report or publication of this study or its results.

PAGES Participant-Level Baseline Questions

Several data items pertaining to individual participant characteristics may be considered sensitive by some program participants, for example, questions about criminal records, disabilities, and limited English proficiency. These data are standard items for other workforce development programs and will allow important comparisons between HPOG and other similar efforts for evaluation and management purposes. In addition, these data are needed to fully identify programs and program characteristics that are most successful for serving the vulnerable populations that HPOG was designed to support and that are a focus of ACF’s assistance programs.

Individual identifying information of a sensitive, personal, or private nature that all HPOG 2.0 grantee applicants will complete includes: (1) last and first name; (2) Social Security number; (3) date of birth, (4) ethnicity and race; (5) marital status; (6) number of children; (7) whether the individual is a TANF or SNAP recipient; (8) disabilities, (9) limited English proficiency, and (10) employment status at program intake and exit.

Reasons why these data items are necessary and specific uses to be made of this information include:

The last and first names and Social Security Numbers of each participant are needed by evaluators to obtain accurate administrative data on individuals’ quarterly earnings and receipt of cash and noncash public benefits. Those data are necessary to measure key impacts. Administrative data matches will only be done for those participants who sign informed consent data agreements.
Other data which are of a sensitive nature—including date of birth, ethnicity, race, marital status, and whether the individual receives public benefits at baseline—are needed to support detailed analyses of the types of individuals enrolled into HPOG programs and the services and education or training they receive.
Some characteristics, such as disability status or limited English proficiency, are likely to be associated with special instructional needs. Performance and outcome analysis for these subgroups of participants may provide particularly useful information about effective training approaches for these populations.
Employment information at program intake and exit will provide essential information about the effectiveness of the HPOG 2.0 programs, particularly whether individuals enter health occupations after training.

Among the baseline data collection items for applicants at the non-Tribal grantees participating in the federal impact evaluation are ones addressing respondents’ employment barriers, personal preferences, motivations and self-efficacy. Some respondents may consider these questions to be sensitive. Because program staff will be collecting the data, we designed the measures to be as neutral and impersonal as possible, framing them as part of an assessment of getting to know clients’ needs and strengths so that the program might best meet their needs and capitalize on their strengths.

The literature provides ample support for including these items to identify barriers to employment (e.g., Matus-Grossman and Gooden, 2002; Fein and Beecroft, 2006; Michalolpoulos and Schwartz, 2001). Including these items is necessary to describe the characteristics of the study population and evaluate their moderating effects on program impacts. Furthermore, questions pertaining to personal preferences, motivations and self-efficacy will be especially useful for identifying the pathways that participants follow through multi-faceted programs, thereby allowing researchers to estimate the impacts of various program models and components. Program staff will remind potential study members during the enrollment process that they may refuse to answer individual items. Potential study members will also be reminded that their responses will be kept private, to encourage their candid responses.

ACF also seeks to combine comparable data from different, related evaluations to enhance the cumulative development of knowledge useful to government policymakers, program operators, and the public. As outlined in the justification table included in Attachment D, these baseline data items are being used in the evaluations of the first round of HPOG grantees. Asking the same questions of HPOG 2.0 applicants will allow for comparative analyses of the first and second round of HPOG programs.

A12: Estimation of Information Collection Burden

A12. 1 Previously Approved Information Collections

Total Burden Previously Approved

The previously approved burden estimates included: 1) burden on grantee staff members who enter grantee-level and ongoing participant-level data into PAGES to complete the HPOG PPRs; 2) burden on HPOG applicants to complete the baseline questions; and 3) burden on grantee staff who enter the baseline data into PAGES.

The total burden for the instruments already approved was estimated to be 6,432 hours annually, or 19,296 hours total.

Burden Remaining from Previously Approved Information Collection

Estimated burden remaining to continue use of the previously approved instruments is 5,058 hours annually, or 15,174 hours over the next three years. Exhibit A12.1 shows the remaining hourly and cost burden.

Exhibit A12.1: Burden Remaining from Previously Approved Information Collection

Instrument	Total Number of Respondents	Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total Annual Cost
Instrument 1: PAGES Grantee- and Participant-Level Data Collection (all grantees)	80	27	2	31.75	1,715	$28.29	$48,517.35
Instrument 1: PAGES Participant-Level Baseline Data Collection (participants at non-Tribal grantees)	19,617	6,539	1	.5	3,270	$3.94	$12,883.80
Instrument 1: PAGES Participant-Level Baseline Data Collection (participants at Tribal grantees)	877	292	1	.25	73	$3.94	$287.62
Estimated Annual Burden Previously Approved					5,058		$61,688.77

A12. 2 Additional Burden for Previously Approved Information Collection

The actual number of expected participants in the National Evaluation over the next three years is now 30,600 participants. This number exceeds the expected enrollment numbers reflected in the original participant burden estimates for the previously approved information collection (25,800). See Supporting Statement B1, for details on the respondent universe. Exhibit A12.2 shows the additional burden estimates in both hours and cost, associated with the higher than expected projected enrollment.

Exhibit A12.2: Additional Burden for Previously Approved Information Collection

Instrument	Total Number of Respondents	Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total Annual Cost
Instrument 1: PAGES Participant-Level Baseline Data Collection (participants at non-Tribal grantees)	4,860	1,620	1	.5	810	$3.94	$3,191.40
Estimated Additional Annual Burden Previously Approved					810		$3,191.40

A12. 3 Newly Requested Information Collections

Exhibit A12.3 presents the reporting burden in both hours and cost for National Evaluation grantee staff who will participate in the descriptive evaluation data collection, and the impact evaluation participants who complete and submit the contact update data collection forms. The burden estimates for Instrument 4: HPOG 2.0 National Evaluation in-person implementation interviews are revised to reflect the decision to conduct in-person implementation interviews from staff in up to ten HPOG 2.0 programs (rather than the originally proposed six programs). The table then shows the burden estimates for grantee staff and study participants who participate in interviews and focus groups as part of the Tribal Evaluation.

Exhibit A12.3: Burden for Newly Requested Information Collection

Instrument	Total Number of Respondents		Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total Annual Cost
HPOG 2.0 National Evaluation
Instrument 2: Screening Interview to identify respondents for the HPOG 2.0 National Evaluation first-round telephone interviews		38	13	1	.5	7	$28.29	$198.03
Instrument 3: HPOG 2.0 National Evaluation first-round telephone interviews with management, staff, partners, and stakeholders		190	63	1	1.25	79	$28.29	$2234.91
Instrument 4: HPOG 2.0 National Evaluation in-person implementation interviews⁴		100	33	1	1.5	49.5	$28.29	$1400.36
Instrument 5a: HPOG 2.0 National Evaluation welcome packet and participant contact update forms		15,750	5,250	1	.1	525	$10.15	$5,328.75
Instrument 5b: HPOG 2.0 National Evaluation letter and participant contact update form		15,750	5,250	3	.1	1,575	$10.15	$15,986.25
HPOG 2.0 Tribal Evaluation
Instrument 6: HPOG 2.0 Tribal Evaluation grantee and partner administrative staff interviews		105	35	1	1	35	$28.29	$990.15
Instrument 7: HPOG 2.0 Tribal Evaluation program implementation staff interviews		150	50	1	1.5	75	$28.29	$2,121.75
Instrument 8: HPOG 2.0 Tribal Evaluation employer interviews		90	30	1	.75	23	$50.99	$1,172.77
Instrument 9: HPOG 2.0 Tribal Evaluation program participant focus groups		405	135	1	1.5	203	$10.15	$2060.45
Instrument 10: HPOG 2.0 Tribal Evaluation program participant completer interviews		300	100	1	1	100	$10.15	$1,015
Instrument 11: HPOG 2.0 Tribal Evaluation program participant non-completer interviews		150	50	1	1	50	$10.15	$507.50
Estimated Annual Burden Total						2,722		$31,161.92

A12.4 Total Burden under OMB # 0970-0462

Exhibit A12.4 shows the estimated annual respondent burden over the next three years is 8,590 hours. This represents the total burden remaining from the previously approved information collection and the additional burden hours for previously approved information collection, due to higher enrollment projections. The table then shows the total annual burden estimates for the new information collection.

Exhibit A12.4: Total Burden under OMB #0970-0462

Instrument	Annual Burden Hours
Burden Remaining from Previously Approved Information Collection	5,058
Additional Burden for Previously Approved Information Collection	810
Burden for New Information Collection	2,722
Total Annual Burden Hours	8,590

Total Annual Cost

To compute the total estimated annual cost reported in Exhibit A12.1, evaluators used the average wage for HPOG 1.0 participants employed at program intake ($10.64) and multiplied that by the proportion of those working at intake (0.37) for an average hourly total of $3.94. Evaluators believe the HPOG 1.0 data provide an accurate basis for estimating wages for HPOG 2.0 study participants for the previously approved information collection under PAGES. The baseline wage was appropriate for the original HPOG Next Generation submission as the PAGES system collects wage information at the time of enrollment. Since, this is a job training program we have revised the cost in the burden table in Supporting Statement A to reflect the loaded federal minimum wage. The loaded federal minimum wage was used in the previously approved information collection requests for the HPOG 1.0 15- and 36-Month Follow-up Surveys and the PACE 15 and 36-Month Follow-up Surveys (OMB Nos. 0970-0394 and 0970-0397 respectively). For the cost to grantees and partner organizations data collection efforts, the total burden costs were multiplied by the average hourly wage, according to the Bureau of Labor Statistics, National Compensation Survey, 2010 ($28.29/hour).⁵ The average hourly wage for the employer interviews is based on Bureau of Labor Statistics code 11-9111, Medical and Health Services Managers ($50.99). The total annual cost burden of collecting this new information is $31,161.92. We estimate that the annual costs for the remaining previously approved data collection is $61,688.77 over the next three years. The costs associated with the additional previously approved information collection is $3,191.90. The total annual cost burden for the two efforts combined is estimated at $96,043.59.

A13: Cost Burden to Respondents or Record Keepers

Not applicable. The proposed information collection activities do not place any new capital cost or cost of maintaining capital requirements on respondents.

A14: Estimate of Cost to the Federal Government

The total cost for the data collection activities under this current request will be $1,788,164 for the National Evaluation descriptive evaluation and $1,225,193 for the Tribal Evaluation, for a total of $3,013,357. The total amount for the previously approved submission was $2,020,248. Thus, the total costs to the Federal government are $5,033,605. Annual costs to the Federal government will be $1,677,868.33 per year for three years.

A15: Change in Burden

This is an additional information collection request under OMB #0970-0462. The burden estimates include estimates for the new information collection that are the subject of this request. It also includes changes to the previously approved burden estimates. The original burden estimates were approved prior to the award of the HPOG 2.0 grants. The National Evaluation grantees selected have higher than anticipated impact evaluation enrollment projections over the next three years. As a result, the number of participants expected to enroll in HPOG 2.0 and complete the baseline intake form (Instrument 1) is higher. There are no changes to the enrollment projections for the Tribal Evaluation.

A16: Plan and Time Schedule for Information Collection, Tabulation and Publication

16.1 Analysis Plan

The evaluation plans for the HPOG 2.0 National Evaluation Implementation and Impact Studies are currently in the design phase. The detailed analysis plans for all data collection protocols associated with those studies will be submitted in a subsequent request for clearance, with the additional data collection protocols.

The HPOG 2.0 National Evaluation team will produce several reports using the data collected for the descriptive evaluation. The reports will include:

Descriptive Evaluation Report. This report will summarize the information on program implementation features, challenges, and best practices using the descriptive evaluation interviews, site visit data, and data from PAGES. This report will include the implementation, outcome, and systems studies. The evaluation will also use results from the implementation study to produce short case study reports on focus areas of specific interest to ACF.
Impact Evaluation Reports. Findings from the implementation study will inform the analysis in the evaluation’s impact evaluation reports. The evaluation is expected to produce reports on results based on 15-month and 36-month follow-up surveys and associated administrative data analysis.

The HPOG 2.0 Tribal Evaluation will use a systematic approach to analyze the data obtained through the interviews and focus groups conducted during and following annual site visits. The evaluation team will use NVivo software to store and analyze the large volume of data collected over the course of the evaluation. NVivo will be used to develop a coding scheme for analyzing these data. The coding scheme will be organized around evaluation topic areas derived from the evaluation questions. The coding scheme will be applied to all data and emergent key themes relating to evaluation topic areas will be identified.

The HPOG 2.0 Tribal Evaluation team will prepare a variety of reports, including site visit reports, practice briefs, and a final report.

Site Visit Reports. These reports will be developed after each annual site visit and summarize the findings from the interviews and focus groups.
Practice Briefs. Practice briefs will be shorter documents that highlight findings from the evaluation and share lessons learned.
Final Report. The final report will reflect the aggregated analysis of all qualitative and quantitative data collected throughout the evaluation

PAGES will provide a platform for grantee representatives monitoring the overall grant implementation to enter semi-annual progress on achieving grant objectives that will be submitted as required by ACF. The system will automatically generate quantitative measures for the federally required semi-annual PPRs, which will include aggregated participant-level data, and will also store narrative-based grantee level performance information. Grantees will print the PPRs from PAGES, sign the paper documents, and submit them to ACF. ACF will use these tables when preparing reports to Congress on the HPOG initiative. HPOG PAGES data collection activities will also support three annual report deliverables that will include information such as characteristics of grantee programs, number and characteristics of participants, and information on program participants’ receipt of training and services and employment and training outcomes. The PAGES team will produce a number of reports using data collected, including the six semi-annual PPRs and three annual reports.

16.2 Time Schedule and Publications

The National Evaluation descriptive evaluation data collection will begin in May2017, pending approval from OMB. Contact updates for those participants in the National Evaluation impact evaluation sample are scheduled to begin in April 2017 and continue throughout the follow-up period. The Tribal Evaluation data collection will occur annually starting in 2017. PAGES data collection will occur as individuals apply for the programs and enrollees receive training and services throughout the three first years of the grant period. Exhibit 16.1 presents an overview of the project schedule for information collection.

Exhibit 16.2 Project Schedule for Data Collection, Analysis, and Publication

Task	Timing
National Evaluation: Descriptive Evaluation
Descriptive evaluation data collection (includes costs, systems, and program implementation)	May 2017-December 2018
National evaluation site visits	October 2017-February 2018
Descriptive evaluation Analysis Plan	August 2017
Descriptive evaluation Report (including implementation, outcome and systems studies)	Draft October 2019; Final March 2020
National Evaluation: Impact Evaluation Participant Contact Updates
Welcome Packets	Monthly, one month after random assignment beginning in April 2017
Contact Update Mailing	Quarterly beginning 3 months after random assignment (June 2017)
HPOG 2.0 Tribal Evaluation
Site visits to tribal grantees (1/year)	Annually spring/summer of 2017, 2018, 2019, 2020
Conduct data analysis	2017-2021
Develop Practice Briefs	Annually September 2017-September 2021
Develop Final Report	September 2021
HPOG 2.0 National and Tribal Evaluation Participant Accomplishment and Grantee Evaluation System (PAGES)
PAGES grantee-level and ongoing participant-level data collection	September 2015 – September 2018
6 Semi-annual PPRs	September 2015 – September 2018
Two annual reports	September 2015 – September 2018

A17: Reasons Not to Display OMB Expiration Date

All instruments created for the HPOG 2.0 National Evaluation and Tribal Evaluation will display the OMB approval number and the expiration date for OMB approval.

A18: Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1 Tribal Evaluation Workgroup. A Roadmap for Collaborative and Effective Evaluation in Tribal Communities. Children’s Bureau, Administration for Children and Families, U.S. Department of Health and Human Services. September 2013.

2 See Chapter 12 of Lynn (2009), in particular, section 12.5 that reviews the effects of incentives in several prominent longitudinal studies.

3 Two Tribal grantees will not collect social security numbers from some or all of their participants. A unique identifier will be assigned for these participants. Two versions of the Tribal informed consent forms were developed, one that includes social security numbers and one for grantees using unique identifiers.

4 This burden estimates reflects the average across all instruments included in Instrument 4.

5 Source: Bureau of Labor Statistics, National Compensation Survey, 2010: Combined average hourly wage across education training and library occupations and community and social services occupations.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Abt Single-Sided Body Template
Author	Katheleen Linton
File Modified	0000-00-00
File Created	2021-01-23

HPOG2 Supporting Statement_Part A_10 18 16_REVISED 4-21-17