June 2016 Updated SSA - Track Changes

This document provides supporting statements for the collection of information for The Evaluation and System Design for Career Pathways Programs: 2nd Generation of Health Profession Opportunity Grants (HPOG Next Gen Design) funded by the U.S. Department of Health and Human Services (HHS), Administration for Children and Families (ACF). The ACF Office of Family Assistance (OFA) has announced plans to award a second round of HPOG grants by September 2015. One of the key objectives of HPOG Next Gen Design is to establish a new data system to support program management and evaluation of the new HPOG grantees. The data system will meet the performance data needs of the HPOG Next Gen grantees and of OFA to monitor the performance of the grants and prepare the report to Congress on the grants, as well as support an impact study, a separate but coordinated Tribal evaluation, and other future research and evaluation efforts sponsored by ACF.

ACF is seeking clearance from OMB for the data collection via the new data system, called the HPOG Next Gen Participant Accomplishment and Grant Evaluation System (PAGES). The information collection proposed is internet-based and includes gathering data from the HPOG Next Gen grantees on: (1) grantee program designs and offerings; (2) intake information on eligible applicants (both treatment and control)¹ through baseline data collection; and (3) individual enrolled program participants’ activities and outcomes.

A1: Necessity for the Data Collection

ACF at HHS seeks approval for the data collection activities described in this request to support the new data system for program management and evaluation of the second round of HPOG grantees. These data collection activities are part of HPOG Next Gen Design.

A1.1 Study Background

The Health Profession Opportunity Grants (HPOG) Program, established by the Patient Protection and Affordable Care Act of 2010 (ACA), funds training in high-demand healthcare professions, targeted to Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals. In September 2010, OFA awarded approximately $67 million in funding to 32 organizations located across 23 states. Grantees include two community based organizations, four state entities, nine local workforce investment boards, one university, two community college districts, nine community colleges, four tribal community colleges, and one tribal council. Those grantees oversee 49 individual HPOG programs.

HPOG programs: (1) target skills and competencies demanded by the healthcare industry; (2) support career pathways, such as an articulated career ladder; (3) result in an employer- or industry-recognized credential (which can include a license, third-party certification, postsecondary educational certificate or degree, as well as a Registered Apprenticeship certificate); and (4) combine supportive services with education and training services to help participants overcome barriers to employment, as necessary.

The fifth and final year of the initial round of HPOG grants will end on September 29, 2015. The Protecting Access to Medicare Act (enacted April 1, 2014) extends HPOG through FY 2015, which enables a new grant competition for the program. It is expected that new grants will be for 60-month project periods with five 12-month budget periods. ACF expects to award approximately 35 grants to grantees serving TANF recipients and other low-income individuals, and approximately five grants to tribes, tribal organizations or tribal colleges or universities. The anticipated project start date for the new grantees is September 30, 2015.²

HPOG’s authorizing legislation calls for a comprehensive evaluation of the funded demonstration projects. Accordingly, ACF plans to evaluate rigorously the effectiveness of funded HPOG Next Gen programs. The federal evaluation activities are intended to expand the career pathways evidence base and to build on what has been learned to date about how to design and implement successful career pathways programs for low-income and low-skilled individuals, and improve the outcomes of individuals who participate in these programs. All grantees will participate in a federal evaluation. The federal evaluation for the non-Tribal HPOG Next Gen grantees will involve random assignment of individual participants and may involve randomization of selected program design components, or “enhancements,” at the grantee or program level. Tribal grantees will participate in a coordinated evaluation that does not involve random assignment.

The proposed HPOG Next Gen PAGES will collect and store uniform data needed for performance management and the federal evaluations, incorporating the required semi-annual grantee performance reports to ACF. These reports include a quantitative section with metrics automatically generated from data in the PAGES system and a narrative section that must be filled out by grantees. The system will also provide necessary data for other future research and evaluation efforts; these projects may require additional data collection efforts, for which separate OMB clearance would be requested.

Abt Associates and the Urban Institute are the contractors leading the HPOG Next Design effort. They are also conducting several other evaluations on behalf of ACF as part of the HPOG research portfolio on the first round of HPOG grantees, for which there are numerous data collections already approved by OMB (see Attachment F for further details). ACF and its contractors are engaged in many efforts to coordinate these evaluation activities and avoid duplication of work. HPOG Next Gen Design will utilize the extensive knowledge generated to date from the research activities on the first round of HPOG and current Pathways for Advancing Careers and Education (PACE) programs to inform the proposed new data collection efforts for the second round of HPOG grantees.

A1.2 Legal or Administrative Requirements that Necessitate the Collection

H.R. 3590, the ACA requires an evaluation of the HPOG demonstration projects (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(B)). The Act further indicates that the evaluation will be used to inform the final report to Congress (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(C)). The Act calls for evaluation activities to assess the success of HPOG in “creating opportunities for developing and sustaining, particularly with respect to low-income individuals and other entry-level workers, a health professions workforce that has accessible entry points, that meets high standards for education, training, certification, and professional development, and that provides increased wages and affordable benefits, including healthcare coverage, that are responsive to the workforce’s needs” (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(B)).

There were two Funding Opportunity Announcements (FOAs) for the second round of HPOG grants—one for non-Tribal grantees (HHS-2015-ACF-OFA-FX-0951) and one for Tribal grantees (HHS-2015-ACF-OFA-FY-0952). Both FOAs required all HPOG Next Gen grantees to participate in a federal evaluation and to follow all evaluation protocols established by ACF or its designated contractors. Participating in the federal evaluations will include, but is not limited to, the use of a data system to collect uniform data elements and, for non-Tribal grantees, the facilitation of random assignment. The FOAs also include an administrative requirement for all HPOG programs to prepare and submit reports on the activities and outcomes of their programs on a semi-annual basis, following the procedures that will be determined by ACF. Grantee reports must use uniform data definitions for enrollment, completion, employment, wages, and other outcomes.

Data collected under HPOG Next Gen’s PAGES described here will be used to automatically generate the federally required semi-annual reports, to inform ACF reports to Congress, to monitor and manage the performance of the grant-funded projects, to inform the HPOG Next Gen impact study and Tribal evaluation, and to inform other future research and evaluation efforts.

A1.3 Study Design

The three key tasks of HPOG Next Gen Design are to: 1) build knowledge and generate recommendations to inform the design of new programs funded under HPOG; 2) design a study to assess the effectiveness of the second round non-Tribal HPOG programs; and, 3) build and maintain a data system for program management and for federal evaluations. This section describes the objectives of the data system and the federal evaluations for the HPOG Next Gen grantees.

HPOG Next Gen’s PAGES

One of the key objectives of HPOG Next Gen Design is to establish a data system for program management and evaluation. The data system will meet the performance data needs of the grantees and of OFA to monitor the performance of the grants and prepare the report to Congress on the grants, as well as support an impact study, evaluation of the Tribal grantees, and other future research and evaluation efforts sponsored by ACF. The information collection proposed—PAGES—is internet-based and includes gathering data from HPOG grantees on: (1) grantee program designs and offerings; (2) intake information on eligible applicants (both treatment and control) through baseline data collection and (3) individual enrolled program participants’ activities and outcomes.

Per the FOAs, all grantees will be required to provide information on their program components and offerings to be included in the data system to simplify later data entry required for grant performance monitoring, reporting, and evaluation. This program level data also is important for future evaluation efforts to help analyze each grantee’s and program’s inputs and outputs and place analytic results into the appropriate context.

Participant baseline data will provide ACF with demographic information needed for grant management and reporting. The demographic data will also be used in the HPOG Next Gen impact study and future research as well. These data will be used to identify balance between the treatment and control groups, to increase the precision of estimates regarding the impact of program components, and to identify subgroups for subgroup impact analysis at follow-up.

In addition, grantees will enter data on participants’ ongoing program participation, outputs, and outcomes, which will enable them to manage program participation and track the progress of program participants. Grantees will use the data collected through the system to generate the required Performance Progress Reports (PPRs) for OFA.

The PPR includes two sections—a narrative section and a quantitative section. In the narrative section, grantees are required to provide OFA updates on their programs including major activities and accomplishments, problems, significant findings and events, dissemination activities, other activities, and activities planned for the next reporting period. This section is taken from the ACF Performance Progress Report, a standard form already approved by OMB under number 0970-0406. The narrative portion of this form will be embedded in PAGES so that grantees can type the narrative directly into the PAGES system (note that we have included the burden for this narrative portion in our burden estimates so that the narrative section will assume the OMB number and expiration date of this information request). In the quantitative section, grantees enter their projections for the grant year on enrollment, basic skills training enrollment and completions, healthcare training enrollment and completions, and employment. The quantitative section also includes metrics that are automatically calculated from data entered in other parts of PAGES—i.e., actual levels of enrollment, completions, and employment, as well as information on participant status, other skill development activity participation, and support service receipt. This part of the PPR also includes narrative fields for grantees to provide information on why projected values differ from actual values. Both the narrative and quantitative sections of the PPR will be able to be saved and printed as one file for submission to OFA by grantees. See the full list of PPR items and a mockup of the PPR in Attachment E.

Overview of the HPOG Next Gen Federal Evaluations

ACF plans to evaluate rigorously the effectiveness of the second round of HPOG grants. OPRE will oversee the federal evaluation activities, which will include an implementation and impact evaluation (with long-term follow-up) of the non-Tribal grant-funded programs, and a separate but coordinated evaluation of the Tribal grantees. The federal evaluation activities are intended to expand the career pathways evidence base and to build on what has been learned to date about how to design and implement successful career pathways programs for low-income individuals, and improve the outcomes of individuals who participate in these programs.

All HPOG Next Gen funded programs are required to be part of a rigorous evaluation of the effectiveness of career pathways-based training. The non-Tribal evaluation will involve random assignment of individual participants and may involve randomization of selected program design components, or “enhancements,” at the grantee or program level. As stated in the FOA, grantees will be required to abide by random assignment procedures and facilitate the random assignment process for individuals, which involves entering eligible HPOG program applicants into a lottery to determine if they will be invited to participate in the program, either in part or in full. Applicants who are not invited to participate will serve as a control group in the evaluation and will not receive HPOG program services, but may enroll in any other program or service for which they are eligible. A period of 120 days after grant award will permit initial planning, after which —if it has not already begun—program intake and randomization of program eligibles will begin. Individuals must complete the application process prior to random assignment; only individuals who have been deemed both eligible and suitable for program participation may be entered into the lottery.

ACF estimates that the non-Tribal HPOG Next Gen programs that enroll at least 300 to 450 individuals over the course of the random assignment period (to be determined after grant award but likely to extend from the end of the 120-day planning period into the last year of the grant period) will be eligible to measure grantee-specific impact results. Enrolling a larger number of individuals would result in more accurate impact estimates. The evaluation study expects to use a 2:1 ratio for random assignment—two participants will be randomly assigned to the control treatment group for every one participant assigned to the treatment control group. For example, a program that enrolls 400 individuals over the course of the random assignment period would need to recruit and assess 200 additional individuals that meet all of the enrollment criteria for the HPOG program (i.e., TANF or low-income individual, citizenship, etc.) to serve in the control group. The overall study sample of at least 600 individuals per grantee will permit estimation of grantee-specific program impacts, although additional research questions hold priority as well, as discussed next.

The HPOG Next Gen Tribal evaluation will employ sound scientific methods and be grounded in sound cultural methods. At the start and throughout the process, the evaluators will engage with tribal leadership or authorized designee(s) to ensure that the evaluation is firmly anchored in questions that are meaningful to local stakeholders and that assist local service providers in better serving their communities. The evaluation of the Tribal HPOG grantees will not involve random assignment.

Once the federal evaluation designs are finalized, ACF will submit for OMB clearance any additional data collection instruments that will be used for the studies. The evaluation data collection included in this request is limited to the information that will be collected through PAGES.

A1.5 Universe of Data Collection Efforts for HPOG Next Gen

To establish a data system for program management and evaluation, the HPOG Next Gen PAGES will collect the following information:

Grantee program designs and offerings;
Intake information on eligible applicants (both treatment and control) through baseline data collection; and
Individual enrolled program participants’ activities and outcomes.

This submission is to request clearance for PAGES, the web-based system needed to collect those data.

A2: Purpose of Survey and Data Collection Procedures

A2.1 Overview of Purpose and Approach

The HPOG Next Gen’s PAGES will collect both grantee- and participant-level data elements from all HPOG Next Gen grantees. The next section provides more information about each of the major data collection components and discusses how each contributes to grantee performance monitoring and the federal evaluations. The complete list of proposed data items is presented in an attachment (see Instrument 1: PAGES Grantee- and Participant-Level Data Items).

Grantee-Level Data Collection

During the initial 120 day planning period, all grantees will be required to provide information on their program components and offerings to be included in the data system. This will simplify later data entry required for grant performance monitoring and reporting by allowing users to make choices from prepopulated lists of activities and services that are limited to what is available in their own HPOG programs as pre-entered during this planning period.

This grantee data will include information on each basic skills training course offered, each occupational training course offered (including vendor, occupational code, credit/non-credit, potential degree), as well as the types of academic supports, types of support services, and types of employment-related activities available through the grant. The grantee-level information is needed to populate fields for data entry required for grant performance management through PAGES. The grantee data will be used in the federal evaluations’ efforts to help analyze each grantee’s inputs and outputs and place analytic results into the appropriate context. The grantee-level data on available services and training courses will be also used to describe program offerings and to allow for measurement of participants’ use of services and training opportunities in the federal evaluations. The data are also important for future evaluation of implementation and systems change.

Participant-Level Baseline Data Collection

All grantees will also collect data on the characteristics of all eligible individual participants at intake. Items from PAGES will provide researchers with important baseline demographic information about study participants including applicants’ demographics, household characteristics, and employment and education experiences. Including these items will allow the federal evaluations to describe the study samples. The data will also help the impact evaluation of the non-Tribal grantees to identify balance between the treatment and control groups; to increase the precision of estimates regarding the impact of program components; and to identify subgroups for subgroup impact analysis at follow-up.

The baseline data also include a child roster, which lists the first name and date of birth of research sample members’ children under 18 who reside with them at least half of the time. This roster will be used to create a sampling frame for follow-up surveys that collect data about child outcomes. The contact information collected at baseline is necessary to enhance researchers’ ability to locate respondents for follow-up surveys needed to measure intervention impacts. As described in the random assignment informed consent form (Attachment B), about every four months researchers will mail to both treatment and control group members a contact information update letter and form to confirm or update the PAGES baseline contact information (see Attachment G).

Applicants at non-Tribal grantees participating in the impact study will also provide baseline data on their expectations for education and employment, barriers to employment, work preferences, and self-efficacy. PAGES will collect these data because prior research on employment and training programs suggests that these factors may be associated with participants’ ability to complete training and to obtain and retain employment (e.g., Matus-Grossman and Gooden, 2002; Fein and Beecroft, 2006; Michalolpoulos and Schwartz, 2001). Understanding people’s experience with prior training and education, their barriers to and preferences for work, and their motivations and self-efficacy will improve researchers’ ability to identify which treatment group members access various components of the HPOG Next Gen programs. Prior work documents that psycho-social questions such as these are important to sorting participants by characteristics that predict outcomes of interest (e.g., Gibson, 2003; Peck, 2007).

Several of the baseline data items pertain specifically to tribal members, particularly those served by the Tribal grantees: tribal affiliation, residence on or off-reservation, employment in a tribal organization or enterprise, and receipt of training by a tribal institution. Non-Tribal HPOG grantees will not enter these data items.

Participant-Level Ongoing Data Collection

Grantee staff will also collect data on participants’ program participation and patterns, outputs and outcomes including education and training received, the services provided to participants, program outputs, intermediate outcomes (such as completed program, acquired credential, etc.), and end outcome results (such as entering employment, employment in the health sector, and employment status six months after program completion). PAGES will enable grantees to manage program participation and track the progress of program participants. In addition, grantees will use the data collected through the system to generate the required PPRs for OFA program management and for federal evaluation activities.

A2.2 Collecting Participant-Level Baseline Data

Grantee staff will administer an informed consent form relating to PAGES data items, administrative data, and follow-up surveys, when individuals who apply to the program are found eligible. All grantees must obtain informed consent from applicants prior to collecting personally identifying data.

All non-Tribal grantees are required to participate in the impact evaluation. During the intake processes for grantees in the study, staff first will describe the impact study and administer the random assignment informed consent form (see Attachment B). All eligible applicants for non-Tribal grantee programs must sign the paper informed consent form during the intake period for the study in order to be part of the study. The informed consent form allows researchers to: (1) access baseline data participants provide at intake and information about the training and services treatment group members receive after enrollment; (2) contact both treatment and control group members for updated contact information and additional follow-up questions for the impact evaluation; and (3) gather administrative data from the National Directory of New Hires (NDNH) and possibly other sources, such as the National Student Clearinghouse (NSC). After the consent process, grantee staff will administer PAGES baseline questions, including the baseline questions on individuals’ expectations for education and employment, barriers to employment, work preferences, and self-efficacy. After individuals complete the informed consent form and the required baseline questions, PAGES will randomly assign them into either the treatment or control group. If a grantee is selected to implement and test a program enhancement as part of the impact study, participants at that grantee will be randomized into one of three experimental groups—the treatment, treatment-enhanced, or control group.

Applicants at the Tribal grantees participating in the federal Tribal HPOG federal evaluation but not participating in the impact study will complete one of two the non-random assignmenttribal informed consent forms (see Attachment B2 and Attachment B3) that asks permission for researchers to access data individuals provide at intake, information about the training and services they receive after enrollment, administrative data, and follow-up surveys. Some grantees will collect social security numbers (SSNs), some will use unique identifiers (instead of collecting SSNs), and one grantee will use SSNssocial security numbers and unique identifiers, depending on whether the participant lives on the reservation or not. One version of the informed consent form notes that SSNssocial security numbers will be collected and the other states that SSNs social security numbers will not be collected. The consent forms have been approved by the Abt Associates IRB and the Tribal Nations Research Group Research Review Board. Eligible applicants to the Tribal programs may enroll in the programs whether or not they consent to being included in the research. After the consent process, grantee staff will administer PAGES baseline questions, excluding the baseline questions on individuals’ expectations for education and employment, barriers to employment, work preferences, and self-efficacy. These baseline questions are excluded as their main purpose is for testing the composition of the treatment and control groups in the random assignment experiment.

The institutional review boards for both Abt Associates and Urban Institute will review and approve all data collection and data security plans prior to the system going online. Data security measures for the data collected through PAGES are described fully in Section A10.

A2.4 Who Will Use the Information

ACF will use the grantee-level and ongoing participant-level information to monitor HPOG Next Gen grantees’ performance. Researchers will have access to and use the baseline data collected as part of the evaluation. In addition, grantees themselves will have access to dashboards, canned reports and flexible reporting tools to monitor and improve their own operations. At the conclusion of HPOG Next Gen, Abt Associates will provide ACF with a restricted-use data set containing individual-level data that are stripped of all personally identifying information. Not collecting the information would constrain ACF’s ability to monitor the grant-funded programs and activities and to fulfill the legislative mandate to conduct evaluations of HPOG.

A3: Improved Information Technology to Reduce Burden

The HPOG Next Gen participant- and grantee-level data will be collected through PAGES, an internet-based data system.

The system is being developed by Green Beacon Solutions, Inc., a subcontractor to Abt Associates. Abt will oversee the development; Urban Institute will provide training and end user support. The system allows entry and secure maintenance of information on grantees and program participants. HPOG grantees will enter these data directly into PAGES.

Key features of PAGES that reduce burden include:

Internet-Based Application. PAGES will be implemented on a secure HPOG website maintained by Green Beacon and Abt Associates. Staff at the grantee or sub-grantee level who are granted authorization to access the system will receive a secure password. Within each organization, access will also be dictated by each user’s role. For example, case managers will be able to enter data, but not see grantee-wide reports. Other roles, such as grantee managers, will not be able to see the personal information unless they are also case managers. HPOG evaluators are able to view all data from participants across all grantees, but private information (such as participant name and Social Security number) will be used only for the purposes identified in informed consent forms signed by the participant.
Efficient and Secure Data Entry Format. PAGES is structured to reduce the burden on grantees while ensuring adequate detail and accuracy. A data upload capability allows some grantee representatives to program their existing information systems to export their data into a standard format and import it into PAGES so that participant data already included in existing grantee or provider management information systems will not have to be manually re-entered directly into PAGES. Directly populating the HPOG system with existing electronic data in this manner reduces data entry burden and minimizes data entry errors. While this upload feature will require manually selecting and importing a file, it will be less burdensome than direct data entry. Data items that cannot be uploaded by a grantee are entered directly by authorized grantee staff into PAGES. Data items that include private information (e.g., Social Security numbers) are automatically encrypted at data entry or at the point of electronic interface.
Reporting Capability. PAGES will provide a mechanism for grantees to enter and manage performance management information. Data on individual participants will be entered at application and intake as part of the initial interaction between the participant and grantee staff. Receipt of training and other services may be recorded at any time in the individual record. Narrative case notes can also be added as text. The system will provide a platform for grantee representatives monitoring the overall grant implementation to enter semi-annual progress on achieving grant objectives that will be submitted as required by ACF. The system will automatically generate quantitative measures for the federally required semi-annual PPRs that includes aggregated participant-level data and will store narrative-based grantee level performance information. This capability will reduce overall reporting burden since grantees will not need to calculate and fill out the quantitative metrics for the PPR reports, but will have these calculated automatically in PAGES.

A4: Efforts to Identify Duplication

The HPOG Next Gen programs have not yet begun operation so no data system currently exists that provides all of the participant- and grantee-level data needed to meet ACF’s Congressionally-mandated requirements. The structure of the new PAGES system does not duplicate, but builds on the structure and data measures used in the Performance Reporting System (PRS) during the first round of HPOG.³ Some data items, however, may exist in other MIS data records at some grantee or program agencies. To minimize duplication of data entry, a data upload capability allows some grantee representatives to program their existing information systems to export their data into a standard format and import it into PAGES so that participant data already included in existing grantee or provider management information systems will not have to be manually re-entered directly into PAGES.

To reduce burden further, data definitions and coding conventions have been made compatible with existing programs most relevant to the HPOG grants, such as those authorized under the Workforce Investment Opportunity Act (WIOA). Where possible, data elements have also been aligned with other key related ACF ongoing data collection efforts, particularly the Implementation, Systems and Outcome Evaluation of the Health Profession Opportunity Grants to Serve TANF Recipients and Other Low-Income Individuals (HPOG ISO), HPOG National Implementation Evaluation (HPOG NIE), HPOG Impact Study (HPOG Impact), and PACE project data collection activities (see Attachment F).

A5: Involvement of Small Organizations

None of the HPOG Next Gen grantees will be small businesses. The primary organizations involved in this study will be community colleges, workforce development agencies and community-based organizations that operate occupational training programs. In addition, the funding announcement informs all grantees of the reporting requirements, and adequate resources have been provided to coordinate the data collection and reporting. There should be no adverse impact for any grantees participating in the study.

A6: Consequences of Less Frequent Data Collection

The consequences of not collecting information through the HPOG data system are that valid and comparable data on participants, activities and outcomes would not be available for the HPOG Next Gen grantees. Ongoing data collection through PAGES offers opportunities to: (1) track the performance and continuous performance improvement of the grantee projects, (2) provide data for the federal evaluations, including the impact study currently being designed and the Tribal evaluation; and (3) ensure data are available for future HPOG Next Gen research and evaluation efforts. Ongoing individual-level data collection is needed for HPOG Next Gen grantees to manage program participation, track the progress of program participants and generate the required PPRs for OFA; for OFA to prepare reports to Congress on the grants; and for the federal evaluation activities. Data collection at less frequent intervals would limit grantees’ abilities to meet their reporting requirements and ACF’s ability to track performance and adapt quickly, if necessary, to improve performance.

The FOAs for the HPOG grants (HHS-2015-ACF-OFA-FX-0951 and HHS-2015-ACF-OFA-FY-0952) also will require all grantees to participate in the federal evaluations and to follow all evaluation protocols established by ACF or its designated contractors. Participating in the federal evaluations will include, but is not limited to, the use of a data system to collect uniform data elements and, for non-Tribal grantees, the facilitation of random assignment. The participant-level baseline data is a one-time data collection. Earlier, we identified a number of critical uses of baseline data. Here, we note briefly how several of these uses would be affected were baseline data not available.

Obtaining baseline data allows the HPOG Next Gen impact study team to describe the research sample’s descriptive characteristics as well as confirming balance between the treatment and control groups. Without baseline data, researchers would lack the ability to confirm that random assignment produced statistically similar treatment and control groups. More importantly, researchers would not be able to describe in sufficient detail the population these innovative programs are serving.
Without baseline values of measures correlated with outcomes, the impact study’s estimates would be less precise. The more precise the estimates are, the more confident evaluators can be in determining which strategies improve outcomes for low-income families.
Lack of baseline data would mean the HPOG Next Gen impact study would be unable to estimate impacts for subgroups of interest to provide insights for improved program targeting.

A7: Special Circumstances

There are no special circumstances for the proposed data collection.

A8: Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13 and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995)), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on Wednesday, March 4, 2015, Volume 80, Number 42, page 11679, and provided a 60-day period for public comment. This notice also included a request to waive future 60-day Federal Register Notices. A copy of this notice is included as Attachment C. During the notice and comment period, the government received no requests for copies of the instruments and no substantive public comments were received.

Consultation with Experts Outside of the Study

The majority of PAGES grantee- and participant-level data items are adapted from previously approved data collection instruments for PACE (clearance number 0970-0343) and HPOG ISO and HPOG-Impact (both under clearance number 0970-0394), as described in Attachment D.

PAGES data items were also developed in consultation with senior methodological and substantive experts, including: Karen Staha, Department of Labor; Yvette Chocolaad, National Association of State Workforce Agencies; Burt Barnow, George Washington University; Tim Harmon, Workforce Enterprise Services, Sung-Woo Cho, Matthew Zeidenberg, and David Fein, Abt Associates; Keith Watson, Lauren Eyster, Alan Dodkowitz, Urban Institute.

A9: Incentives for Respondents

No incentives to respondents are proposed for the data collection via PAGES but we seek approval to offer $2 for each round of the participant contact information update forms. .Researchers will send the $2 to participants with the updated information request as a gesture of appreciation to respondents. This request is based on the literature documenting the effectiveness of prepayments in interviewer-mediated surveys. Singer, Van Hoewyk, and Maher (2000) report that prepaid incentives were effective in reducing nonresponse to the Survey of Consumer Attitudes in a series of experiments in which promised incentives were not. An earlier meta-analysis by these authors (Singer et al. 1999) found that prepaid incentives were effective in increasing response rates and, although there was no significant difference between prepaid and promised incentives, noted that “the direction of the differences is in accord with conventional wisdom – i.e., prepayment appears to be more effective than promised payments. When we look only at experiments that hold size of payment constant and compare prepayment and promised payment, find in every case that prepayment yields higher response rates than the promised payment condition” (p. 223). Similarly, Cantor, O’Hare, and O’Connor’s (2008) review of incentive experiments in telephone surveys found consistently significant effects for prepaid incentives between $1 and $5, with response rate increases of 2.2 to 12.1 percentage points. The effectiveness of prepaid incentives for interviewer-mediated surveys builds upon the extensive literature documenting the effectiveness of prepaid incentives for mail surveys (c.f. Church 1993; Edwards et al. 2002; Yammarino, Skinner, and Childers 1991).

A10: Privacy of Respondents

The HPOG Next Gen baseline data collection will include individual identification data collected through PAGES. Respondent privacy will be protected to the extent allowed by law. ACF recognizes that HPOG grantees serve vulnerable populations (per the authorizing legislation), and that grantees must protect those populations from any risks of harm from the research and evaluation activities. Accordingly, HPOG Next Gen Design will obtain informed consent forms from all study participants. This informed consent will ensure that participants understand the nature of the research and evaluation activities being conducted. The random assignment informed consent form and non-random assignment informed consent form are included as Attachment B.

As a part of informed consent, the following rationale for data collection and privacy assurances will be provided to HPOG participants by grantees:

Research is being conducted to see if and how HPOG makes a difference in people’s lives by helping them complete training and get healthcare jobs. This program and research are funded by HHS, and HHS may fund other research on this program in the future.
In this program, grantees will collect some personal information from individuals, such as their name, date of birth, Social Security number, and involvement in other programs. The researchers studying the program for the government also need this information. Researchers will use data security procedures to keep all of the study data private and to protect individuals’ personal information. All of the information collected for the program or for the research studies will be kept completely private to the extent allowed by law, and no one’s name will ever appear in any report or discussion of the evaluation results.
Researchers may contact applicants at grantees participating in the impact study in the future. Individuals may refuse to answer any of their specific questions at any time.

In accordance with the requirements of the Privacy Act of 1974, as amended (5 U.S.C. 552a), ACF published a Federal Register Notice (80 FR 17893) on April 2, 2015 that included the announcement of establishment of a system of records for OPRE. The new system is title: 09-80-0361 OPRE Research and Evaluation Project Records, HHS/ACF/OPRE. This Notice will become effective 30 days after publication, unless ACF makes changes based on comments received.

OPRE is also published a Privacy Impact Assessment (PIA) to ensure that information handling conforms with applicable legal, regulatory, and policy requirements regarding privacy; determine the risks of collecting and maintaining PII; assists in identifying protections and alternative processes for handling PII to mitigate potential privacy risks; and communicates an information system’s privacy practices to the public. This PIA, titled Participant Accomplishment and Grant Evaluation System (PAGES), was approved on October 9, 2015 and will be available online through HHS at http://www.hhs.gov/pia/#System.

The HPOG Next Gen PAGES will be developed using the highest standards of technology and data security. Data for grantee-level and individual-level records will be stored securely in an SQL server database. The web interface for data entry and reporting will be built on the industry leading Microsoft Dynamics customer relationship management (CRM) platform. The system will be hosted and maintained on Microsoft Dynamics CRM Online Government, a highly secure Federal Information Security Management Act (FISMA) Moderate compliant cloud-based Software as a Service (SaaS) solution.

Based on the architecture of Microsoft Dynamics CRM Online Government, PAGES will meet U.S. government security demands by providing:

Business continuity with true geographic redundancy, including a financially guaranteed uptime of 99.9%;
Data, applications, and hardware located in the continental U.S.;
Physically isolated datacenter and network;
Operated by screened personnel and Technical Support provided by U.S. Citizens;
International Organization for Standardization (ISO) 27001 certified, one of the best security benchmarks available across the world;
Compliance with the requirements of the Health Insurance Portability and Accountability (HIPAA) Act/ Health Information Technology for Economic and Clinical Health (HITECH) Act; and
A regulatory compliance roadmap including FISMA Moderate/FedRamp.

Accounts on the web server will be protected with dual factor authentication, to include a password and an additional means of authentication. Passwords will be at least eight characters long and contain at least one special character and one number and will not contain dictionary words. These requirements are enforced upon account creation. Passwords will expire every 90 days and users will have to create new passwords that fulfill the requirement of the password policy. Dynamics CRM uses HTTPS with the SSL/TLS protocol providing encrypted communication and secure identification of the network web server. The platform is heavily utilized in other Federal Government organizations with externally facing instances and has undergone and passed all Authority to Operate (ATO) and security protocols within these organizations. All data is filtered using the security model so records and fields containing Personally Identifiable Information (PII) data are removed for users that do not have authorization. Any logging or output files will not contain private data and will be limited to generic system and error data.

PAGES will support field-level security so users without authorization to specific data do not see the data on forms, views or reports. Thus, private participant data such as Social Security numbers will be entered into the system and encrypted at the field level, but will not be visually displayed or downloadable by system users. User-identifiable participant-specific data will be stored separately from grantee-level data and will be available for updating only by the grantee representative who originally entered the data. Grantee-specific data will be available to the project team in specific extracts and reports once the information has been entered and submitted.

Grantees will receive detailed guidelines and training on data entry and security procedures. Clearly defined variables and labeled fields will specify how to enter each data element. Supporting guidance on data formats and definitions will be provided. Training on the data system will be provided prior to its implementation and technical assistance on the system will be available to grantees throughout their grant period of performance.

A11: Sensitive Questions

Participant-Level Baseline Questions

Several data items pertaining to individual participant characteristics may be considered sensitive by some program participants, for example, questions about criminal records, disabilities, and limited English proficiency. These data are standard items for other workforce development programs and will allow important comparisons between HPOG and other similar efforts for evaluation and management purposes. In addition, these data are needed to fully identify programs and program characteristics that are most successful for serving the vulnerable populations that HPOG was designed to support and that are a focus of ACF’s assistance programs.

Individual identifying information of a sensitive, personal, or private nature that all HPOG Next Gen grantee applicants will complete includes: (1) last and first name; (2) Social Security number; (3) date of birth, (4) ethnicity and race; (5) marital status; (6) number of children; (7) whether the individual is a TANF or SNAP recipient; (8) disabilities, (9) limited English proficiency, and (10) employment status at program intake and exit.

Reasons why these data items are necessary and specific uses to be made of this information include:

The last and first names and Social Security Numbers of each participant are needed by evaluators to obtain accurate administrative data on individuals’ quarterly earnings and receipt of cash and noncash public benefits. Those data are necessary to measure key impacts. Administrative data matches will only be done for those participants who sign informed consent data agreements.
Other data which are of a sensitive nature—including date of birth, ethnicity, race, marital status, and whether the individual receives public benefits at baseline—are needed to support detailed analyses of the types of individuals enrolled into HPOG programs and the services and education or training they receive.
Some characteristics, such as disability status or limited English proficiency, are likely to be associated with special instructional needs. Performance and outcome analysis for these subgroups of participants may provide particularly useful information about effective training approaches for these populations.
Employment information at program intake and exit will provide essential information about the effectiveness of the HPOG Next Gen programs, particularly whether individuals enter health occupations after training.

Among the baseline data collection items for applicants at the non-Tribal grantees participating in the federal impact evaluation are ones addressing respondents’ employment barriers, personal preferences, motivations and self-efficacy. Some respondents may consider these questions to be sensitive. Because program staff will be collecting the data, we designed the measures to be as neutral and impersonal as possible, framing them as part of an assessment of getting to know clients’ needs and strengths so that the program might best meet their needs and capitalize on their strengths.

The literature provides ample support for including these items to identify barriers to employment (e.g., Matus-Grossman and Gooden, 2002; Fein and Beecroft, 2006; Michalolpoulos and Schwartz, 2001). Including these items is necessary to describe the characteristics of the study population and evaluate their moderating effects on program impacts. Furthermore, questions pertaining to personal preferences, motivations and self-efficacy will be especially useful for identifying the pathways that participants follow through multi-faceted programs, thereby allowing researchers to estimate the impacts of various program models and components. Program staff will remind potential study members during the enrollment process that they may refuse to answer individual items. Potential study members will also be reminded that their responses will be kept private, to encourage their candid responses.

ACF also seeks to combine comparable data from different, related evaluations to enhance the cumulative development of knowledge useful to government policymakers, program operators, and the public. As outlined in the justification table included in Attachment D, these baseline data items are being used in the evaluations of the first round of HPOG grantees. Asking the same questions of HPOG Next Gen applicants will allow for comparative analyses of the first and second round of HPOG programs.

A12: Estimation of Information Collection Burden

Exhibit A12.1 presents the reporting burden on grantee staff who enter grantee-level and ongoing participant-level data to complete the HPOG PPRs, on the HPOG applicants completing the baseline questions, and on the grantee staff who enter those baseline data. At the non-Tribal grantees participating in the impact study, we estimate that the baseline data collection will take applicants approximately 30 minutes to complete, and we anticipate 8,600 applicants per year. At the Tribal grantees, we anticipate that the baseline data collection will take applicants 15 minutes to complete, and we anticipate 400 applicants per year. Response times were estimated based on prior experience of the contractors with similar data collection on the first round of HPOG grants.

Exhibit A12.1: Annual Information Collection Activities and Cost

Instrument	Total Number of Respondents		Annual Number of Respondents		Number of Responses Per Respondent		Average Burden Hours Per Response		Annual Burden Hours	Average Hourly Wage	Total Annual Cost
PAGES Grantee- and Participant-Level Data Collection (all grantees)		96		32		2		31.75	2,032	$28.29	$57,485.28
PAGES Participant-Level Baseline Data Collection (participants at non-Tribal grantees participating in impact study)		25,800		8,600		1		.5	4,300	$3.94	$16,942
PAGES Participant-Level Baseline Data Collection (participants at Tribal grantees)		1,200		400		1		.25	100	$3.94	$394
Total Annual									6,432		$74,821.28

Sample members participating in the baseline data collection will be individuals applying for access to HPOG to pursue education and training for healthcare sector employment. To compute the total estimated annual cost reported in Exhibit A12.1, the average wage for HPOG participants employed at program intake ($10.64) was multiplied by the proportion of those working at intake (0.37) for an average hourly total of $3.94. For the cost to grantees entering data, the total burden costs were multiplied by the average hourly wage, according to the Bureau of Labor Statistics, National Compensation Survey, 2010 ($28.29/hour). The total annual cost burden of that data collection is $57,485.28.⁴

A13: Cost Burden to Respondents or Record Keepers

Not applicable. The proposed information collection activities do not place any new capital cost or cost of maintaining capital requirements on respondents.

A14: Estimate of Cost to the Federal Government

This OMB submission seeks approval for information collection from HPOG grantees for three years, from September 2015 through September 2018. In the case the grants exceed this time period, a contract and OMB extension will be sought at the appropriate time. The total cost for these data collection activities will be $224,463.84. Total cost to the Federal government will be $2,020,248 for the proposed data collection. The estimated annual cost to the Federal government will be $673,416.

A15: Change in Burden

This submission to OMB is a new request.

A16: Plan and Time Schedule for Information Collection, Tabulation and Publication

16.1 Analysis Plan

The HPOG Next Gen PAGES will provide a platform for grantee representatives monitoring the overall grant implementation to enter semi-annual progress on achieving grant objectives that will be submitted as required by ACF. The system will automatically generate quantitative measures for the federally required semi-annual PPRs, which will include aggregated participant-level data, and will also store narrative-based grantee level performance information. Grantees will print off the PPRs from PAGES, sign the paper documents, and submit them to ACF. ACF will use these tables when preparing reports to Congress on the HPOG initiative. HPOG Next Gen data collection activities will also support three annual report deliverables that will include information such as characteristics of grantee programs, number and characteristics of participants, and information on program participants’ receipt of training and services and employment and training outcomes.

16.2 Time Schedule and Publications

Data collection will occur as individuals apply for the programs and enrollees receive training and services throughout the three first years of the grant period. Exhibit 16.2 presents an overview of the project schedule for information collection. It also identifies deliverables associated with each major data collection activity.

Exhibit 16.2 Overview of Project Data Collection Schedule

Data Collection Activity

Timing

Associated Publications

PAGES grantee-level and ongoing participant-level data

September 30, 2015 – September 29, 2018

6 Semi-annual PPRs

Three annual reports

A17: Reasons Not to Display OMB Expiration Date

All instruments created for HPOG Next Gen will display the OMB approval number and the expiration date for OMB approval.

A18: Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1 The HPOG Next Gen Tribal grantees are not required to participate in random assignment so the data system will capture data on only program enrollees (as opposed to both treatment and control group study participants).

2 The FOAs can be found: http://www.acf.hhs.gov/grants/open/foa/index.cfm?switch=foa&fon=HHS-2015-ACF-OFA-FX-0951; http://www.acf.hhs.gov/grants/open/foa/index.cfm?switch=foa&fon=HHS-2015-ACF-OFA-FY-0952

3 The PRS was built specifically for and only intended for use by the first round of HPOG grantees; it will be shut down once the first round of HPOG grantees have concluded their period of performance.

4 Source: Bureau of Labor Statistics, National Compensation Survey, 2010: Combined average hourly wage across education training and library occupations and community and social services occupations.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Abt Single-Sided Body Template
Author	Katheleen Linton
File Modified	0000-00-00
File Created	2021-01-21

June 2016 Updated SSA - Track Changes

HPOG2 Supporting Statement_Updated Part A_June 2016 - TRACK CHANGES.docx

Evaluation and System Design for Career Pathways Programs: 2nd Generation of HPOG (HPOG Next Gen Design)