Supporting Statement Part A Revised 9_28_FINAL-3 PM

Implementation, Systems and Outcome Evaluation of the Tribal and Low-Income Health Profession Opportunity Grants (HPOG)

OMB Clearance Request

Supporting Statement Part A

June 8, 2011

Part A: Justification

This section provides supporting statements for each of the eighteen points outlined in Part A of the Office of Management and Budget (OMB) guidelines, for the collection of information for the Health Profession Opportunity Grants (HPOG) program, funded by the U.S. Department of Health and Human Services (HHS), Administration for Children and Families (ACF). The grants fund programs that provide Temporary Assistance for Needy Families (TANF) recipients, other low-income individuals, and members of Native American tribes with training and support needed to find and keep employment in health care occupations and fill the growing demand for skilled health care workers. Thirty-two grants were awarded in September 2010 to government agencies, community-based organizations, post-secondary institutions, and tribal-affiliated organizations to conduct these activities.

A multi-pronged research and evaluation approach for the HPOG program is being implemented by ACF to better understand and assess the activities conducted and their results. Currently, ACF has contracted with Abt Associates and its partner, The Urban Institute, for one core project of this approach. As part of the Implementation, Systems and Outcome Evaluation of the Health Profession Opportunity Grants to Serve TANF Recipients and Other Low-Income Individuals, Abt Associates and The Urban Institute will design and operate an HPOG Performance Reporting System and will design a national evaluation of the HPOG grants to assess the implementation, systems changes, and outcomes of the HPOG program. The Performance Reporting System will collect data to meet the performance data needs of the grantees and OFA program office to monitor the performance of the grants and report to Congress on the grants, as well as support the national evaluation and other future research and evaluation efforts sponsored by ACF. Grantee administrators and/or staff will enter data on participants and key program features directly into the Internet-based secure HPOG data system. ACF is seeking clearance from OMB for the performance management data collection via the HPOG Performance Reporting System, noting that these data will also be used in other current and future evaluation efforts, thereby greatly reducing the overall burden and cost of data collection for HPOG-related research.

Other HPOG-related current and future research and evaluation activities include a separate evaluation of the Tribal HPOG grants currently being conducted by NORC at the University of Chicago for ACF. A separate clearance request is being submitted to OMB for the HPOG Tribal Evaluation project (OMB clearance package forthcoming; 60-day Federal Register Notice published on April 18, 2011, Vol. 76, No. 74,). Additionally, the Tribal grantees’ programs and the evaluation will use the HPOG data system presented in this package to assure that where there is overlap, data are being collected in the same way and are only being collected once to meet these multiple needs. ACF also anticipates funding university-based research grants to examine and assess various aspects of the HPOG initiative, and other future HPOG evaluations and analyses. In addition, up to three of the HPOG grantee programs will be a part of the Innovative Strategies for Increasing Self-Sufficiency (ISIS) project to study the impacts of these programs on participant outcomes. A separate clearance request is being submitted to OMB for the ISIS project (OMB # 0970-0343), which is an ACF-funded evaluation of career pathways programs training low-income individuals for various occupations, including, in some sites, health care jobs. A high level of coordination among all of these related activities is underway to eliminate duplication of effort and minimize the data collection burden on grant recipients and their program participants, promote cross-site comparability of data items, and enhance the cumulative development of knowledge useful to government policy makers and to the public.

A1. Circumstances Making the Collection of Information Necessary

ACF seeks approval for the collection of data through the HPOG Performance Reporting System described in this request to support HPOG research and evaluation activities and ACF’s responsibility for managing the performance of the HPOG grants. As mentioned, ACF is funding a multi-pronged strategy to examine and evaluate the HPOG initiative. Abt Associates Inc., in partnership with The Urban Institute, is conducting the Implementation, Systems, and Outcome Evaluation of HPOG which includes: 1) designing and implementing the Performance Reporting System, an ongoing tracking and management information system, and 2) developing designs appropriate for evaluating the implementation, systems change and outcomes of the HPOG programs, also referred to as the National HPOG Evaluation. The data collected through the Performance Reporting System will serve multiple purposes, including: 1) facilitating ongoing performance management by ACF and by the HPOG grantees; 2) supporting the National Evaluation being designed under the current Abt/Urban Institute contract; 3) supporting the ongoing Tribal evaluation; 4) avoiding some duplication of data collection burden across the National HPOG Evaluation and the ISIS project; and 5) supporting prospective research to be conducted under the anticipated HPOG university partnership grants and other research projects currently being developed by ACF.

As part of the landmark Patient Protection and Affordable Care Act (PAPCA) of 2010, Congress authorized funds for the HPOG program “to conduct demonstration projects that provide eligible individuals with the opportunity to obtain education and training for occupations in the healthcare field that pay well” (Grant Announcement HHS-2010-ACF-OFA-FX-0126). These demonstration projects are thus intended to address two pervasive and growing problems: the increasing shortfall in the supply of healthcare professionals in the face of expanding demand, and the increasing requirement for a post-secondary education to secure a job with a living wage for families. If ACF does not take the unique advantage these demonstrations present to build knowledge about whether and how such projects are feasible and effective, a great opportunity may be lost. Ongoing performance data collection and rigorous research and evaluation activities are essential parts of building this knowledge for HPOG grantees, similar service and education providers, and for the federal government.

Legal or Administrative Requirements that Necessitate the Collection

H.R. 3590, the Patient Protection and Affordable Care Act, stipulates that grantees must submit both periodic and final reports on activities carried out under the HPOG initiative (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(A)). The Act also mandates an evaluation of the demonstration projects (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(B)). The Act further indicates that the evaluation will be used to inform the final report to Congress (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(C)).

Accordingly, the FOA for the HPOG grants (HHS-2010-ACF-OFA-FX-0126) includes an administrative requirement for successful grantees to submit semi-annual Performance Progress Reports (SF-PPR) on activities carried out under the project and include assessments and documentation of performance measured against grantees’ planned milestones. Data collected under the HPOG Performance Reporting System described here will be used to automatically generate the federally required semi-annual reports, to inform ACF reports to Congress, to monitor and manage the performance of the grant-funded projects, and to inform the design of the National Evaluation and the evaluation of the Tribal HPOG grantee programs.

Study Objectives

The Implementation, Systems and Outcome Evaluation of the Health Profession Opportunity Grants to Serve TANF Recipients and Other Low-Income Individuals has three major components: 1) develop and maintain a Performance Reporting System that will collect information from the 32 HPOG grantees about their participants, their service delivery activities, and the outputs and outcomes of the services; 2) develop a design for evaluating HPOG implementation, systems and outcomes; and 3) coordinate multiple HPOG research and evaluation efforts. The evaluation being designed will examine and assess the implementation, systems changes, and outcomes to build knowledge regarding innovative strategies to train and educate recipients of Temporary Assistance for Needy Families (TANF) benefits and other low-income individuals, including Tribal members, to become health care professionals. A separate evaluation of the Tribal HPOG grantees is being conducted by NORC at the University of Chicago. Additionally, at least three HPOG grantee programs will be included in the ISIS evaluation project. Future research and evaluation activities may be carried out through grants to university-based researchers and as part of other evaluation efforts. The proposed HPOG Performance Reporting System will collect and store uniform data needed for performance management, incorporating the required semi-annual grantee performance reports to ACF, and for the various HPOG evaluations, including those conducted in the future. Although the HPOG Performance Reporting System will provide necessary data for other research and evaluation efforts as noted, those other projects may require additional data collection efforts for which separate OMB clearance would be requested.

The HPOG Performance Reporting System objectives include:

• Monitor grantee progress towards established goals.

• Measure grantee performance on key HPOG indicators, such as: enrollment, participation in training, completion of training, obtaining employment in health care (See Instrument A.1).

• Collect and maintain data ACF needs to prepare required reports to Congress.

Questions the National HPOG Evaluation design must address that will rely on data collected through this system include:

• How are health professions training programs being implemented across the grantee sites?

• What changes to the service delivery system are associated with program implementation?

• What are the characteristics of individuals targeted and served by grantee sites?

• What individual-level outputs and outcomes occur (for example: recruitment, enrollment, retention, completion (accreditation/certification), job entry, employment retention and advancement, and earnings)?

• What can be learned about how best to implement these programs for this population (what implementation and/or systems components are related to programs outputs and outcomes)?

• What key components appear necessary or contribute to the success of these programs?

A2. Purpose and Use of the Information Collection

As part of the Implementation, Systems and Outcome Evaluation of the Health Profession Opportunity Grants to Serve TANF Recipients and Other Low-Income Individuals, ACF has contracted with Abt Associates and its partner, The Urban Institute to design and operate an HPOG Performance Reporting System (PRS). The PRS will collect data to meet the performance data needs of the grantees and OFA program office to monitor the performance of the grants and report to Congress on the grants. As such – this system will create an administrative data set containing all HPOG participants that captures basic programmatic information (including program enrollment and completion and what services were received). These data may be useful to other future evaluations that seek to answer more detailed questions about program implementation, outputs, outcomes and impacts.

Data will be collected through this system on participants in all HPOG Grantee sites - both those funded under The Health Profession Opportunity Grants to Serve TANF and Other Low-Income Individuals (HHS-2010-ACF-OFA-FX-0126) and the Health Profession Opportunity Grants for Tribes, Tribal Organizations or Tribal College or University (HHS-2010-ACF-OFA-FY-0124).

These data will also support future evaluation needs. Specifically it is anticipated that the data collected through this system will be used in the national evaluation (currently being designed by the Abt/UI team as part of Implementation, Systems and Outcome Evaluation of the Health Profession Opportunity Grants to Serve TANF Recipients and Other Low-Income Individuals contract). This evaluation is intended to measure the implementation, systems changes, and outcomes of the HPOG program and is specific to the 27 Grantees funded under HHS-2010-ACF-OFA-FY-0124.

Additionally, the data will be used to supplement a separate primarily qualitative evaluation of the Tribal HPOG grants (those funded under HHS-2010-ACF-OFA-FY-0124). This evaluation is currently being conducted by NORC at the University of Chicago for ACF (ICR Ref # 201107-0970-002).

ACF also anticipates that these data may be used to supplement other components of its HPOG strategy which includes random assignment impact studies conducted under contract to ACF in a sub-set of HPOG sites and research and evaluation studies conducted under cooperative agreements between ACF and university-based researchers and scholars.

Instrument A.1 presents the HPOG Performance Reporting System Basic (“Common Core”) Participant Data Elements. The Common Core consists of data elements including characteristics of individual participants that will be collected at application and enrollment, assessment results used to determine eligibility and/or placement, education and training received, the services provided to participants, program outputs, and intermediate outcomes (such as completed program, acquired credential, etc.), and end outcome results (such as entering employment, employment in the health sector, and employment status six months after program completion). Several of the data items pertain specifically to tribal members, particularly those served by the Tribal grantees: tribal affiliation, residence on or off-reservation, employment in a tribal organization or enterprise, and receipt of training by a tribal institution. Non-Tribal HPOG grantees that enroll tribal members will also enter these data items. The data items included in the Common Core were determined to be necessary to provide: 1) performance-related information, including performance indicators, needed semi-annually (see Instrument A.1); 2) variables likely to be required to address critical policy and program effectiveness questions in future evaluations of individual outcomes, program implementation, and systems change; and 3) information required by NORC at the University of Chicago for the Tribal HPOG evaluation.

Thus, in addition to micro-data on HPOG participants, the HPOG Performance Reporting System will also be a repository of information about each grantee’s program or programs. The program-level information is important for future evaluation of implementation and systems change, and will be compiled from grantees’ initial applications for funding and the biannual reports grantees submit to ACF and information obtained by NORC on Tribal HPOG grantees. The program and grantee data will be used in the National Evaluation and Tribal Evaluation, and likely other future evaluation efforts, to help analyze each grantee program’s inputs and outputs and place analytic results into the appropriate context.

Informed consent needed to collect personal identifying data will be obtained by grantees from all participants when they are enrolled in HPOG programs. Suggested language for grantees to use for informed consent is provided in Attachment A.1. The institutional review boards for both Abt Associates and The Urban Institute will review and approve all data collection and data security plans prior to the system going online. Data security measures for the data collected through the HPOG Performance Reporting System are described fully in Section A3 below.

Not collecting the information would constrain ACF’s ability to monitor the grant-funded programs and activities and to fulfill the legislative mandate to conduct evaluations of HPOG.

A3. Use of Improved Information Technology and Burden Reduction

The information will be collected through an Internet-based management information system, the HPOG Performance Reporting System. The system is currently under development by Abt Associates and The Urban Institute. The system will be maintained by The Urban Institute and will be designed to allow entry and secure maintenance of information on participants and on the grantees and their programs. Grantee- and program-level information for the non-Tribal grantees will be entered into the system by Abt Associates and Urban Institute staff, extracted from grantee applications, project descriptions, ongoing progress reports, program modification and continuation applications to enter data on participant characteristics, program experiences and outcomes, either by direct entry or by streaming data from existing grantee data systems. Tribal grantees will work with NORC to enter program data into the system in the same way.

Key IT features of the system include:

• Internet-Based Application. The HPOG data system will be implemented on a secure HPOG website maintained by The Urban Institute. Staff at the grantee or sub-grantee level who are granted authorization to access the system will receive a secure password and will be able to enter and/or view data on their program’s participants (but not those in programs operated by other grantees). HPOG evaluators will be able to view all data from participants across all grantees, but private information (such as participant name and Social Security number) will be accessible only for the purposes identified in informed consent forms signed by the participant.

• Efficient and Secure Data Entry Format. The data system will be structured to reduce the burden on grantees while ensuring adequate detail and accuracy. A data streaming capability will be built into the secure web-based system, allowing authorized grantee representatives to program their existing information systems to interface with the HPOG system. This structure will facilitate an interface that allows HPOG participant data that is already on existing grantee or provider management information systems to be uploaded directly into the HPOG data system. Uploading and directly populating the HPOG system with existing electronic data in this manner reduces data entry burden and minimizes data entry errors. Data items that cannot be directly streamed by a grantee will be entered directly by authorized grantee program staff into the HPOG data system. Data items that include private information (e.g., Social Security number) will be automatically encrypted at data entry or at the point of electronic interface.

• Reporting Capability. The HPOG data system will provide a mechanism for grantees to enter and manage performance management information. Data on individual participants will be entered at application and intake as part of the initial interaction between the participant and grantee staff. Receipt of training and other services may be recorded at any time in the individual record. Narrative case notes can also be added as text. The system will provide a platform for grantee representatives monitoring the overall grant implementation to enter semi-annual progress on achieving grant objectives that will be submitted as required by ACF. The system will automatically generate the federally required semi-annual Performance Progress Report (SF-PPR) that includes aggregated participant-level data and narrative-based grantee level performance information. This capability will reduce overall reporting burden since grantees will not need to fill out SF-PPR reports manually.

The HPOG Performance Reporting System will be developed using the highest standards of technology and data security. Data for grantee-level and individual-level records will be stored securely in an SQL server database. The web interface for data entry and reporting will be programmed in ColdFusion. The system will be maintained on a highly secure Internet Information Server (IIS) web server at The Urban Institute, physically located in The Urban Institute’s server room in Washington, DC. The data will supplement the standard PPR data submitted semi-annually by grantees, and it is assumed that grantee staff will provide the HPOG supplemental data items via the performance tracking system.

The database software will be MySQL (current version 5.1.36). When stored, all project files containing private data will be encrypted using PGP software, a tool for encrypting storage media and for creating encrypted compressed files.

The web application will reside on a standard Windows server running a web service such as IIS or Apache that will be physically located in a monitored access controlled secure server room at The Urban Institute in Washington, DC. The web server has been hardened using a best practice security hardening checklist. The web server will be dedicated to the HPOG Performance Reporting System application such that no applications, information, or data pertaining to other projects will be stored on this server. Administrator access to the database server will be restricted to an authorized Urban Institute server administrator. Accounts on the web server will be password protected. Passwords will be at least 8 characters long and contain at least 1 special character and number and will not contain dictionary words. These requirements are enforced upon account creation. Passwords will expire every 90 days and users will have to create new passwords that fulfill the requirement of the password policy. A dedicated Secure Site License (SSL) will be employed for this project and all non-encrypted access to the MIS application will be restricted. Any logging or output files will not contain private data and will be limited to generic system and error data.

All data transmitted from the grantee sites to the application server will take place over the encrypted SSL connection. Data extracts for use by the project team will be available in files encrypted with the PGP software and available to project team members and other approved researchers through The Urban Institute’s Secure FTP (SFTP) site.

The system will segregate user data into private participant data, user identifiable participant data, and project/program level data. Private participant data such as Social Security numbers will be stored in a separate database table containing a system-generated ID number with the Social Security number stored in encrypted form. Private participant data will be entered into the system but will not be visually displayed or downloadable by system users. User-identifiable participant-specific data will be stored separately from project/program-level data and will be available for updating only by the grantee representative who originally entered the data. Project/program-specific data will be available to the project team in specific extracts and reports once the information has been entered and submitted.

Grantees will receive detailed guidelines for data entry and security procedures. Clearly defined variables and labeled fields will specify how to enter each data element. Supporting guidance on data formats and definitions will be provided. Training on the data system will be provided prior to its implementation and technical assistance on the system will be available to grantees throughout their grant period of performance.

A4. Efforts to Identify Duplication and Use of Similar Information

No data system currently exists that provides all of the participant-level data required to meet ACF’s Congressionally mandated requirements and that will be needed to evaluate outcomes. Some data items, however, may exist in other MIS data records at some grantee or program agencies. To minimize duplication of data entry, the HPOG Performance Reporting System is being designed with the capacity to interface electronically with existing data systems used by the grantees or their partners where possible, as discussed in the previous section.¹ To reduce burden further, data definitions and coding conventions have been made compatible with existing programs most relevant to the HPOG grants, such as those authorized under the Workforce Investment Act. Where possible, data elements have also been aligned with other key related ACF proposed data collection efforts, particularly the Tribal HPOG evaluation and the ISIS project. Specifically, the Tribal evaluation will be using data from the Performance Reporting System in its descriptive analysis of the Tribal grants and their accomplishments. As noted earlier, up to three of the HPOG grantees will be included in the ISIS study, and aligning the data elements across the two projects will avoid some duplication of data collection.

A5. Impact on Small Businesses or Other Small Entities

None of the HPOG grantees are small businesses. At the time the grants were awarded, ACF informed all grantees of the reporting requirements and adequate resources have been provided to coordinate the data collection and reporting. There should be no adverse impact even in small grantee organizations.

A6. Consequences of Collecting Information Less Frequently

The FOA for the HPOG grants (HHS-2010-ACF-OFA-FX-0126) includes an administrative requirement for grantees to “submit interim reports, on a semi-annual basis” to ACF. Information in the required semi-annual report for the period beginning September 30, 2011 (the start of the second year of grant funding) will come from the HPOG Performance Reporting System, through the report generation capacity being programmed into to the system. Therefore ACF anticipates that the HPOG Performance Reporting System will be operational starting September 30, 2011.

The consequences of not collecting information through the HPOG data system is that valid and comparable data on participants, activities and outcomes would not be available for all HPOG grantees. Ongoing data collection through the Performance Reporting System offers opportunities to: 1) track the performance and continuous performance improvement of the grantee projects, and 2) ensure that data are available for the National Evaluation currently being designed, the ongoing Tribal HPOG evaluation and future HPOG evaluations. Data collection at less frequent intervals would limit ACF’s ability to track performance and adapt quickly, if necessary, to improve performance and would not provide timely enough data for reporting to Congress.

A7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances for the proposed data collection.

A8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. No. 104-13 and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on January 27, 2011, Volume 76, Number 18, page 4912, and provided a 60-day period for public comment. A copy of this notice is included as Attachment A.2.

During the notice and comment period, the government received no requests for copies of the instrument and no public comments were received.

The development of the data items has been informed by clarifications provided by grantees via email, telephone conversations and site visits, as well as by workplans and progress reports submitted by the grantees to OFA. Additionally, the Abt Associates and Urban Institute project team has assembled a Technical Working Group (TWG) composed of national leaders in relevant knowledge areas and research disciplines, including, for example: health economics, labor economics, statistical modeling, and performance management system design. The TWG represents the range of expertise needed to assist and advise ACF and the HPOG contractors in its two major tasks: the development and implementation of a grantee performance tracking system and the development of a design for the National Evaluation. A TWG meeting was held on May 4, 2011; the following experts attended:

• Joshua Wiener, Ph.D.

Distinguished Fellow and Program Director, Aging, Disability, and Long-Term Care Program

RTI International

• Robert I. Lerman, Ph.D.

Institute Fellow and Professor of Economics

The Urban Institute and American University

• James A. Riccio, Ph.D.

Director, Low-Wage Workers and Communities Policy Area

MDRC

• Kevin Hollenbeck, Ph.D.

Vice President and Senior Economist

Upjohn Institute

• Chris Hulleman, Ph.D.

Assistant Professor of Psychology

James Madison University

• Maureen Conway

Executive Director

Economic Opportunities Program

The Aspen Institute

• Maria P. Aristigueta, Ph.D.

Professor and Director, School of Public Policy and Administration

University of Delaware

• Christine Kovner, Ph.D.

Professor, College of Nursing

New York University

• Olivia Golden, Ph.D.

Institute Fellow, Center on Labor, Human Services, and Population

The Urban Institute

• John Holahan, Ph.D.

Center Director, Health Policy Center

The Urban Institute

• Howard Rolston, Ph.D.

Principal Associate

Abt Associates Inc.

• Stephen Bell, Ph.D.

Principal Scientist

Abt Associates Inc.

The TWG members provided discussion and feedback on the Performance Reporting System plans and data elements. Based on the discussions at the TWG meeting, some data items were revised and others were added on the basis of the TWG group’s assessment of the requirements of the Performance Reporting System and the prospective National Evaluation design.

NORC is also working with an advisory committee for guidance on tribal-specific measures needed for its evaluation. To identify and develop tribal-specific measures, the Tribal evaluation team consulted with members of its Tribal Advisory Committee and received clarifications from Tribal HPOG grantees on information contained in the applications they submitted to OFA in response to the grant FOA. Based on this input, several key measures were identified that reflect unique cultural, population, and implementation factors among Tribal HPOG grantees. Examples include tribal affiliation, living on or off reservation, first generation college student, and employed by a tribal organization.

A9. Explanation of Any Payment or Gift to Respondents

No payments to respondents are proposed for this information collection.

A10. Assurance of Privacy Provided to Respondents

Because individual identification data will be collected for the Performance Reporting System, respondent privacy will be protected to the fullest extent allowed by law. ACF recognizes that the HPOG grantees will be serving vulnerable populations (per the authorizing legislation) and must protect them from any risks of harm from the research and evaluation activities. Informed consent from participants will be obtained to ensure that they understand the nature of the research and evaluation activities being conducted. To assist the grantees in obtaining informed consent, the Abt Associates and Urban Institute project team will provide HPOG grantees with guidance and language for informed consent (see Attachment A.1)

As a part of informed consent, the following rationale for data collection and privacy assurances will be provided to HPOG participants by grantees:

Research is being conducted to see how well different approaches to training for health care jobs work. This program and research are funded by the U.S. Department of Health and Human Services, and they may fund other research on this program in the future…

In this program, we will collect some personal information from you, such as your name, date of birth, Social Security number, and your involvement in other programs. The researchers studying the program for the government also need this information… All of the information about you collected for the program or for the research studies will be kept completely private to the fullest extent allowed by law, and no one’s name will ever appear in any report or discussion of the evaluation results…

As part of the study, researchers may contact some of you in the future. You may refuse to answer any of their specific questions at any time.

Researchers and program staff using the information collected must take all necessary actions to protect your information and they will pledge their agreement to protect privacy.

A11. Justification for Sensitive Questions

Several data items pertaining to individual participant characteristics may be considered sensitive by some program participants, for example, questions about ex-offender status, disabilities, and limited English proficiency. These data are standard items for other workforce development programs and will allow important comparisons between HPOG and other similar efforts for evaluation and management purposes. In addition, these data are needed to fully identify programs and program characteristics that are most successful for serving the vulnerable populations that HPOG was designed to support and that are a focus of ACF’s assistance programs.

Individual identifying information of a sensitive, personal, or private nature includes: (1) last and first name; (2) Social Security number; (3) date of birth, (4) ethnicity and race; (5) marital status; (6) number of children; (7) whether the individual is a TANF or SNAP recipient; (8) disabilities, (9) limited English proficiency, and (10) employment status at program exit.

Reasons why these data items are necessary and specific uses to be made of this information include:

• The last and first names and Social Security Numbers of each participant are needed by evaluators to obtain administrative data on quarterly earnings and receipt of cash and noncash public benefits of participants necessary to estimate key outcomes.² Administrative data matches will only be done for those participants who sign informed consent data agreements.

• Other data which are of a sensitive nature—including date of birth, ethnicity, race, marital status, and whether the individual is a TANF or SNAP recipient—are needed to support detailed analyses of the types of individuals enrolled into HPOG programs and the services and education or training they receive.

• Some characteristics, such as disability status or limited English proficiency, are likely to be associated with special instructional needs. Performance and outcome analysis for these subgroups of participants may provide particularly useful information about effective training approaches for these populations.

• Employment information at program exit will provide essential information about the effectiveness of the HPOG programs, particularly whether individuals enter health occupations after training.

A12. Estimates of Annualized Burden Hours and Costs

Exhibit A12.1 summarizes the reporting burden on grantee respondents using the HPOG Performance Reporting System to submit the Semi-Annual HPOG Program Performance Report to ACF. The burden estimates are based on our developing knowledge of which data categories and items will represent additional collections over what HPOG grantees already collect as part of their administration of the grantee programs. 32 Specifically, the three categories of data items that might involve additional grantee effort to obtain from participants are: baseline characteristics; participant status at program exit; and participant status six months after program exit. Additional reporting burden for these types of data is specified below and summarized in Exhibit A12.1.

At Intake

All grantees already collect considerable information at intake. Some (but not all) grantees may not be routinely recording the following data: D19 (at intake, whether the person is pregnant or an expectant father), D23 (at intake, whether the person is a first-generation college student), D25 and D27 (at intake, whether the person had ever trained for a health occupation and the type of occupation), D33 (at intake, whether the person is an Unemployment Insurance claimant), D34 (at intake, whether the person is homeless), D37 and D38 (at intake, whether the person had ever trained for a health occupation). Nearly all the grantees are either workforce investment agencies (WIA), One-Stop Career Centers or WIA contractors, or community colleges. Those associated with WIA are most likely to be collecting all of the demographic data at intake, including the items listed here, while community colleges may not routinely collect the items listed. If all 8 of these items are added at intake, we estimate it would take a staff person conducting an intake interview no more than 11 minutes per registrant to record them, and a conservative estimate is that about half of the grantees (for 3,000 participants annually) might need to add these items to their intake process.

At exit

All grantees are recording if individuals are employed at program exit, if it is a health care job, and the hourly earnings. Some (but not all) grantees may not be routinely recording the following data in O8 (if employed in health care, the SOC code; Grantees associated with WIA will already be routinely recording SOC code; others are being asked to add the SOC code for the occupation at exit); O11 (if the person has access to employer-sponsored health insurance; and O4 (if the person left the program early, reason for leaving early). Grantees are being asked to add the SOC code for the occupation at exit and whether the person has access to health insurance to the information they already collect. We estimate that this should add no more than about 3 minutes per person who exits with a job for a staff person to record these items, and we expect all grantees may have to add these items to their exit information collection. (Based on other education and training programs such as WIA, we estimate about 65% of participants will exit with a job).

At Follow-up

In their grant applications, all programs indicated they would be conducting follow-up on former participants. To standardize the follow-up periods, we set the time for post-program follow-up at 6 months after the last service was received (this will not add burden since it is just a shift in the timing). All grantees also indicated they would be asking if the person is employed at follow-up. The following are the items that some (but not all) grantees may not have already been collecting at follow-up: E2a (hourly wage), E2b (hours worked last week), E4 (SOC code), E5 (promoted since leaving the program), E6 (access to employer-sponsored health insurance), and E7 (currently in education). Our estimate is that 80% of grantees will have to add these 8 items to their follow-up information collection and that collecting all eight will take about 10 minutes per person contacted at follow-up who is employed at that time. We also estimate that about 60% of those employed at exit would be employed at follow-up and that grantees would reach 80% of exited participants.

Over the five-year life of the grants, the 32 grantees combined plan to serve about 30,000 individuals. Assuming that participation extends on average for 12 months, grantees will be serving 6,000 participants annually, or 187.5 per grantee. Based on our revised calculations above, it is estimated that the time added to usual data collection that each grantee needs to complete the semiannual data collection is about 9.98 minutes/participant, or 19.96 minutes/ participant each year. The average burden hours per respondent are thus 62.4 hours per year (31.2 hours for each semiannual report per grantee), and the total annualized burden is expected to be 1996.8 hours.

The following table recalculates the burden estimate based on the updated information above.

Exhibit A12.1 Annual Information Collection Activities and Additional Respondent Burden for the Semi-Annual HPOG Program Performance Report
Collection Point	Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response3	Total Burden Hours
Program Intake	32	2	17.2	1100.8
Program Exit	32	2	6.1	390.4
Program Follow-up	32	2	7.8	499.2
Total	32	2	31.2	1996.8

Estimates of Annualized Costs

Exhibit A12.2 summarizes the annualized cost estimate for the information collection activities. Report respondents will be project directors or designated staff in grantee organizations (institutions of higher education, workforce investment boards, state agencies, tribal councils, and community-based organizations). To compute the total estimated annual cost, the total burden hours were multiplied by the average hourly wage, according to the Bureau of Labor Statistics, National Compensation Survey, 2005 ($29.81/hour). The total estimated annual cost is $59,531.⁴

Exhibit A12.2 Annualized Cost Estimate
Instrument	Total Burden Hours	Average Hourly Wage	Total Annual Cost
Semi-annual HPOG Program performance Report*	1997	$29.81	$59,531

A13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

Not applicable. The proposed information collection activities do not place any new capital cost or cost of maintaining capital requirements on respondents. All grantees will use their existing computers to access the Internet and enter data on the secure Internet site.

A14. Annualized Cost to the Federal Government

The information collection activity and associated instruments have been developed by the evaluation contractor, Abt Associates Inc. The proposed data collection will take place beginning on September 30, 2011, continuing through September 30, 2014. The annual cost of this data collection to the government is estimated to be $166,997. These projected costs were based on the contractor’s detailed budget analysis of the labor time and other costs needed to meet the specifications for this component of the procurement.

A15. Explanation for Program Changes or Adjustments

This submission to OMB is a new request for approval.

A16. Plans for Tabulation and Publication and Project Time Schedule

A16.1 Analysis Plan

The HPOG Performance Reporting System will be programmed to allow the contractor and ACF to produce regular semi-annual HPOG performance summary reports across all grantees, for the overall national initiative. The specific reports will include: 1) HPOG Participation Report, 2) HPOG Performance Measures Report, and 3) HPOG Grantee Program Services Summary. Semi-annual summary tables will be prepared for ACF, tabulating the enrollments, services, activities and outcomes, and performance (project to date and compared to targets set by grantees), based on data entered by grantees into the data system. These summary performance tables will be programmed into the data system and will be automatically produced from the data items in the system. ACF will use these tables when preparing reports to Congress on the HPOG initiative, including the Tribal grantee programs.

As mentioned, the Tribal evaluation contractor (NORC at the University of Chicago) will use data from the Performance Reporting System in its evaluation project. Following is a citation from the NORC evaluation plan outlining their approach to analysis of the system data:

Quantitative program operations data collected through the HPOG PMS will be analyzed on a semi-annual basis to track the progress and success of the Tribal HPOG grantees. The functionality of the HPOG Performance Reporting System will allow for aggregate reporting of data elements and performance indicators. The Tribal HPOG Evaluation Team will coordinate with the Performance Reporting System team to receive individual level data without identifiers for the Tribal HPOG grantees. The Tribal HPOG Evaluation Team will also request aggregated performance indicator across all Tribal HPOG grantees, by grantee program, and by grantee implementation site, if available. These data will be requested and analyzed on a semi-annual basis beginning in October, 2011.

Aggregate reports generated from the Performance Reporting System will allow the Tribal HPOG Evaluation Team to report descriptive statistics and performance indicators for each Tribal HPOG grantees and across all Tribal HPOG grantees. This analysis will allow us to assess grantees’ progress toward meeting goals. Descriptive statistics and performance indicators for the Tribal HPOG grantee portfolio will be compared to entire portfolio of HPOG grantees to highlight successes and challenges uniquely encountered by the Tribal HPOG grantees.

Additionally, the Tribal HPOG Evaluation Team will conduct descriptive analyses on the individual-level data to determine across the Tribal HPOG programs which factors contribute to the success of the program. We will assess the extent to which select program outcomes variables are statistically associated with factors, such as demographic characteristics of program participants and program-related variables, such as receipt of employment or supportive services.⁵

In addition to the currently funded evaluation for the Tribal Grantees, and as part of the same contract under which Abt Associates and the Urban Institute are developing and implementing the HPOG Performance Reporting System, Abt Associates and theUrban Institute are designing a future evaluation that will answer the following questions:

1. How are health professions training programs being implemented across the grantee sites?

2. What changes to the service delivery system are associated with program implementation?

3. What individual level outputs and outcomes occur (for example: recruitment, enrollment, retention, completion (accreditation/certification), job entry, employment retention and advancement, and earning)?

4. What can be learned about how best to implement these programs for this population (what implementation and/or systems components are related to programs outputs and outcomes)?

5. What key components appear necessary or contribute to the success of these programs?

The data collected under this ICR that will be particularly useful in answering questions c, d and e above.

A16.2 Time Schedule and Publications

ACF is seeking OMB approval for three years beginning September 30, 2011. In the case the grants exceed the three years for OMB approval, an extension will be sought at the appropriate time.

The current contract for the Implementation, Systems and Outcome Evaluation of HPOG began September 30, 2010 and goes through September 30, 2014. This OMB submission seeks approval for information collection from HPOG grantees for three years, from September 30, 2011 through September 30, 2014. In the case the grants exceed this time period, a contract and OMB extension will be sought at the appropriate time. The summary performance tables noted in the previous section will be programmed into the system and will be automatically produced from the data items in the data system. ACF will use these tables when preparing reports to Congress.

The report schedule for the Tribal Evaluation is as follows:

An Interim Report will be prepared in Year 3 that presents key findings to date related to the key evaluation questions.

A Final Report will build upon the structure of the Interim Report and will be prepared in Year 5.

Practice Briefs will be prepared on an annual basis beginning in Year 2.

As discussed above, future HPOG evaluations and research will have access to data in the HPOG Performance Reporting System, and the reports from those studies will be specified in future design reports for those projects.

A17. Reason(s) Display of OMB Expiration Date is Inappropriate

All performance measurement data system instruments for the Health Profession Opportunity Grants (HPOG) National and Tribal Evaluations will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1 Consideration was given to providing interface capability with data from other agencies not partnering with HPOG (e.g., human services, child care), but the costs of developing the programming linkages and obtaining legal clearances for data sharing seem prohibitive. Establishing the data linkages with HPOG and partnering agencies will adequately provide data items needed for performance management and evaluation purposes.

2 Only the last four digits of the SSN will be collected for the Tribal grantee participants to further protect privacy, as requested by Tribal councils. The use of the last four digits, along with name and date of birth, will affect the ability to match with quarterly wage records only minimally.

3 Averaged across 32 grantees

4 Source: Bureau of Labor Statistics, National Compensation Survey, 2005: Average hourly wage of college teachers, social scientists, legislators, and public administration officials.

5 Draft Report: Tribal Health Professions Opportunities Grants: Evaluation Plan. Chicago: NORC, May, 2011.

Part A: Justification A-2