NextGen Supporting Statement_Part B_5.11.15

Supporting Statement for OMB Clearance Request

Part B

Evaluation and System Design for Career Pathways Programs: 2nd Generation of HPOG (HPOG Next Gen Design)

May 2015

Submitted by:

Office of Planning,
Research & Evaluation

Administration for Children & Families

U.S. Department of Health
and Human Services

Federal Project Officer:

Hilary Forster

Table of Contents

Part B: Statistical Methods 1

B.1 Respondent Universe and Sampling Methods 1

B.2 Procedures for Collection of Information 1

B.3 Methods to Maximize Response Rates and Deal with Non-response 2

B.4 Tests of Procedures 3

B.5 Individuals Consulted on Statistical Aspects of the Design 4

Attachments:

Instrument 1: PAGES Grantee- and Participant-Level Data Items and Definitions

Attachment A: References

Attachment B: Informed Consent Forms

Attachment C: 60 Day Federal Register Notice

Attachment D: Sources and Justification for PAGES Grantee- and Participant-Level Data Items

Attachment E: Performance Progress Reports (PPRs) Data List and Mockup

Attachment F: First Round of HPOG Grantees Research Portfolio

Attachment G: Participant Contact Information Update Letter and Form

Part B: Statistical Methods

Part B of the Supporting Statement for The Evaluation and System Design for Career Pathways Programs: 2nd Generation of Health Profession Opportunity Grants (HPOG Next Gen Design)—sponsored by the Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services (HHS)—considers the issues pertaining to Collections of Information Employing Statistical Methods. Abt Associates (Abt) is the prime contractor for the study and data system design. The federal evaluations of the HPOG Next Gen grantees will evaluate postsecondary career pathway programs, focused on the healthcare sector, that target economically disadvantaged families and individuals.

This submission seeks clearance for one data collection instrument—the HPOG Next Gen Participant Accomplishment and Grant Evaluation System (PAGES)—which includes information from all HPOG Next Gen grantees on their program designs and offerings, intake information on eligible applicants (both treatment and control) through baseline data collection, and a record of participants’ activities and outcomes.

B.1 Respondent Universe and Sampling Methods

The HPOG Next Gen PAGES will include the applicant population of the anticipated 40 organizations receiving HPOG funding.¹ As discussed, the system will provide data at the grantee and individual levels. Thus, data will be collected from the expected 40 grantees on their program designs and offerings, from all eligible applicants on their baseline characteristics, and from all of the individuals the grantees serve on their individual participation and outcomes.

We anticipate that approximately 32,700 individuals will complete the baseline data collection across the expected 40 grantees during the HPOG Next Gen grant period. We expect the study sample to include up to 31,500 individuals who apply to participate in the HPOG programs operated by approximately 35 non-Tribal HPOG Next Gen grantees participating in the impact evaluation (15,750 control and 15,750 treatment). We expect up to 1,200 individuals to apply to participate in the HPOG programs operated by approximately five HPOG Next Gen Tribal grantees. Among the total sample, the grantees will continue to collect ongoing participant-level data on up to 16,950 HPOG enrollees. No sampling techniques will be employed for HPOG PAGES data collection.

B.2 Procedures for Collection of Information

1. HPOG Program Performance Report Based On Grantee-Level and Ongoing Participant-Level Data

During the initial 120 day planning period after grant award, all grantees will be required to provide information on their program components and offerings to be included in the data system. All HPOG grantees must update the grantee-level and ongoing participant-level data at least semi-annually, but may enter data at any time either manually or by uploading data from existing data systems into an internet-based data system (PAGES) constructed for the purpose of collecting data for the HPOG Next Gen programs. PAGES will be managed by Abt Associates.

2. Participant-Level Baseline Data Collection

Grantee staff will administer an informed consent form relating to PAGES data items, administrative data, and follow-up surveys, when individuals who apply to the program are found eligible. All HPOG Next Gen grantees must obtain informed consent from applicants prior to collecting personally identifying data.

All non-Tribal HPOG Next Gen grantees are required to participate in the impact evaluation. During the intake processes for individuals applying to grantees in the study, staff first will describe the impact study and administer the random assignment informed consent form (see Attachment B). All eligible applicants for HPOG during the intake period for the study must sign the paper informed consent form to be part of the study. The informed consent form allows researchers to: (1) access baseline data participants provide at intake and information about the training and services treatment group members receive after enrollment; (2) contact both treatment and control group members for updated contact information and additional follow-up questions for the impact evaluation; and (3) gather administrative data from the National Directory of New Hires (NDNH) and possibly other sources, such as the National Student Clearinghouse (NSC). After the consent process, grantee staff will administer PAGES baseline questions, including the baseline questions on individuals’ expectations for education and employment, barriers to employment, work preferences, and self-efficacy. After individuals complete the informed consent form and the required baseline questions, a secure, web-based software program will randomly assign them into either the treatment or control group. If an HPOG Next Gen grantee is selected to implement and test a program enhancement as part of the impact study, participants at that grantee will be randomized into one of three experimental groups—the treatment, treatment-enhanced, or control group.

These requests for contact information updates (Appendix G) provide HPOG Next Gen participants the opportunity to update their contact information and provide alternative contact information. Participants can send back the updated information in an enclosed self-addressed stamped envelope. Researchers will send participants $2 as a as a gesture of appreciation to respondents for completing and mailing back the contact information update form. A research team will send contact update letters about every four months after random assignment, pending OMB approval.

The HPOG Next Gen Tribal grantees are participating in a federal evaluation that will not require random assignment. Applicants at the Tribal grantees will complete the non-random assignment informed consent form (see Attachment B) that asks permission for researchers to access data individuals provide at intake, information about the training and services they receive after enrollment, administrative data, and follow-up surveys. Eligible applicants to the Tribal programs may enroll in the programs whether or not they consent to being included in the research. After the consent process, grantee staff will administer PAGES baseline questions, excluding the baseline questions on individuals’ expectations for education and employment, barriers to employment, work preferences, and self-efficacy. These baseline questions are excluded as their main purpose is for testing the composition of the treatment and control groups in the random assignment experiment.

3. Procedures with Special Populations

To ensure participants can understand each of the documents, the informed consent forms and PAGES data elements are designed at an 8th-grade readability level. The HPOG Next Gen Design team will provide a Spanish version of the informed consent forms and will work with grantees on ways staff can assist where translation of other data collection instruments may be needed.

B.3 Methods to Maximize Response Rates and Deal with Non-response

We expect to obtain a 100 percent response rate from the HPOG Next Gen grantees in collecting both the grantee- and participant-level data, all of which will have committed to provide the required performance data and reports to ACF as part of the cooperative agreements made in accepting their grants. However, it is expected that some grantees may have more technical challenges than others in using PAGES, depending on the staff that will be entering data. Thus, Abt Associates and the Urban Institute will provide grantees training and technical assistance on PAGES to assist grantees in complying with the data requirements.

PAGES will have several features for addressing missing data. First, on each form, certain key fields will be required, forcing the user to enter the data prior to saving the form. However, it is not always possible to make a field required, since users may not have all information available at the same time. Thus, PAGES will issue alerts when those data are not complete, but still allow users to save. For fields that are associated with warnings, PAGES will also maintain a queue of missing data available from each user’s dashboard, allowing user to easily see which data are missing and quickly add that information. Finally, there will be data quality reports, for Urban Institute staff to monitor for missing data at the grantee level.

We anticipate that offering training and ongoing technical assistance on PAGES to the grantees will also improve the reliability of the data. In addition to clearly labeling the data fields and providing definitions and instructions within the system, a separate PAGES Guidance Manual for grantees will be developed to provide a detailed explanation of each data element, definitions, and how to enter the data. Other training activities and materials, potentially including webinars and targeted, brief “how-to” documents, will also be provided. Documents and recorded webinars will be available to grantees electronically so users can readily access them whenever needed.

Additionally, Urban Institute staff will be available to provide technical assistance to grantees via a free telephone number. These staff will be able to answer data collection and data entry related questions, and can refer the caller to other staff as needed for assistance of a more technical nature. Users will also be able to submit issues and questions directly through an issue tracking system accessible within PAGES itself. Urban Institute staff will respond to questions related to data collection. IT staff will respond to questions of a technical nature.

As for baseline data items, all eligible individuals who apply to the HPOG Next Gen programs must complete the informed consent forms and all required baseline data collection in order to have the opportunity to enroll in HPOG programs. Therefore, a response rate of 100 percent is expected for required baseline elements.

With the participant contact information update form, offering appropriate monetary gifts to study participants in appreciation for their time can help ensure a high response rate, which is necessary to ensure to improve researchers’ ability to locate respondents for follow-up surveys needed to measure intervention impacts. About every four months researchers will mail to both treatment and control group members a contact information update letter and form to confirm or update the PAGES baseline contact information. Researchers will send the $2 to participants with the updated information request as a gesture of appreciation to respondents.

B.4 Tests of Procedures

Abt Associates is subcontracting with Green Beacon Solution, a leading Microsoft Gold Certified software development firm to create the HPOG Next Gen PAGES in consultation with ACF. Federal staff and evaluation team members had informal discussions with six first round HPOG grantees on possible data system designs and data elements. Prior to the initial launch of PAGES there will be two distinct environments, one for development and one for testing. Development and testing will be conducted in multiple stages. Major modules will be built in four to six week increments. Green Beacon staff will first develop and test each major module created in the development environment. These modules will then be deployed to the test environment. Abt Associates professional software testers will then test the module and submit issues. Once these issues are resolved, the broader team of program staff will review the module and offer feedback. In the month prior to the release of the system, Abt technology staff will re-test all parts of the system to ensure that it meets the stated functional requirements and is free of bugs. Abt Associates, Urban Institute and ACF program staff will also conduct user acceptance testing.

After the system moves into full operations mode (anticipated September 2015), three environments will be maintained: Development, Test, and Production. Any changes to the system will have to move through the Development and Testing environments and procedures prior to being deployed in the live production environment. The Test environment will also serve as the Training environment, allowing for trainers and users to enter fictional data without affecting the Production system. Using the Test environment as a training environment also allows users to be trained on any new features developed before they are released into the live system. These user training sessions also serve to identify any outstanding issues prior to Production release. All three environments will be maintained throughout the life of the project.

The baseline questions are either identical or similar to questions used in previous Abt Associates or national surveys. As such, they have been thoroughly tested on large samples.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

With ACF oversight, Abt Associates and Urban Institute are responsible for developing the HPOG Next Gen PAGES and providing HPOG Next Gen grantees with support for using the system to produce the required semi-annual reports and collecting the needed baseline data.

Statistical analyses of the data for annual program performance reports will be limited to descriptive tabulations included in the contractor’s annual reports to ACF (see A.16). Other as yet unspecified statistical analyses may be planned for the impact evaluation currently being designed and other future research efforts.

The individuals listed in Exhibit B5.1 below made a contribution to the statistical aspects of the design of the data system.

Exhibit B5.1. Contributors

Name	Role in HPOG Next Gen	Organization/Affiliation
Julie Strawn	Project Director	Abt Associates
Dr. Laura R. Peck	Co-Principal Investigator	Abt Associates
Dr. Pam Loprest	Co-Principal Investigator	Urban Institute
Dr. Eleanor Harvill	Evaluation Design Task Lead	Abt Associates
Brian Sokol	Data System Task Lead	Abt Associates
Dr. Alan Werner	Key staff on HPOG Next Gen project	Abt Associates
Jennifer Buell	Deputy Project Director	Abt Associates
Dr. Howard Rolston	Project Quality Advisor	Abt Associates
Karen Gardiner	Project Quality Advisor	Abt Associates

Inquiries regarding the statistical aspects of the design of the data system should be directed to:

Julie Strawn, Project Director

Abt Associates

4550 Montgomery Ave #800N

Bethesda, MD 20814

(301) 347-5853

The HHS project officer, Hilary Forster, has overseen the design process and can be contacted at:

Hilary Forster

Office of Planning, Research & Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

370 L'Enfant Promenade, SW, Washington, DC 20447

(202) 619-1790

1 The anticipated project start date for the new grantees is September 30, 2015. The FOAs can be found: http://www.acf.hhs.gov/grants/open/foa/index.cfm?switch=foa&fon=HHS-2015-ACF-OFA-FX-0951; http://www.acf.hhs.gov/grants/open/foa/index.cfm?switch=foa&fon=HHS-2015-ACF-OFA-FY-0952