HPOG NIE and Impact Supporting Statement_Part B 4 24 13 revised 7 26 13_rev 8.7.13

Supporting Statement for OMB Clearance Request

Part B

National Implementation Evaluation of the Health Profession Opportunity Grants (HPOG) to Serve TANF Recipients and Other Low-Income Individuals and HPOG Impact Study

0970-0394

April 24, 2013

Revised July 26, 2013

Submitted by:

Office of Planning,
Research & Evaluation

Administration for Children & Families

U.S. Department of Health
and Human Services

Federal Project Officers:

Molly Irwin and Mary Mueggenborg

Table of Contents

Part B: Statistical Methods 1

B.1 Respondent Universe and Sampling Methods 1

B.1.1 HPOG Grantees, Partners, Stakeholders, and Employers 1

B.1.2 HPOG Participants and Control Group Members 2

B.1.3 Target Response Rates 3

B.2 Procedures for Collection of Information 3

B.2.1 Sample Design 3

B.2.2 Estimation Procedures 6

B.2.3 Degree of Accuracy Required 7

B.2.4 Who Will Collect the Information and How It Will Be Done 9

B.2.5 Procedures with Special Populations 12

B.3 Methods to Maximize Response Rates and Deal with Non-response 12

B.3.1 Grantees, Management and Staff, Partners and Stakeholders, and Employers 12

B.3.2 HPOG Participant Survey 12

B.4 Tests of Procedures 14

B.4.1 Grantee, Management and Staff, Stakeholder/Network and Employer Surveys 14

B.4.2 15-Month Participant Follow-Up Survey 14

B.5 Individuals Consulted on Statistical Aspects of the Design 14

References 15

Appendices

A. HPOG Logic Model

B. HPOG-NIE Sampling Questionnaire for the HPOG Surveys

C. HPOG-NIE Follow-Up Phone Call Protocol for the Stakeholder/Network Survey

D. HPOG-NIE Grantee Survey

E. HPOG-Impact Implementation Interview Guides

F. HPOG-NIE Management and Staff Survey

G. HPOG-NIE Stakeholder/Network Survey

H. HPOG-NIE Employer Survey

I. HPOG-Impact and HPOG-NIE 15-month Participant Follow-Up Survey

J. HPOG-Impact 15-month Control Group Member Follow-Up Survey

K. OMB 60-Day Notice

L. Consent Forms for Participant Surveys

M. Information on the Full ISIS Sample Frame

N. Previously Approved Supplementary Documents

O. HPOG-Impact and HPOG-NIE Tracking Letter and Contact Update Form

P. HPOG-Impact and HPOG-NIE Advance Letter

Part B: Statistical Methods

In this document, we discuss the statistical methods to be used in the initial data collection activities for the Health Profession Opportunity Grants National Implementation Evaluation (HPOG-NIE) and for follow-on data collection activities for the HPOG Impact Study (HPOG-Impact). Both studies are sponsored by the Office of Planning, Research and Evaluation (OPRE) in the Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services (HHS).

B.1 Respondent Universe and Sampling Methods

Thirty-two HPOG grants were awarded to government agencies, community-based organizations, post-secondary educational institutions, and tribal-affiliated organizations to conduct these activities. Of these, 27 were awarded to agencies serving TANF recipients and other low-income individuals. All 27 participate in HPOG-NIE. Twenty of these grantees participate in HPOG-Impact. Three of these grantees participate in the Innovative Strategies for Increasing Self-Sufficiency (ISIS) project.

For HPOG-NIE and HPOG-Impact, there are two major respondent universes: (1) HPOG grantees, partners, stakeholders, and employers; and (2) HPOG participants and potential participants, including HPOG-Impact treatment and control group members and participants from HPOG-NIE grantees that are not participating in HPOG-Impact or ISIS.

B.1.1 HPOG Grantees, Partners, Stakeholders, and Employers

Below we describe each of the HPOG-NIE and HPOG-Impact respondent subgroups and respective data collection strategies.

1. HPOG Grantees

For HPOG-NIE and HPOG-Impact, the universe includes all of the grantees participating in either of these studies. The research team will ask the total universe of HPOG grantees to respond to two major data collection activities that will support both HPOG-NIE and HPOG-Impact. First they will respond to the on-line Grantee survey (Appendix D). This on-line survey includes sections related to general program context and structure. It also includes sections about intake and enrollment, education and training, and support services. HPOG staff answering the Grantee survey may reach out to their partner organizations for additional information necessary to complete this survey. The number of respondents will vary according to the size of the grantee and individual organizational roles and responsibilities. On average, the team expects one to three respondents to answer one or more modules of the Grantee survey, for a total of 54 respondents.

Second, staff from HPOG-Impact grantees will participate in a round of site visits. During the site visits, the research team will interview relevant site staff, including core HPOG staff such as case managers, placement specialists, and training and education instructors. The team will also interview other staff as appropriate for program structure, such as support service coordinators (Appendix E). Again, the number of HPOG staff participating in the site visits will depend on grantee size and organizational structure. On average we expect between 10 and 11 respondents per site, for a total of 220 research participants.

Finally, HPOG-Impact grantees participating in the experimental (three-arm) test of one of the enhanced program components (as described in Part A of this request) will participate in an additional round of site visits (see Appendix E). We expect that 13 grantees will be part of the experimental test of a program enhancement, for a total of 100 potential respondents.

2. HPOG Management and Staff

We define HPOG management and staff as the universe of staff from HPOG grantees and partners, including intake staff, case managers, and counselors. We also include staff who combine participant support with instruction, such as in soft skills workshops, and management staff who interact directly and regularly with HPOG participants.

The research team will ask all HPOG management and staff, as defined here, to participate in the on-line Management and Staff survey (Appendix F). It is necessary to survey all management and staff because of their different roles in the program. A sample of the relatively small number of staff in many grantees (averaging around 20 per grantee) will lead to some roles being represented in the responses while others are not. This will also make any comparisons across grantees difficult. Subsampling within roles is not possible given the small numbers. An average of 20 staff per grantee will be asked to participate, with the actual number varying by grantee size, for a total of 540 potential respondents.

3. HPOG Partners and Stakeholders

We define HPOG partners as entities that participate in HPOG operations, such as by referring prospective HPOG participants, providing data to HPOG programs useful for program recruitment and implementation, offering opportunities for work-based learning or other work-based experiences, and providing other services or trainings. HPOG stakeholders include other individuals and organizations with an active interest in the 27 grantees participating in HPOG-NIE and their results, whether or not they participate directly in program operations. The NIE includes a network analysis based on network partners’ and stakeholders’ reported roles in and perspectives on the network. To accomplish this analysis, the team will interview all partners and stakeholders in a grantee HPOG network. Sampling among these partners and stakeholders would not allow development of the full picture of the network, particularly because partners and stakeholders hold very different roles. Approximately 500 partners and stakeholders will participate in the HPOG survey across all grantees.

4. HPOG Employers

The universe of HPOG employers for this study includes two groups: 1) those who could be considered HPOG partners, i.e., those who were involved in program design, development, and implementation of HPOG; and 2) those not directly involved as partners but who are active in hiring or who have been contacted about hiring HPOG participants. A purposive sample of employers who are most active in hiring HPOG participants will be surveyed. The research team will interview employers from these two groups by telephone or through an on-line survey. Employers who are in both of these groups will use a single login to respond to the Stakeholder/Network and Employer surveys at one time. Employers only in the second group will complete an interview by telephone. Approximately 200 employers will participate in the Employer survey.

B.1.2 HPOG Participants and Control Group Members

For HPOG-Impact, the universe of potential respondent participants consists of those adults eligible for services who actively seek out training for a healthcare profession from an HPOG grantee. Program staff will recruit these individuals and determine eligibility. For those individuals who are deemed eligible for the program and who furthermore agree to be in the study, program staff will obtain informed consent (Appendix L). If individuals do not agree to be in the study, they are not eligible for HPOG services.

For HPOG-Impact, the target enrollment for the study is a total of 10,500 would-be students across the 20 participating grantees. (This sample will be supplemented with a sample of 2,220 would-be students from the three HPOG sites participating in ISIS. This data collection is included in OMB submission 0970-0397.) HPOG grantees vary greatly in size; the target amount of study sample from each grantee is proportional to the grantee’s annual numbers of HPOG participants.

For HPOG-NIE, the sample will be approximately 600 HPOG student participants from among the four HPOG-NIE grantees who are participating neither in HPOG-Impact nor in ISIS.

B.1.3 Target Response Rates

Overall, we expect response rates to be sufficiently high in this study to produce valid and reliable results that can be generalized to the universe of the study. For HPOG-NIE and HPOG-Impact, we expect the following response rates:

• HPOG grantees and partners. Grantees and partners have agreed to participate in the evaluation as a condition of receiving HPOG grant funding. Therefore, we expect a 100 percent response rate;

• HPOG management and staff. Similarly, we expect a very high response rate (at least 80 percent) among grantee and partner staff.

• HPOG stakeholders. Based on similar research, we expect an 80 percent response rate among stakeholders.

• HPOG employers. Also based on similar research, we expect an 80 percent response rate among HPOG employers.

• Follow-up survey of HPOG-Impact participants and HPOG-NIE participants. We expect an 80 percent response rate, which is based on experience in other studies with similar populations and follow-up intervals.

B.2 Procedures for Collection of Information

B.2.1 Sample Design

The sample frame includes all of the HPOG-NIE grantees, including the HPOG grantees who are participating either in HPOG-Impact or in ISIS. (For information on the full ISIS sample frame, please see Appendix M.) This section first describes the sample design related to organizations and HPOG staff involved in HPOG (e.g., grantees, management and staff, stakeholders, and employers). We then describe the sample frame for HPOG-Impact treatment and control group members.

5. Grantees, Partners, Stakeholders, and Employers

Below is a summary of the sample design for each of these groups.

• HPOG grantees. The research team will collect data from all HPOG-NIE grantees on the major aspects of the program (i.e., recruitment, intake, assessments, academic and non-academic supports, basic education, training, employment assistance, etc.).

• HPOG management and staff. The research team will collect data from all HPOG-NIE staff who interact directly with HPOG participants, with the exception of general instructional staff, as well as from all of their direct supervisors. Respondents include intake staff, case managers, counselors, and staff who combine participant support with general skill-building instruction (e.g., soft skills workshops).

• HPOG partners and stakeholders. Partners and stakeholders will include, to the greatest extent possible, the universe of partners and stakeholders in the communities where HPOG-NIE grantees are located.

• HPOG employers. Respondents will include (1) all of the employers who would be considered “partners” and directly involved in the HPOG programs, and (2) employers not directly involved as partners but active in hiring HPOG graduates or who have been contacted by programs as potential employers of HPOG participants.

6. HPOG Study Participants—HPOG-Impact

The universe of HPOG study participants consists of those adults eligible for services who actively seek out training for a healthcare profession from an HPOG grantee. For those individuals who consent to participate in HPOG-Impact, HPOG staff collect baseline data. (OMB approved these forms under previous requests for clearance.) Program staff enter this information into the HPOG Performance Reporting System (PRS). Evaluation staff then use the system to randomly assign these individuals to the treatment or control group.

In sites that agree to test an approved program enhancement, individuals will be randomly assigned to a control group and one of two treatment groups: a basic HPOG treatment group, and an enhanced HPOG treatment group. (See Appendix L for the consent form for sites implementing enhancements.)

Therefore, the baseline sample will include:

• 3,500 individuals in the no-HPOG control group;

• 7,000 total individuals in an HPOG treatment group, including:

• Approximately 4,900 individuals in the HPOG basic treatment group; and

• Approximately 2,100 individuals in the enhanced HPOG treatment group, located in the grantees that agree to test the enhancement selected for the study.

Those assigned to the treatment group are offered HPOG services. Those assigned to the control group are not offered HPOG services but can access other services in the community. Exhibit B-1 summarizes the general process described above.

Exhibit B-1: HPOG-Impact Study Participant Recruitment and Random Assignment Process

7. HPOG Study Participants—HPOG-NIE

In addition, the research team will interview HPOG participants from the four HPOG grantees serving TANF and other low-income individuals not participating in either HPOG-Impact or ISIS. For these four grantees, the team will sample individuals who consent to participate in any federally funded research study of HPOG for the follow-up survey. This is the consent process implemented in September 2011 (prior to the implementation of HPOG-Impact) to cover all federally funded research on HPOG and for which HPOG participants have the discretion to withhold consent.¹ HPOG staff collect baseline data from individuals who consent to participate in HPOG with forms approved by OMB under previous requests for clearance (see Appendix N). Program staff enter this information into the HPOG PRS. The final sample size, taking into account attrition, will be 600 from among these agencies. The sample will include all of the HPOG participants from these agencies who enroll in HPOG services during the three-month time period when the HPOG-NIE Grantee survey will be fielded, so that their experiences correspond closely to the time period during which we are also collecting implementation data.

B.2.2 Estimation Procedures

8. Procedures for HPOG-NIE

Estimation procedures will be, for the most part, very simple.  Much of the data collection will be on a census basis, removing the need for survey weights.  There will be one Grantee survey per grantee or subgrantee. We will survey all HPOG management and staff using the Management and Staff survey, as described above. The Stakeholder/Network survey will cover all partners and stakeholders in the HPOG grantee network.  Additionally, the Employer survey will use a purposive sample, so no survey weights will be needed.  For these data systems, estimation will mostly involve reducing the rich data elements to a set of key indicators and then calculating simple means and proportions. 

Estimation will be slightly more complicated for the 15-month Participant Follow-Up survey.  Although this survey is primarily conducted under the HPOG-Impact contract, data on HPOG participants (but not control subjects) will also be used in HPOG-NIE analyses.  For HPOG-NIE analyses, the randomized HPOG participants will be combined with a sample of participants at the four grantees which do not participate in HPOG-Impact or in ISIS.  At these four grantees, the sample will be a census and data will be collected on participants enrolled in a narrow time window.  Therefore, the combined sample will include two censuses of participants that are defined by specific time windows for HPOG enrollment.  As censuses, the research team could analyze these without weights.  Nonetheless, the team will create sampling weights for this aspect of the study.  These weights will have two purposes: (1) to reduce any bias due to nonresponse; and (2) to cause the participants who enter into HPOG during the specified time windows to resemble more closely the entire group of participants who are in the PRS, which includes individuals who enrolled in HPOG during the approximately one and a half years prior to the beginning of HPOG-Impact.  Both purposes will be served by use of a weighting technique called “raking.” With this technique, profiles of the HPOG participant population are built in terms of data from the PRS and then weights are created by a series of iterated adjustments that cause the sample to match PRS profiles on a weighted basis. The characteristics to be used for the profile will be those that appear to be related to nonresponse and/or late participation.  Once the weights are prepared, we will use the survey-sensitive procedures within the SAS system for all analyses to ensure that standard errors reflect the effects of weighting.

9. Procedures for HPOG-Impact

No survey weights will be used for HPOG-Impact.  For overall treatment impact estimation, the research team will use multivariate regression.  The team will include covariates to improve the power to detect impacts. The research team will pre-select the covariates to avoid steerage of findings.  The team will pool primary findings across sites and will prepare them in a matter appropriate for ITT (intention to treat) analysis, and may also prepare effects of treatment on the treated (TOT).  In general, analyses will use everyone who gives informed consent during the randomization period for HPOG-Impact. Nonresponse will not be an issue for analyses based on NDNH data.  Although analyses based on Participant Follow-Up survey data will have to deal with nonresponse, including covariates in the regressions will reduce the risk of nonresponse bias as effectively as preparing nonresponse-adjusted weights using those same covariates. 

In addition to the straightforward experimental impact analysis (or the two- and three-arm trials), the research team will use additional analytic methods to attempt to determine which program components are more effective.  The team plans to use innovative procedures that will exploit the fact that many of the grantees will be implementing a three-arm test.  While the main impact analysis will be experimental, innovative analytic methods—capitalizing on individual- and site-level variability—will exploit the experimental design to estimate the impact of specific program components.  As detailed in the project’s Evaluation Design Report, the research team will examine the extent to which varied methods can/do lead us to reach the same conclusion as the experimental analysis. This exercise has the potential to increase the confidence placed in non-experimental analyses in other areas too. All assumptions will be carefully spelled out and appropriate caveats will accompany findings in all reports. 

B.2.3 Degree of Accuracy Required

The research team considers the implication of the sample size for two selected impact estimates. The first focal impact, quarterly earnings, relies on data from the National Directory of New Hires (NDNH) and therefore the full sample of respondents, including those the team cannot successfully reach for the 15-month Participant Follow-Up survey. If the team can randomize 7,000 individuals in selected sites to HPOG services (either the basic HPOG program or the enhanced HPOG program) and 3,500 to the control group, then estimates suggest that the study will be able to detect an average impact of HPOG participation of $156 in quarterly earnings (see Exhibit B-2). In addition, the sample size will detect the following impacts on quarterly earnings when comparing quarterly earnings between HPOG participants receiving standard HPOG services and those receiving enhanced services, assuming the sample sizes in Exhibit B-3:²

• Contrast between HPOG-enhanced program/peer support with HPOG basic program: $301;

• Contrast between HPOG-enhanced program/emergency assistance with HPOG basic program: $674; and

• Contrast between HPOG-enhanced program/non-cash incentives with HPOG basic program: $434.

These sample sizes are sufficient to yield policy-relevant findings regarding the enhancements, especially given we expect to use both experimental results and natural variation in our analysis.  If 2,500 individuals can be randomized in selected sites to the HPOG program or the enhanced HPOG program with peer support, as planned, then our estimates suggest that the study will be able to detect an average impact of the program enhancement on quarterly earnings of $301 per person using data just from the three-arm tests alone.  If natural variation on these same enhancement components can also be added from other sites, the MDE of the enhancement will decrease from what we report here. To respond to the concern about these MDEs relative to findings from the ITA demonstration, the relative effects estimated in that demonstration are somewhat greater than $301 (D’Amico, Salzman & Decker, 2004). 

The experimental comparison between enhanced and standard HPOG participants will also contribute to bias reduction in the study’s larger analysis of natural variation in program components across sites.  Refinements to the natural variation impact model (from Bloom et al., 2003) that move the natural variation-based impact estimate of the effect of an enhancement feature closer to the experimental estimate of that effect have been shown to reduce the bias of all natural variation-based estimates (Bell, 2013) even when sample sizes for the experimental estimates are not larger enough to yield separate policy estimates.  Hence, data from all three randomized enhancements will contribute to policy findings on other program components, irrespective of their own sample sizes.

To put these estimated impacts on quarterly earnings into perspective, consider that the relative effects of various approaches to training estimated for the U.S. Department of Labor’s Individual Training Account (ITA) Demonstration are somewhat greater than $301 (McConnell et al., 2006).  For example, that evaluation found that providing intensive case management and educational counseling, relative to simply offering individuals a training voucher and the opportunity to choose a training program, produced earnings impacts of $328 per quarter during the first two quarters after randomization.  This is on par with the minimum detectable effects (MDEs)³ for the peer support program enhancement to be tested in this study. With the smaller sample sizes for the other enhancement tests, the research team can detect only larger MDEs.

ACF selected these particular enhancements to test with the possibility of being able to detect their effects and also to learn more about what components and features of career pathways programs (which by definition include a number of different components and features) are more influential. ACF will be able to use the information gained to understand if programs that do include a given component or use an implementation practice produce better participant outcomes and are “worth” the added cost of doing so. Connecting treatment explicitly to costs will allow for a better understanding of the implication for policy and practice—i. e., are the incremental effects of a given enhancement worth the cost of adding it to the standard program?

Also note that this is a study of an existing grant program where the legislation mandated that grantees use a career pathway approach and grantees were offered choices in how they put together different programmatic components and features and how they implement their programs. This study will allow ACF to understand if this investment in sector-based career pathways programs (which we currently have very little evidence about) is effective overall, and if specific components and features are more influential (and improve outcomes over and above the "standard" treatment). The research team has some ability to try to encourage grantees to include a specific enhancement and to try to increase sample in order to better detect effects; the team is using that ability to increase power as much as possible so that these important questions can be answered.

Exhibit B-2. Minimum Detectable Effects for Average Quarterly Earnings

Treatment Type	Average Quarterly Earnings
7,000 Standard HPOG Treatment group (20 HPOG-Impact grantees): 3,500 Control group (20 HPOG-Impact grantees)	$156
8,375 Standard HPOG Treatment group (23 HPOG grantees, including HPOG-Impact and ISIS) 4,875 Control group	$135
MDE for Enhanced HPOG Treatment
1,250 Enhanced HPOG Treatment group assigned to Peer Support: 1,250 Standard HPOG Treatment group (5 grantees)	$301
250  Enhanced HPOG Treatment group assigned to Emergency Assistance: 250 Standard HPOG Treatment group (3 grantees)	$674
600 Enhanced HPOG Treatment group assigned to Non-cash Incentives: 600 Standard HPOG Treatment group (5 grantees)	$434

Note: MDEs based on 80 percent power with a 5 percent significance level in a two-tailed test, assuming estimated in model where baseline variables explain 20 percent of the variance in the outcome (Nisar, Juras & Klerman, 2012). MDEs for earnings are based on standard deviations using data for adult women from the National JTPA study.⁴

Exhibit B-3 shows best estimates for the MDEs on credential receipt. The data source for this will be the 15-Month Participant Follow-Up survey and therefore sample sizes are 20 percent smaller than for quarterly earnings (assuming an 80 percent survey response rate). For this impact, if the treatment group sample, receiving either basic or enhanced HPOG services, includes 5,600 individuals and the control group sample includes 2,800, then estimates suggest that the study will be able to detect an average impact of HPOG participation of 2.7 percentage points in the earnings of credentials, from an assumed base of 30 percent. Alternatively expressed, the power will be adequate to detect a boost in the percentage of the population who earn credentials from 30.0 to 32.7 percent (which corresponds to a 9 percent effect size). In addition, the sample size will permit detecting the following impact of credential receipt between HPOG participants receiving standard HPOG services and those receiving enhanced services, assuming the sample sizes in Exhibit B-3, as follows:

• Contrast between HPOG-enhanced program/peer support with HPOG basis program: 5.1 percentage points;

• Contrast between HPOG-enhanced program/emergency assistance with HPOG basic program 11.5 percentage points, and

• Contrast between HPOG-enhanced program/non-cash incentives with HPOG basic program: 7.4 percentage points.

Exhibit B-3: Minimum Detectable Effects for Credential Receipt

	Impact on Credential Receipt (Percentage points)
5,600 Standard HPOG Treatment group (20 HPOG-Impact grantees): 2,800 Control group (20 HPOG-Impact grantees)	2.7
6,700 Standard HPOG Treatment group (23 HPOG grantees, including HPOG-Impact and ISIS) 3,900  Control group	2.3
MDE for Enhanced HPOG Treatment
1,000 Enhanced HPOG Treatment group assigned to Peer Support: 1,000 Standard HPOG Treatment group (5 grantees)	5.1
200 Enhanced HPOG Treatment group assigned to Emergency Assistance: 200 Standard HPOG Treatment group (3 grantees)	11.5
480 Enhanced HPOG Treatment group assigned to Non-cash Incentives: 480 Standard HPOG Treatment group (5 grantees)	7.4

Note: MDEs based on 80 percent power with a 5 percent significance level in a two-tailed test, assuming that baseline variables explain 20 percent of the variance in the outcome (Nisar, Juras & Klerman, 2012). MDEs for credential attainment are based on the standard deviations around the share of the control group (30%) with an educational degree or training credential using data from the NEWWS.

To put these estimated impacts on credentials into perspective, consider the effects of Job Corps estimated for the U.S. Department of Labor’s national study. That evaluation found that providing comprehensive and consistent services produced large effects on receiving a vocational certificate: 38 percent of the treatment group received a vocational certificate compared to 15 percent of the control group, an estimated impact of 23 percent (and more than twofold increase in the receipt of credentials) (Schochet et al., 2008). This is much greater than even the MDEs for the smallest program enhancement to be tested in this study, and so the research team is comfortable with the level of power in this study to detect relative increase on this outcome of interest.

B.2.4 Who Will Collect the Information and How It Will Be Done

10. Grantee, Management and Staff, Stakeholders and Employers

    1. HPOG-NIE

The research team will collect most of the data for HPOG-NIE from respondents at grantees and other organizations via web-based surveys. The exception is the Employer survey. All respondents from HPOG employers will be offered an option of responding to the survey on-line. However, it is expected that many surveys will be conducted by telephone, with the interviewer using the web-based survey instrument.

To help with the data collection, the evaluation team will ask each of the grantees to appoint one to three site liaisons to aid with various data collection activities. One of the liaisons’ primary tasks is to assist in assembling the sample frames for the Stakeholder/Network and Employer surveys. As soon as OMB clearance is obtained, the team will contact the site liaisons and ask them to use the sampling questionnaire to provide contact information (including email addresses) for staff from the grantee agency and staff connected to them (other partners, employers, and stakeholders). The team will also use individual telephone calls to provide grantee-specific guidance, as needed.

• Grantee survey: HPOG staff identified by the grantee liaison will complete the Grantee survey on-line. The grantee liaison is expected to identify the correct staff to provide information needed to respond to the different sections described above and will be responsible for coordinating the entry of responses into the Grantee survey. The liaison will also be responsible for reaching out, if necessary, to HPOG partners to gather or check on any needed pieces of information for the survey.

• Management and Staff survey: The survey will target intake workers, case managers and supervisors, and other line staff; managers; counselors; job placement staff; and other relevant staff. These Management and Staff survey respondents will be identified by the site liaison.

• Stakeholder/Network survey: The sample of respondents for the Stakeholder/Network survey will start with those partners and stakeholders identified by the site liaison in the sampling survey. Because the liaison may not be familiar with all stakeholders, the evaluation team will carry out a brief follow-up telephone survey of the initial list of stakeholders identified by the liaison. The interviewer will first provide the definition and examples of stakeholders, such as post-secondary training institutions serving HPOG participants, advocacy groups for low-wage workers, or healthcare employer associations. Then the interviewer will ask each respondent to identify any stakeholders that are not already listed. These will be added to the sample list for the Stakeholder/Network survey.

• Employer survey: The Employer survey will target two types of employers: (1) employers who are part of the partnership network and may have been involved in program design, development, and implementation of HPOG; and (2) employers not directly involved as partners but active in hiring HPOG graduates or who have been contacted by the program as potential employers of HPOG participants. The site liaison will identify these employers using the Sampling Questionnaire for the HPOG surveys. The liaison may reach out to other HPOG staff (for example, employment developers) for names or contact information of employers involved in hiring HPOG participants.

As stated above, the primary data collection will be hosted on the Internet and accessed via a live secure web-link. This approach is particularly well-suited to the needs of these surveys in that respondents can easily stop and start if they are interrupted or cannot complete the entire survey in one sitting and review and/or modify responses in a previous section. A complicating factor for the primary data collection is the fact that grantees have unique organization and staffing structures and operate in very different geographical contexts. Therefore, the data collection approach for HPOG-NIE must be tailored to each grantee and its community to ensure that the appropriate respondents are approached for each aspect of the data collection.

The data collection period will be approximately three months in length. Once the sample frame has been developed, the research team will send each respondent an email inviting him or her to log in and respond to the survey. To ensure that questions are answered in a timely manner and that accurate data are collected, the team will establish an in-house “survey support desk” with an email address and a toll-free telephone number to assist respondents with completing the survey. The support desk will carefully monitor response rates and data quality on an ongoing basis. It will also serve as a point of contact when respondents have questions. If concerns arise that are applicable to all respondents, the team will send emails to all appropriate respondents. The support desk will also be responsible for contacting non-respondents as the survey deadline approaches. In addition to providing a reminder, this contact also can be used as an opportunity for the respondent to complete the instrument over the phone, if desired. The phone number and email address of the support desk will be displayed on the survey website. The team will accommodate respondents who do not have computer access by completing the survey with them over the phone.

2. HPOG-Impact

For HPOG-Impact’s qualitative implementation study, the evaluation team will collect data on program operations on site. The site visits will be led by senior evaluation team members with expertise in the HPOG program and in implementation research. The senior team member will be accompanied by a more junior member.

The research team will conduct two rounds of site visits for the HPOG-Impact study. The first round will include all 20 HPOG-Impact grantees. The second round will include a subset of these grantees—those that are implementing program enhancements (peer mentoring, emergency assistance, non-cash incentives) alongside their standard HPOG programs. Approximately 13 grantees will participate in one of the three enhancements.

11. HPOG Participant Survey

    1. HPOG-Impact

After consent is obtained and random assignment is conducted, the evaluation staff will conduct follow up activities with treatment and control group participants:

• Send HPOG participants periodic contact information and tracking requests. These tracking forms (Appendix O) provide HPOG participants the opportunity to update their contact information and provide alternative contact information. Participants can send back the updated information in an enclosed self-addressed stamped envelope. Participants will be offered a $5 token of appreciation for updating their contact information. The research team will send tracking letters 4, 8, and 12 months following baseline.

• Conduct a 15-month Participant Follow-Up survey. The HPOG data collection team will contact study participants with an advance letter (see Appendix P) reminding them that they will soon receive a call from an HPOG interviewer who will want to interview them over the telephone. The letter will remind the sample member that their participation in the survey is voluntary and that they will receive a $30 token of appreciation upon completion of the interview. Centralized interviewers using computer-assisted interview (CATI) software will conduct the follow-up survey. Interviewers will be trained in the study protocols and their performance will be regularly monitored. The interviewers will first try to reach the sample member by calling the specified contact numbers. For sample members who cannot be reached at the original phone number, interviewers will attempt to locate new telephone numbers by calling secondary contacts and doing on-line directory searches. Once the centralized interviewers have exhausted all leads, cases will be transferred to field locators to find the sample member in person. When field staff succeeds in finding a sample member and convinces him or her to answer the survey, the field staff will establish contact with the centralized interviewer on a company cell phone. The centralized interviewer will then conduct the interview while the field interviewer waits nearby. With this approach, the team hopes to minimize mode effects and training requirements for field staff.

2. HPOG-NIE

For the HPOG participants from the four grantees who are not participating in HPOG-Impact or ISIS, the data collection approach will be parallel. However, the consent process will be different. Specifically, the research team will sample HPOG participants who have consented to be included in any federally funded research study of HPOG for this follow-up survey. This is the consent process implemented in September 2011 to cover all federally funded research on HPOG and for which HPOG participants have the discretion to withhold consent. Through January 2013, over 92 percent of participants have consented.

B.2.5 Procedures with Special Populations

All study materials designed for HPOG participants will be available in English and Spanish. Interviewers will be available to conduct the Participant Follow-Up survey interview in either language. Persons who speak neither English nor Spanish, deaf persons, and persons on extended overseas assignment or travel will be ineligible for follow-up, but we will collect information on reasons for ineligibility. Persons who are incarcerated or institutionalized will be eligible for follow-up only if the institution authorizes contact with the individual.

B.3 Methods to Maximize Response Rates and Deal with Non-response

B.3.1 Grantees, Management and Staff, Partners and Stakeholders, and Employers

The in-house “survey support desk” will carefully monitor response rates and data quality on an ongoing basis. The support desk will be responsible for contacting non-respondents if a survey has not been opened within the first two weeks, via an email reminder and/or telephone call, and then more frequently as the survey deadline approaches. In addition to providing a reminder, this contact also presents an opportunity for the respondent to complete the instrument over the phone, if desired. The survey website and hard-copy survey forms will display the phone number and email address of the support desk.

In addition, the research team will contact grantee site liaisons if respondents for the Grantee and Management and Staff survey have not started their surveys within the first six weeks of the data collection period and will ask them to follow up with these staff.

B.3.2 HPOG Participant Survey

All individuals who agree to participate in the evaluation must complete all baseline data collection in order to have the opportunity to be randomly assigned to the HPOG program. Therefore, a response rate of 100 percent is expected at baseline. For the 15-month follow-up, the following methods to maximize response will include:

• Participant tracking and locating;

• Tokens of appreciation; and

• Sample control during the data collection period.

12. Participant Tracking and Locating

The HPOG team will develop a comprehensive participant tracking system to maximize response. This multi-stage locating strategy blends active locating efforts (which involve direct participant contact) with passive locating efforts (which rely on various consumer database searches). At each point of contact with a participant (through tracking letters and at the end of the survey), the research team will collect updated name, address, telephone and email information. In addition, the team will also collect at baseline contact data for up to three people who do not live with the participant, but will likely know how to reach him or her. Interviewers only use secondary contact data if the primary contact information proves to be invalid—for example, if they encounter a disconnected telephone number or a returned letter marked as undeliverable. Appendix O shows a copy of the tracking letter. The research team proposes sending tracking letters at 4, 8, and 12 months after baseline data collection and will offer a $5 token of appreciation to individuals who return tracking information.

In addition to direct contact with participants, the research team will conduct several database searches to obtain additional contact information. Passive tracking resources are comparatively inexpensive and generally available, although some sources require special arrangements for access.

13. Tokens of Appreciation

Offering appropriate monetary gifts to study participants in appreciation for their time can help ensure a high response rate, which is necessary to ensure unbiased impact estimates. Study participants will be provided $30 after completing the 15-month follow-up survey. As noted above, in addition to the survey, at three time points between the baseline survey and 15-month follow-up survey (4, 8, and 12 months following baseline) the participants will receive a tracking letter with a contact update form that lists the contact information they had previously provided. The letter will ask them to update this contact information by calling a toll-free number or returning the contact update form in the enclosed postage-free business reply envelope. Study participants will receive $5 for updating their contact information, in appreciation for their time.

14. Sample Control during the Data Collection Period

During the data collection period, the research team will minimize non-response levels and the risk of non-response bias in the following ways:

• Using trained interviewers who are skilled at working with low-income adults and skilled in maintaining rapport with respondents, to minimize the number of break-offs and risk of non-response bias.

• Using a tracking letter and contact update form to keep the sample members engaged in the study and to enable the research team to locate them for the follow-up data collection activities. (See Appendix O for a copy of the tracking letter.)

• Using an advance letter that clearly conveys to study participants the purpose of the survey, the tokens of appreciation, and reassurances about privacy, so they will perceive that cooperating is worthwhile. (See Appendix P for a copy of the advance letter.)

• Providing a toll-free study hotline number to participants, which will be included in all communications to them, will allow them to ask questions about the survey, to update their contact information, and to indicate a preferred time to be called for the survey.

• Taking additional tracking and locating steps, as needed, when the research team does not find sample members at the phone numbers or addresses previously collected.

• Using an automated sample management system that will permit interactive sample management and electronic searches of historical tracking and locating data.

B.4 Tests of Procedures

In designing the follow-up survey, the research team included items used successfully in previous studies or in national surveys. Consequently, many of the survey questions have been thoroughly tested on large samples.

B.4.1 Grantee, Management and Staff, Stakeholder/Network and Employer Surveys

The Grantee, Stakeholder/Network, and Employer surveys were pretested with fewer than nine respondents from the grantees serving TANF recipients and other low-income individuals, The research team pre-tested the Grantee survey with three grantees. Grantees who completed the survey during the pre-test will be given their completed surveys to review and update when the full survey is fielded to reduce burden while ensuring all responses are accurate and up-to-date. The Management and Staff survey was pretested with staff from two non-HPOG career pathways programs to avoid duplication of effort. The team pretested the Stakeholder/Network survey with three organizations identified by grantees, but not all the partners in the grantee network. Stakeholders and Partners who completed the survey during the pretest will also be given their completed surveys to review and update when the full survey is fielded to reduce burden while ensuring all responses are accurate and up-to-date. The team pre-tested the Employer survey with employers who will not be contacted again to complete the survey.

Experienced interviewers called each respondent after they completed the survey to discuss their perceptions of the clarity and flow of survey items, ease of completion, and time requirements. After pretesting, we revised the instruments based on the feedback and trimmed, as needed, to stay within the proposed administration time. Changes made to the instruments are included in this revised clearance request for OMB to review.

B.4.2 15-Month Participant Follow-Up Survey

The research team pretested the follow-up survey instrument with six present and past participants from two grantees. None of these participants will be randomly assigned as part of the Impact study. Experienced interviewers conducted the pretest, and the team held a debriefing with them to discuss their perceptions of the clarity and flow of survey items, ease of completion, and time requirements. After pretesting, the team revised the questionnaire based on the feedback and trimmed, as needed, to stay within a 45-minute average administration time, including time to update contact information for possible future follow-up activities. Changes made to the instrument are included in this revised clearance request for OMB to review.

B.5 Individuals Consulted on Statistical Aspects of the Design

The individuals listed in Exhibit B-4 below made a contribution to the design of the evaluation.

Exhibit B-4: Individuals Consulted

Name	Role in Study
Dr. Maria Aristigueta	HPOG-ISO Technical Working Group member
Dr. Stephen Bell	Impact Study Principal Investigator
Ms. Maureen Conway	HPOG-ISO Technical Working Group member
Dr. David Fein	Key staff on ISIS evaluation
Dr. Olivia Golden	HPOG-ISO Technical Working Group member
Dr. Larry Hedges	Impact Study Technical Working Group member
Dr. Carolyn Heinrich	NIE and Impact Study Technical Working Group member
Dr. John Holahan	HPOG-ISO Technical Working Group member
Dr. Kevin Hollenbeck	HPOG-ISO Technical Working Group member
Dr. Chris Hulleman	HPOG-ISO Technical Working Group member
Mr. David Judkins	Key staff on NIE Impact Study Project Quality Advisor
Dr. Christine Kovner	HPOG-ISO Technical Working Group member
Dr. Robert Lerman	HPOG-ISO Technical Working Group member
Ms. Karin Martinson	Key staff on ISIS evaluation
Dr. Rob Olsen	Impact Study Team member
Dr. Laura R. Peck	Impact Study Lead Analyst
Dr. James Riccio	HPOG-ISO Technical Working Group member
Dr. Howard Rolston	Key staff on ISIS evaluation
Dr. Jeff Smith	Impact Study Technical Working Group member
Dr. Alan Werner	NIE Co-Principal Investigator Impact Study Project Director
Dr. Joshua Wiener	Implementation, Systems and Outcome Evaluation of the Health Profession Opportunity Grants to Serve TANF Recipients and Other Low-Income Individuals (HPOG-ISO) Technical Working Group member

Inquiries regarding the statistical aspects of the study’s planned analysis should be directed to:

Dr. Alan Werner Project Director, HPOG-Impact, and Principal Investigator, HPOG-NIE

Ms. Robin Koralek Project Director, HPOG-NIE

Dr. Stephen Bell Principal Investigator, HPOG-Impact

Dr. Molly Irwin Federal Contracting Officer’s Representative (COR), HPOG-Impact &

HPOG-NIE, Administration on Children and Families, HHS

References

McConnell, S., Stuart, E., Fortson, K., Decker, P., Perez-Johnson, I., Harris, B., & Salzman, J. 2006. Managing Customers’ Training Choices: Findings from the Individual Training Account Experiment. Washington, DC: Mathematica Policy Research.

Nisar, H., Klerman, J., & Juras, R. 2012. Estimation of Intra Class Correlation in Job Training Programs. Unpublished draft manuscript. Bethesda, MD: Abt Associates Inc.

Schochet, P. Z, Burghardt, J., & McConnell, S. 2008. Does Job Corps Work? Impact Findings from the National Job Corps Study. The American Economic Review 98(5), 1864–1886.

1 As per previously approved OMB data collections, consent to participate in ISIS and HPOG-Impact is not discretionary; only individuals agreeing to be included in those studies are allowed to move forward to random assignment.

2 Note that when examining the effect of enhancements we are looking at the effect of something additive (not at competing stand alone treatments).  Therefore, the enhanced version will need to be relatively more effective than the basic version to permit us to detect the estimated effect of that enhancement.

3 Minimum detectable effects are the smallest impacts that the experiment has a strong chance of detecting if such impacts are actually caused by HPOG.

4 The standard deviations for the female populations in the P/PV Sectoral Employment study and the Welfare-to-Work Voucher evaluation were higher and lower respectively; thus the data from the National JTPA study was around the average from the previous two studies noted above. We use the adult women subgroup from the JTPA study because it more closely aligns with HPOG’s target participants.