Supporting Statement for OMB Clearance Request
Part B
National and Tribal Evaluation of the 2nd Generation of the Health Profession Opportunity Grants
0970-0462
October 2016
Revised April 2017
Submitted by:
Office of Planning,
Research & Evaluation
Administration for Children & Families
U.S. Department of
Health
and Human Services
Federal Project Officers:
Hilary Forster
Nicole Constance
Amelia Popham
Table of Contents
B.1 Respondent Universe and Sampling Methods 1
B.2 Procedures for Collection of Information 5
B.3 Methods to Maximize Response Rates and Deal with Non-response 10
B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 15
Attachments:
Previously Approved Instruments
Instrument 1: PAGES Grantee- and Participant-Level Data Items List
New Instruments Included in this Request
Instrument 2: HPOG 2.0 National Evaluation Screening Interview
Instrument 3: HPOG 2.0 National Evaluation first-round telephone interview Protocol
Instrument 4: HPOG 2.0 National Evaluation in-person implementation interviews
Instrument 4A HPOG 2.0 National Evaluation In-Person Implementation Interview
Instrument 4B HPOG 2.0 National Evaluation In-Person Implementation Interviews Basic Skills Training
Instrument 4C HPOG 2.0 National Evaluation In-Person Implementation Interviews Career Pathways
Instrument 4D HPOG 2.0 National Evaluation In-Person Implementation Interviews Work-Readiness
Instrument 4E HPOG 2.0 National Evaluation In-Person Implementation Interviews Sustainability
Instrument 5: HPOG 2.0 National Evaluation welcome packet and participant contact update forms
Instrument 5a: HPOG 2.0 National Evaluation welcome packet and contact update form
Instrument 5b: HPOG 2.0 National Evaluation participant contact update letter and form
Instrument 6: HPOG 2.0 Tribal Evaluation grantee and partner administrative staff interviews
Instrument 7: HPOG 2.0 Tribal Evaluation program implementation staff interviews
Instrument 8: HPOG 2.0 Tribal Evaluation employer interviews
Instrument 9: HPOG 2.0 Tribal Evaluation program participant focus groups
Instrument 10: HPOG 2.0 Tribal Evaluation program participant completer interviews
Instrument 11: HPOG 2.0 Tribal Evaluation program participant non-completer interviews
Attachment A: References
Attachment B: Informed Consent Forms
Attachment B2: Tribal Evaluation informed consent form A (SSNs)
Attachment B3: Tribal Evaluation informed consent form B (Unique identifiers)
Attachment C: 60 Day Federal Register Notice
Attachment D: Sources and Justification for PAGES Grantee- and Participant-Level Data Items
Attachment E: Final Updated Attachment E PPR Data List and Mockup
Attachment F: First Round of HPOG Grantees Research Portfolio
Attachment G: Participant Contact Information Update Letter and Form
Attachment H: HPOG Logic Model
Attachment I: Focus group participant consent form
Attachment J: Interview Verbal Informed Consent Form
Part B of the Supporting Statement for The Health Profession Opportunity Grants 2.0 (HPOG 2.0) National and Tribal Evaluation —sponsored by the Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services (HHS)—considers the issues pertaining to Collections of Information Employing Statistical Methods. Abt Associates (Abt) is the prime contractor for the study. Abt and its partners MEF Associates, the Urban Institute and Insight Policy Research are conducting the HPOG 2.0 National Evaluation and partner NORC is conducting the Tribal Evaluation. Abt and partners the Urban Institute and Green Beacon are responsible for the Participant Accomplishment and Grantee Evaluation System (PAGES) design and implementation. PAGES is a data collection system for collecting information from all HPOG grantees on their program designs and offerings, intake information on eligible applicants (both treatment and control group members) through baseline data collection, and a record of participants’ activities and outcomes.
The federal evaluations of the HPOG 2.0 National and Tribal grantees will evaluate postsecondary career pathway programs, focused on the healthcare sector, that target Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals. This submission seeks clearance for the HPOG 2.0 National Evaluation descriptive evaluation protocols, the HPOG 2.0 National Evaluation impact evaluation participant contact update form, adding a non-monetary token of appreciation to the welcome packet for the impact evaluation, and the HPOG 2.0 Tribal Evaluation data collection protocols beyond the previously approved information collection instruments (Instruments 2-11).
Thirty-two HPOG grants were awarded to government agencies, community-based organizations, post-secondary educational institutions, and tribal-affiliated organizations in September 2015. Of these, 27 were awarded to non-tribal entities and five were awarded to tribal organizations.
All 32 grantees will participate in this federally sponsored evaluation. There is no statistical sampling required for the HPOG 2.0 National Evaluation descriptive evaluation or the HPOG 2.0 Tribal Evaluation. Evaluators will work closely with grantees to identify participants in the respective studies. Under the National Evaluation impact evaluation, the evaluators will select up to 15,000 study participants beginning with the cohort of participants enrolled in March 2017 for inclusion in the follow-up survey sample. Pending OMB approval, the evaluators will send welcome packets to all study participants (including treatment and control group members) selected for survey data collection, beginning in April 2017 (for those participants enrolled in the March 2017 cohort). The evaluators waited to begin survey sample selection until the March 2017 cohort in order to maximize efficiency for the survey data collection effort and allow all programs time to complete start-up activities and reach steady-state operations. Allowing time for all programs to mature helps to alleviate some of the challenges typically associated with early enrollment cohorts on random assignment studies, such as very small monthly enrollment cohorts, or grantees modifying eligibility criteria or intake processes. Compressing the length of the field period was the most efficient way to ensure that evaluators could meet the survey sample size requirements within the available resources. The evaluators will rely on baseline equivalency testing to ensure that there are no significant differences in participant characteristics between those enrolled prior to March 2017 and those enrolled after.
Study participants will receive contact update requests every three months. As the first follow-up survey gets closer, the HPOG 2.0 National Evaluation team will develop a sampling plan. That plan, along with the survey instrument for the 15-month follow-up survey, will be submitted in a subsequent information collection request package.
All five tribal grantees will participate in the federally sponsored HPOG 2.0 Tribal Evaluation. For the HPOG 2.0 Tribal Evaluation, there are two major respondent universes: (1) Tribal HPOG 2.0 grantees, partners, and employers; and (2) Tribal HPOG participants, including program completers and non-completers. Exhibit B.1.1 presents the sampling methods and target response rates for each of the HPOG 2.0 National and Tribal Evaluation respondent subgroups.
Exhibit B1.1. HPOG 2.0 National and Tribal Evaluation Respondents
Respondent Universe |
Respondent Subgroup |
Sampling Methods and Target Response Rates1 |
Data Collection Strategies |
National HPOG 2.0 Evaluation |
|
||
Grantees, partners, and employers |
Grantees |
Evaluation team members review the topics of interest with grantees using the HPOG 2.0 Screening Interview to identify appropriate respondent(s) based on who is most knowledgeable about the topics of interest. (See Instrument 2). Grantees have agreed to participate in the evaluation as a condition of receiving HPOG grant funding. Therefore, the team expects a 100 percent response rate. |
Semi-structured telephone interviews |
|
Managers and staff |
A very high response rate (at least 80 percent) is expected among grantee managers and staff. |
Semi-structured in-person interviews |
|
Partners |
A very high response rate (at least 80 percent) is expected among grantee partners. |
Semi-structured in-person interviews |
|
Employers |
A very high response rate (at least 80 percent) is expected among employers. |
Semi-structured in-person interviews |
Impact evaluation participants selected for follow-up survey sample |
A sample of participants (up to 15,000) beginning with those enrolled in March 2017 |
Up to 15,000 study participants, beginning with those enrolled in March 2017 will be part of the participant contact update efforts. The team expects that 35 percent of the respondents will respond to each quarterly participant contact update effort.2 |
Contact updates by mail, online portal, or telephone |
Tribal HPOG 2.0 Evaluation |
|||
Grantees, partners, and employers |
Grantees |
Grantees have agreed to participate in the evaluation as a condition of receiving HPOG grant funding. Therefore, the team expects a 100 percent response rate. |
Semi-structured in-person interviews |
|
Management and Staff |
A very high response rate (at least 80 percent) is expected among grantee staff. |
Semi-structured in-person interviews |
|
Partners |
Partners have agreed to participate in the evaluation as a condition of receiving HPOG grant funding. Therefore, the team expects a 100 percent response rate. |
Semi-structured in-person interviews |
|
Employers |
A very high response rate (at least 80 percent) is expected among HPOG employers. |
Semi-structured in-person interviews |
Participants |
Program participants (current) |
The tribal evaluation team will work with the grantees to recruit participants during the annual site visit planning period. The team expects a 25-50 percent response rate from current program participants. |
In-person focus groups
|
|
Program completers |
The tribal evaluation team will work with the grantees to recruit participants during the annual site visit planning period. The team expects a 25-50 percent response rate from program completers. |
Semi-structured in-person interviews
|
|
Program non-completers |
The tribal evaluation team will work with the grantees to recruit participants during the annual site visit planning period. The team expects a 10-25 percent response rate from program non-completers. |
Semi-structured in-person interviews
|
HPOG National and Tribal Evaluation Participant Accomplishment and Grantee Evaluation System (PAGES) |
|||
Participants |
National Evaluation (Non-Tribal) HPOG Participants |
No sampling techniques will be employed for PAGES data collection. A 100 percent response rate is expected. |
Baseline and ongoing participant level data |
|
Tribal HPOG Participants |
No sampling techniques will be employed for PAGES data collection. A 100 percent response rate is expected. |
Baseline and ongoing participant level data |
PAGES includes the applicant population of the anticipated 32 organizations that received HPOG funding. As discussed, the system provides data at the grantee- and individual- level. Thus, data is collected and will continue to be collected from the 32 grantees on their program designs and offerings, from all eligible applicants on their baseline characteristics, and from all of the individuals the grantees serve on their individual participation and outcomes.
Approximately 31,800 individuals are expected to complete the baseline data collection across the 32 grantees during the HPOG 2.0 grant period. The evaluation team expects the impact evaluation sample to include up to 29,100 individuals who apply to participate in the HPOG programs operated by 27 non-Tribal HPOG 2.0 grantees participating in the impact evaluation (9,700 control and 19,400 treatment). These projected enrollment numbers suggest an additional 4,860 National Evaluation participants will complete the PAGES baseline intake form than was originally estimated. Approximately 1,500 participants from the first round of HPOG grants are expected to receive additional services under HPOG 2.0. Thus, the total National Evaluation sample is estimated at 30,600 participants. Further, it is anticipated that up to 1,200 individuals will apply to participate in the HPOG programs operated by five HPOG 2.0 Tribal grantees. Among the total sample, the grantees will continue to collect ongoing participant-level data on up to 22,100 HPOG enrollees. No sampling techniques will be employed for PAGES data collection.
This section describes the data collection procedures for the HPOG 2.0 National Evaluation descriptive evaluation, then the impact evaluation contact updates. The section concludes with a description of the HPOG 2.0 Tribal Evaluation data collection procedures.
HPOG 2.0 National Evaluation Descriptive Evaluation Data Collection Procedures
The sample frame for the HPOG 2.0 National Evaluation descriptive evaluation includes all 27 non-tribal grantees. No statistical methods will be used for stratification and sample selection. The descriptive evaluation exclusively uses purposive sampling since it is a descriptive study. The primary data collection approach for the descriptive evaluation will be two rounds of semi-structured interviews conducted by telephone and up to two sets of in-person site visits with program directors, case managers and other relevant grantee staff. The first round of telephone interviews will focus on early implementation efforts. The second round will update the earlier round and will collect information to help lay the groundwork for the systems and cost studies. Site visits will be to programs implementing promising approaches to program components of specific interest to ACF. Telephone and site visit data collection will be supplemented with data from PAGES and other existing site-specific materials developed earlier by the national evaluation team.
Grantee Telephone Interviews
The national evaluation team will conduct telephone interviews to gain insight from grantee program directors, case managers and other relevant staff on program administration, outreach and recruitment; eligibility and intake; basic skills training; healthcare occupational training; academic and career counseling; career pathway training opportunities; work-readiness training; academic supports; personal support services; employer assistance and work-based learning opportunities; employer engagement; control group services; and sustainability (Instrument 3). Prior to conducting these interviews, the national evaluation team will conduct a planning call with each grantee to review the purpose of the interviews, provide an overview of the topics to be covered, and ask the grantee to complete a spreadsheet indicating who would be the best person to respond to the specific topics of interest (Instrument 2). The team will use this sheet to schedule interviews with grantee staff as appropriate. There will be a second round of implementation interviews later in the evaluation period. The content of those interviews will be included in a subsequent information collection request.
Site Visits
Site visits will take place at five to ten grantees participating in the National Evaluation during the first round of the descriptive evaluation. The research team will rely on data from the first round of descriptive evaluation telephone interviews and other extant data, (such as from PAGES and other existing site-specific materials developed earlier by the national evaluation team) to identify up to ten grantees that are implementing programs with innovative approaches to program components of interest to ACF. The national evaluation team will conduct two-person site visits to conduct this phase of the data collection (Instrument 4).
The national evaluation team has developed semi-structured interview protocols for the collection of first round implementation telephone interviews and evaluation site visits (Instruments 3 and 4). All protocols begin with a brief introductory script that summarizes the overall evaluation, the focus of each interview, how respondent privacy will be protected, and how data will be aggregated.
National Evaluation Impact Evaluation Participant Contact Update Request Procedures
The participant contact update efforts for the impact evaluation begin with the study participants who are randomly assigned to the study beginning in March 2017. All study participants enrolled during this timeframe will be included in this effort. The participant contact update form will be self-administered (Instrument 5b). The form will be mailed to sample members quarterly, beginning three months after random assignment. Participants will be encouraged to respond by returning the form by mail, through a secure online portal, or they can update their contact information by phone. Participants can indicate that the information is correct or they can make any necessary changes in contact information.
National Evaluation Impact Evaluation Token of Appreciation
In an effort to strengthen the participant’s engagement, the study team will send each study participant a welcome packet the month after enrollment. (See Instrument 5a for the welcome packet and Instrument 5b for the contact update form; Attachment G is obsolete and has been replaced by Instrument 5b.) We propose including a non-monetary incentive, such as a portable cell phone charger or comparably valued item, to all participants as part of the welcome packet. This item will be branded with the HPOG study logo. This token of appreciation is a tool to remind the participant about the study (rather than the program). The chargers are a practical item that can be used in a computer’s USB port or in a charging block for a cell phone or other portable device. The fact that the charger is practical will encourage participants to use it, and thus to see a regular reminder of their participation in the HPOG study. The team anticipates that this will be particularly important for the control group since most members of the treatment group will have many reminders of HPOG as a result of the training they will be receiving. For more rationale on the addition of this token, please see A.9 of Part A of the Supporting Statement.
Tribal Evaluation Implementation Study Data Collection Procedures
The sample frame for the HPOG 2.0 Tribal Evaluation includes all five tribal grantees. No statistical methods will be used for stratification and sample selection. The HPOG 2.0 Tribal Evaluation exclusively uses purposive sampling since it is a descriptive study. The tribal evaluation team will use multiple sources of data for the process and outcome evaluation, primarily centered around annual site visits to Tribal HPOG grantees: semi-structured in-person interviews with grantee and partner administrative staff, program implementation staff, and local employers; focus groups and follow-up interviews with program participants, including program completers and non-completers; and program operations data collected through PAGES.
Site Visits
Site visits will take place at each of the five tribal grantees and their program sites on an annual basis. The tribal evaluation team will conduct four in-person site visits at each grantee in Years 2-5 of the evaluation. The tribal evaluation team will discuss logistics for the site visit (e.g., scheduling, travel, where to host focus groups and interviews) with each grantee. In addition, we will work closely with each grantee to recruit respondents.
The evaluation team has developed semi-structured interview and focus group protocols for the collection of qualitative data during the initial evaluation site visits and follow-up site visits (Instruments 6-11). All protocols will begin with a brief introductory script that summarizes the overall evaluation, the focus of each interview, how respondent privacy will be protected, and how data will be aggregated. The evaluation team will obtain written informed consent in-person prior to participant focus groups (Attachment I) and verbal informed consent from interview participants (Attachment J). The senior member of the evaluation team will be responsible for seeking consent from participants. The tribal evaluation team will collect data from the interviews and focus groups via extensive detailed notes.
Grantees, partners, and employers
The evaluation team will conduct in-person interviews on an annual basis to gain insight from grantee and partner administrative staff on high-level program strategies, program development, and lessons learned (Instrument 6). Partners may include public and private health care employers, education and training organizations, community-based organizations, labor organizations, and national, state or local foundations that provide assistance to American Indians and Alaska Natives (AI/ANs). The team anticipates that interviews will focus on the overall strategic approach of the program as well as processes used to develop the program curricula; any evidence behind the structure of the program and special considerations and modifications for tribal populations; and on program modifications, overall challenges and successes, as well as lessons learned which, at a high level, will inform the evaluation team’s assessment of the success of the program as a whole. The team anticipates that the total number of grantee and partner administrative staff will be limited (3-7) for each grantee program (total of 15-35 across all Tribal HPOG grantees annually).
The evaluation team will conduct in-person interviews with staff responsible for coordinating and implementing the program at each site (Instrument 7). These individuals may include, but are not limited to, program instructors, recruitment and orientation staff, and providers of supportive services. These interviews will focus on program processes including recruitment, orientation and program implementation and on program modifications, overall challenges and successes in implementation, as well as lessons learned which, at a high level, will inform the evaluation team’s assessment of the success of the program as a whole. Depending on the structure of the program, multiple staff may be interviewed at each implementation site. The evaluation team anticipates that the number of implementation staff will vary by grantee program and expects to interview at least 3-10 staff per grantee (for a total of 15-50 across all Tribal HPOG grantees annually).
The tribal evaluation team will work with grantee sites to identify potential employers in the region and conduct interviews with appropriate staff at these facilities. These interviews will be used to assess employers’ general impressions of program graduates, their degree of awareness of the program as a whole, and their views on the extent to which programs are making an impact on the creation of a workforce that is equipped to meet the demands of the current health care needs of AI/AN communities (Instrument 8). The evaluation team plans to target at least 3-6 employers per grantee (for a total of 15-30 across all Tribal HPOG grantees annually); however, this total may vary based on the number of employers located and identified by grantees.
Participants
The evaluation team will conduct focus groups with program participants during the annual site visits (Instrument 9). The focus groups will gather program participants’ perceptions around the following key evaluation topics: program design and curriculum; supportive services; recruitment and orientation; quality of instruction; participant educational attainment; and satisfaction with the HPOG program. To prepare for the focus groups, the team will discuss with each grantee the best and most culturally appropriate recruitment techniques as well as whether or not they would prefer evaluation team staff to contact potential participants or if they would prefer to contact participants themselves. If the grantee provides a list of student participants, the tribal evaluation team will reach out to the potential participants using recruitment letters and follow-up phone calls if necessary. Should the grantee prefer to have potential participants contacted by program staff, the team will provide recruitment materials to facilitate outreach activities. In addition, the team will further consult with each grantee about how best to conduct the focus group in a culturally competent manner that stimulates discussion and full participation (e.g., allowing opportunities for self-reflection and privacy in composing responses to questions). The team anticipates that there will be 5-9 participants in each focus group and between 1-3 focus groups per grantee, depending on the number of locations where students are enrolled at each grantee (total of 25-135 focus group participants across all Tribal HPOG grantees annually). Once the evaluation team has obtained consent, the facilitator (a member of the tribal evaluation team) will introduce participants to the overall purpose and structure of the gathering. In addition, the facilitator will re-emphasize to the assembled group that their comments will be aggregated in the site visit summaries and reports and not directly attributable to them. The tribal evaluation team will collect data from the focus groups via extensive detailed notes. First names of focus group participants will be collected upon registration and will be used for note-taking purposes.
In order to obtain information on key program outcomes related to educational attainment and employment, the evaluation team will conduct in-person interviews with participants who have completed a training program (Instrument 10). The purpose of these interviews is to assess the current employment status of the participants who have completed training programs and to capture their voice and perspectives on whether the program adequately prepared them for employment in the health sector and, if applicable, to serve AI/AN communities. The team anticipates that some participants in the annual focus groups will also participate in the interviews; others may have had no previous contact with the tribal evaluation team. The tribal evaluation team will work with the grantees to recruit participants during the annual site visit planning period and conduct interviews in-person during the site visit. The team anticipates conducting interviews with approximately 20 students per grantee annually (total of 100 across all Tribal HPOG grantees annually).
In order to understand factors that led to non-completion, the evaluation team will conduct in-person interviews with participants who did not complete a training program through HPOG (Instrument 11). The tribal evaluation team will work with the grantees to recruit participants during the annual site visit planning period and conduct interviews in-person during the site visit. The team anticipates conducting interviews with approximately 10 students per grantee annually (total of 50 across all Tribal HPOG grantees annually). Topics to be addressed in the interview include: reasons for leaving the program; challenges experienced; elements of the program that were effective or non-effective; identification of any short-term outcomes resulting from program participation; how the program could be improved; and whether the non-completer plans to re-enroll in the program or pursue an alternative course of study.
Program Operations data via PAGES
Tribal grantee staff will administer informed consent and collect participant data at intake and throughout a participant’s enrollment via PAGES. Data that will be collected through PAGES includes participant demographics at intake; educational enrollment and completion; number and type of supportive services received; employment status at intake and after training; and changes to participant wages. The tribal evaluation team will be able to view and analyze aggregate data but are not able to see any personally identifiable information.
HPOG Program Performance Report Based On Grantee-Level and Ongoing Participant-Level Data
During the initial 120 day planning period after grant award, all grantees were required to provide information on their program components and offerings to be included into a previously approved internet-based data system (PAGES). All HPOG grantees update the grantee-level and ongoing participant-level data at least semi-annually, but may enter data at any time either manually or by uploading data from existing data systems into an internet-based data system (PAGES) constructed for the purpose of collecting data for the HPOG 2.0 programs. PAGES is managed by Abt Associates.
Participant-Level Baseline Data Collection
Grantee staff administer a previously approved informed consent form relating to PAGES data items, administrative data, and follow-up surveys, when individuals who apply to the program are found eligible (Instrument 1). All HPOG 2.0 grantees must obtain informed consent from applicants prior to collecting personally identifying data.
All non-Tribal HPOG 2.0 grantees are required to participate in the impact evaluation, which includes randomly assigning eligible HPOG applicants to either be invited to receive HPOG services or to serve in the control group. However, prior HPOG and Pathways for Advancing Careers and Education (PACE) project participants who were randomly assigned in the first impact studies will not be subject to random assignment again and will be allowed to enroll in the program3. These applicants complete the previously approved non-random assignment informed consent form (see Attachment B) that asks permission for researchers to access data individuals provide at intake, information about the training and services they receive after enrollment, and administrative data, but not the follow-up surveys. After the consent process, grantee staff administer PAGES baseline questions to all applicants.
During the intake processes for individuals applying to grantees in the study who are subject to random assignment, staff first describe the impact evaluation and administer the previously approved random assignment informed consent form (see Attachment B). All eligible applicants for HPOG during the intake period for the study must sign the paper informed consent form to be part of the study. The informed consent form allows researchers to: (1) access baseline data participants provide at intake and information about the training and services treatment group members receive after enrollment; (2) contact both treatment and control group members for updated contact information and additional follow-up questions for the impact evaluation; and (3) gather administrative data from the National Directory of New Hires (NDNH) and possibly other sources, such as the National Student Clearinghouse (NSC). After the consent process, grantee staff administer PAGES baseline questions, including the baseline questions on individuals’ expectations for education and employment, barriers to employment, work preferences, and self-efficacy. After individuals complete the informed consent form and the required baseline questions, a secure, web-based software program randomly assigns them into either the treatment or control group.
These requests for contact information updates (Attachment G) provide HPOG 2.0 participants the opportunity to update their contact information and provide alternative contact information. Participants can send back the updated information in an enclosed self-addressed stamped envelope, through a secure online portal, or by telephone. Researchers will send participants $5 as a gesture of appreciation to respondents for providing updated contact information. The research team will send contact update letters about every three months after random assignment (Instrument 5b), pending OMB approval.
The HPOG 2.0 Tribal grantees are participating in a federal evaluation that will not require random assignment. Applicants at the Tribal grantees complete the previously approved non-random assignment informed consent form (see Attachment B) that asks permission for researchers to access data individuals provide at intake, information about the training and services they receive after enrollment, administrative data, and follow-up surveys. After the consent process, grantee staff administer PAGES baseline questions, excluding the baseline questions on individuals’ expectations for education and employment, barriers to employment, work preferences, and self-efficacy.
To ensure participants can understand each of the documents, the National Evaluation welcome packet and participant contact update forms, as well as the previously approved informed consent forms and PAGES data elements were designed at an 8th-grade readability level. The HPOG 2.0 team will provide a Spanish version of the informed consent forms and will work with grantees on ways staff can assist where translation of other data collection instruments may be needed. To ensure the instruments are culturally responsive, the Tribal Evaluation protocols were reviewed by the Tribal HPOG 2.0 grantees and consultants with expertise conducting research in tribal communities.
This section describes the methods to maximize response rates for the HPOG 2.0 National Evaluation descriptive evaluation, the contact-update request for the impact evaluation, and the HPOG 2.0 Tribal Evaluation.
HPOG 2.0 National Evaluation Descriptive Evaluation
The HPOG 2.0 National Evaluation data collection efforts are heavily dependent on gaining cooperation, buy-in, and collaboration with grantees. For this data collection effort, the evaluation team believes that the specialized nature of the respondent groups – grantees and partners, and in some cases implementation staff and employers - that received funding from ACF and who are interested in supporting HHS efforts to design and implement health profession training programs will increase their propensity to respond, resulting in a response rate of at least 80 percent. It is important to note that the HPOG grantees are aware of the federal evaluation and have agreed to participate in it as part of their cooperative agreement with ACF.
Contact Update Request for the HPOG 2.0 National Evaluation Impact Evaluation
The HPOG 2.0 National Evaluation impact evaluation anticipates two rounds of follow-up survey data collection. The evaluation team developed a comprehensive participant contact update system in order to maximize response to the two follow-up surveys. This multi-stage locating strategy blends active locating efforts (which involve direct participant contact) with passive locating efforts (which rely on various consumer database searches). The survey data collection instruments and plans to maximize response to the surveys and minimize non-response bias will be submitted under a separate request for clearance. This request covers the participant contact update efforts only.
All impact evaluation participants will be included in the administrative data collection. However, in order to maximize efficiency for the survey data collection effort and allow all programs time to mature, the evaluators will select up to 15,000 study participants beginning with those enrolled in March 2017, for inclusion in the follow-up survey sample. The plans to maintain updated participant contact information for the HPOG 2.0 National Evaluation impact evaluation begin with a welcome packet, sent to all sample members within the first month of enrollment and quarterly contact updates thereafter. Pending OMB approval, the evaluation team will begin sending the welcome packets in April 2017 for those participants enrolled in March 2017. The welcome letter and study brochure provide comprehensive information about the study requirements, the contact update efforts, and survey data collection activities.
The contact update form will capture updates to the respondent’s name, address, telephone and email information, and preferred method of contact. It will also collect new or updated contact data for up to three people who do not live with the participant, but who will likely know how to reach him or her. Contact update forms will be sent to study participants every three months between random assignment and the second follow-up survey. Interviewers will only use secondary contact data if the primary contact information proves to be invalid (e.g., they encounter a disconnected telephone number or a returned letter marked undeliverable). Instrument 5b shows a copy of the contact update form.
To boost the response rate to the contact update requests, researchers will offer three ways for participants to provide updated data. Participants can return the contact update form by mail, they can respond online, or they can call in and provide their updated information. Offering multiple options to respond helps to ensure that participants can respond in the mode most convenient for them.
In addition, the study team wants to offer a token of appreciation valued at $5 for each address update. The token of appreciation is a way to thank the participant for returning the form and for remaining engaged in the study. Participants will receive the token of appreciation after they provide updated contact information. Incentives at one or more phases of data collection have been used successfully, on a number of similar federally sponsored surveys such as PACE (OMB control number 0970-03970) and the HPOG 1.0 impact study (OMB control number 0970-0394.) The planned incentive amount is comparable to what was offered for the participant contact update efforts leading up to the 15-month follow-up survey efforts for both of those studies.
As noted in section A.9 in Part A of the Supporting Statement, the research team is unaware of any prior research on the utility of incentives for obtaining more address updates much less any indirect effects on long-term response rates or nonresponse bias in long-term follow-up data, but nonetheless feels that they are an important feature for the eventual acceptance of the entire study as providing strong evidence of the effects of HPOG 2.0.
ACF will provide information about proposed tokens of appreciation for the follow-up survey data collection efforts as well as plans to reduce non-response bias in a subsequent information collection request for this project.
HPOG 2.0 Tribal Evaluation
The HPOG 2.0 Tribal Evaluation data collection efforts are heavily dependent on gaining cooperation, buy-in, and collaboration with the tribal grantees. For this data collection effort, the evaluation team believes that the specialized nature of the respondent groups – grantees and partners, and in some cases implementation staff and employers - that received funding from ACF and who are interested in supporting HHS efforts to design and implement health professions training programs will increase their propensity to respond, resulting in a response rate of at least 80 percent. It is important to note that the Tribal HPOG grantees are aware of the federal evaluation and have agreed to participate in it as part of their cooperative agreement with ACF.
In the case of the participant focus groups and interviews, the tribal evaluation team will work with the Tribal HPOG grantees to recruit these individuals. Given that the participants are likely involved with the Tribal HPOG program or are receiving its services, it will be effective to collaborate with the Tribal HPOG grantees to secure the participation of these individuals.
In addition, the tribal evaluation team will use a number of proven methods to maximize participation and cooperation in the study:
Recruitment of focus group participants: One potential challenge with grantee staff recruiting focus group participants is that it may result in a disproportionate number of HPOG participants who are more engaged with the program. However, during the Tribal HPOG 1.0 evaluation, the evaluation team found that this recruitment method provided increased focus group participation and a diverse range of perspectives.
Recruitment of interview participants: Based on experience with the Tribal HPOG 1.0 evaluation and per recommendations from grantees and technical workgroup members, the evaluation team plans to recruit program completers and program non-completers to participate in in-person interviews during the annual site visit.
Administration of data collection: Focus groups and interviews conducted during the site visits will be scheduled at the respondents’ convenience. Focus groups will be conducted during the lunch hour or in the evening and light refreshments will be provided.
Assurances of privacy: Respondents will be assured that reported data are aggregated and not attributable to individuals or organizational entities.
Tokens of Appreciation: A token of appreciation valued at $50 will be provided to program participants for their participation in an in-person focus group (90 minutes) or in-person completer or non-completer interview (60 minutes). Each grantee will be consulted to determine the most appropriate method of honorarium (e.g., gift certificate to a local grocery store). The evaluation team believes this will maximize response rates based on experience from the HPOG 1.0 Tribal Evaluation and minimize non-response bias. Based on experience from the HPOG 1.0 Tribal Evaluation, we learned that there is the potential for non-response bias due to circumstances experienced by tribal HPOG participants, including substantial family commitments, longer travel time to the interview or focus group location, expenses associated with research participation (e.g. transportation, childcare), or reluctance to participate in research. Participants experiencing these circumstances would be underrepresented in data collection activities, thereby resulting in non-response bias. Therefore, offering an incentive to participate in the study will address these circumstances and offset the potential of non-response bias.
PAGES
The evaluation team expects to obtain a 100 percent response rate from the HPOG 2.0 grantees in collecting both the grantee- and participant-level data in PAGES. However, it is expected that some grantees may have more technical challenges than others in using PAGES, depending on the staff who will be entering data. Thus, Abt Associates and the Urban Institute have provided and will continue to provide grantees training and technical assistance on PAGES to assist grantees in complying with the data requirements.
PAGES has several features for addressing missing data. First, on each form, certain key fields are required, forcing the user to enter the data prior to saving the form. However, it is not always possible to make a field required, since users may not have all information available at the same time. Thus, PAGES issues alerts when those data are not complete, but still allow users to save. For fields that are associated with warnings, PAGES maintains a queue of missing data available from each user’s dashboard, allowing users to easily see which data are missing and quickly add that information once received. Finally, there are data quality reports, for Urban Institute and evaluation site team staff to monitor for missing data at the grantee level.
In addition to clearly labeling the data fields and providing definitions and instructions within the system, a separate PAGES Guidance Manual for grantees was developed to provide a detailed explanation of each data element, definitions, and how to enter the data. Documents and recorded webinars are and will continue to be available to grantees electronically so users can readily access them whenever needed.
Additionally, Urban Institute staff are available to provide technical assistance to grantees via a toll-free telephone number. Grantees can also submit help requests through PAGES ‘ticket’ system.
As for baseline data items, all eligible individuals who apply to the HPOG 2.0 programs must complete the informed consent forms and all required baseline data collection in order to have the opportunity to enroll in HPOG programs. Therefore, a response rate of 100 percent is expected for required baseline elements.
HPOG 2.0 National Evaluation Descriptive Evaluation
The data collection instruments for the HPOG 2.0 National Evaluation descriptive evaluation (Instruments 2-4) were developed and reviewed by Federal staff and evaluation team members. Many questions were either taken from or modified from instruments successfully used in the HPOG 1.0 evaluation. The research team pre-tested the screening interview and first-round telephone interview instruments with three non-tribal HPOG grantees. Grantees that completed the survey during the pre-test will be given their completed surveys to review and update when the full survey is fielded to reduce burden while ensuring all responses are accurate and up-to-date. Experienced interviewers conducted the interviews and discussed respondents’ perceptions of the clarity and flow of survey items, ease of completion, and time requirements. After pretesting, we revised the instruments based on the feedback. Changes made to the instruments are included in this revised clearance request for OMB to review.
HPOG 2.0 National Evaluation Impact Evaluation
The participant contact update form for the HPOG 2.0 National Evaluation impact evaluation (Instrument 5b) draws largely from the contact update forms previously approved for other ongoing career pathways studies, particularly the Pathways for Advancing Careers and Education (PACE) and the first round of the HPOG Impact studies, OMB control numbers 0970-0397 and 0970-0394 respectively. Based on our experiences, minor modifications were made to the HPOG 2.0 contact update form. The first modification allows participants to give consent for researchers to text any new phone numbers provided on the update form. The second modification allows the participants to indicate their preferred mode of contact, which will allow researchers to tailor their approach to the participant preferences.
HPOG 2.0 Tribal Evaluation
The data collection instruments for the HPOG 2.0 Tribal Evaluation have been reviewed by (1) ACF staff, (2) a Technical Working Group comprised of consultants with expertise in workforce development and tribal research, and (3) all five Tribal HPOG grantees. Their comments were incorporated into the final versions.
PAGES
Abt Associates subcontracted with Green Beacon Solution, a leading Microsoft Gold Certified software development firm to create the HPOG 2.0 PAGES in consultation with ACF. Federal staff and evaluation team members had informal discussions with six first round HPOG grantees on possible data system designs and data elements. Prior to the initial launch of PAGES, there were two distinct environments, one for development and one for testing. Development and testing was conducted in multiple stages. Abt Associates professional software testers tested major modules that were deployed to the test environment and submitted issues. Once these issues were resolved, the broader team of program staff reviewed the module and offered feedback. In the month prior to the release of the system, Abt technology staff re-tested all parts of the system to ensure that it met the stated functional requirements and is free of bugs. Abt Associates, Urban Institute and ACF program staff also conducted user acceptance testing.
Since the launch of the system, three environments are maintained: Development, Test, and Production. Any changes to the system will have to move through the Development and Testing environments and procedures prior to being deployed in the live production environment. All three environments will be maintained throughout the life of the project.
The baseline questions are either identical or similar to questions used in previous Abt Associates or other national surveys. As such, they have been thoroughly tested on large samples.
With ACF oversight, Abt and its partners MEF Associates, the Urban Institute and Insight Policy Research are responsible for conducting the HPOG 2.0 National Evaluation; Abt and the Urban Institute are responsible for developing and maintaining PAGES and providing HPOG 2.0 grantees with support for using the system to produce the required semi-annual reports and collecting the needed baseline data. NORC, under subcontract to Abt Associates, is responsible for conducting the HPOG 2.0 Tribal Evaluation.
With ACF oversight, Abt Associates and Urban Institute are responsible for developing the HPOG 2.0 PAGES and providing support for using the system to produce the required semi-annual reports and collecting the needed baseline data.
Statistical analyses of the data for annual program performance reports will be limited to descriptive tabulations included in the contractor’s annual reports to ACF. Other as yet unspecified statistical analyses may be planned for the impact evaluation currently being designed and other future research efforts. Such analyses will be the subject of a later request for clearance.
The individuals listed in Exhibit B5.1 below made a contribution to this request for clearance.
Exhibit B5.1. Contributors
Name |
Role in HPOG 2.0 National and Tribal Evaluation |
Organization/Affiliation |
Gretchen Locke |
National Evaluation Project Director |
Abt Associates |
Robin Koralek |
National Evaluation Deputy Project Director |
Abt Associates |
David Judkins |
National Evaluation Director of Impact Analysis |
Abt Associates |
Debi McInnis |
National Evaluation Site Coordinator |
Abt Associates |
Michael Meit |
Tribal Evaluation Project Director |
NORC |
Kate Fromknecht |
Tribal Evaluation Project Manager |
NORC |
Emily Phillips |
Tribal Evaluation Senior Research Analyst |
NORC |
Julie Strawn |
PAGES Project Director |
Abt Associates |
Dr. Laura R. Peck |
PAGES Co-Principal Investigator |
Abt Associates |
Dr. Pam Loprest |
PAGES Co-Principal Investigator |
Urban Institute |
Dr. Eleanor Harvill |
PAGES Evaluation Design Task Lead |
Abt Associates |
Brian Sokol |
PAGES Data System Task Lead |
Abt Associates |
Dr. Alan Werner |
Key staff on HPOG 2.0 project |
Abt Associates |
Jennifer Buell |
PAGES Deputy Project Director |
Abt Associates |
Dr. Howard Rolston |
PAGES Project Quality Advisor |
Abt Associates |
Karen Gardiner |
PAGES Project Quality Advisor |
Abt Associates |
Inquiries regarding the statistical aspects of the design of the data system should be directed to:
Julie Strawn, Project Director
Abt Associates
4550 Montgomery Ave #800N
Bethesda, MD 20814
(301) 347-5853
Inquiries regarding the statistical aspects of the HPOG 2.0 National Evaluation design should be directed to:
Gretchen Locke, Project Director
Abt Associates
55 Wheeler St
Cambridge, MA 02138
(617)-349-2373
Inquiries regarding the statistical aspects of the HPOG 2.0 Tribal Evaluation design should be directed to:
Michael Meit, Project Director
NORC at the University of Chicago
4350 East West Highway, 8th Floor
Bethesda, MD 20814
(301)-634-9324
The Evaluators also consulted a team of outside experts on the design on the HPOG 2.0; recruitment strategies and data collection instruments for the HPOG 2.0 Tribal Evaluation; the grantees will also contribute to data collection via the PAGES system:
Technical Working Group (Consultants)
Mark Doescher, MD, MSPH, Stephenson Cancer Center, University of Oklahoma
Rick Haverkate, MPH, Deputy Director, Indian Health Service
Loretta Heuer, PhD, RN, FAAN, School of Nursing, North Dakota State University
Joan LaFrance, Ed.D, Mekinak Consulting
Myra Parker, JD, MPH, PhD, Center for the Study of Health and Risk Behaviors, University of Washington
Tribal Health Professions Opportunities Grants 2.0 grantees
Mark Hiratsuka, Cook Inlet Tribal Council
Phillip Longie, Cankdeska Community College
Irene BearRunner, Turtle Mountain Community College
Scott Baker, Ute Mountain Ute Tribe
Kathleen Thurman, Great Plains Tribal Chairmen’s Health Board
The following HHS staff, including the HHS project officers Hilary Forster, Nicole Constance, and Amelia Popham, have overseen the design process and can be contacted at:
Hilary Forster
Office of Planning, Research & Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
330 C Street S.W., 4th Floor, Washington, D.C. 20201
(202) 619-1790
Nicole Constance
Office of Planning, Research and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
330 C Street S.W., 4th Floor, Washington, D.C. 20201
(202) 401-7260
Amelia Popham, MSW
Office of Planning, Research and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
330 C Street S.W., 4th Floor, Washington, D.C. 20201
202-401-5322
1 Response rate expectations are based on a variety of factors. Grantees have agreed to participate in the evaluation as a condition of receiving HPOG funding, so grantee, partner, and employer response rates are expected to be very high. Participation in the evaluation studies is voluntary for HPOG participants, so response rates are expected to be lower. Previous experience with similar populations indicates that response rates are expected to be lower for participants who do not complete the program than those who do.
2 The projected response rate for the contact update form is based on prior experience with similar approaches on studies of comparable populations—primarily the PACE and HPOG 1.0 Impact study samples (OMB No. 0970-0397 and 0970-0394 respectively).
3 There were some first round HPOG grantees that also operated PACE programs. Participants from these HPOG/PACE sites are subject to the same bypass criteria as HPOG 1.0 participants,
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Abt Single-Sided Body Template |
| Author | Katheleen Linton |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |