Instrument 3: HPOG 2.0 National Evaluation first-round telephone interviews with management, staff, partners, and staffstakeholders

As part of this study we are conducting phone interviews with management, staff, partners, and stakeholders to learn more about how programs are designed and implemented as well as the training and support services they provide.

Use the “Grantee Preparation Guide for the HPOG 2.0 National Evaluation First Round Telephone Interviews” to help management, staff, partners, and stakeholders prepare for the telephone interviews.

We expect that the telephone interview will take approximately 90 minutes to complete depending on the module in the interview protocol and role of key respondents involved in [name of local HPOG program].

The deadline to complete all telephone interviews with management, staff, and stakeholders is [date]. Please follow the 11 steps below to ensure all data collection materials are completed for the HPOG 2.0 Implementation Study.

Before the telephone interview:

Complete the “Screening Interview to Identify Respondents for the HPOG 2.0 National Evaluation First-Round Telephone Interviews.” Schedule a meeting with management and staff to identify the most appropriate HPOG program staff, key partners, and stakeholders to interview for each module in the interview protocol (e.g. Program Administration, Outreach and Recruitment, etc.).
Schedule telephone interviews with respondents. Once you have identified the appropriate respondents to interview, work with management, staff, partners, and stakeholders to schedule dates and times to complete the telephone interviews.
Pre-fill interview questions and responses, if available, with information from required data sources in advance of each telephone interview. The junior site team member/interviewer should pre-fill the interview protocol in the online survey and on paper to help prepare for the telephone interview. Please follow the “Interviewer/HPOG Site Team Instructions” on which data sources to prefill for each interview question and selected tables. Data sources may include, but are not limited, to the following:
- Evaluation Design and Implementation Plan (EDIP) Reports

Target population and recruitment strategies
Eligibility criteria and application processes
Control group services and conditions

PAGES
1. Screenings and assessments during eligibility
2. Basic skills and other skills development activities
3. Healthcare occupational training
4. Employment
5. Program Supports (e.g., Academic Supports, Personal/Logistical Supports, Employment Assistance Supports, and Work-Based Learning Opportunities)
Site Team Monitoring Call Notes
1. Updates on marketing and recruitment strategies
2. Treatment group use of services and training (e.g., percentage enrolled in remedial/pre-training activities, percentage enrolled in training, etc.)
3. Documented program changes
4. Implementation experiences
5. Updates on control group services

For interview questions that require information from PAGES: Junior site team members should follow instructions on how to extract information from PAGES using the spreadsheet saved here: [File Path]. Please note that information from PAGES will vary by grantee and program. Refer to the EDIP and site monitoring call notes to help pre-fill questions or responses in cases where information is limited from PAGES or leave blank until the telephone interview.

Delete all “Interview/HPOG Site Team instructions.” Once you have pre-filled the questions and responses, please remember to delete all the “Interview/HPOG Site Team Instructions” before emailing the “Grantee Preparation Guide for the HPOG 2.0 National Evaluation First Round Telephone Interviews” to respondents.
Email the pre-filled “Grantee Preparation Guide for the HPOG 2.0 National Evaluation First Round Telephone Interviews.” We are requesting management, staff, partners, and stakeholders to review selected questions and pre-filled responses to prepare for the telephone interview. Please advise all respondents not to submit any changes to the responses by phone or email. Respondents will have an opportunity to change responses during the telephone interview.
Familiarize yourself with the skip patterns in each module of the interview protocol. Some interview questions can be skipped based on how the respondent answers a particular question.

During the telephone interview:

Use the online survey to document all responses during the telephone interview. The junior site team member will record all responses in the online survey during the interview. Please record all changes to any pre-filled responses that management, staff, or stakeholders identify as incorrect in the online survey. This is an internal survey, so please do not share the link with the respondent.
Share screens to reduce burden (optional). Arrange a WebEx meeting to share screens and allow respondents to view each question and the responses during the telephone interview.
Complete the HPOG 2.0 Site Team Program Module. We have identified a few questions that site teams can complete on their own using information from the EDIP, site monitoring call notes, or other data sources. This includes information about outreach and recruitment, eligibility and intake, and control group services.

After the telephone interview:

Review all responses to ensure they are complete and accurate. The lead interviewer will review all closed and open-ended responses to ensure they are completed before submission.
Thank the respondents. Don’t forget to thank your respondents for completing these interviews and providing important information for the study.

Grantee Preparation Guide

Sample Email to HPOG Management and Staff

Health Profession Opportunity Grants (HPOG) 2.0 Implementation Study

Sample Email to HPOG Management and Staff

Dear [Name of HPOG Program Director/Staff],

As you may know, [name of local HPOG program] is participating in a national evaluation of the Health Profession Opportunity Grants. The second round of grants were awarded in 2015 (referred to as HPOG 2.0) and are sponsored by the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services (DHHS). The evaluation is being conducted by Abt Associates, the Urban Institute, MEF Associates, NORC, and Insight Policy Research, Inc. It is studying all HPOG-funded education and training programs across the country and examining how they help low-income individuals, including Temporary Assistance for Needy Families (TANF) recipients, secure well-paying healthcare jobs. I am writing to enlist your support and assistance in this project.

We plan to conduct interviews by phone with all HPOG Program grantees to better understand the structure, operations, and services of their programs. In particular, we are asking about your program background and context, organizations with which you collaborate, including their roles and responsibilities, and program activities such as outreach and recruitment, eligibility and intake, training, and support services. The interview should take approximately 60 to 90 minutes to complete depending on the role of key staff in [name of local HPOG program]. With your help, we have identified the most appropriate staff, partners, and stakeholders to talk to us about each of the topic areas.

In preparation for this telephone interview, we are asking all HPOG Program grantees to review the responses to questions from our telephone interview protocol in the attached document. We selected responses using information from the Evaluation Design and Implementation Plan (EDIP) report, weekly and monthly site monitoring reports, and PAGES. Please do not submit any changes to the questions or responses in this document. We simply ask that you review the information in preparation of the telephone interview. You will have an opportunity to share changes and corrections during the telephone interview. Thank you in advance for your assistance for participating in these interviews and providing important information for the study. With your help, we will have better information about the practices of HPOG programs across the nation.

Sincerely,

HPOG site team

The Paperwork Reduction Act Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0462 and it expires xx/xx/xxxx. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to [Contact Name]; [Contact Address]; Attn: OMB-PRA (0970-0462).

Sample Email to HPOG Partners and Stakeholders

Health Profession Opportunity Grants (HPOG) 2.0 Implementation Study

Sample Email to HPOG Partners and Stakeholders

Dear [HPOG Partner or Stakeholder],

We plan to conduct interviews by phone with all HPOG Program grantees to better understand the structure, operations, and services of their programs. In particular, we are asking about how partners and stakeholders collaborate with HPOG programs, including their roles and responsibilities, and program activities such as education and healthcare occupational training and program supports. We have worked with representatives from [name of local HPOG program] to identify partners and stakeholders that have been involved in their HPOG program. The interview should take approximately 60 to 90 minutes to complete.

In preparation for these telephone interviews, we are asking all HPOG partners and stakeholders to review the responses to questions from our telephone interview protocol in the attached document. We selected responses using information from the Evaluation Design and Implementation Plan (EDIP) report (a brief report that documents the HPOG program’s target population, recruitment strategies, eligibility criteria and application processes for enrollment and services available to individuals randomly assigned to the control group), weekly and monthly site monitoring reports with grantees and programs, and the Participant Accomplishment and Grant Evaluation System (PAGES, the management information system that supports program management, performance reporting, and data collection for the evaluation). As you review, please do not submit any changes to the questions or responses in this document. We simply ask that you review the information in preparation of the telephone interview.

Thank you in advance for your assistance for participating in these interviews and providing important information for the study. With your help, we will have better information about the practices of HPOG programs across the nation.

Sincerely,

HPOG site team

Instructions for Management, Staff, Partners, and Stakeholders

Thank you in advance for your assistance in completing these interviews and providing important information for the study. As part of this study we are conducting phone interviews with management, staff, partners and stakeholders at each HPOG program to learn more about how programs are designed and implemented as well as the training and support services they provide. Please follow the steps below in preparation for your telephone interview with Abt on [date of interview].

Review the questions and pre-filled responses. To help save time during the telephone interview, we have pre-filled responses with information collected from the Evaluation Design and Implementation Plan (EDIP) report (a brief report that documents the HPOG program’s target population, recruitment strategies, eligibility criteria and application processes for enrollment and services available to individuals randomly assigned to the control group), site monitoring call notes, and Participant Accomplishment and Grant Evaluation System (PAGES, the management information system that supports program management, performance reporting, and data collection for the evaluation). Please review the questions and pre-filled responses included in this document to ensure the information is correct and up-to-date. Please note that responses to questions are left blank when data are not available.
Do not submit any changes. If the information is incorrect, use the “Grantee Preparation Guide for First Round Telephone Interviews” to document any changes that need to be made. You will have an opportunity to report all changes during the telephone interview. We simply ask that you only review the questions and pre-filled responses in preparation for your telephone interview. Some response options are left blank in cases where we had limited data from PAGES or other data sources. You will have an opportunity to provide additional responses during the telephone interview.

Review data sources in advance. To reduce time during the telephone interview, please be prepared to discuss the following information:

Roles of partners currently involved in your program;
Eligibility criteria, including thresholds to determine financial eligibility (e.g., percent of federal poverty level, household or individual income, etc.);
Delivery of basic skills training and other skill development activities;
Delivery of healthcare occupational training and program supports (e.g., academic, personal/logistical, employment assistance supports and work-based learning opportunities);
Type and accessibility of control group services; and
Plans for sustainability.

Contact your HPOG Site Team if you have questions or concerns. If you have any questions or concerns, please do not hesitate to contact your HPOG site teams members:________________ and ___________________ at the following email addresses:_________________________________________.

Program Administration

The following questions focus on how your HPOG program is organized and how you and your partners deliver services and training opportunities to participants.

1.1. Based on information collected by your HPOG study team, we have compiled a list of organizations that are involved with or have supported [name of local HPOG program]. This list currently includes basic skills and healthcare occupational training providers. Do you have any changes or updates to the organizations listed in the table? What type of organization is each partner? If partners include employers, please consider only employers who are involved beyond hiring graduates.

Interviewer/HPOG Site Team Instructions: Select only one answer for each organization.

Question 2.1 in telephone interview protocol

Data Source(s): PAGES [Vendors: Columns Selected (Vendor ID, Name, Type of Organization)]

EDIP and Site monitoring call notes for new organizations

Grantee/HPOG Program Instructions: Information about organizations involved with your program was collected from PAGES, the EDIP, or site monitoring call notes. Please review each of the organizations listed in the table below. If an organization does not appear below or is no longer involved with [name of local HPOG program], please be prepared to discuss during the telephone interview. Select only one answer for each organization.

Organization	A	B	C	D	E	F	G	I	J
Organization	State government agency	Local government agency	Work Investment Board	One-Stop Career Center/ American Job Center	Community/ Technical college	Community and faith-based organizations (Non-profits)	For-profit or proprietary service/ training provider	Healthcare Employer	Other (Please specify: )
a. [Organization Name]	☐	☐	☐	☐	☐	☐	☐	☐	☐
b. [Organization Name]	☐	☐	☐	☐	☐	☐	☐	☐	☐
c. [Organization Name]	☐	☐	☐	☐	☐	☐	☐	☐	☐

Source: HPOG-NIE Grantee Survey, Questions 1.1 and 5.1 and Master Interview Guide, Question 26

Go to question 1.2

1.2. What is each organization’s current role in [name of local HPOG program]?

Interviewer/HPOG Site Team Instructions: Select all that apply.

Question 2.2 in telephone interview protocol

Data Source(s): PAGES [Training & Services: Columns Selected (Category, Vendor, Training/Support Type)]

EDIP and Site monitoring call notes for new organizations

Grantee/HPOG Program Instructions: Information about organizations was collected from PAGES, the EDIP, or site monitoring call notes. Please review each of the partners organizations listed in the table below. Please refer to the key examples listed below if a category is unclear. If an organization does not appear below or is no longer involved with [name of local HPOG program], please be prepared to discuss during the telephone interview. Select all that apply.

Organization	A	B	C	D	E	F	G	H	I	J	K
Organization	Planning and design of HPOG grant activities	Referral of applicants for services provided by [name of local HPOG program]	Marketing and Outreach	Curriculum Development	Healthcare Occupational Training	Basic and Other Skill Development Activities (non-occupational)	Academic Support	Personal/ Logistical Support	Employment Assistance Support	Work-Based Learning Opportunities	Direct Financial Support
a. [Organization Name]	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐
b. [Organization Name]	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐
c. [Organization Name]	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐
d. [Organization Name]	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐

Source: HPOG Stakeholder/Network Survey, Question 14

Key Examples:

^A Planning and design of HPOG grant activities (e.g., grant writing, letter of commitment, member of advisory/steering committee)

^B Referral of applicants for services provided (e.g., formal referral arrangement, initial screening of applicants, referral of current employees)

^C Marketing and outreach (e.g., printed materials available on-site, information available on partner’s website, mentions during presentations to

stakeholders, mentions during orientation for organization’s services, mentions during assessment and counseling session)

^D Curriculum development (e.g., offering examples of relevant curricula, providing feedback on draft curricula, writing modules for curriculum)

^E Healthcare occupational training (e.g., operation of training program, provision of faculty/instructors, provision of training space, provision of

equipment, provision of learning technologies, provision of work-based learning opportunities—e.g., internships, clinicals)

^F Basic and other skill development activities (e.g., Adult basic education, Adult secondary education (GED, HiSET, TASC), English language acquisition, College

developmental education, Introduction to Healthcare Career Workshop, Work Readiness, College Readiness, Digital Literacy, CPR Training, Mandatory work readiness

with peer support)

^G Academic support (e.g., case management, academic advising, tutoring, mentoring, peer support, post eligibility assessments training related costs assistance (other

than tuition), laptop for blended learning, internet access for blended learning)

^HPersonal/Logistical support (e.g., emergency, non-emergency food, child/dependent care, transportation, housing support/assistance)

^I Employment Assistance Support (e.g., job development activities (including job readiness workshops, job search skills training, individual job search assistance, job

coach navigator, group job search support, post-placement and retention support), job placement activities (including obtaining and screening job listings for HPOG

participants, screening HPOG participants for suitability for a position, scheduling interviews for a job candidate, and providing interview space), and job retention

services (including counseling for specific job-related issues, incumbent worker career advancement counseling, and job-specific workplace behavior counseling))

^J Work-based Learning Opportunities (e.g., work experience, job shadowing opportunities, registered apprenticeships, unpaid internships or externships, or on-the-

Job training)

K Direct Financial Support (e.g., grants, loans, work-study funds, etc.)

Include this question only for returning HPOG 1.0 grantees

1.3. Which of the following organizations are new or returning partners from the HPOG 1.0 grant?

Interviewer/HPOG Site Team Instructions: Select only one answer for each row.

Data Source(s): PAGES [Vendors: Columns Selected (Vendor ID and Name)]

Results from the National Implementation Evaluation (NIE) of the HPOG Program Survey.

Grantee/HPOG Program Instructions: Information about organizations was collected from PAGES and previous results from the National Implementation Evaluation (NIE) of the HPOG Program Survey. Please review each of the organizations listed in the table below. Please be prepared to discuss any changes during the telephone interview. Select only one answer for each row.

Organization	New Organization	Returning Organization from HPOG 1.0 grant
a. [Organization Name]	☐	☐
b. [Organization Name]	☐	☐
c. [Organization Name]	☐	☐
d. [Organization Name]	☐	☐

Go to question 1.4

1.4. Are you currently collaborating with (or planning to collaborate with) your partners in one or more of the following ways to improve the service delivery system in which your program operates?

Interviewer/HPOG Site Team Instructions: Select all that apply

Question 2.5 in telephone interview protocol

Grantee/HPOG Program Instructions: Please review the responses below. If the information is incorrect or left blank, please be prepared to discuss during the telephone interview. Select all that apply.

☐ Creation of a logic model or theory of change that is being used across organizations to shape practice and track performance

☐ Curriculum developed for HPOG participants that can be shared with and used for

training services by partners

☐ Development of new credentials that are recognized by employers

☐ Development of a new or improvement of a current referral system across partners

☐ Development or improvement of case management practices across partners to

reduce duplication

☐ Outreach efforts to employers to help change their perspectives on and hiring practices for the target groups served by the

HPOG program

☐ Increased capacity across partners for training and other services that HPOG participants and others in need can access

☐ Articulation agreements between education and training programs to support advancement on a career pathway

☐ Creation of a sector partnership where employers identify training priorities and lead program design

☐ Creation of a collaborative group across service and training providers to better coordinate activities

☐ Sharing of funding or resources or joint identification of new resources by partners

☐ Data sharing on participants or program operations

☐ Joint outreach efforts to market training and career opportunities in the healthcare industry to target populations such as youth

☐ Other (Please specify: )

Go to “Healthcare Occupational Training”

Healthcare Occupational Training

2.1. I am going to name the healthcare occupational training courses that our records show you offer, and for each, I would like you to tell me if each course is open only to HPOG participants or to any non-HPOG students, including the control group. By healthcare occupational training, we mean the combination of one or more classes that are necessary for you to consider that a participant has completed the healthcare occupational training. For example, a six-week training for nursing assistants would be one healthcare occupational training activity. The coursework necessary to become a registered nurse would also be listed as one healthcare occupational training activity, even though it is made up of many individual classes.

Interviewer/HPOG Site Team Instructions: Select all that apply.

Question 5.7 in telephone interview protocol

Data Source(s): PAGES [Training & Services: Columns Selected (Category

(Health Occupation Training), Occupational Code, Vendor, Training/Support

Type, and Career Pathway Level)]

Availability to HPOG and non-HPOG students is not available in PAGES
Supplement with EDIP or site monitoring call notes

Grantee/HPOG Program Instructions: Please review the responses below. If the

information is incorrect or left blank, please be prepared to discuss during the

telephone interview. Select all that apply.

Healthcare Training Course (Pre-fill from PAGES)	Available ONLY to HPOG Participants Only	Available to HPOG and other students
[Training Course]	☐	☐
[Training Course]	☐	☐
[Training Course]	☐	☐
Pre-fill with additional trainings as needed	☐	☐

Source: HPOG Grantee Survey, Question 8.8a-b

Go to “Work-Readiness Training Provision”

Work-Readiness Training Provision

The following questions focus on the type of skill development training, activities, or workshops provided to improve participants’ soft and/or work-related skills, included when they are offered as part your HPOG program.

3.1a. Some HPOG programs offer skill development training, activities, or workshops to improve participants’ soft and/or work-related skills such as orientation/introduction to healthcare careers, work readiness, college readiness, and others. Does your HPOG program currently offer the following skill development training and activities to prepare participants for healthcare training and employment?

Interviewer/HPOG Site Team Instructions: Select all that apply

Question 5.17a in telephone interview protocol

Data Source(s): PAGES [Training & Services: Columns Selected (Category

(Other Skill Development Activities) and Training/Support Type)]

Grantee/HPOG Program Instructions: Information about skill development training, activities, or workshops was collected from PAGES. Please review the responses below. If the information is incorrect or left blank, please be prepared to discuss during the telephone interview. Select all that apply.

Key Definitions:

Introduction to Healthcare Careers: A workshop that provides information in a group setting about a variety of healthcare careers, including necessary educational and other requirements, day-to-day work activities, and career pathways.
Work Readiness: A course or workshop that focuses on world-of-work awareness and addresses the interpersonal and intrapersonal skills (or “soft skills”) individuals need to be successful in the workplace.
Digital Literacy: A course or workshop that educates participants on the use of digital technology, communication tools or networks to locate, evaluate, use and create information; the ability to understand and use information in multiple formats from a wide range of sources when it is presented via computers; how to read and interpret media; how to reproduce data and images through digital manipulation; and how to evaluate and apply new knowledge gained from digital environments.
CPR Training: Course of instruction in cardiac pulmonary resuscitation (CPR) that follows a nationally recognized program such as those of the American Heart Association or Red Cross and those approved by the Occupational Safety and Health Administration (OSHA) or state license boards for medical professionals.

☐ Introduction to Healthcare Career workshop

☐ Work readiness (e.g. positive work habits, attitudes, and behavior, etc.)

☐ College readiness

☐ Digital literacy

☐ CPR training

☐ Other (Please specify: )

3.1c. Which courses, activities, or workshops are mandatory or voluntary for HPOG participants?

Interviewer/HPOG Site Team Instructions: Select one answer for each course,

activity, or workshop offered.

Question 5.17c in telephone interview protocol

Data Source(s): PAGES [Training & Services: Columns Selected (Category

(Other Skill Development Activities), Vendor, and Training/Support Type)]

Requirements (e.g. mandatory or voluntary) for courses, activities, or workshops are not available in PAGES

Grantee/HPOG Program Instructions: Information about courses, activities, and

workshops was collected from PAGES, the EDIP and site monitoring call notes.

Please review the responses in the table below. If the information is incorrect or left

blank, please be prepared to discuss these changes during the telephone interview.

Select one answer for each course, activity, or workshop offered.

	Mandatory for HPOG Participants	Voluntary for HPOG Participants
Introduction to Healthcare Career workshop	☐	☐
Work readiness (e.g. positive work habits, attitudes, and behavior, etc.)	☐	☐
College readiness	☐	☐
Digital literacy	☐	☐
CPR training	☐	☐
Other (Please specify: )	☐	☐

Go to “Program Supports”

Program Supports

The following questions focus on the type of academic, personal/logistical, and employment assistance supports you and your partners offer and what ways you believe they help participants complete training and obtain employment in the healthcare field.

4.1. What type of case management and counseling services are delivered to participants? Who among your staff provides these services? Are any of the services delivered by partners (at a cost to the HPOG program or not) or other programs or community service agencies?

Interviewer/HPOG Site Team Instructions: For each service offered, please indicate whether HPOG staff, partners, or other programs or community service agencies provide the service. Select all that apply.

Question 6.5 in telephone interview protocol
If services are provided by partners, identify whether the service is provided at a cost to the program or not.

Data Sources(s): EDIP and site monitoring call notes

Grantee/HPOG Program Instructions: Information about case management and counseling services was collected

from the EDIP and site monitoring call notes. Please review the responses in the table below. If the

information is incorrect or left blank, please be prepared to discuss during the telephone interview. Select all that

apply.

If services are provided by partners, identify whether the service is provided at a cost to your program or not.

Case Management and Counseling Services	Services Provided by [name of local HPOG program]				Services provided by partners		Services provided by other programs or community service agencies	Not Offered
Case Management and Counseling Services	Case Manager	Academic/ Career Advisor	Job Developer	Other HPOG staff	At a cost to the HPOG program	At no cost to the HPOG program		Not Offered
Participant monitoring (e.g., assessing participants’ progress in training or needs for program supports)	☐	☐	☐	☐	☐	☐	☐	☐
Academic counseling (e.g., course advising)	☐	☐	☐	☐	☐	☐	☐	☐
Career counseling (e.g., reviewing careers or career pathways)	☐	☐	☐	☐	☐	☐	☐	☐
Counseling to identify personal and supportive service needs	☐	☐	☐	☐	☐	☐	☐	☐
Financial counseling (e.g., helping with financial aid or related income support or budget matters)	☐	☐	☐	☐	☐	☐	☐	☐
Job search assistance	☐	☐	☐	☐	☐	☐	☐	☐
Job placement assistance	☐	☐	☐	☐	☐	☐	☐	☐
Job retention services	☐	☐	☐	☐	☐	☐	☐	☐
Other (Please Specify: )	☐	☐	☐	☐	☐	☐	☐	☐

Source: HPOG-NIE Grantee Survey, Question 9.2

Key Definitions:

^f. Job Search Assistance: Provision of assistance, one-on-one or in a group, in job search including information on labor markets, occupational information, and job search

techniques (resumes, interviews, applications, and follow-up letters). Resulting job search is self-directed by participants. This assistance does not include job

placement assistance.

^g. Job Placement Assistance: Referral of individuals to jobs matching their abilities and interests. Staff may interview and assess/test participants for the purpose of

achieving suitable job placements where there is a good match between management needs and employee qualifications. This is separate from job search assistance

that results in self-directed job search.

^h. Job Retention Services: Various practices that help a person maintain employment or change jobs without a period of unemployment. The services in this category

must take place while the participant is employed and differ from services that focus on work readiness training prior to finding employment. Examples of job

retention services include counseling for specific job-related issues, incumbent worker career advancement counseling, and job-specific workplace behavior

counseling.

Go to question 4.2

4.2. Please describe the type of advising services that are routinely offered by [name of local HPOG program]. Are these services provided directly by your program, your partners (at a cost to the HPOG program or not) , or other programs or community service agencies?

Question 6.6 in telephone interview protocol
If services are provided by partners, identify whether the service is provided at a cost to the program or not.

Data Sources(s): EDIP and site monitoring call notes

Grantee/HPOG Program Instructions: Information about advising services was

collected from PAGES, the EDIP, and site monitoring call notes. Please review the

responses in the table below. If the information is incorrect or left blank, please

be prepared to discuss during the telephone interview. Select all that apply.

If services are provided by partners, identify whether the service is provided at a cost to your program or not.

Advising Services	Services provided by [name of local HPOG program]	Services provided by Partners		Services provided by other programs or community service agencies	Not Offered
Advising Services	Services provided by [name of local HPOG program]	At a cost to the HPOG program	At no cost to the HPOG program		Not Offered
Advise on graduation requirements	☐	☐	☐	☐	☐
Advise on college entrance requirements or prerequisites	☐	☐	☐	☐	☐
Assistance developing and tracking career goals in healthcare	☐	☐	☐	☐	☐
Advise on course selection	☐	☐	☐	☐	☐
Assistance with license/certification examination preparation	☐	☐	☐	☐	☐
Other (Please specify: )	☐	☐	☐	☐	☐

Source: HPOG-NIE Grantee Survey, Question 8.15

Go to question 4.3

4.3. Some academic support services are designed to connect participants in a social setting or with other individuals, including mentors or peers. Does your organization and/or any of your partners provide academic support services such as mentoring or peer support to participants? If so, are they provided directly by your program, your partners (at a cost to the HPOG program or not), or other programs or community service agencies?

Interviewer/HPOG Site Team Instructions: For each service offered, please indicate whether it is provided by the program, partner, or other programs or community service agencies. Select all that apply.

Question 6.7 in telephone interview protocol

If services are provided by partners, identify whether the service is provided at a cost to the program or not.

Data Source(s): PAGES [Training & Services: Columns Selected (Category

(Academic Supports), Training/Support Type, and Provider)]

Select only if Provider is “HPOG grantee”
Supplement with EDIP or site monitoring call notes

Grantee/HPOG Program Instructions: Information about academic support

services was collected from PAGES, the EDIP, and site monitoring call notes.

Please review the responses in the table below. If the information is incorrect or

left blank, please be prepared to discuss during the telephone interview. Select

all that apply.

If services are provided by partners, identify whether the service is provided at a cost to your program or not.

Key Definitions:

Mentoring: Advice and counseling to a participant by a person other than case manager/program staff who has already achieved goals that are the same as or similar to the goals that a participant has set for his/herself, based on the mentor’s personal experiences. This involves an ongoing relationship that may be formal or informal.
Peer Support: Activities that foster social and emotional connections among a consistent cohort or group of participants with the intention of enabling mutual assistance and shared accountability and commitment to program retention and completion.

Academic Support Services	Services provided by [name of local HPOG program]	Services provided by Partners		Services provided by other programs or community service agencies	Not Offered
Academic Support Services	Services provided by [name of local HPOG program]	At a cost to the HPOG program	At no cost to the HPOG program		Not Offered
Mentoring activities	☐	☐	☐	☐	☐
Peer support activities	☐	☐	☐	☐	☐
Other (Please specify: )	☐	☐	☐	☐	☐

Source: HPOG-NIE Grantee Survey, Question 9.8 and HPOG Master Interview Guide 75

Go to question 4.4

4.4. According to our information, [name of local HPOG program] offers the following personal/logistical supports: [Interviewer/HPOG Site Team Instructions: Pre-fill with information from PAGES]. Have there been any changes? Are these services provided directly by your program, your partners (at a cost to the HPOG program or not), or other programs or community service agencies?

Question 6.8 in telephone interview protocol

If services are provided by partners, identify whether the service is provided at a cost to the program or not.

Data Source(s): PAGES [Training & Services: Columns Selected (Category

(Personal/Logistical Supports), Training/Support Type, and Provider)]

Select only if Provider is “HPOG grantee”
Supplement with EDIP or site monitoring call notes

Grantee/HPOG Program Instructions: Information about personal/logistical

supports was collected from PAGES, the EDIP, and site monitoring call notes.

Please review the responses in the table below. If the information is incorrect

or left blank, please be prepared to discuss during the telephone interview. Select

all that apply.

If services are provided by partners, identify whether the service is provided at a cost to your program or not.
Please see glossary of terms below.

Personal/Logistical Supports	Services provided by [name of local HPOG program]	Services provided by Partners		Services provided by other programs or community service agencies	Not Offered
Personal/Logistical Supports	Services provided by [name of local HPOG program]	At a cost to the HPOG program	At no cost to the HPOG program		Not Offered
Emergency assistance	☐	☐	☐	☐	☐
Non-emergency food assistance	☐	☐	☐	☐	☐
Child/dependent care assistance	☐	☐	☐	☐	☐
Transportation assistance	☐	☐	☐	☐	☐
Housing support/assistance	☐	☐	☐	☐	☐
Other (Please specify: )	☐	☐	☐	☐	☐

Source: HPOG-NIE Grantee Survey, Question 9.11 and 9.19 and HPOG Master Interview Guide, Question 83

Key Definitions:

^A Emergency Assistance: Usually a one-time payment for an unexpected and atypical expense for which a

participant’s current resources are inadequate, and that if not met causes the participant to be at significant risk

of ending program participation or employment due to hardship or practical necessity. These include but are not

limited to expenses for rent, utilities, food, or car repairs when the lack of payment would result in a loss of

housing, utilities, or necessary transportation.

^B Non-Emergency Food Assistance: Payments or other assistance that provide food for an HPOG participant, as part

of an HPOG training program or activity on a non-emergency basis.

^C Child/Dependent Care Assistance: Payments or other financial assistance for direct care for children or dependent

family members that enable a participant to participate in education and training, other HPOG services, or

attaining and retaining employment. Care provider must meet the child/dependent care requirements of state

and local law.

^D Transportation Assistance: Payments or other assistance that enable the participant to travel to and from

education and training, other HPOG services, or employment that do not meet the definition of emergency

assistance. These include but are not limited to bus/subway cards, gas vouchers/cards, or van/carpool

arrangements.

^E Housing Support/Assistance: Payments or other assistance that enable a participant to attain or maintain housing

or a temporary accommodation that do not meet the definition of emergency assistance. These include but are not

limited to first month’s rent, security deposit, housing during training, and utility payments.

Go to question 4.5

4.5. What type of training-related costs and assistance are delivered to participants? Are these services provided directly by your program, your partners (at a cost to the HPOG program or not), or other programs or community service agencies?

Question 6.9 in telephone interview protocol
If services are provided by partners, identify whether the service is provided at a cost to the program or not.

Data Sources(s): EDIP or site monitoring call notes

Grantee/HPOG Program Instructions: Information about training-related costs and

assistance was collected from PAGES, the EDIP, and site monitoring call notes.

Please review the responses in the table below. If the information

is incorrect or left blank, please be prepared to discuss during the telephone

interview. Select all that apply.

If services are provided by partners, identify whether the service is provided at a cost to your program or not.

Training-Related Costs and Assistance	Services provided by [name of local HPOG program]	Services provided by Partners		Services provided by other programs or community service agencies	Not Offered
Training-Related Costs and Assistance	Services provided by [name of local HPOG program]	At a cost to the HPOG program	At no cost to the HPOG program		Not Offered
Book costs	☐	☐	☐	☐	☐
College fees (e.g., course fees, tuition support)	☐	☐	☐	☐	☐
Licensing and certification fees	☐	☐	☐	☐	☐
Exam/exam preparation fees	☐	☐	☐	☐	☐
Work/training uniforms, supplies, tools	☐	☐	☐	☐	☐
Computer/technology equipment (e.g., laptops, software, etc.)	☐	☐	☐	☐	☐
Other (Please specify: )	☐	☐	☐	☐	☐

Source: HPOG-NIE Grantee Survey, Question 9.17

Go to question 4.6a

4.6a. What type of job search and placement assistance does your organization and/or any of your partners provide to [name of local HPOG program] participants? Are they provided directly by your program, your partners (at a cost to the HPOG program or not), or other programs or community service agencies?

Question 6.11 in telephone interview protocol
If services are provided by partners, identify whether the service is provided at a cost to the program or not.

Data Sources(s): EDIP or site monitoring call notes

Grantee/HPOG Program Instructions: Information about job search and placement

assistance was collected from PAGES, the EDIP, and site monitoring call notes.

Please review the responses in the table below. If the information

is incorrect or left blank, please be prepared to discuss during the telephone

interview. Select all that apply.

If services are provided by partners, identify whether the service is provided at a cost to your program or not.

Job Search and Placement Assistance	Services provided by [name of local HPOG program]	Services provided by Partners		Services provided by other programs or community service agencies	Not Offered
Job Search and Placement Assistance	Services provided by [name of local HPOG program]	At a cost to the HPOG program	At no cost to the HPOG program		Not Offered
Job search skills workshops (e.g., group workshops on writing resumes and cover letters, conducting a job search, interviewing, etc.)	☐	☐	☐	☐	☐
Identifying job openings for program graduates	☐	☐	☐	☐	☐
Meeting with employers to identify job openings for graduates	☐	☐	☐	☐	☐
One-on-one job search assistance	☐	☐	☐	☐	☐
Advising on career and job choices	☐	☐	☐	☐	☐
Operating or providing referrals to job fairs	☐	☐	☐	☐	☐
Job screening (e.g., screen for suitability for a job)	☐	☐	☐	☐	☐
Other (Please specify: )	☐	☐	☐	☐	☐

Source: HPOG-NIE Grantee Survey, Question 9.21 and 9.24

Go to question 4.6b

4.6b. When do you provide these job search and placement assistance to [name of local HPOG program] participants? Are they offered before healthcare training, during healthcare training, during the job search, after job placement, or all of the above?

Interviewer/HPOG Site Team Instructions: Select all that apply for each service offered.

Question 6.11b in telephone interview protocol

Data Sources(s): EDIP or site monitoring call notes

Grantee/HPOG Program Instructions: : Information about job search and

placement assistance was collected from PAGES, the EDIP, and site monitoring call

notes. Please review the responses in the table below. If the information is

incorrect or left blank, please be prepared to discuss t during the telephone

interview. Select all that apply.

	Before healthcare training	During healthcare training	During job search	After Job Placement
Job search skills workshops (e.g., group workshops on writing resumes and cover letters, conducting a job search, interviewing, etc.)	☐	☐	☐	☐
Identifying job openings for program graduates	☐	☐	☐	☐
Meeting with employers to identify job openings for graduates	☐	☐	☐	☐
One-on-one job search assistance	☐	☐	☐	☐
Advising on career and job choices	☐	☐	☐	☐
Operating or providing referrals to job fairs	☐	☐	☐	☐
Job screening (e.g., screen for suitability for a job)	☐	☐	☐	☐
Other (Please specify: )	☐	☐	☐	☐

Source: HPOG-NIE Grantee Survey, Question 9.21 and 9.24

Go to question 4.7

4.7. What type of job retention (e.g., after job placement) services does your organization and/or any of your partners provide to [name of local HPOG program] participants? Are they provided directly by your program, your partners (at a cost to the HPOG program or not), or other programs or community service agencies?

If services are provided by partners, identify whether the service is provided at a cost to the program or not.

Data Sources(s): EDIP or site monitoring call notes

Grantee/HPOG Program Instructions: Information about job retention services was

collected from PAGES, the EDIP, and site monitoring call notes. Please review the

responses in the table below. If the information is incorrect or left blank, please be

prepared to discuss during the telephone interview.

If services are provided by partners, identify whether the service is provided at a cost to your program or not.

Job Retention Services	Services provided by [name of local HPOG program]	Services provided by Partners		Services provided by other programs or community service agencies	Not Offered
Job Retention Services	Services provided by [name of local HPOG program]	At a cost to the HPOG program	At no cost to the HPOG program		Not Offered
In-person meetings with participant	☐	☐	☐	☐	☐
Phone check-ins with participant	☐	☐	☐	☐	☐
Phone calls or meetings with participant’s supervisor	☐	☐	☐	☐	☐
Email check-ins with participant	☐	☐	☐	☐	☐
Social media (e.g., Facebook groups, LinkedIn)	☐	☐	☐	☐	☐
Host events for and/or group meetings with program graduates?	☐	☐	☐	☐	☐
Other (Please specify: )	☐	☐	☐	☐	☐

Source: HPOG-NIE Grantee Survey, Question 9.22

Thank you for your assistance reviewing the responses in preparation of the telephone interview. We greatly appreciate your participation in the pilot. With your help, we will have better information about the practices of participating HPOG programs across the nation.

C. First-Round Telephone Interviews

Study Overview and Consent

Health Profession Opportunity Grants (HPOG) 2.0 Implementation Study

Study Overview and Consent

As part of the HPOG study, we are asking grantee staff, partners, and stakeholders to participate in an interview conducted by phone to help us better understand the structure, operations, and services of their programs. The interview should take approximately 60 to 90 minutes to complete. It asks about your program background and context, organizations with which you collaborate, including their roles and responsibilities, and program activities such as outreach and recruitment, eligibility and intake, training, and support services. With your help we have identified the most appropriate staff, partners, and stakeholders to talk to us about each of the topic areas.

Your answers will be kept private. Information you provide will not be shared with other staff at your program or organization. Only the evaluation team will have access to the information you provide through this interview. Names will not be listed in any reports published and comments will not be attributed to grantees. Instead, your information will be combined with information provided by other grantees. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to a particular grantee. Your responses to these questions are also completely voluntary. We hope you will choose to complete all of the questions in the interview, but you may choose to skip any question you do not feel comfortable answering.

If you have any questions or concerns about participating in the interview, please let me know. Your verbal decision to participate in the interview will be considered as your consent. Do I have your verbal permission to participate in this interview?

☐ Yes

☐ No

Staff Positions and Roles

Before we get started, do you have any questions? To begin, could you/each person on the phone please introduce himself or herself?

Respondent Characteristics and Roles

1.1. What is/are your title(s)/position(s) as part of [name of local HPOG program, or name of partner organization working with the local HPOG program]?

Interviewer/HPOG Site Team Instructions: Select all that apply. If a title/position is not defined, please include it in the “Other” category below.

Programming Instructions: Expand the table to include up to thirteen respondents

	Respondent 1	Respondent 2	Respondent 3
Program Director	☐	☐	☐
Program Coordinator	☐	☐	☐
Case Manager	☐	☐	☐
Career Advisor/counselor (e.g., assistance with career and training, etc.)	☐	☐	☐
Academic advisor (e.g., assistance with course selection, tutoring, etc.)	☐	☐	☐
Marketing/recruitment specialist	☐	☐	☐
Intake coordinator/staff (e.g., eligibility verification and assessment, etc.)	☐	☐	☐
Job developer (e.g., assistance with job readiness, job search, job placement, etc.)	☐	☐	☐
Instructor (e.g., healthcare occupational training, basic skills, work readiness courses, etc.)	☐	☐	☐
Employer	☐	☐	☐
Community service organization/partner	☐	☐	☐
Academic Dean/Director of Healthcare Training(s)	☐	☐	☐
Other (Please specify: [textbox, line type=single, size=150 characters])	☐	☐	☐

Source: HPOG Master Interview Guide, Question 1 and 3

Go to question 1.2

1.2. How long have you been working in this/these position(s)?

Programming Instructions: Expand the table to include up to thirteen respondents

	Respondent 1	Respondent 2	Respondent 3
Years
Months

Source: HPOG Master Interview Guide, Question 2

Go to question 1.3

1.3. How long have you been working at [name of grantee institution]?

Programming Instructions: Expand the table to include up to thirteen respondents

	Respondent 1	Respondent 2	Respondent 3
Years
Months

Source: HPOG Master Interview Guide, Question 2

Go to question 1.4

1.4. How long have you been working as part of [name of local HPOG program]?

Programming Instructions: Expand the table to include up to thirteen respondents

	Respondent 1	Respondent 2	Respondent 3
Years
Months

Source: HPOG Master Interview Guide, Question 2

Go to “Program Administration”

Program Administration

Let’s now discuss how your HPOG program is organized and how you and your partners deliver services and training opportunities to participants.

Partnership Characteristics and Roles

2.1. Based on information collected by your HPOG study team, we have compiled a list of organizations that are involved with or have supported [name of local HPOG program]. This list currently includes basic skills and healthcare occupational training providers. Do you have any changes or updates to the organizations listed in the table? What type of organization is each partner? If partners include employers, please consider only employers who are involved beyond hiring graduates.

Interviewer/HPOG Site Team Instructions: This question and list of organizations was shared with respondents in advance of the telephone interview. Please note that PAGES only provides a list of organizations involved with basic skills or healthcare occupational trainings. If the respondent includes employers, please advise that they consider only employers who are involved beyond simply hiring graduates (e.g., select employers who provide work-based learning opportunities such as job shadowing, clinicals, etc.). If the respondent has any changes or updates, please delete or include additional organizations. Select only one answer for each organization.

Data Source(s): PAGES [Vendors: Columns Selected (Vendor ID, Name, Type of Organization)]

EDIP and Site monitoring call notes for new organizations

Organization	A	B	C	D	E	F	G	H	I
Organization	State government agency	Local government agency	Work Investment Board	One-Stop Career Center/ American Job Center	Community/ Technical college	Community and faith-based organizations (Non-profits)	For-profit or proprietary service/training provider	Healthcare Employer	Other (Please specify: [textbox, line type= single, size=50 characters])
a. [Organization Name]	☐	☐	☐	☐	☐	☐	☐	☐	☐
b. [Organization Name]	☐	☐	☐	☐	☐	☐	☐	☐	☐
c. [Organization Name]	☐	☐	☐	☐	☐	☐	☐	☐	☐

Source: HPOG-NIE Grantee Survey, Questions 1.1 and 5.1 and Master Interview Guide, Question 26

Go to question 2.2

2.2. What is each organization’s current role in [name of local HPOG program]?

Interviewer/HPOG Site Team Instructions: This question and list of partner organizations was shared with respondents in advance of the telephone interview. If the respondent has any changes or updates, please include them in the table below and select all that apply.

Data Source(s): PAGES [Training & Services: Columns Selected (Category, Vendor, Training/Support Type)]

EDIP and Site monitoring call notes for new organizations

Programming Instructions: Auto-populate names of organizations from question 2.1. Allow interviewers/HPOG site teams to hover over the terms in columns A-K (e.g., “planning and design of HPOG grant activities”) for examples

Organization	A	B	C	D	E	F	G	H	I	J	K
Organization	Planning and design of HPOG grant activities	Referral of applicants for services provided by [name of local HPOG program]	Marketing and Outreach	Curriculum Development	Healthcare Occupational Training	Basic and Other Skill Development Activities (non-occupational)	Academic Support	Personal/ Logistical Support	Employment Assistance Support	Work-Based Learning Opportunities	Direct Financial Support to [name of local HPOG program]
a. [Organization Name]	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐
b. [Organization Name]	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐
c. [Organization Name]	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐
d. [Organization Name]	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐	☐

Source: HPOG Stakeholder/Network Survey, Question 14

Key Examples:

^A Planning and design of HPOG grant activities (e.g., grant writing, letter of commitment, member of advisory/steering committee)

^B Referral of applicants for services provided (e.g., formal referral arrangement, initial screening of applicants, referral of current employees)

^C Marketing and outreach (e.g., printed materials available on-site, information available on partner’s website, mentions during presentations to

stakeholders, mentions during orientation for organization’s services, mentions during assessment and counseling session)

^D Curriculum development (e.g., offering examples of relevant curricula, providing feedback on draft curricula, writing modules for curriculum for different parts of the

program)