HPOG-Impact Implementation interview guide for HPOG program staff

Appendix E-4 OMB # 0970-0394

Expiration Date xx/xx/xxxx

Appendix E-4: HPOG-Impact Implementation Interview Guide for Staff

HPOG Impact Evaluation

Interview Guide for Staff

Introductory statement to respondents: We are members of the HPOG Research Team. We are visiting all of the HPOG grantees included in an impact study to document in greater detail program design and implementation. Today we’d like to ask about your activities and opinions as staff for [name of HPOG program]. The major purpose of this interview is to gather more nuanced and detailed information about your role. In preparation for our discussion today we have reviewed the information that we have available from prior site visits, grant applications, your program’s annual reports to HHS, and the recent Staff and Grantee surveys that you may see. We have taken this step to personalize the information we will be discussing regarding the program in which you are involved. This will allow us to reduce the amount of information you will need to provide today.

The interview will take approximately 60 minutes to complete. Before we begin, I would like to assure you that all of your responses will be kept private and used only for this research study. Your name will not appear in any written reports we produce. Also, the interview is voluntary and you may choose not to answer any specific question.

According to the Paperwork Reduction Act (PRA), an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0394 and it expires xx/xx/xxxx. If you have comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, please send them to [Contact Name]; [Contact Address]; Attn: OMB-PRA (0970-0397). Do you have any questions before we begin?

I. Staff Position and Role

We’d like to begin by asking you about your role, responsibilities, and experiences at [name of local HPOG program].

1. What is your staff title? ______________________

2. How long have you been working in this position of [title from Q1] at [name of local HPOG program]?

_____ years _____ months

3. What is your educational background (e.g., degree(s), content area)?

4. What is your prior work experience?

5. What are your responsibilities as part of [name of local HPOG program] in the following areas?

• Marketing and recruitment

• Intake and enrollment

• Academic advising (e.g., assistance with course selection, tutoring, etc.)

• Non-academic advising (e.g., assistance with personal/financial supports and guidance)

• Career advising (e.g., assistance with career and employment choices)

• Employment assistance (e.g., job readiness, job search, job placement)

• Other (Please specify):______________________________________

a. Please briefly describe your typical day-to-day activities.

6. Do you believe you receive the support and training you need to successfully carry out these responsibilities?

a. What kinds of supports are available to you?

b. What kinds of professional development opportunities are available to you?

7. Do you believe [name of local HPOG program] is managed in a way that staff work well together to provide the necessary services to program participants?

a. What are the things that work well in your program and why?

b. What are the things that could use improvement and why?

II. Program Recruitment, Eligibility, and Intake

Now, we’d like to ask you questions about how the [name of local HPOG program] program is designed and implemented. Let’s begin with [name of local HPOG program]’s eligibility requirements, how participants are recruited, and the intake process.

Outreach and recruitment

1. According to information we have already collected through site visits and surveys, we understand your target population is [*] and your eligibility requirements for [name of local HPOG program] are [*]. Has anything about this information changed? If yes, how, when, and why did they change?

2. According to our information, your organization used the following strategies to inform your community and potential participants about the program: [*]. If there are changes, please describe how your outreach and recruitment strategies have changed, when they changed, and why.

3. Which strategies do you believe have been most/least effective (and why)?

4. Do you believe [name of local HPOG program] has the adequate resources to meet the marketing and recruitment needs? (If not, what more do you feel is necessary)?

5. Is there anything more your partner organizations could do to provide additional referrals (if yes, what)?

Eligibility and intake

1. The study team has worked with [name of local HPOG program] extensively to understand your intake and enrollment process and to modify it for the impact study. Has anything changed about your intake or enrollment processes since you started random assignment? If yes, please explain.

2. What characteristics or attributes do you believe successful participants possess? (Probe: academic record and achievement, existing support systems, high motivation, vehicle for transportation, employment experience, etc.)

3. Do you think [name of local HPOG program]’s intake process and eligibility criteria lead to enrollment of the most suitable participants to your program? Is there anything you would change? If yes, what would you change and why?

4. What are the primary reasons that result in interested applicants not completing the full intake process?

Comprehensive assessments

1. Your intake process requires applicants to complete the following comprehensive assessments: [*]. Have there been any changes? If yes, when did the changes occur, what were they and why were they made?

2. Do you believe these assessments effectively screen applicants? If not, why?

3. How are assessment findings used by staff to tailor participant activities, training, education, and support services? Do you think the assessments work for this purpose? If not, why?

4. If you work with participants on academic and career plan development during the intake process, how does it work?

a. If you do not work with participants on academic and career plan development during the intake process, do participants receive this training, and if yes, how?

III. Program Components

The study team has worked with [name of local HPOG program] staff to describe and understand your program and has previously collected data about your program components through site visits and surveys. We’d like to quickly review each of your program components to ask if anything has changed since the beginning of random assignment. (Interviewer will have information from existing data sources (e.g., NIE surveys) to confirm information on each program component.)

Core curriculum—basic skills instruction

1. [Name of local HPOG program] offers the following basics skills trainings: [*]. Have there been any changes to the basic skills instruction/activities you offer? If yes, when did the changes occur, what changed and why?

2. Do you believe the offered basic skills trainings successfully improve participants’ basic skills proficiency levels; would you change (expand/eliminate) any of the basic skills offerings if you could? If yes, in what area(s)?

3. Have there been any barriers to the provision of basic skills instruction? If yes, what were/are they?

Core curriculum—vocational training (ask questions below if appropriate to the staff member’s position, role and knowledge)

1. [Name of local HPOG program] offers the following vocational trainings: [*]. If there are changes, please describe how vocational trainings changed, when they changed, and why.

2. Over the course of the HPOG grant, has your program changed in terms of the healthcare occupations targeted? If yes, please explain how it has changed and why.

3. In your provision of vocational training are you using any of the following instructional strategies? If yes, please explain:

a. articulated career pathways /ladders

b. contextualization

c. acceleration

d. flexible delivery

e. active learning

f. off-site training

g. modularization

4. Do you believe the offered vocational trainings effectively prepare participants for the local labor market? Why or why not?

Support services

1. [Name of local HPOG program] offers the following support services: [*]. If there are any changes, please describe how support services changed, when they changed and why.

2. How is participant need for support services identified?

3. Once a participant is receiving support services, how are they monitored (what is the process to determine when supports need to be added/eliminated/modified)?

4. Among the support services offered, which do you believe have the greatest impact on participants’ training retention? Why?

5. If you could add or expand the availability of one support service to your HPOG training services what would it be? Why?

6. Have there been any barriers to use or provision of specific support services?

Academic and personal counseling

1. According to our information, [name of local HPOG program] provides these academic counseling and tutoring services: [*]. Please describe any changes to this list and explain when the change was made, and why.

2. How are these counseling services delivered?

3. How effective do you think the academic and career planning counseling services are in guiding participants toward successful outcomes?

4. Would you change how academic and career planning counseling are delivered at [name of local HPOG program]? If yes, what would you change and why?

5. According to our information, [name of local HPOG program] provides these personal counseling services: [*]. Please describe any changes to this list and explain when the change was made, and why.

6. How are these counseling services delivered?

7. How effective do you think the personal counseling services are in helping participants to complete training?

8. Would you change how personal counseling services are delivered at [name of local HPOG program]? If yes, what would you change and why?

Employment development and post-employment services

1. [Name of local HPOG program] offers the following employment development services: [*]. If there are changes, please describe how, when, and why employment development services changed.

2. Please describe how you establish and maintain contacts with employers. What are the most effective strategies for building and maintaining employer connections?

3. What do you believe are the most effective job placement strategies for your target population? Why?

4. What have been the most important challenges to placing your participants into the jobs they have trained for?

5. We understand that you offer the following services to your participants after they are placed in jobs [*]. If there are changes, please describe how, when, and why these services changed.

6. Have these services been effective in helping your graduates retain their jobs?

7. If there are instances in which your graduates either fail to find jobs to match their training or lose their jobs after leaving [name of local HPOG program], what do you think might be some of the reasons?

IV. Program Enhancements

Now, we’d like to learn about the [*] program enhancement that you designed and implemented for the research study.

1. What was the rationale for selecting the specific enhancement you are currently implementing?

2. Have you introduced any changes to your enhancement since receiving approval of your proposal? If yes, what changes were made and why?

3. Have you been implementing all the components of your approved enhancement?

[NOTE: Ask the following question in sites that have implemented peer support groups.]

4. Do all the individuals selected for the peer support enhancement group participate?

a. If not, why did participants decline to participate in the enhancement?

b. What was your policy for participants who declined to participate in the enhancement?

5. In your opinion, what are the strengths and weaknesses of the enhancement? How have program participants reacted to the enhancement?

6. Were there any barriers to implementing the enhancement? If yes, what were they and how were they handled?

V. Changes to Program Design, Components, or Operation

1. Please describe any significant changes or evolutions in your program since its inception and since random assignment began that we have not yet discussed.

VI. Control Group Services

Now, we’d like to ask you about alternative services in the community available to control group members.

1. According to information we have already collected we understand the services available to control group members are [*]. Has anything about this information changed? If yes, what changes were made, when were they made, and why?

2. Do you have a sense of how likely it is that control group members are eligible for and will attempt to access alternative services?

3. What are the major barriers to accessing those alternative services for control group members?

4. What steps have been taken to ensure separation of core program components by treatment and control groups? Under what circumstances and for what services might treatment and control group members access the same services?

5. Have partner or stakeholder organizations had to assume more responsibilities for serving eligible control group members? How do you know/what have they done, to your knowledge? Has this caused problems in [name of local HPOG program]’s relationship with partner or stakeholder organizations? If yes, explain.

VII. Employer & Labor Market Contexts

Now, we’d like to discuss in some detail the nature of the employer and labor market context and how it affects [name of local HPOG program] and its participants.

1. Do you think the skills participants gain in [name of local HPOG program] effectively match employer needs in the area? If not, why?

2. In general, what feedback about your employed participants do you get from their employers?

3. Do you think your graduates are faring well in the current healthcare labor market? If not, why?

VIII. HPOG Program Successes, Challenges, and Lessons Learned

Now, we’d like to hear from you about [name of local HPOG program]’s successes, challenges, and lessons learned.

1. How do you monitor participant retention and completion?

2. What are participants’ biggest barriers to program completion?

3. What measures do you take to improve participant retention and completion and how do you assess the effectiveness of these measures?

4. Overall, what do you believe are the program’s biggest strengths/weaknesses?

5. What do you think are the factors contributing to challenges/successes?

6. In your opinion, is there anything more that could/should be done to address program challenges?

IX. Evaluation

Finally, we’d like to ask you about your experiences participating in the HPOG-Impact Study.

1. What were the greatest challenges associated with participation in the study (e.g., random assignment, changes to intake, etc.)?

2. What did program staff do to ensure the integrity of assignment to treatment vs. control groups (and if appropriate, between treatment vs. treatment + enhancement)? What were the main challenges?

3. Do you believe study participation hindered program operations or success? If yes, how and why?

4. Do you feel staff members at [name of local HPOG program] were able to communicate with the study site team about any issues/problems that arose? What were the challenges?

I’d like to close our discussion by asking you 1) if there are ways in which you think [name of HPOG program] can improve anything about the overall design, trainings, or services and 2) if you have any general advice for those designing programs like [name of HPOG program].

Thank you for your time and your thoughtful responses. Please feel free to contact us if you have additional information you’d like to communicate.

NOTE to Interviewer: Provide respondent sheet with contact information

▌pg. 1