HPOG-Impact Implementation interview guide for HPOG program management

Appendix E-3 OMB # 0970-0394

Expiration Date xx/xx/xxxx

Appendix E-3: HPOG-Impact Implementation Interview Guide for Program Management

HPOG Impact Evaluation

Interview Guide for Management

Introductory statement to respondents: We are members of the HPOG Research Team. We are visiting all of the HPOG grantees included in an impact study to document in greater detail program design and implementation. Today we’d like to ask about your activities and opinions as management staff for [name of HPOG program]. The major purpose of this interview is to gather more nuanced and detailed information about your role. In preparation for our discussion today we have reviewed the information that we have available from prior site visits, grant applications, your program’s annual reports to HHS, the recent Staff and Grantee surveys that you may have seen, and what we have learned over time from our ongoing monitoring calls.  We have taken this step to personalize the information we will be discussing regarding the program in which you are involved.  This will allow us to reduce the amount of information you will need to provide today.

The interview will take approximately 90 minutes to complete. Before we begin, I would like to assure you that all of your responses will be kept private and used only for this research study. Your name will not appear in any written reports we produce. Also, the interview is voluntary and you may choose not to answer any specific question.

According to the Paperwork Reduction Act (PRA), an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0394 and it expires xx/xx/xxxx. If you have comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, please send them to [Contact Name]; [Contact Address]; Attn: OMB-PRA (0970-0397). Do you have any questions before we begin?

1. Staff Position and Role

We’d like to begin by asking you about your role, responsibilities, and experiences at [name of local HPOG program].

1. What is your title? ______________________

2. How long have you been working in this position of [title from Q1] at [name of local HPOG program]?

_____ years _____ months

3. What is your educational background (e.g., degree(s), content area) and prior work experience?

4. What are your responsibilities as part of [name of local HPOG program]?

   1. Can you give me a brief description of your day-to-day activities?

2. HPOG Program Socio-Economic Context

We would now like to focus on some aspects of the socio-economic environment in which [name of local HPOG program] was developed and operates.

1. According to our information the [name of local HPOG program] primarily serves [INSERT GEOGRAPHIC AREA(s) SERVED]. Please confirm these are the local areas the HPOG program targets.

2. If there has been a change to the geographic area served, please describe these changes and why they occurred.

3. According to our information, the demographics of the population primarily served by [name of the local HPOG program] are [INSERT DEMOGRAPHICS OF POPULATION(S) SERVED].

4. If there has been a change to the demographics of the populations served, please describe these changes.

3. Employer & Labor Market Contexts

Now, we’d like to discuss in some detail the nature of the employer and labor market context and how it affects [name of local HPOG program] and its participants.

1. Please describe the general availability of healthcare training before the HPOG grant.

2. Outside of any additional healthcare training made available through HPOG funding, has the general availability of non-HPOG healthcare training changed over the grant period (and since random assignment began)?

3. According to our information, the local labor markets for your area(s) at the time of the HPOG grant application [fall 2010] were [INSERT INFORMATION ABOUT LOCAL LABOR MARKETS].

4. How has the labor market changed over the course of the grant period (and since random assignment began)?

4. Institutional Context, Program Administration, and Staffing

We would also like to know about the institutional context, program administration, and staffing at [name of local HPOG program].

Institutional framework

1. Please describe your organization’s experience, if any, implementing healthcare training and/or other sectoral training programs before HPOG?

2. Please describe your organization’s experience, if any, providing support services to TANF and other low-income populations before HPOG?

Partner/stakeholder networks

[Note to interviewer: Refer to summary table of partnering organizations sent to the respondent before the visit.] As you know, all HPOG grantees run their programs in collaboration with partnering organizations. This table summarizes our understanding of the organizations participating in your HPOG activities and the roles and responsibilities that each partnering organization has. Do you have any changes or updates to the table? If so, please describe how partnerships changed and why they changed.

1. Please describe your organization’s experience working with these organizations prior to HPOG.

2. If you could redesign your HPOG program, how would you change the structure of your partnerships with other organizations?

3. Are there specific aspects of your partnership arrangements that you believe have supported or facilitated the overall implementation of your HPOG program? If yes, please explain these factors/aspects and how they facilitated implementation.

4. Conversely, are there specific aspects of your partnership arrangements that you believe have hindered the overall implementation of your HPOG program? If yes, please explain these factors/aspects and how they hindered implementation.

Program Administration

1. How are program activities coordinated across the partner organizations?

2. How many people within your organization work on HPOG activities including those who have other non-HPOG responsibilities?

3. Were any of these people hired specifically to conduct HPOG activities? If yes, why (to increase existing staff capacity, or to add a new skill set to your organization’s capacity)?

4. Has your organization offered any professional development services to staff involved in HPOG? If yes, please describe the services offered and the purpose of each service.

5. Program Structure, HPOG Planning, and Start Up

We would now like to discuss in more detail the nature of the planning and start-up of [name of local HPOG program] and its structure.

1. The HPOG grant application required grant applicants to describe their objectives and need for assistance. Your organizations grant application listed: [INSERT NEED AND OBJECTIVES FROM GRANT APPLICATION]. Have the program’s objectives or the population’s need for assistance changed over the period of the grant? If yes, how and when did they change?

2. What was the process for designing your HPOG strategy?

1. Partnerships

2. Recruitment / population targeted

3. Enrollment and outcome goals

3. What factors most influenced the design of your HPOG program? (Probe: previous experience, grant requirements, resources)

6. Program Recruitment, Eligibility, and Intake

Now, we’d like to ask you questions about how the [name of local HPOG program] program is designed and implemented. Let’s begin with [name of local HPOG program]’s eligibility requirements, how participants are recruited, and the intake process.

Outreach and recruitment

1. [Note to Interviewer: If changes to target population and eligibility have not been covered in recent monitoring calls, ask:] According to information we have already collected through site visits and surveys, we understand your target population is [*] and your eligibility requirements for [name of local HPOG program] are [*]. Has anything about this information changed? If yes, how, when and why did they change?

2. According to our information, your organization used the following strategies to inform your community and potential participants about the program: [*]. [Note to Interviewer: If changes to outreach and recruitment have not been covered in recent monitoring calls, ask:] If there are changes, please describe how your outreach and recruitment strategies have changed, when they changed, and why.

3. Which strategies do you believe have been most/least effective (and why)?

4. Do you believe [name of local HPOG program] has the adequate resources to meet the marketing and recruitment needs? (If not, what more do you feel is necessary)?

5. Is there anything more your partner organizations could do to provide additional referrals (if yes, what)?

Eligibility and intake

1. [Note to Interviewer: If changes to intake or enrollment have not been covered in recent monitoring calls, ask:] The study team has worked with [name of local HPOG program] extensively to understand your intake and enrollment process and to modify it for the impact study. Has anything changed about your intake or enrollment processes since you started random assignment? If yes, please explain.

2. What characteristics or attributes do you believe successful participants possess? (Probe: academic record and achievement, existing support systems, high motivation, vehicle for transportation, employment experience, etc.)

3. Do you think [name of local HPOG program]’s intake process and eligibility criteria lead to enrollment of the most suitable participants to your program? Is there anything you would change? If yes, what would you change and why?

Comprehensive assessments

1. Your intake process required applicants to complete the following comprehensive assessments: [*]. [Note to Interviewer: If changes to assessments have not been covered in recent monitoring calls, ask:] Have there been any changes? If yes, when did the changes occur, what were they and why were they made?

2. Do you believe these assessments effectively screen applicants? If not, why?

3. How are assessment findings used by staff to tailor participant activities, training, education, and support services? Do you think the assessments work for this purpose? If not, why?

7. Program Components

The study team has worked with [name of local HPOG program] staff to describe and understand your program and has previously collected data about your program components through site visits and surveys. We’d like to quickly review each of your program components to ask if anything has changed since the beginning of random assignment. (Interviewer will have information from existing data sources – (e.g., NIE surveys) to confirm information on each program component.)

Core Curriculum - basic skills instruction

1. [Name of local HPOG program] offers the following basics skills trainings: [*].[Note to Interviewer: If changes to basic skills instruction have not been covered in recent monitoring calls, ask:] Have there been any changes to the basic skills instruction/activities you offer? If yes, when did the changes occur, what changed and why?

2. Do you believe the offered basic skills trainings successfully improve participants’ basic skills proficiency levels; would you change (expand/eliminate) any of the basic skills offerings if you could? If yes, in what area(s)?

3. Have there been any barriers to provision of basic skills instruction? If yes, what were/are they?

Core Curriculum - vocational training

1. [Name of local HPOG program] offers the following vocational trainings: [*]. If there are changes, please describe how vocational trainings changed, when they changed, and why.

2. Over the course of the HPOG grant, has your program changed in terms of the healthcare occupations targeted? If yes, please explain how it has changed, when it changed, and why.

3. Are you using any of the following strategies, if yes, please explain:

1. articulated pathways

2. contextualization

3. acceleration

4. flexible delivery

5. active learning

6. off-site training

7. modularization

4. Do you believe the offered vocational trainings effectively prepare participants for the local labor market? Why or why not?

Support Services

1. [Note to Interviewer: If changes to supports services have not been covered in recent monitoring calls, ask:] [Name of local HPOG program] offers the following support services: [*]. If there are any changes, please describe how support services changed, when they changed, and why.

2. How is participant need for support services identified?

3. Once a participant is receiving support services, how are they monitored (what is the process to determine when supports need to be added/eliminated/modified?

4. Among the support services offered, which do you believe has the greatest impact on participants training retention? Why?

5. If you could add or expand one support service to your HPOG training services what would it be? Why?

6. Have there been any barriers to use or provision of specific support services?

Academic and personal counseling

1. According to our information, [name of local HPOG program] provides these academic counseling and tutoring services: [*]. [Note to Interviewer: If changes to academic and personal counseling have not been covered in recent monitoring calls, ask:] Please describe any changes to this list and explain when the change was made, and why.

2. How are these counseling services delivered?

3. How effective do you think the academic and career planning counseling services are in guiding participants toward successful outcomes?

4. Would you change how academic and career planning counseling are delivered at [name of local HPOG program]? If yes, what would you change and why?

5. According to our information, [name of local HPOG program] provides these personal counseling services: [*]. Please describe any changes to this list and explain when the change was made and why.

6. How are these counseling services delivered?

7. How effective do you think the personal counseling services are in helping participants to complete training?

8. Would you change how personal counseling services are delivered at [name of local HPOG program]? If yes, what would you change and why?

Employment development and post-employment services

1. [Name of local HPOG program] offers the following employment development services: [*]. [Note to Interviewer: If changes to employment development and post-employment services have not been covered in recent monitoring calls, ask:] If there are changes, please describe how, when, and why employment development services changed.

2. Please describe how you establish and maintain contacts with employers. What are the most effective strategies for building and maintaining employer connections?

3. What do you believe are the most effective job placement strategies for your target population? Why?

4. What have been the most important challenges to placing your participants into the jobs they have trained for?

5. We understand that you offer the following services to your participants after they are placed in jobs [*]. If there are changes, please describe how, when, and why these services changed.

6. Have these services been effective in helping your graduates retain their jobs?

7. If there are instances in which your graduates either fail to find jobs to match their training or lose their jobs after leaving [name of local HPOG program], what do you think might be some of the reasons?

8. Program Enhancements¹

Now, we’d like to learn about the [*] program enhancement that you designed and implemented for the research study.

1. What was the rationale for selecting the specific enhancement you are currently implementing?

2. Have you introduced any changes to your enhancement since receiving approval of your proposal? If yes, what changes were made and why?

3. Have you been implementing all components of your approved enhancement?

[NOTE: Ask the following question in sites that have implemented peer support groups.]

4. Do all individuals selected for the peer support enhancement group participate?

   1. If not, why did participants decline to participate in the enhancement?

   2. What was your policy for participants who declined to participate in the enhancement?

5. In your opinion, what were the strengths and weaknesses of the enhancement? How have program participants reacted to the enhancement?

6. Were there any barriers to implementing the enhancement? If yes, what were they and how were they handled?

9. Changes to Program Design, Components, or Operation

1. Please describe any significant changes or evolutions in your program since its inception and since random assignment began that we have not yet discussed.

10. Control Group Services

Now, we’d like to ask you about alternative services in the community available to control group members.

1. According to information we have already collected we understand the services available to control group members are [*]. [Note to Interviewer: If control group services have not been discussed recently in monitoring calls, ask:] Has anything about this information changed? If yes, what changes were made, when were they made, and why?

2. Do you have a sense of how likely it is that control group members are eligible for and will attempt to access alternative services?

3. What are the barriers to accessing those alternative services for control group members?

4. What steps have been taken to ensure separation of core program components by treatment and control groups? Under what circumstances and for what services might treatment and control group members access the same services?

5. Have partner or stakeholder organizations had to assume more responsibilities for serving eligible control group members? How do you know/what have they done, to your knowledge? Has this caused problems in [name of local HPOG program]’s relationship with partner or stakeholder organizations? If yes, explain.

11. HPOG Program Successes, Challenges, and Lessons Learned

Now, we’d like to hear from you about [name of local HPOG program]’s successes, challenges, and lessons learned.

1. Overall, what do you believe are the program’s biggest strengths/weaknesses?

2. What do you think are the factors contributing to challenges/successes?

3. What strategies has your organization implemented to address these challenges?

4. In your opinion, is there anything more that could/should be done to address program challenges?

12. Systems Change and Sustainability

Now, I’d like to ask you some questions about whether and how [name of HPOG program] has facilitated any changes in the systems by which low-income workers are trained for career ladder jobs. We are going to discuss aspects of any of these systems changes (to the training of low-income workers) that the HPOG program has experienced.

1. Please describe any systems changes that have resulted from the HPOG grant (e.g., partnership network, training availability, etc.)

2. Please describe any efforts your organization is engaging in to sustain HPOG programming after the current HPOG funding for the effort ends.

3. Do you believe you will be able to sustain the program? Why or why not?

13. Evaluation

Finally, we’d like to ask you about your experiences participating in the HPOG-Impact Study.

1. What were the greatest challenges associated with participation in the study (e.g., random assignment, changes to intake, etc.)?

2. What did program staff do to ensure the integrity of assignment to treatment vs. control groups (and if appropriate, between treatment vs. treatment + enhancement)? What were the main challenges?

3. Do you believe study participation hindered program progress or implementation? If yes, how and why?

4. Do you feel staff members at [name of local HPOG program] were able to communicate with the study site team about any issues/problems that arose? What were the challenges?

I’d like to close our discussion by asking you 1) if there are ways in which you think [name of HPOG program] can improve anything about the overall design, trainings, or services and 2) if you have any general advice for those designing programs like [name of HPOG program].

Thank you for your time and your thoughtful responses. Please feel free to contact us if you have additional information you’d like to communicate.

NOTE to Interviewer: Provide respondent sheet with contact information

1 This module should only be asked of grantees implementing enhancements

▌pg. 1