Instrument 4: HPOG 2.0 National Evaluation in-person implementation interviews

Supporting Statement for OMB Clearance Request

Instrument 4: HPOG 2.0 National Evaluation In-Person Implementation Interviews

National and Tribal Evaluation of the 2nd Generation of the Health Profession Opportunity Grants (HPOG)

0970-0462

September 2016

Submitted by:

Office of Planning,
Research & Evaluation

Administration for Children & Families

U.S. Department of Health
and Human Services

Federal Project Officers:

Nicole Constance, Hilary Forster, and Amelia Popham

Instrument 4-A: Employer Engagement

Site Visit Preparation for Interviewers

• Review responses from telephone interviews. The telephone interviews with grantees include information about the HPOG program’s approach to employment assistance and work-based learning opportunities and employer engagement. The exhibit below highlights the interview questions associated with both topics in the telephone interview protocol.

Exhibit 1: Interview Topics to Review in Preparation of Site Visits

Interview Topic Areas	Questions
Employment Assistance and Work-Based Learning Opportunities	6.11 – 6.14f
Employer Engagement	6.15a - 6.19b

• Review the Screening Interview to Identify Respondents for the HPOG 2.0 National Evaluation First-Round Telephone interviews. Create a list of the local HPOG program staff, key partners, and stakeholders with knowledge and expertise about “Employment Assistance and Work-Based Learning Opportunities” and “Employer Engagement”. Always interview the program director and include others on the list as well. This may include program coordinators, job developers, curriculum developers, instructors, employers or other related partners that have a role with respect to employer engagement (e.g., industry associations, staffing firms, mayor’s office staff, etc.).

• Select guide (staff or employer/partner) and customize interview topics by the type of respondent. All topic areas should be covered when interviewing program directors. Based on your list of respondents, tailor the topics and/or questions to the role and level/type of involvement of each respondent (e.g., instructors should not receive the same questions as job developers).

• Customize interview topics with information collected from the telephone interviews. Use the italicized notes to customize selected topics in the employer engagement guide to prepare in advance of the site visit.

HPOG 2.0 Implementation Study

Site Visit Guide on Employer Engagement

(for HPOG program staff)

Introductory statement to [Pre-fill with Respondent Type]: Thank you for taking the time to talk to us today. As you may know, the Administration for Children and Families (ACF) of the U.S. Department of Health and Human Services has funded the 2^nd Generation Health Profession Opportunity Grants (HPOG 2.0). ACF has also funded an evaluation study of HPOG 2.0; we are members of the research team. We are visiting HPOG 2.0 grantee programs to document in greater detail program design and implementation. The major purpose of this site visit is to gather more information about how your HPOG program engages employers. By “employer engagement,” we mean the involvement of employers in the oversight, development, operation, service delivery and/or hiring of program graduates. We plan to use the information in our research reports that describe the HPOG 2.0 initiative and analyze its results and effectiveness.

In preparation for our discussion today, we have worked with representatives from your program to identify the most appropriate staff, partners, and stakeholders to talk to us about employer engagement. We have also reviewed responses about employer engagement from telephone interviews we previously conducted with [name of local HPOG program] staff, management and partner organizations in [month and year of telephone interview], as well all other information about the [name of local HPOG program] program we already have available.

The interview will take about 60 minutes to complete. Interview questions are structured around six key topics:

1. Staffing,

2. Employer engagement activities,

3. Curricular and program design,

4. Work-based learning opportunities,

5. Hiring/placement, and

6. Lessons for the field

Before we begin, I would like to assure you that all of your responses will be kept private and used only for this research study. Your name will not appear in any written reports we produce. Also, the interview is voluntary and you may choose not to answer any specific question and may stop the interview at any time.

According to the Paperwork Reduction Act (PRA), an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0462 and it expires on xx/xx/xxxx. If you have comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, please send them to [Contact Name]; [Contact Address]; Attn: OMB-PRA (0970-0462).

Do you have any questions before we begin? I would first like to start by giving you and/or each individual a minute to introduce yourselves. Please tell me:

• Your name

• Your role within [name of local HPOG program]

I have some topics I would like to cover, but these interviews are meant to be exploratory. Please feel free to bring up additional points that you would like to make about employer engagement and how it contributes to your HPOG program.

1. Staffing

Let’s first talk about the staff involved in employer engagement.

1. Staff responsible

2. Other staff involved

3. Familiarity or experience with employer engagement

4. Familiarity or experience with curricular design, work-based learning, or hiring

3. Employer Engagement Activities

Let’s now talk about the ways you engage employers as part of [name of local HPOG program]. According to information collected from the telephone interviews, we understand that your employers provide the following activities or services: [Pre-fill with information from question 6.15a from the telephone interview protocol under Employer Engagement and information from PAGES]. Let’s discuss:

5. Activities that employers are involved in

6. Strategies for engaging employers

   1. Identifying employers

   2. Building relationships

   3. Engaging employers in programs

7. Timing of strategies, either as part of the grant application, pre-implementation program design and planning, or at distinct times during program operations

8. Goals for engaging employers

9. Changes in employer engagement strategies and activities since HPOG award

10. Reasons for changes

4. Curricular and Program Design

Now let’s discuss how employers are involved with curricula and program design. Based on information collected from the telephone interviews, employers have provided input into curricula or program design in the following ways: [Pre-fill with information from question 6.15c from the telephone interview protocol under Employer Engagement].

11. Strategies and timing for receiving employer input (informal and formal)

12. Examples of how

    1. employers help to design programs or curricula

    2. employers provide input and/or change programs or curricula

    3. employers provide feedback on program graduates

13. Other strategies for alignment with employer needs

14. Curricular requirements

    1. State mandates regarding program and curriculum design

    2. State requirements for making changes to programs

    3. Institutional requirements regarding program and curriculum design

    4. Institutional requirements regarding changes to programs

5. Work-Based Learning Opportunities

[Name of local HPOG program] provides the following types of work-based learning opportunities for HPOG participants: [Pre-fill with information from PAGES on available work-based learning opportunities].

15. About work-based learning offered

1. Type

2. Structure

3. Design for specific programs and participants

4. Timing

16. How established and maintained

17. Goals of work-based learning opportunities

18. Changes under HPOG

19. Relationship to licensure requirements

6. Hiring/Placement

We learned from the telephone interviews that [name of local HPOG program] provides the following job search and placement assistance to HPOG participants: [Pre-fill with information from question 6.12a from the telephone interview protocol under Employer Assistance and Work-Based Learning Opportunities].

20. Approach to engagement for hiring

21. Staffing hiring/placement

22. Difference in approach based on type of training, participant, or other factors

23. Biggest challenges

24. Most successful strategies

7. Lessons for the Field

I would like to end our discussion today on lessons learned for the field. We learned previously that your program has implemented the following strategies and/or innovative practices to engage employers: [Pre-fill with information from questions 6.17, 6.19a, and 6.19b from the telephone interview protocol under Employer Engagement].

25. Employer engagement outcomes

    1. Employer demand for trainees

    2. Employer feedback on trainees

    3. Repeat employer customers

    4. Comparison to other programs

26. Successful strategies for

1. Identifying potential partners

2. Building and maintaining relationships

3. Moving relationships to deeper partnerships

27. Challenges for

    1. Identifying potential partners

    2. Building and maintaining relationships

    3. Moving relationships to deeper partnerships

28. Areas for improvement

29. Advice for designing programs like [name of local HPOG program]

Thank you for your time and your thoughtful responses. Please feel free to contact us if you have additional information you’d like to communicate.

HPOG 2.0 Implementation Study

Site Visit Guide on Employer Engagement

(for employers and related partners)

Introductory statement to [Pre-fill with Employer or Partner]: Thank you for taking the time to talk to us today. As you may know, the Administration for Children and Families (ACF) of the U.S. Department of Health and Human Services has funded the 2^nd Generation Health Profession Opportunity Grants (HPOG 2.0). ACF has also funded an evaluation study of HPOG 2.0 and of HPOG 2.0 Research Team. We are members of the HPOG 2.0 Research Team. We are visiting HPOG grantees and partners to document in greater detail program design and implementation. The major purpose of this site visit is to gather detailed information about your organization’s involvement with [name of local HPOG program]. Today we’d like to ask about the ways [employer/partner name] has been involved with [name of local HPOG program]. We plan to use the information in our research reports that describe the HPOG 2.0 initiative and analyze its results and effectiveness.

In preparation for our discussion today, we have worked with representatives from [name of local HPOG program] to identify partners like [employer/partner name] that have been involved in their HPOG program.

The interview will take about 60-90 minutes to complete. Interview questions are structured around six key topics:

1. Staffing,

2. Employer engagement activities,

3. Curricular and program design,

4. Work-based learning opportunities,

5. Hiring/placement, and

6. Lessons for the field

Before we begin, I would like to assure you that all of your responses will be kept private and used only for this research study. Your name will not appear in any written reports we produce. Also, the interview is voluntary and you may choose not to answer any specific question and may stop the interview at any time.

According to the Paperwork Reduction Act (PRA), an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0462 and it expires on xx/xx/xxxx. If you have comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, please send them to [Contact Name]; [Contact Address]; Attn: OMB-PRA (0970-0462).

Do you have any questions before we begin? I would first like to start by giving you and/or each individual a minute to introduce yourselves. Please tell me:

• Your name

• Your role within [employer/partner name]

• How you have worked with [name of local HPOG program]

I have some topics I would like to cover, but these interviews are meant to be exploratory. Please feel free to bring up additional points that you would like to make about employer engagement and how it contributes to the HPOG program.

8. Staffing

Let’s talk first about the staff involved with [name of local HPOG program]

1. Staff from [employer/partner name] that are involved with [name of local HPOG program]

2. Roles/responsibilities

3. Main contact at [name of local HPOG program]

4. Other staff at [name of local HPOG program]

9. Employer Engagement Activities

Let’s now talk about the ways [employer/partner name] is involved in [name of local HPOG program]. According to information we collected from telephone interviews previously conducted with managers and staff at [name of local HPOG program], we understand that [employer/partner name] is involved in the following activities: [Pre-fill with information from question 6.15a from the telephone interviews under Employer Engagement and information from PAGES].

5. How employer/partner is involved in the program

6. Reasons and goals for involvement

7. History of involvement (e.g., length, how began, etc.)

8. Changes as a result of HPOG award

9. Context of involvement

   1. Shortages

   2. Local labor market conditions

   3. Changes in skills required

   4. Other examples

10. Recruitment and human resource challenges with respect to entry-level workforce

10. Curricular and Program Design

Now let’s discuss how [employer/partner name] is involved with curricula and program design. Based on information we collected from the telephone interviews, [employer/partner name] provided input into curricula or program design in the following ways: [Pre-fill with information from question 6.15c from the telephone interview protocol under Employer Engagement].

11. Input provided by [employer/partner name]

12. Frequency of input

13. Structure for input

14. Examples of suggested changes made

15. Feedback on HPOG graduates

16. State mandates regarding skills and training

11. Work-Based Learning Opportunities

We understand that [employer/partner name] provides the following types of work-based learning opportunities for HPOG participants: [Pre-fill with information from PAGES].

17. About work-based learning opportunities offered

    1. Type

    2. Structure

    3. Design for targeted programs and participants

    4. Length and intensity

    5. Supervision

    6. Licensing requirements

18. Inception and history

19. Number of opportunities

    1. At any given time

    2. Ever offered

    3. For HPOG program participants

20. Goals and reasons for offering opportunities

21. Changes under HPOG

12. Hiring/Placement

We learned that [employer/partner name] has hired [Pre-fill with number of HPOG graduates using information from PAGES] graduates. Let’s discuss:

22. Structure of hiring relationship

23. Length of hiring relationship

24. Frequency and nature of contact related to hiring

25. Difference in approach based on type of training and/or job position being hired for

26. Difference in relationship compared to other training programs

13. Lessons for the Field

I would like to end our discussion today on reflections and lessons learned from your involvement with [name of local HPOG program].

27. Satisfaction with program graduates

28. Missing skills

29. Difference in quality of the graduates compared to other programs

30. Challenges working with [name of local HPOG program]

31. Recommended changes and/or program improvements

32. Future plans for hiring from [name of local HPOG program]

33. Lessons learned from involvement with [name of local HPOG program]

34. Areas for improvement

35. Advice for those designing programs like [name of HPOG program]

Thank you for your time and your thoughtful responses. Please feel free to contact us if you have additional information you’d like to communicate.