HPOG-NIE Performance Measurement Semi-Structured Discussion Guide

Supporting Statement for OMB Clearance Request

Appendix F: HPOG-NIE Semi-Structured Discussion Guide

National Implementation Evaluation of the Health Profession Opportunity Grants (HPOG) to Serve TANF Recipients and Other Low-Income Individuals and HPOG Impact Study

0970-0394

November 2014

Submitted by:

Office of Planning,
Research & Evaluation

Administration for Children & Families

U.S. Department of Health
and Human Services

Federal Project Officers:

Hilary Forster and Mary Mueggenborg

Appendix F: HPOG-NIE Semi-Structured Discussion Guide

Advance Invitation for Interview

Grantees’ Use of Performance Measurement Information in the HPOG Program

Dear [Name of grantee representative]:

Thank you for recently filling out our questionnaire for the Health Profession Opportunity Grants (HPOG) evaluation giving feedback about how your program has reached decisions to make program changes to achieve your HPOG objectives.

After reviewing all grantees’ questionnaire responses and additional evaluation materials, your grantee has been selected for a follow-up telephone interview to find out more about your organization’s decision making process related to making changes to your program to achieve your HPOG objectives.

Your answers will be kept private to the extent permitted by law. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you. Only the HPOG evaluation team will have access to the information you provide through this interview. Your name will not be listed in any reports published, and comments will not be attributed to you. Your participation in this interview is completely voluntary, but it is important that we have as much input as possible to ensure accurate evaluation of these programs.

To guide our conversation about your decision-making process for making program changes, the study team is sending information –attached to this email– taken from your semiannual program performance report (PPR) and elsewhere in the performance reporting system (PRS), other program and evaluation documents, and the recent questionnaire. This information identifies some changes to your HPOG program that we would like to discuss during the telephone interview.

A member of the Urban Institute study team will be contacting you to schedule a time for the interview, but if you have any questions about the interview or the study, please contact xxxx at xxx@urban.org (202-261-xxxx).

Thank you in advance for your assistance in participating in this interview and providing important information to the study. We appreciate all you and your organization have done to assist in the evaluation of HPOG. With your help, we will have better information about the practices of participating HPOG programs across the nation.

Sincerely,

HPOG Evaluation Team

The Paperwork Reduction Act Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is xxxx-xxxx, and it expires xx/xx/xxxx. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to [Contact Name]; [Contact Address]; Attn: OMB-PRA (xxxx-xxxx).

Draft Telephone Interview Guide

Grantees’ Use of Performance Measurement Information in the HPOG Program

Introduction:

Thank you for taking the time to talk with us today.

I am ____________, from the Urban Institute, part of the team conducting the national evaluation of the HPOG Program for the U.S. Department of Health and Human Services. We want to thank you for taking the time to talk with us today.

We want to talk with you today about the decision making process used to make changes to [name of HPOG program] to achieve HPOG objectives. In the email we sent you to schedule this call, we identified some changes in your program’s targets or practices. These were taken from information you provided in the recent questionnaire we sent you as well as from review of your program performance reports (PPRs) and other information you have provided or submitted previously.

This interview will take approximately 60 minutes. Before we begin, I would like to assure you that all of your responses will be kept private to the extent permitted by law and used only for this research study. Information you provide will not be shared with other program staff and will not be attributed to particular individuals or specific HPOG programs. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you. Your participation in this interview is completely voluntary, but it is important that we have as much input as possible to ensure accurate evaluation of these programs.

We ask you to answer the interview questions as accurately as possible. Feel free to ask me to repeat a question or define a term. Do you have any questions before we begin?

Semi-Structured Discussion Guide

Q1. First, we’d like to hear about what you consider to be important changes made to your HPOG program since it began. Please identify some examples of important changes.

[Interviewer: for each change in turn, probe for type of change made, why they feel it was an important change, the information used/decision making process that led to the change, what happened after the change was made, and whether the change had the desired effect – as in Q2a - Qc]

Q2. We identified a few changes made to your HPOG program from the information we have, and would like to find out more about why these changes were made.

Q2a. [Interviewer: identify a change and approximate timeframe and whether it was identified from their response to the screening questionnaire or from review of PPRs or other documents.] What led you to make that change? [Interviewer: probe for specific types of information that led to making that change.]

Q2b. [If not mentioned in response to 2a] Did you use performance information in deciding to make that change? If so, what kind of performance information did you use?

Q2c. Did this change have the desired effect? Why do you say that? [probe for what happened after the change was made and evidence of desired effect, or reason they feel it did not have the desired effect]

[Interviewer - After obtaining response for first change, repeat sequence of questions for subsequent pre-identified changes.]

Q3. Are there any other important changes that were made to program practices, policies, program focus, program structure, goals or target numbers that we haven’t talked about yet? If yes:

Q3a. What was the change? [For each change mentioned, ask:]

Q3b. What led you to make that change? [Interviewer – probe as in Q2 for types of information used]

Q3c. [If not mentioned in response to 3b] Did you use performance information in deciding to make that change? If so, what kind of performance information did you use?

Q3d. Did this change have the desired effect? Why do you say that? [Seek information about what happened after the change was made and evidence of desired effect, or reason they feel it did not have desired effect]

[Interviewer: after discussing first additional change mentioned, ask if there are any others and obtain same information for each.]

Q4: Has your program used the information obtained from the PPR or PRS for any of the following purposes: [Interviewer: screen out any of the following topics that have already been covered in previous questions.]

Q4a. Helping motivate your staff?

Q4b. Providing information about your program to your Board or the public?

Q4c. Fund raising/sustainability efforts?

Q4d. Developing internal targets for your program?

Q4e. Keeping track of participant’s progress?

[Interviewer: For each of the above uses identified by the respondent, ask for specifics as to how the information was used and how useful the information was in bringing about the desired effect.]

Q5: Has the way you or other HPOG program staff used performance information changed since the start of your HPOG program?

[If yes]:
Q5a. In what way has it changed? Has it been for the better or the worse? [Interviewer: Probes if needed – is it used more, or less? Used in different ways? More or different staff involved in using it?]

Q5b. Are there other ways you would like to change the way performance information is used? [explain]

[If no:]
Q5c.Would you like to change the way you or your colleagues use performance information? [If yes:] What changes would you like to make?

Q6. Thinking back to your past experience in programs other than HPOG, how does the usefulness of the HPOG performance information system compare to other systems you have used? [Interviewer – probe for why they said that].

Q6a. Has your previous experience with performance information systems affected the way you have used performance information in HPOG? [If yes:] In what ways? Has it affected use for the better or the worse?

Q7: What other performance information that the PRS does not currently collect would be helpful for program management? How would that information be helpful?

Q8: Is there anything else you’d like to tell me about your experience using performance information to make program changes or for program management?

On behalf of ACF, thank you for taking the time to participate in this interview!