Consent and guidance

OMB Control Number: 0990-0421

Expiration Date: October 12, 2020

Attachment B: Semi-Structured Discussion Guides, including Consent Scripts

Semi-Structured Interviews with Program Administrator/Managers and Frontline Staff: Consent Script and Discussion Guides

Script for Verbal Informed Consent for both Program Administrators/Managers and Frontline Staff

Thank you for taking the time to speak with us today. The U.S. Department of Health and Human Services, referred to as HHS, has contracted with our company, Mathematica, to study virtual human services delivery. By virtual, we mean something other than in person – such as video, phone, text, email, etc.

The purpose of the study is to explore implementation and lessons learned from a wide range of human services programs that moved to virtual service delivery during the COVID-19 pandemic. We want to begin to understand the circumstances, programs, and service types for which virtual delivery may be as (or even more) effective as in-person delivery, and those for which it is not.

We are asking for your participation in this 60-minute interview so that we can learn about your professional perspective on [ORGANIZATION’S/AGENCY’S] virtual delivery of human services.

For this study, Mathematica researchers will also interview [FRONTLINE STAFF/PROGRAM ADMINISTRATORS AND MANAGERS] and conduct focus groups with program participants. We will analyze the information that we collect and summarize our findings in a memo to HHS. HHS may then publish a brief highlighting the key themes that emerged from the interviews and focus groups, including challenges, successes, and lessons in virtual human services delivery. It will not include your name or other personally identifying information. It will include only generic professional roles.

There are no known risks to respondents in this study. Participating in the study may not provide a direct benefit to you, but the information you provide will promote a better understanding of virtual service delivery for programs across the country like yours.

Participation in the study is completely voluntary, and you may stop your participation in the study at any time or refuse to answer specific questions. Your refusal to participate or your withdrawal from the study will not result in any penalty. There is no compensation for participating.

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-0421. The time required to complete this information collection is estimated to average 60 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information.

We would also like your consent to record the interview to ensure that we have captured your responses accurately for our analysis. If you do not agree to the recording, you can still participate in the interview. All recordings will be stored on Mathematica’s secure network, and only the Mathematica research team will have access to them. We will destroy the audio recording at the end of this study.

Do you have any questions about the study?

If you think of questions about the study after the interview, you can contact me [INTERVIEWER] at [CONTACT EMAIL AND PHONE NUMBER] at any time. Before we can begin, I need to confirm your participation by asking you two questions:

• Do you agree to participate in the study? [YES/NO]

• [If yes] Do you agree to the audio recording of the interview? [YES/NO]

Semi-Structured Discussion Guide for Program Administrators/Managers

Instructions for interviewers

Tailor the following interview guide by (1) skipping sections or questions not relevant for the respondent; or (2) modifying questions as needed to reflect his/her role, his/her professional perspective/knowledge, or information obtained before the interview. Use the follow-up questions or probes as needed to obtain additional information or clarification. If you are running out of time, prioritize the most important questions.

You should be familiar with basic information available publicly online about the program and agency/organization, so you should select and/or tailor questions, as relevant, to probe for the respondent’s insight into their experiences and lessons with virtual human services delivery.

Before you begin the interview, read the verbal consent script (see above) to describe the study to respondents, answer any questions they have about the study, and ask for their consent to participate in the study and to audio-record the interview.

A. Roles and Responsibilities

1. Can you please tell us about your position and role with [ORGANIZATION/AGENCY]?

B. Virtual Service Delivery Overview

2. What specific programs is [ORGANIZATION/AGENCY] currently delivering virtually?

3. Focusing on [ONE PROGRAM BEING DELIVERED VIRTUALLY], what aspects of the service delivery continuum for those programs are you delivering virtually?

Prompt if needed:

1. Outreach

2. Intake

3. Screening/assessment

4. Referrals

5. Case management

6. Services/intervention

7. Follow-up

8. Prevention

1. Can you tell us how [ORGANIZATION/AGENCY] is delivering services virtually? What mechanism(s) are you using?

Prompt if needed:

1. Mode - video, phone, text, email, other

2. Platform – e.g. FaceTime, Zoom, custom platform, etc.

1. To what extend did you offer any of these services virtually before COVID-19?

2. Are there any services that your agency delivered remotely during your State’s or community’s stay at home order but are now starting to deliver in person again?

C. Policies

7. What new or altered policies have you put in place to address the shift to virtual service delivery?

Prompt if needed: Policies related to, for example:

1. Documentation/signatures

2. Staffing and caseloads

3. Resource allocation

4. Monitoring and supervision

5. Frequency/length of service interactions/appointments

6. Training

7. Health, safety, and privacy protections

1. To what extent have you had challenges keeping policies relevant and flexible? Have certain policies been more challenging to design or implement?

2. Do you expect to keep any of the new or updated policies in place in the long term?

D. Successes and Challenges

10. How do you think about or define success in delivering services virtually?

11. What would you say have been [ORGANIZATION’S/AGENCY’S] biggest successes in delivering human services virtually?

Prompt if needed: e.g.

1. Service quality/content

2. Participant engagement/participation

3. Participant satisfaction

4. Service impacts/outcomes for participants

5. Staff satisfaction

6. Staff engagement

7. Impacts/outcomes for the organization/agency

1. What has contributed to these successes?

2. Which strategies for delivering services virtually appear to have been most cost-effective or efficient?

3. What would you say have been the major challenges for [ORGANIZATION/AGENCY] in delivering services virtually?

Prompt if needed: e.g.

1. Service quality/content

2. Participant engagement/participation

3. Participant satisfaction

4. Service impacts/outcomes for participants

5. Staff satisfaction

6. Staff engagement

7. Impacts/outcomes for the organization/agency

1. What contributed to these challenges?

2. How has your [ORGANIZATION/AGENCY] sought to overcome or mitigate any of these challenges?

E. Evidence of Effectiveness

17. Which services do you see as most effective in a virtual context, compared with in person?

Prompt if needed: [LIST ANY MENTIONED UNDER SECTION B]

18. Which services do you see as less effective when delivered virtually compared to in person? Similarly effective?

19. Which aspects of service delivery do you see as most effective in a virtual context, compared with in person?

Prompt if needed: e.g.

1. Outreach

2. Intake

3. Screening/assessment

4. Referrals

5. Case management

6. Services/intervention

7. Follow-up

8. Prevention

1. Which aspects of service delivery do you see as least effective in a virtual context? Similarly effective?

   1. Should some services always be delivered in person if possible?

2. For which populations do you think virtual services are most effective? Least effective?

3. Are there certain contexts or circumstances under which you think virtual services are more effective than in person? Less effective?

Prompt if needed: e.g.

1. Geography (e.g. rural vs. urban vs. suburban vs. tribal)

2. Family characteristics (e.g. having small children, substance use disorder)

3. Staff demographics or experience

1. What are the downsides to delivering virtual services for families/clients? Upsides?

2. On what information are you basing these assessments?

Prompt if needed: e.g.

1. Staff perceptions

2. Program outcome data

3. Participation rates

4. Other

1. Do you have any plans for or interest in trying to measure anything further in the medium or long term about virtual services?

F. Lessons Learned

26. To what extent has delivering services virtually had an impact on [ORGANIZATION’S/AGENCY’S] ability to engage in cross-program collaboration? How so?

27. What adjustments have you made (or do you plan to make) to [ORGANIZATION’S/AGENCY’S] initial delivery of human services virtually?

28. What recommendations would you make for other [COMMUNITY/STATE]-level agencies about how to best support virtual services?

29. Do you have any other lessons learned about delivering services virtually?

G. Next Steps

30. To what extent do you view virtual delivery of human services as a long-term trend vs. a short-term response to COVID-19? Why?

    1. Which, if any, aspects of service delivery do you hope to continue offering virtually in the long term?

31. What additional research would be most helpful to you as you think about ways to design and deliver services virtually in the future?

32. Do you have recommendations for others we should consider speaking with, such as colleagues who work as program managers or frontline staff, or participants receiving virtual services?

33. Is there anything else you’d like to add?

Semi-Structured Discussion Guide for Frontline Staff

Instructions for interviewers

The following semi-structured discussion guide is designed to be tailored as appropriate to the professional perspective and knowledge of respondents. Possible probes for questions will be used as needed to obtain additional information or clarification. Due to time constraints, you may prioritize and skip some questions.

In a few sites, where there are several practitioners whom we would like to include in our study, we may conduct small group interviews instead of individual interviews. In those cases, prioritize the most relevant questions for discussion.

You should be familiar with basic information available publicly online about the program and agency/organization, so you should select and/or tailor questions, as relevant, to probe for the respondent’s insight into their experiences and lessons with virtual human services delivery.

Before beginning, read the verbal consent script (see above) to provide information about the study, answer any questions from participants about the study, and ask consent questions for participation in the study and for audio-recording the interview.

1. Work Experience, Current Responsibilities, and Community

1. Can you please tell us about your position and role/responsibility with [EMPLOYER NAME]?

Prompt if needed:

1. How long have you worked in this position?

2. What is your caseload? How many families or clients do you work with?

3. What are your responsibilities? What types of services do you provide to the [FAMILIES/CLIENTS/CHILDREN/YOUTH] you work with?

1. Please tell us about your educational background and how many years you have worked in this field.

2. Please tell us about the community you work in and the population you serve.

Prompt if needed:

1. Is the community you work in considered rural, urban, suburban?

2. What would you say are the defining characteristics of the community?

3. Who are the clients or families your agency serves?

B. Virtual Human Services

4. What specific services do you currently deliver virtually/remotely? How does this compare with service delivery prior to the COVID-19 health emergency and during any community or State stay-at-home orders?

Prompt if needed:

1. What types of services do you deliver remotely? For example, outreach, prevention, intake, screening/assessment, referrals, case management, intervention, follow-up.

2. Can you describe a typical remote service delivery session, for example for [e.g. SCREENING, INTERVENTION]? [depending on the program, probe for more specific details on the services]

3. Prior to the COVID-19 health emergency were you providing any of these services remotely?

4. Are there any services that you delivered remotely during your State’s or community’s stay at home order but are now starting to deliver in person again?

5. How do you deliver these services remotely?

1. What technology and devices do you use to provide services remotely? For example, telephone, smartphones, computers, tablets, etc. Has this been a problem for any clients?

2. Do you and the family/client need reliable internet/broadband for remote service delivery? Has this been an issue?

3. How often do you remotely connect with the families/clients you work with?

4. How long is a typical remote service delivery session? [could ask for a specific time or a range]

5. Overall, would you say delivering services remotely is saving you time (so you could serve more families/clients), taking about the same amount of time, or taking more time for each family/client?

C. Benefits and Challenges with Virtual Human Services Delivery

6. What are the 3 or 4 major benefits and successes in delivering services virtually?

Prompt if needed:

1. What do you gain (successes, benefits) by delivering services virtually? What do the families/clients you serve gain?

2. What has contributed to these successes?

7. What are the 3 or 4 major challenges with delivering human services virtually? Have you found any strategies to help overcome these challenges?

Prompt if needed:

1. What have been the biggest challenges for you in delivering services remotely?

2. What do you lose when delivering services virtually? What do families/clients lose?

3. What has contributed to these challenges?

4. Have you identified any strategies to try to overcome or minimize these challenges?

7. Has providing services virtually had an impact on your ability to collaborate with other human services programs in your community? If so, how?

Prompt if needed:

1. Do you regularly connect with service providers from other programs? If yes, please describe.

2. Do you regularly refer the families or clients you serve to other programs? If yes, please describe.

3. Has moving to more virtual service delivery had a positive or negative impact on your collaboration with other programs? Please share an example or two.

D. Effectiveness of Virtual Human Service Delivery

7. How do you and/or your agency think about/define success with your service delivery?

Prompt if needed:

1. What are the outcomes you hope to achieve? E.g. number of clients served, quality of services delivered, specific outcomes for your clients, etc.

2. What outcomes are you held accountable for?

7. How do you or your agency assess the effectiveness of virtual service delivery?

8. Which services that you provide do you consider to be more effective when delivered virtually? Less effective? Similar? Most cost-effective?

Possible prompts if needed: [could probe for different aspects of service delivery including outreach, prevention, intake, screening/assessment, referrals, case management, intervention, follow-up]

1. Are there services that should always be delivered virtually?

2. Are there services that should never be delivered virtually?

3. Are there services that it really doesn’t matter if they are delivered virtually or in-person?

7. Are there clients/families or specific contexts where virtual service delivery is more or less effective?

Prompt if needed:

1. Are there clients or families that you should maximize or always provide services virtually?

2. Are there clients or families that you should minimize or never provide services virtually?

3. Are there certain situations where virtual service delivery is preferable and effective?

4. Are there certain situations where virtual service delivery is not effective?

E. Lessons Learned from Providing Services Virtually

13. What lessons would you share with other services providers about virtual service delivery?

Prompt if needed:

1. What adjustments did you make when you initially changed to virtual service delivery?

2. What recommendations would you make to other agencies or programs considering expanding or keeping virtual human services?

3. Do you have any recommendations for the U.S. Department of Health and Human Services or other federal agencies about how to best support virtual human services?

14. Which, if any, types or aspects of service delivery do you hope to continue offering virtually long into the future?

15. Do you have any questions about virtual service delivery that research could help answer?

Prompt if needed:

1. How could research help you make informed decisions about your approach to service delivery?

F. Final Thoughts

16. Is there anything else we should know related to your shift to providing services virtually?

Semi-Structured Focus Groups with Program Participants: Consent Script and Discussion Guide

Script for Verbal Informed Consent

[Directions for Interviewers]: Read the script to the participant(s) and ask each participant to respond to the two requests for consent: (1) for study participation and (2) for permission to record the focus group. Do not start recording until after the participants consent. Do not record the focus group if you do not have the participants’ consent to so. In that case, be prepared for the notetaker to take detailed notes.]

Thank you for taking the time to speak with us today. The U.S. Department of Health and Human Services (referred to as HHS) has contracted with our company, Mathematica, to study the transition to virtual service delivery.

This focus group discussion will help HHS understand lessons about virtual service delivery (getting services through video, phone, text, email, IM, web, etc. – in other words, not in person). We want to learn about your experiences receiving virtual services like child protective services, Head Start, TANF cash assistance, home visiting, healthy marriage and parenting services. HHS wants to understand which types of human services can work well virtually.

We will analyze the information that we collect from today’s focus group and other focus groups and summarize it in a report for HHS. This report will highlight the key themes that came out of the focus groups, such as the benefits and challenges delivering human services virtually. The report will not include your name or other personally identifying information. We’ll ask you to use “fake names” for the conversation today, so the report will not use your real name in any way.

There are no known risks to joining this study. Participating and completing the focus group will result in a $25 gift card for you. Participation in the focus group is voluntary, and you may leave the focus group at any time or refuse to answer any specific questions and you will still receive the $25 gift card. Your refusal to participate or your withdrawal from the focus group will not result in any penalty.

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-0421. The time required to complete this information collection is estimated to average 75 minutes per response (90 minutes in total, including the 15 minute Demographic Questionnaire, Attachment C), including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information.

We would also like your permission to record the focus group to be sure that we accurately remember what everyone says. If you do not agree to the recording, you can still participate in the focus group. All recordings and transcriptions will be stored on Mathematica’s secure network, and only the Mathematica research team will have access to them. If HHS requests the transcriptions, the Mathematica research team will mask all identifying information before sharing any files with ASPE. We will destroy the audio recording and the transcription at the end of this study.

Do you have any questions about the study?

If you think of questions about the study after the interview, you can contact me [INTERVIEWER] at [CONTACT EMAIL AND PHONE NUMBER] or ________, the study’s lead researcher at Mathematica, at _________, at any time.

Before we can begin, I need to confirm your participation by asking you two questions:

• Do you agree to participate in the study? [YES/NO]

• [If yes] Do you agree to the audio recording of the interview? [YES/NO]

Semi-Structured Focus Group Discussion Guide for Program Participants

Instructions for interviewers

You will be conducting a focus group interview. You may need to skip some questions in order to focus on the questions that are most relevant to the respondents.

Use the follow-up questions or probes as needed to obtain additional information or clarification. If you are running out of time, prioritize the most important questions.

Before you begin the interview, read the verbal consent script (see above) to describe the study to respondents, answer any questions they have about the study, and ask for their consent to participate in the study and to audio-record the interview.

Draw their attention to the link in the focus group invitation for the demographic questionnaire and ask them to open it, and to complete and save it prior to beginning the group. Walk them through the questions as needed and answer any questions. Proceed with the focus group.

A. Ground Rules

1. [Facilitator reads the following] Now I just need to go over a few ground rules.

1. A focus group is just a guided discussion. As the facilitator, I have a set of questions that I’d like to cover today, but we encourage open conversation.

1. This discussion will last about an hour from here; we will not take a formal break.

1. We are recording this session to be sure we can accurately remember what everyone says. Only people working on the project will have access to the recordings, and your responses will remain anonymous. [NOTETAKER] will also be taking notes today just in case the recordings aren’t clear. Again, these notes will be kept private and will not identify anyone by their real name.

2. Please speak one at a time, in a voice as loud as mine, and avoid side conversations. This will help me follow the discussion and it will also help us with the recording and notes.

3. We need to hear from everyone today, but you don’t have to answer every question.

4. There may be times in the discussion where you feel differently from other people and we want to hear about that. Even though you may feel differently than people in this room, you represent others who aren’t here today who may have similar feelings.

5. We want to hear the good and the bad – there are no right or wrong answers here, and we respect differences of opinion.

6. I have a lot of questions to cover, so I may need to interrupt you to keep the conversation moving. Please don’t take it personally; it’s simply to ensure we cover all our questions today.

8. This session is confidential; your names will not be associated with anything you say. We ask that you respect each other’s privacy as well once we end the focus group– in other words, what happens in the focus group stays in the focus group!

2. Does anyone have any questions about the ground rules or about this discussion? [Facilitator addresses any final questions; check with notetaker that all paperwork has been collected and is in order; turn on recorder]

B. Introductions

3. To maintain confidentiality, I am asking you to pick a pseudonym or fake to use for the discussion today. Let’s go around the table and have everyone share:

1. Their “name” for the day and

2. [SOMETHING AS WARM UP, NOT NECESSARILY THIS] A fictional character (like from a T.V. show, book, or movie) that has a job you’d like to have

C. History of Virtual Service Delivery

4. Have you received any aspects of services virtually? If so, which ones?

Prompt if needed [tailor based on services offered by the program]:

1. Signing up for services

2. Figuring out if you’re eligible for the program

3. Case management or service coordination (i.e. working with your case manager, social worker)

4. Referrals to other services or programs in the community

5. Other

5. How did you access these virtual services?

Prompt if needed:

1. telephone

2. text

3. email

4. instant message

5. web

6. videoconferencing

7. other

1. Has your service provider connected you virtually with other [PARENTS/INDIVIDUALS/

YOUTH] receiving services?

1. IF YES: To what extent did that make you feel less isolated?

D. Opinions about Receiving Services Virtually

7. How do you feel about those virtual services?

   1. Did you like them?

   2. How did you feel about the technology (e.g. video call, text, etc.) you used?

8. Had you been getting or participating in any of these services or programs before the past few months?

   1. IF YES: Were they virtual or in person before?

   2. IF IN PERSON: How did you find out that the services you were receiving in-person were transitioning to virtual? Can you describe the process and what happened?

E. Access to the Internet and Devices

9. Do you need reliable internet/broadband to access services?

   1. IF YES: Do you have reliable internet/broadband and the devices you need? How well does the device you use (your phone, a tablet, laptop, etc.) work for remote services?

   2. IF NO: Has your service providers helped you gain access?

10. How often do you remotely connect with service providers?

11. How long is a typical remote service session?

12. Have you liked having to receive services virtually? How does it compare with in person? (better, worse, similar)

13. What do you feel has been good about receiving services virtually? Why?

14. What has been hard or bad about receiving services virtually? Why?

F. Effectiveness of Receiving Services Virtually Compared to In Person

15. Do you think virtual services work better, worse, or the same as in-person services? Why?

16. Think about the services you have received virtually since COVID-19. How would you rate those services on a scale of 1 – 5 with 5 being very satisfied and 1 being totally dissatisfied?

17. What aspects of virtual service delivery do you like? Why?

Prompt if group does not have anything to say at first or to spur discussion:

1. Outreach

2. Intake

3. Screening/assessment

4. Referrals

5. Case management

6. Services/intervention

7. Follow-up

8. Prevention

1. What aspects of virtually service delivery do you dislike? Why?

2. Do you think that services should continue to be offered virtually? If so which ones and why?

G. Recommendations

20. What recommendations would you give caseworkers and managers here or in other communities or states about how to improve virtual service delivery?

H. Wrap Up

21. We are almost done, but I just a few more questions to make sure we didn’t miss anything.

    1. Is there anything else that affects your work decisions that we haven’t talked about today?

    2. Is there anything else that affects your benefits that we haven’t talked about today?

    3. Is there anything else that affects your services by video or phone that we haven’t talked about today?

    4. Is there anything else that affects your services that we haven’t talked about today?

22. Thank you all for sharing your opinions and experiences with me today. This has been very helpful.

    1. If you know others who you think might want to participate in a focus group as part of our study, please have them contact us at ______________.

    2. Please contact me directly at ________ if there is anything you would like discuss or if you have any concerns.

17