Site Visit Discussion Guide for Staff from Lead and Partner Organizations – Follow-Up Interviews (Instrument 7)

OMB Control No. 0970-0541
Expiration Date: 02/28/2023

Instrument 7:

Site Visit Discussion Guide for

Staff from Lead and Partner Organizations – Follow-up Interviews

Introduction to Discussion Guide 3

Discussion Guide for Staff from Lead and Partner Organizations – Follow-up Interviews 4

Introduction to Discussion Guide

Introduction: We are from the research firms Abt Associates and Child Trends and we are conducting a study of the Child Welfare Community Collaborations to Strengthen and Preserve Families (CWCC) grant initiative. The study is overseen by the Office of Planning, Research, and Evaluation (OPRE), in collaboration with the Children’s Bureau, in the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS). We are conducting a cross-site process evaluation study to learn how grantees receiving funding under the CWCC initiative are approaching their collaborative work.

As part of this study, we are talking with leaders and staff at the lead and partner organizations involved in [insert name of local CWCC-funded initiative]¹ to learn more about how grantees and their partners are collaborating to prevent child abuse and neglect in their communities. Discussion topics include: partnership approaches; data use; implementation of CWCC-funded activities or strategies; factors influencing implementation of your CWCC grant; and sustainability plans beyond the CWCC grant.

Before beginning our discussion, we want to thank you for agreeing to talk with us today. We know you are busy and we appreciate the valuable time you are spending with us today. The interview will take about 75 minutes. Your participation is voluntary, and there are no penalties for choosing not to take part in the interview. You can refuse to answer any questions or stop the interview at any time. Our aim is to learn from your insights and experience. There are no right or wrong answers. Your answers will be kept private to the extent permitted by law. Information you provide will not be shared with other staff at your agency or organization. Only the study team will have access to the information you provide through this interview. Your name will not be listed in any published reports, and comments will not be attributed to you. Instead, your information will be combined with information provided by others. However, because of the relatively small number of agencies/organizations participating in the study, there is a possibility that a response could be correctly attributed to you. We may be required to share your de-identified responses with ACF and authorized researchers. Your data will only be used for research purposes. However, you may request a transcript of your own interview from your site visit team in order to share your responses with the local evaluator, if you are comfortable, which could reduce time needed during any interviews with them.

We would like to record this discussion with a digital recorder so we can listen to it later when we write up our notes. No one besides our research team and the transcription vendor will listen to the recording. If you want to say anything that you don’t want recorded, please let us know and we will be glad to pause the recorder. Do you have any objections to being part of this interview or to us recording our discussion?

The Paperwork Reduction Act Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0541, which expires 02/28/2023. Please send comments about this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Abt Associates, 10 Fawcett Street, Cambridge, MA, 02138, Attn: Michelle Blocklin OMB-PRA 0970-0541. If you have questions or concerns about your rights as a participant in this evaluation, please call Abt’s Institutional Review Board at 877-520-6835.

Do you have any questions before we begin?

Discussion Guide for Staff from Lead and Partner Organizations – Follow-up Interviews

First, I’d like to get an idea of the background of your organization and your work in it.

Background

1. Have there been any changes to the role [name of agency/organization] plays in [CWCC grant] since we last spoke in [Month, Year]?

2. Have there been any changes to your level of involvement within the [CWCC grant], including the specific roles or responsibilities that you perform, since we last spoke in [Month, Year]?

Now, we have some questions about partnerships in [CWCC grant].

Partnership Approaches

3. Are there any types of organizations that don’t exist in your community but you see a need for in order to prevent child maltreatment? (Probe: For example, is there a need for more advocacy organizations versus service providers? Or providers of different types of services? Are there any organizations in your community that focus on preventing child maltreatment but have waiting lists?)

4. How would you describe your experience partnering with other organizations on the [CWCC grant]?

5. Have there been any changes to the structures or processes of the [CWCC grant] since we last spoke in [Month, Year]?

1. Which of those structures or processes, if any, do you think have been the most helpful in serving families at risk of child maltreatment?

2. In what ways, if any, have you and your colleagues been supported to implement the new structures or processes you just described? (Probe for training on collaboration, working with others who may have different definitions for similar activities)

3. To what extent, if at all, do you believe the voices of frontline staff have been heard or represented in the work of the [collaborative]?

6. Are there clear expectations about what frontline staff and data managers at each partner organization are expected to do?

1. Do you think you have you received adequate training (and cross-training, if applicable) on your role and tasks within the [collaborative]?

7. Which partners (or types of partners) do you consider to be most essential to the work of the [CWCC grant], and why? Are there any partners who seem less essential to this work? (If needed, clarify: Essential could mean that these partners provide many services to families or provide direct services to client families as opposed to partners outside of this core set.)

The next few questions are about data use and data management.

Data Use

8. Since we last spoke in [Month, Year], has your organization made any changes to the data sources, assessments, and screening tools that are used to determine family and community needs?

1. Who are you screening?

2. When are screenings and assessments conducted?

3. Do you know whether screening is conducted systematically across partners? (Probe: If not, are there any efforts to make screening more systematic across partners?)

4. How, if at all, are assessment or screening results used? (Probe: Are these data shared across partners? If so, in what ways are the shared data used?)?

9. When, if at all, does your organization enter screening and assessment data collected from participants?

1. Who is responsible for data collection and entry?

2. What are some of the challenges your organization faces, if any, in collecting and entering data in a timely manner?

10. To what extent, if any, does your organization use needs assessment processes to prioritize which services participants receive?

1. Are needs assessment processes used to determine the order in which participants receive services? (Probe: Why or why not?)

11. To what extent, if any, does [CWCC grant] use data to inform its work?

1. Does data inform [collaborative] strategies and activities? (Probe: Why or why not?)

2. Are some strategies and activities more data-informed than others?

3. In what ways, if any, has the [CWCC grant]’s use of data changed since the [CWCC grant] began? (Probe for explanations and examples)

4. To what extent, if any, does your organization use data to inform its work within the [CWCC grant]? Has this changed over time?

12. Data Manager Only: Is there an expectation that partners participating in [CWCC grant] share data with [name of lead agency’s organization] as the lead organization?

1. If so, what are the purposes or goals of partners sharing data with the lead organization?

2. Are partners also expected to share data with other partner organizations? (Probe: Why or why not?)

3. Have expectations around data sharing changed since the [CWCC grant] began? (Probe: In what ways and why?)

13. Data Manager Only: What is the process, if any, for partner organizations to share data (either with the [name of lead agency’s organization] as lead organization or with other partner organizations)?

1. Do partners share data summaries or raw data?

2. Are partners entering data into an integrated system (available across multiple partner organizations)?

• If so, who is responsible for hosting/maintaining the integrated data system?

• Are data shared at the individual participant level (e.g., the parent/caregiver or child level) or in the aggregate (e.g., all caregivers, all children)?

• How frequently are data shared?

• What kind of training, if any, did staff receive related to data collection or entry?

• What kinds of security measures, if any, are in place to safeguard the data?

3. Have data sharing processes changed since the [CWCC grant] began? (Probe: In what ways and why?)

14. Data Manager Only: Since we last spoke in [Month, Year], have there been any changes to the way data are linked collected, shared, and used by you or your partner organizations? (Probe: What changes were made and why?)

1. In what ways are data linked?

2. In what ways are data and used to do the following:

• identify families in need of child abuse and neglect (CAN) prevention services

• identify the specific needs of families

• make informed decisions about service provision

• inform continuous quality improvement efforts

• track outcomes

15. Data Manager Only: Are there particular processes or tools that have helped facilitate data sharing and use? (Listen for information about data systems, training/support to use those systems, templates, clear expectations about data sharing and use, regularly scheduled opportunities to share and discuss data)

1. Have partners received guidance on how to interpret data and apply it to their work?

16. Data Manager Only: What are some of the barriers, if any, that hinder efforts for partners to share and use data? (Probe for barriers related to cost, man power, data security, legislation on data sharing/use, barriers to sharing across programs, across systems, etc.)

1. Are there any barriers that your organization has faced?

2. How are the barriers faced by you and other partner organizations being addressed?

Next, we have some questions about implementation of the [CWCC grant]-funded activities.

Implementation of CWCC-funded Activities

17. Have any of the [CWCC grant] priorities shifted since we last spoke in [Month, Year]? (Probe: If so, in what ways and why?)

18. Thinking about [CWCC grant] as a whole, which services or activities, if any, are provided through collaborative efforts? (For tribal grantees, probe for information about cultural or spiritual activities and interventions included as part of their programming.)

1. Does this way of delivering services or activities differ from how services were delivered prior to the [CWCC grant]? (Probe: If so, how and why?)

2. Are there any services or activities that your organization started delivering collaboratively as a result of the [CWCC grant]?

   • If so, what are some of the pros and cons of working collaboratively to deliver these services? (Probe for issues related to organizational identity, changes in mission and/or scope of work)

   • Have these new approaches required changes in terms of activities, staffing, reallocation of resources, etc.?

3. Are staff on board with these changes?

19. Which services or activities offered by [CWCC grant], if any, are delivered by a single agency (as opposed to multiple agencies)?

1. Does this way of delivering services or activities differ from how services were delivered prior to the [CWCC grant]? (Probe: If so, how and why?)

2. Are there any services or activities that your organization delivers as a single agency?

3. If so, what are some of the pros and cons of being the sole partner responsible for delivering these services?

20. To date, are there any promising practices that have emerged from implementation of [CWCC grant]?

1. What implications, if any, might these practices have for the future of this particular [CWCC grant]?

2. What implications, if any, might they have for your organization?

3. What implications, if any, might they have for other grantees or sites interested in developing a CWCC initiative?

21. Are there any lessons you would like to share regarding challenges or pitfalls that future CWCC initiatives should avoid?

1. What advice, if any, do you have for overcoming those challenges and pitfalls?

22. To date, have any new or modified policies (legislation or internal organizational procedures/policies) emerged from implementation of [CWCC grant]?

1. If so, which policies?

2. Have these policies been created or changed within particular agencies or organizations (state, local, or tribal)?

3. To what extent, if any, have these policies made their way into practice?

4. In what ways, if any, do these policies impact your organization and your partner organizations?

5. In what ways, if any, do these policies impact the families and children served by your organization and your partner organizations?

Now I’d like to ask a few questions about some of the factors that may be influencing the implementation of [CWCC grant].

Factors Influencing Implementation of the CWCC Initiative

23. Since we last spoke in [Month, Year] have there been changes to the specific services and/or preventative interventions offered by your organization?

1. What informed or led to these changes?

2. When were these changes made?

3. Were these changes adopted by all partner organizations, or just some of them?

4. Do you anticipate any additional changes to specific services and/or preventative interventions? Why or why not?

24. Since we last spoke in [Month, Year], have you noticed any shifts in the array of social service organizations available to support families in your community?

1. If so, please describe the changes. (Probe for changes in the number of service providers, types of services offered, target populations.)

2. In what ways, if any, have these changes impacted the work of your organization?

In these last few questions, we would like to know more about your perceptions of the progress your team is making towards achieving the project goals and also lessons learned thus far.

Perceptions of Progress Towards Goals

25. Since we last spoke in [Month, Year], have there been any changes to the ultimate goals and outcomes of the [CWCC grant]?

1. Have there been any changes in how the [initiative] tracks or measures progress towards meeting its goals?

26. In what ways, if any, do you see your day-to-day work contributing to the goals and outcomes of the [CWCC grant]?

27. To what extent do you believe the [CWCC grant] is meeting its goals? (Probe for specific examples)

1. Can you describe some of the early “wins” or successes from the [collaborative]?

2. Are there any goals or outcomes that the initiative is falling short in at this time? (Probe for specific examples)

28. In your opinion, is there anything that has happened since our last site visit in [Month, Year] that would not have happened without the grant?

Lessons Learned

29. To date, what lessons, if any, have you learned about cross-agency collaboration?

1. In what ways, if any, have these lessons confirmed that the approach to collaboration was on-track?

2. In what ways, if any, have these lessons led you to change how you and your partners are approaching the CWCC work?

3. What advice would you give to other organizations preparing to launch or participate in a CWCC effort?

Finally, we are interested in your planning for sustaining [CWCC grant].

Sustainability Plans and Influential Factors

30. Knowing what you know about and have experienced with the work of [CWCC grant] so far, which aspects of the work do you expect to continue after the [CWCC grant] ends?

1. Which strategies, activities, and/or services, if any, should be sustained? Why?

2. Which strategies, activities, and/or services, if any, do you think your organization will continue? Why?

3. Which strategies, activities, and/or services, if any, do you think your organization will discontinue? Why?

4. Which partner organization(s), if any, would you expect to continue these strategies, activities, and/or services? Why? Which, if any, might not continue them? Why not, do you suppose?

5. Which strategies, activities, and/or services, if any, should not be sustained? Why?

31. How many [CWCC grant] strategies, activities, and/or services, if any, would need resources to be sustained?

1. Which ones (activities and/or services)?

2. What kind of resources would be necessary?

3. To what extent are these resources available?

32. Are the necessary resources (such as staff and organizational capacity and training) available to continue to support [CWCC grant] strategies, activities, and/or services after the grant ends?

33. Are the existing partnership coordination processes and relationships strong enough to continue to support the [CWCC grant] strategies, activities, and/or services after the grant ends?

1. If no, is there anything that could strengthen partnership processes and relationships?

1 Throughout this document, wherever terms are in brackets, the grantee’s own term will replace the bracketed terms in order to facilitate comprehension. Potential terms include: partnership, collaboration, coalition, collective impact, collaborative, collective, program, or project.

Abt Associates  ▌pg. 11