13. MUSE Local Evaluator Interview Questions

July 2020 OMB Control No.: 0970-0521

Expiration Date: 12/31/2021

Length of time for instrument: 1 hour, 30 minutes

MULTI-SITE IMPLEMENTATION EVALUATION OF TRIBAL HOME VISITING (MUSE)

LOCAL EVALUATOR INTERVIEW

This collection of information is voluntary. Public reporting burden for this collection of information is estimated to average 90 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: 0970-0521, Exp: 12/31/2021. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Kate Lyon, James Bell Associates; 3033 Wilson Blvd. Suite 650, Arlington, VA 22201; MUSE.info@jbassoc.com.

Qualitative Interview – Local Program Evaluator

Thank you for taking part in today’s interview. The purpose of this study is to learn about tribal home visiting programs and the experiences of families receiving home visiting services.

We have asked you to take part in this interview because you are an evaluator for one of the home visiting programs participating in MUSE. We are interested in hearing about your experience working as an evaluator with [PROGRAM NAME]. We will ask you questions about [Program Name] planning and implementation and your role on the project.

Your participation in this interview is voluntary. If you choose to participate, it will take about 1 hour and 30 minutes. Please let me know if you need clarification on a question or if you need to take a break at any time.

Your answers will be kept private. Only the MUSE study team will have access to this information. What you tell me here today will not be shared with anyone at your program or any other agencies. We will not report information collected in this study in a way that could identify you or your program.

Please note that there are no right or wrong answers to the questions I am going to ask. I am interested in learning about your particular thoughts and experiences.

Universe of Possible Questions—Local Program Evaluator

1. Please tell me about what led you to enter the field of home visitation.

   1. How long have you been doing evaluation in this field?

   2. What other fields have you done evaluation work in?

2. How did you become an evaluator with [program/organization]?

   1. How did you originally get connected with the Tribe/Tribal organization?

   2. What other projects had you work on in this community or with this agency before you began this project?

   3. What were your original expectations for the project and how have those expectations panned out?

   4. Please tell me about any other projects you work on in this community?
      
      

3. Can you please describe how you got oriented to the Tribal MIECHV requirements?

   1. What has best supported you in getting oriented to this initiative (Probe: in person meetings, TEI?)

4. What prior experience did you have doing evaluation work in tribal communities?

   1. How does this evaluation project differ from other projects you have done with non-Native communities?

   2. How did your education or training prepare you for this work?

   3. What else besides what you learned in your formal education or training have you had to learn and how did you go about learning it?

   4. What would you suggest an evaluator with no experience working in tribal communities do to prepare if they want to get into this kind of work?

5. If you were going to help another THV site hire an evaluator, what qualities would you suggest they look for? Any specific qualifications? Required experience?

   1. Which of your personal skills do you feel like are most valuable in your current role?

   2. What are the most critical skills for a THV evaluator to have?

6. There are many different kinds of things evaluators do when they work with a program, what do you see as your primary role(s) on this particular project?

   1. How is this role different than what is described in your contract/position description?

   2. How is this role different than roles you have played on other evaluation projects?

   3. When do you feel most successful in this role? Least successful?

   4. What makes the role difficult?

   5. What do you like best about your role?

   6. How have your roles and responsibilities on this project changed over time?

7. What supports do you rely on to do your best in this role?

   1. How has TEI supported you in your work? How could TA for evaluation be enhanced or improved?

   2. What about evaluators from other THV programs, how have you connected with your peers and has that been helpful? How would this type of peer to peer support be best facilitated?

8. Could you describe some of the more challenging aspects of integrating evaluation into program implementation?

   1. How has the general idea of evaluation been received by program leadership? Staff? Families?

   2. How have people’s impressions of evaluation changed over time?

   3. How do you think being an evaluator impacts your relationships with program staff? Community members?

   4. How does being associated with data impact your interactions with program staff?

9. Each THV site has a slightly different way of structuring their evaluation positions. For example, some have in-house staff and others contract with outside evaluators. Could you please briefly describe the way evaluation roles are structured for this program? (Probe: ask about FTE, how contracting has worked)

   1. How would you characterize your position on the program team? (Probe: do you function by and large externally to the team? Do you feel like part of the team?

   2. Do you generally find that the FTE you have on this project is adequate for the work you need to do?

   3. If you were to consult with a new THV program and advise them in how to structure their evaluation position, what would you recommend?

   4. How does [institution evaluator works for] support you in doing your best on this project? (Probe: flexibility during demanding periods; allocating enough time for relationship building; institutional experience working with tribal communities, contracting, content support)

10. We are really interested in capturing the range of things evaluators do across THV sites and the experience of evaluators as they complete various tasks. Please describe the primary tasks you have on this project. (Probe: implementation plans, quarterly reports, CQI activities, home visitor trainings, community activities, advisory board presentations/participation)

    1. For each task mentioned consider asking:

       1. What were your responsibilities on this project?

       2. How effective did you feel in this work?

       3. What factors most contributed to you feeling this way?

11. How have you been involved with the program as they adjusted to COVID-19?

12. How do you communicate with program staff? (Probe: formal formats such as meetings, informal and on-going such as unplanned calls).

    1. When do you wish you had more communication with program staff?

13. Can you please describe the data system that you currently have in place?

    1. How did you come about using this data system?

    2. What was your role in getting the data system set up (Probe: research systems, build system etc.)?

    3. How comfortable did you feel in this role?

    4. What kinds of institutional support were you able to take advantage of? (Probe: existing data systems, consultants, other staff support)

    5. How did you figure out what the program needed and how to get that into the system? What about the data system have you changed over time?

    6. How satisfied are you with your current data system? Why?

    7. Would you recommend the data system to another program? Why or why not?

    8. If you could design the perfect data system, what would it look like?

14. How is data used in program decision making at [program]? (Probe: To monitor caseload? Set program priorities? Tracing family progress/outcomes? For CQI?)

    1. How is data shared with program staff?

    2. Are staff encouraged/seem comfortable asking questions about data?

    3. Can you tell me about any instances of changes or adjustments in implementation that were based on data? (Probe: what data? How was data shared? Who was part of the process…)

    4. How could program data be better used and what would the program need to do this?

15. How confident do you think home visiting staff are in using the data systems your program has in place? (Probe: data collection, data entry and storage)

    1. What do you think accounts for this confidence/lack of confidence?

    2. How are staff trained in using data systems? (Probe: by whom? How often? Who decides?)

    3. What kind of on-going support do home visitors get in using the data system?

16. What about this program do you wish was better captured in the data?

    1. How would you go about capturing that if you could?

17. Can you describe the model selection process? (Probe: who selected the model, what other models were considered? Any resources or TA used in the selection process?)

    1. Why was [model/curriculum] selected? (Probe: what kind of evidence?)

    2. How is [model/curriculum] supplemented or adapted here? (Probe: please describe that process; why was this necessary; who advocated for this and how? How has the supplement/adaptation gone so far? What is your role in assessing the supplement/adaptation?)

18. What were your initial impressions of [model/curriculum]?

    1. How did you first become aware of [model/curriculum]?

    2. How did you learn what you needed to know about the [model/curriculum] to make evaluation decisions?

    3. Did you think the model was going to be difficult or easy for home visitors to use? Why?

    4. What other expectations did you have?

    5. How long have you been evaluating implementation of this model?

    6. How have your impressions changed over time?

    7. Would you choose the model for this program again?

    8. What kinds of communities/programs would you recommend use this model?

    9. What resources have model developers provided to support implementation through evaluation or data?

    10. How have the model developers supported you in your work?

19. What would you change about [model/curriculum]? Program?

    1. Why would make these changes?

20. How has program planning activities such as writing the implementation plan, impacted actual implementation?

    1. How do program leadership use the implementation plans in decision making?

    2. What has gone basically according to plan?

    3. What changes have needed to be made?

    4. How have you tracked changes?

    5. How do you think program implementation would be different without having written the implementation plan?

21. How did larger systems, including Tribal governments, state home visiting programs, education, health and mental health systems impact program decision making?

    1. How do these same systems impact day to day implementation?

22. Can you describe any CQI activities that you have been a part of on this program?

    1. How did the team pick a problem to work on?

    2. What was the process for identifying the data needed?

    3. Who analyzed/prepared the data for the study? What was your role?

    4. How were decisions made about what to “do?” (Probe: Who identified potential solutions?)

23. How does the work you do support home visitors in their work with families?

    1. What trainings if any have you done for program staff? What was the goal of these trainings? To what degree did you achieve that goal? Why?

    2. How were you involved in setting up necessary data systems for the program? What were you hoping for in terms of functionality of that system? How did the system work out?

    3. Anything that people might not think of as evaluation work?

24. What do you see as the most important impact of your work as an evaluator on this project?

    1. What are the things that make it easier for your work to have this impact? Harder?

    2. What keeps you invested in this project?

25. What do you hope families get out of the home visitation services they receive from [program/organization]?

    1. How does what you do as an evaluator help families get these benefits?

26. What most influences your ability to feel good about your work as an evaluator with [program]?

    1. What do you like best about working with on this project?

    2. What is the hardest part?

27. Is there anything that we have not talked about regarding your experience as an evaluator with [name of program/organization], the [model/curriculum] or the home visitation services you provide that you would like to share?

Multi-Site Implementation Evaluation of Tribal Home Visiting OMB Supporting Documents: Evaluator Interview Questions

12