Form SOCEA

Attachment 8c – Form B OMB No. xxxx-xxxx

Expires: xx/xx/201x

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

SYSTEM OF CARE EXPANSION ASSESSMENT

PROJECT DIRECTOR

INTRODUCTION

Thank you for your willingness to complete the system of care expansion assessment. The purpose of this interview is to identify the strategies and mechanisms implemented at the local service system level, and examine the extent to which these mechanisms are being implemented from a project director’s perspective.

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of the federal system of care expansion grant program.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 1 hour.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

Your responses to these questions will be kept confidential and will not be shared outside of the evaluation team. In any of our reports, your responses will be combined with other people’s responses, so your answers will never be attributed to your name. However, we must tell you that during the course of the interview, if you tell us about any intent to abuse or cause harm to yourself or others we have to report it to SAMHSA.

In any of our reports, your responses will be combined with other people’s responses, so your answers will never be attributed to your name.

[OBTAIN INFORMED CONSENT] At this point, do you agree to continue with this interview?

[If yes, continue]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION] Do you agree to this session being recorded?

[If yes, continue]

INSTRUCTIONS

The SOCEA interview focuses on the local-level management, structure, and procedures as well as direct service delivery level involves activities and practices in the delivery of direct services.

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

OMB No. xxxx-xxxx

Expires: xx/xx/201x

A. Project Director

Overview

1. Please briefly describe the overall goal and mission of (name of grant program).

What are the eligibility criteria for participation in (name of grant program)? Have these changed since CMHS grant funds were received? Please describe.

2. What are the primary service components (e.g., outpatient, school-based, case management, etc.)?

3. In what specific areas does (name of grant program) plan to expand?

• Expand population of focus?

• Broaden funding approaches?

• Integrate additional agency(ies)?

• Extend outreach?

• Expand service locations?

• Expand cultural and linguistic accessibility?

• Other?

How far along would you say those expansion efforts have progressed since CMHS grant funds were received?

[Data entry: code expansion areas]

4. How many children, youth, and families have been served since grant funds were received?

What shifts, if any, has the system of care experienced in the number of children, youth, and families served per year since grant funds were received? Please explain.

Entry into the System of Care

5. Since grant funds were received, how have children and youth been referred to (name of grant program) for services? [Probe for client flow into (name of grant program) .]

6. Which public (or quasi-public) agencies have referred children, youth, and families to (name of grant program) ? (D.6.a.)

1 = Mental Health 4 = Juvenile Justice 7 = Intellectual Disabilities

2 = Education 5 = Public Health 8 = Other (describe)

3 = Child Welfare 6 = Substance Abuse

[circle all that apply]

What was the distribution of referrals received from these agencies?

5=Five or more agencies refer children, youth, or families to the grant

4=Four agencies refer children/families to the grant

3=Three agencies refer children/families to the grant

2=Two agencies refer children/families to the grant

1=One agency refers children/families to the grant

7. Are other agencies in the child-serving systems (for example, mental health, education, child welfare, health, juvenile justice, substance abuse, intellectual disabilities) able to conduct or initiate intake into (name of grant program)? If yes, please describe the process. (D.6.b.)

Are there any mechanisms in place to ACTIVELY involve other agencies in the intake process for the (name of grant program) (e.g., schools provide space for program staff to conduct intake process, other agency staff conduct intake process and complete intake forms, routine follow-up to make sure family was linked with grant program, etc.)?

[Note: Passive referrals, for example, sharing grant phone number and address with families, leaving it to the family to make contact with the (name of grant program) , are not considered active involvement in intake.]

1 = Mental Health 4 = Juvenile Justice 7 = Intellectual Disabilities

2 = Education 5 = Public Health 8 = Other (describe)

3 = Child Welfare 6 = Substance Abuse

[circle all that apply]

5=Five or more agencies were ACTIVELY involved in intake into the grant program

4=Four agencies were ACTIVELY involved in intake into the grant program

3=Three agencies were ACTIVELY involved in intake into the grant program

2=Two agencies were ACTIVELY involved in intake into the grant program

1=One agency was ACTIVELY involved in intake into the grant program

Outreach

8. Since grant funds were received, have there been any outreach efforts to inform children, youth and families about (name of grant program) and its services? If yes, please describe. (D.8.a.)

[Data entry: code outreach efforts]

How effective have your outreach efforts been? For example, have you seen an increase in calls to (name of grant program) or an increase in awareness or interest in the community? Explain.

To what extent have these efforts been sufficient? Has (name of grant program) been able to reach everyone?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

9. Since grant funds were received, have there been any outreach efforts to specific cultural groups or populations (e.g., language, religion, historical trauma, gender identity, alternative lifestyle, etc.)? (D.5.a.)

If yes, please describe those efforts.

[Data entry: code outreach efforts]

How effective would you say these efforts have been? For example, have you seen an increase in interest or awareness?

To what extent have these efforts been sufficient to reach all of the specific populations you have targeted your efforts towards?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

10. Since grant funds were received, have there been any outreach efforts to inform other agencies, community-based providers, private providers, family organizations, youth organizations, primary health care providers, etc. about the (name of grant program) and its services? (D.7.a.)

If yes, please describe those efforts.

[Data entry: code outreach efforts]

How effective have these efforts been, and in what way?

To what extent have these efforts been sufficient to ensure that all providers and organizations have been made aware of (name of grant program) ?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Service Planning and Delivery

11. Please describe the primary functions of care coordinators (e.g., case managers, case coordinators, facilitators).

Does (name of grant program) rely on one or more care management entities (CMEs)? If yes, please describe.

Are there different levels or types of care management/coordination? If yes, please describe

Do care management/coordination services differ based on the needs of a child or family? If so, how does care management/coordination differ? [Probe for levels and types of services, role of intensity of child/youth challenges and other factors]. (F.7.b.)

Are care coordination services sufficiently flexible and varied to meet children’s specific individual care coordination needs?

[Interviewer to rate the efforts made to vary the services based on the needs of a child or family]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

12. Are there any mechanisms in place or have efforts been made to facilitate the coordination of services across agencies, organizations, and providers (for example, interagency team meetings, joint staff treatment team meetings, interagency case management meetings, etc.)? (A.7.b.)

If yes, please describe theme.

[Data entry: code mechanisms]

To what extent have these efforts been effective? Please describe?

To what extent have these efforts been sufficient—is there anything else that could have been done to enhance coordination of services across agencies, organizations, and providers?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

13. Since grant funds were received, what has been done to ensure that services and supports in (name of grant program)’s service array have sufficient capacity such that all children, youth, and families who should have access to these services and supports can get them (for example, expanding network of contract providers, increasing contracts to increase capacity)? [Probe for capacity across all services in the array.] (B.8.a.)

[Data entry: code mechanisms]

How effective have these efforts been, and in what ways?

To what extent have these efforts been sufficient? Are there any services and supports that some children, youth, and/or families cannot get because capacity is limited? If yes, what are they? [Probe for which specific services.]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

14. What efforts are made to ensure that services provided through (name of grant program) have evidence of effectiveness?

What evidence-supported services or treatment models are provided through (name of grant program)? (B.4.a.)

-Family services and supports?

-Child/youth services and supports?

[Data entry: code evidence-supported models, treatments, approaches]

What proportion of services provided through (name of grant program) has evidence supporting their effectiveness?

5=All or almost all of the services and supports are evidence-supported

4=Most of the services and supports are evidence-supported

3=Some of the services and supports are evidence-supported

2=A few of the services and supports are evidence-supported

1=No, or almost no, services and supports are evidence-supported

What proportion of children, youth, and families receive evidence-supported services through (name of grant program)?

5=All or almost all children, youth, and families receive evidence-supported services and supports

4=Most children, youth, and families receive evidence-supported services and supports

3=Some children, youth, and families receive evidence-supported services and supports

2=A few children, youth, and families receive evidence-supported services and supports

1=No, or almost no, children, youth, and families receive evidence-supported services and supports

15. What efforts are made to ensure that evidence-supported services and treatments are provided with good fidelity and quality?

15.a. Are practitioners trained on evidence-supported services and treatments? (B.4.b.)

• If yes, who is trained?

• How are they trained (e.g., learning collaborative model)?

How effective have these efforts been to ensure practitioners have the skills to implement evidence-supported services and supports? Explain.

To what extent have these efforts been sufficient? Are there any additional efforts that should be made to ensure that practitioners provide evidence-supported models? [Probe for what more should be done.]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

15.b. How is fidelity to evidence-supported models and approaches maintained in practice (e.g., manuals provided to practitioners, booster training, on-going professional development)? (B.4.c)

How effective have these efforts been to support faithful implementation of evidence-supported services and supports? If yes, explain.

To what extent have these efforts been sufficient? Are there any additional efforts that should be made to ensure that practitioners faithfully implement evidence-supported models? [Probe for what more should be done.]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

15.c. Is there any effort to monitor fidelity to models and approaches across (name of grant program) and over time? If yes, please describe. (C.4.a.)

Have these efforts been effective to ensure that evidence-supported services and supports are implemented faithfully across the (name of grant program) and over time? Explain.

To what extent have these efforts been sufficient? Are there any additional efforts that should be made to ensure that evidence-supported services and supports are implemented faithfully across the (name of grant program) and over time? [Probe for what more should be done.]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

16. What has (name of grant program) done to make services more accessible in terms of scheduling of services or locations where services were provided? [Probe for flexible hours including after hours and weekends; services offered in settings such as home, schools, community). Probe for applicability across all services in the array.] (B.8.b.)

[Data entry: code mechanisms]

How effective have these efforts been, and in what ways?

Have these efforts been sufficient? What else could be done to make services more accessible?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

17. Since grant funds were received, have there been any efforts to make services more affordable and to minimize financial barriers? If yes, please describe. [Probe for sliding scale fees, scholarship funds, flexible funding, use of health insurance navigators, process for enrollment in Healthcare.gov or other health exchange, etc.] (B.8.c.)

[Data entry: code mechanisms]

How effective have these efforts been? In what ways?

Overall, to what extent have efforts been sufficient? Has cost continued to be a barrier for some families to access needed services in the array?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

18. Please describe how flexible funds are used in (name of grant program)?

Probe for what are the funds used for?

[Data entry: code uses]

What is the process for accessing flexible funds?

For what proportion of families are flexible funds used? Do all families who need flexible funds receive them?

18.a To what extent are flexible funds used effectively to meet the needs of children and youth? (B.3.b)

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

18.b. To what extent are flexible funds used effectively to meet the needs of caregivers and families? (B.1.b)

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

19. What efforts have been made to minimize the need for children, youth, and families to travel out of their home communities for services? [If response is “not a problem” (i.e., all services received in the community), probe for what allows them to serve all children/families in the community.] (A.9.a.)

[Data entry: code mechanisms]

How effective have these efforts been, and in what ways?

To what extent have these efforts been sufficient to ensure that all children, youth, and families are served in the community? If not, what else could be done?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

20. What efforts have been made to reduce the number of children and youth who are served in settings that separate them from typically developing children and youth and/or their families? [If response is “not a problem” (e.g., children are never served in restrictive environments), probe for what allows them to serve children and youth in the least restrictive settings.] (A.10.a.)

[Data entry: code mechanisms]

How effective have these efforts been, and in what ways?

To what extent have these efforts been sufficient to eliminate the use of restrictive settings/service options?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

20.a. If a child or youth is placed in a restrictive setting, what policies, structures or mechanisms have been implemented to ensure that children/youth and their families maintain regular contact with (name of grant program)? (A.10.b.)

[Data entry: code mechanisms]

How effective have these efforts been, and in what ways?

Have these efforts been sufficient? What else could be done to maximize continuity of contact?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

20.b. What policies, structures or mechanisms have been implemented to ensure that children/youth do not experience a disruption of services/supports upon return to their home following a restrictive placement? (A.10.c.)

[Data entry: code mechanisms]

How effective have these efforts been, and in what ways?

Have these efforts been sufficient? Are all children/youth immediately re-entered into (name of grant program) services/supports upon return to their homes?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

21. Since grant funds were received, what policies, structures or mechanisms has (name of grant program) implemented to promote the provision of individualized, person-centered services and supports? (For example, use of flexible funds to meet unique needs, development of specific services, training on the provision of individualized care) (A.3.a.)

[Data entry: code mechanisms]

How effective have these efforts been, and in what ways?

Have these efforts been sufficient? What other things that could be done to maximize the provision of individualized care?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

22. Since grant funds were received, what policies, structures or mechanisms have been  (name of grant program) implemented to promote family-driven service delivery? (For example, use of flexible funds to provide unique services and supports, development of specific services, training on family-driven service delivery, efforts to maximize family empowerment and engagement) (A.1.a.)

[Data entry: code mechanisms]

How effective have these efforts been, and in what ways?

To what extent have these efforts been sufficient? What other things that could be done to maximize the provision of family-driven service delivery?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

23. What efforts have been made to promote the provision of culturally and linguistically competent care? (A.5.a.)

Probe for:

• Training on community characteristics, traditions, culture, language, religion, historical trauma, gender identity, alternative lifestyles?

• Effort to meet language needs of clients. Are formal policies in place that address language needs?

• Training on how to build on family and youth culture in the delivery of services

How effective have these efforts been, and in what ways?

Overall, to what extent have efforts been sufficient to ensure the provision of cultural and linguistically competent service delivery? Are more efforts needed?

Which specific efforts do you feel are lacking?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

24. Since grant funds were received, how has the cultural background (e.g., traditions, culture, language, religion, historical trauma, gender identity, alternative lifestyles) of the children, youth, and families you serve been considered in developing the service array? (B.5.a.)

Since grant funds were received, have cultural organizations, community groups, etc. been involved in efforts such as providing services, developing the service array, advising providers, etc.? If yes, please describe.

Since grant funds were received, have you added or modified any services or supports to address the cultural and/or linguistic needs of specific groups?

How effective have these efforts been, and in what ways?

Have efforts been sufficient to address the cultural needs of all groups? Are some groups’ needs still unmet? [Probe for specific groups.]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Staffing

25. Since grant funds were received, have you had any difficulty hiring or retaining staff? If yes, for which positions and why?

26. Have any efforts been made to hire staff who reflect the cultural background (for example, race, ethnicity, language, gender, sexual orientation, disabilities, lifestyle) of the children, youth, and families you serve? (A.5.b.)

How effective have these efforts been? Explain.

Have efforts been sufficient to hire the number or type of staff needed to meet the cultural needs of populations served? Please describe.

[Note: If staff are already in place (i.e., no hiring was necessary) probe for diversity of staff relative to population served.]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

27. What mechanisms have been used to integrate staff across core child-serving agencies including any of the following? [Probe for descriptions of each mechanism and agencies involved.] (A.6.b.)

1. Joint training (i.e., staff from multiple agencies are trained together)?

1=no, 2=yes

b. Shared staff (i.e., more than one agency funds one staff position)?

1=no, 2=yes

c. Outstationing or outposting staff (i.e., staff from one agency is housed in another agency’s office or service locations)?

1=no, 2=yes

d. Co-location of staff (i.e., staff from several agencies are housed in central location)

1=no, 2=yes

e. Other efforts? Please describe.

1=no, 2=yes

[Data entry: code mechanisms]

How effective have these efforts been? Explain.

Have efforts been sufficient. That is, to what extent have they reduced the barriers for staff across agencies to work together?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Management and Operations

28. Are there any mechanisms in place or have efforts been made to involve family members in program operations such as designing and implementing programs, providing training, working as staff, helping with staff recruitment, developing the service array, attending management meetings, serving on oversight committees, etc.? (A.1.b.)

If yes, please tell me all of the different ways they have been involved.

[Note: Do not count involvement in the SOC evaluation.]

[Data entry: code ways]

Have family members been involved in any other ways?

[Continue to probe for examples until the respondent reports no more.]

5=Four or more examples of family member involvement in grant operations

4=Three examples of family member involvement in grant operations

3=Two examples of family member involvement in grant operations

2=One example of family member involvement in grant operations

1=No examples of family member involvement in grant operations

29. Since grant funds were received how have youth been involved in program operations? (A.2.a.)

If yes, please tell me all of the different ways they have been involved.

[Note: Do not count involvement in the SOC evaluation.]

[Data entry: code ways]

Have youth been involved in any other ways?

[Continue to probe for examples until the respondent reports no more.]

5=Four or more examples of youth involvement in grant operations

4=Three examples of youth involvement in grant operations

3=Two examples of youth involvement in grant operations

2=One example of youth involvement in grant operations

1=No examples of youth involvement in grant operations

30. Since grant funds were received, have there been any shared administrative processes across child-serving agencies such as jointly developing staff training materials; holding joint staff meetings; hiring/recruiting staff together; using the same administrative forms, unified case records, or integrated MIS; etc.?

30.a. If yes, what are all of the different shared processes that are in place?? (A.6.a.)

[Data entry: code ways]

[Continue to probe for examples until the respondent reports no more.]

5=Four or more examples of shared administrative processes across child-serving agencies

4=Three examples of shared administrative processes across child-serving agencies

3=Two examples of shared administrative processes across child-serving agencies

2=One example of shared administrative processes across child-serving agencies

1=No examples of shared administrative processes across child-serving agencies

30.b. Which agencies participated in these shared processes? (A.6.a.)

1 = Mental Health 4 = Juvenile Justice 7 = Intellectual Disabilities

2 = Education 5 = Public Health 8 = Other (describe)

3 = Child Welfare 6 = Substance Abuse

[circle all that apply]

5=Five or more agencies

4=Four agencies

3=Three agencies

2=Two agencies

1=No shared processes

31. Are there de-categorized funds that are pooled, blended, or shared across agencies (other than for shared staff positions)? If yes, please describe.

[Data entry: code mechanism]

31.a. If yes, which agencies contribute to the blended funding? (A.6.c)?

1 = Mental Health 4 = Juvenile Justice 7 = Intellectual Disabilities

2 = Education 5 = Public Health 8 = Other (describe)

3 = Child Welfare 6 = Substance Abuse

[circle all that apply]

5=Five agencies contribute

4=Four or more agencies contribute

3=Three agencies contribute

2=Two agencies contribute

1=No blended funding

31.b. For what purpose(s) (if any) are funds blended, pooled, or integrated across different sources? (A.6.c)

To what extent have the blended, pooled, or funds been effective for these purposes?

To what extent is the total amount of blended funds sufficient? Are individual agency contributions sufficient?

What, if anything, could be done to blend funds better or more fully?

[If 31.=1, then 31.a.=666]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

666=No blended funding

32. Since grant funds were received, how have you shared information about program operations (for example, changes in eligibility criteria, referral processes, service/program components) with supervisors and direct line staff in the different agencies, and organizations with whom you work? (A.7.a.)

[Data entry: code mechanisms]

Have these mechanisms been effective? Please describe.

To what extent have these efforts been sufficient to ensure that all parties have been aware of procedures and operations?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Lessons Learned

33. Since grant funds were received, what have been the most effective strategies/activities undertaken to develop and implement this system of care?

34. Since grant funds were received, what have been the least effective strategies/activities undertaken to implement this system of care?

Achievement of Goals

35. Since grant funds were received, what have been (name of grant program) ’s most important achievements?

How has your community benefited from the system/program?

36. To what extent would you say (name of grant program) has had an influence on the larger system across all child-serving sectors? In what ways?

37. To what extent would you say (name of grant program) has worked to move toward the establishment of a jurisdiction-wide system of care?

38. To what extent would you say (name of grant program) has collaborated with other state efforts to transform mental health services during the last year?

39. What have been the major obstacles to expanding the system-of-care principles and philosophies beyond the scope of (name of grant program) ?

Sustainability

40. Please tell me about the status of your grant community’s efforts to sustain itself as grant funds decrease over time. [Probe to determine how they are responding to increased matching funds requirements.]

Have any aspects of the system/program been changed, reduced, or eliminated due to decreases in CMHS funding? If so, what has occurred?

41. What aspects of the system/program do you anticipate will be sustained over time? [Probe for family-driven, individualized/person-centered and youth-guided services/supports, culturally competent, interagency involvement, accessible services, coordinated, community-based and least restrictive care, etc.]

What barriers exist that may hinder sustainability?

End of Instrument:

Thank you for participating in the System of Care Expansion Assessment portion of the National Systems of Care Expansion Evaluation.

Attachment 8c – Form B OMB No. xxxx-xxxx

Expires: xx/xx/201x

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

SYSTEM OF CARE EXPANSION ASSESSMENT

cORE aGENCY rEPRESENTATIVE

INTRODUCTION

Thank you for your willingness to complete the system of care expansion assessment. The purpose of this interview is to identify the strategies and mechanisms implemented at the local service system level, and examine the extent to which these mechanisms are being implemented from a core agency representative’s perspective.

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of the federal system of care expansion grant program.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 1 hour.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

Your responses to these questions will be kept confidential and will not be shared outside of the evaluation team. In any of our reports, your responses will be combined with other people’s responses, so your answers will never be attributed to your name.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

The SOCEA interview focuses on the local-level management, structure, and procedures as well as direct service delivery level involving activities and practices in the delivery of direct services.

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

OMB No. xxxx-xxxx

Expires: xx/xx/201x

B. Representative of Core Agency

Introduction

1. What agency/organization do you represent?

1=Mental health public agency 6= Substance abuse public agency 11 = Private org. representing education

2=Public health agency 7= Intellectual disabilities public agency 12 = Private org. representing juvenile justice

3=Child welfare public agency 8=Private org. representing mental health 13= Private org. representing substance abuse

4=Public education 9=Private org. representing public health 14= Private org. representing intel disabilities

5=Juvenile justice public agency 10= Private org. representing child welfare 15=Other: Specify _____________________

Describe briefly the type of clientele served by (name of grant program) and its key goals.

2. What is your title/position? Do you have supervisory responsibilities? Do you provide services directly?

1=Exec Director/Chief 4=Supervisor

2=Deputy or Assistant Exec Director/Chief 5=Front-line staff

3=Middle manager 6=Other, specify ___________________________

Management and Operations

3. Since grant funds were received, how has (name of grant program) informed supervisors and line staff in your agency about program operations such as changes in eligibility criteria, referral processes, service/program components? (A.7.a.)

[Data entry: code mechanisms]

On a scale of 1 to 5 (with 5 being best), how effective have these efforts been in ensuring that all parties have been aware of program procedures and operations?

5=Very effective

4=Moderately effective

3=Somewhat effective

2=Minimally effective

1=Not effective

4. What mechanisms have been used to integrate staff across core child-serving agencies including any of the following? [Probe for descriptions of each mechanism and agencies involved.] (A.6.b.)

2. Joint training (i.e., staff from multiple agencies are trained together)?

1=no, 2=yes

b. Shared staff (i.e., more than one agency funds one staff position)?

1=no, 2=yes

c. Outstationing or outposting staff (i.e., staff from one agency is housed in another agency’s office or service locations)?

1=no, 2=yes

d. Co-location of staff (i.e., staff from several agencies are housed in central location)

1=no, 2=yes

e. Other efforts? Please describe.

1=no, 2=yes

[Data entry: code mechanisms]

How effective have these efforts been? Explain.

Have efforts been sufficient. That is, to what extent have they reduced the barriers for staff across agencies to work together?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

5. Does your agency and (name of grant program) have any shared administrative processes such as jointly developing staff training materials; holding joint staff meetings; holding joint planning meetings; having interagency work groups/committees; hiring/recruiting staff together; using the same administrative forms, unified case records, or integrated MIS; etc.? (A.6.a)

If yes, what are all of the different shared processes put in place?

[Data entry: code ways]

[Continue to probe for examples until the respondent reports no more.]

5=Four or more examples of shared administrative processes across child-serving agencies

4=Three examples of shared administrative processes across child-serving agencies

3=Two examples of shared administrative processes across child-serving agencies

2=One example of shared administrative processes across child-serving agencies

1=No examples of shared administrative processes across child-serving agencies

6.a. Are there de-categorized funds that are pooled or blended across agencies (other than for shared staff positions)? If yes, please describe. (A.6.c.)

If so, which agencies contribute to the blended funding?

1 = Mental Health 4 = Juvenile Justice 7 = Intellectual Disabilities

2 = Education 5 = Public Health 8 = Other (describe)

3 = Child Welfare 6 = Substance Abuse

[circle all that apply]

5=Five or more agencies actively involved

4=Four agencies actively involved

3=Three agencies actively involved

2=Two agencies actively involved

1=No blended funding

6.b. For what purpose(s) are funds blended?

How have the blended or pooled funds been effective for these purposes?

To what extent is the total amount of blended funds sufficient? To what extent are individual agency contributions sufficient? (A.6.c.)

What more could be done to blend funds?

[If 6.a. =1, then 6.b.=666]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

666= No blended funding

Service Delivery

7. Are there any mechanisms in place or have any efforts been made to facilitate the coordination of services across agencies, organizations, and providers (for example, interagency team meetings, joint staff treatment team meetings, interagency case management meetings, etc.)? (A.7.b.)

If yes, please describe.

[Data entry: code mechanisms]

How effective have these efforts been and in what ways?

To what extent have these efforts been sufficient—is there anything else that could have been done to enhance coordination of services across agencies, organizations, and providers?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

8. How has (name of grant program) worked to minimize the need for children, youth, or families to travel out of their home communities for services? [If response is “not a problem” (i.e., all services received in the community). probe for what allows them to serve all children, youth, or families in the community.] (A.9.a.)

[Data entry: code mechanisms]

How effective have these efforts been and in what ways?

To what extent have these efforts been sufficient to ensure that all children, youth, and families are served in the community? What else could have been done?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

8. How has (name of grant program) worked to reduce the number of children and youth who are served in settings that separate them from typically developing children and youth and/or their families? [If response is “not a problem” (e.g., children are never served in restrictive environments), probe for what allows them to serve children in the least restrictive settings.] (A.10.a.)

[Data entry: code mechanisms]

How effective have these efforts been, and in what ways?

To what extent have these efforts been sufficient to eliminate the use of restrictive settings/service options?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

8. Since grant funds were received, what has been done to ensure that services and supports in (name of grant program) ’s service array have sufficient capacity (for example, expanding network of contract providers, increasing contracts to increase capacity)? [Probe for capacity across all services in the array.] (B.8.a.)

[Data entry: code mechanisms]

How effective have these efforts been, and in what ways?

To what extent have these efforts been sufficient? Are there any services and supports that some children, youth, and/or families cannot get because capacity is limited?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

8. Since grant funds were received, what has (name of grant program) done to make services more accessible? [Probe for flexible hours including after hours and weekends; services offered in settings such as home, schools, community, accessible transportation, child care. Probe for applicability across all services in the array.] (B.8.b.)

[Data entry: code mechanisms]

How effective have these efforts been, and in what ways?

To what extent have these efforts been sufficient? What else could be done to make services more accessible?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Achievement of Goals

8. Since grant funds were received, what have been (name of grant program) ’s most important achievements?

How has your community benefited from (name of grant program) ?

8. How has your agency’s involvement in the (name of grant program) affected the way your agency does business, (for example, how it serves children, youth, and families, its management and operations)? (A.6.d.)

Please provide examples of specific things you/your staff are doing differently. [Probe for behavior changes beyond just knowledge acquisition and change in attitude or philosophy.]

5=Agency has changed program operations and methods of service delivery in two or more ways to incorporate system-of-care principles

4=Agency has changed program operations and methods of service delivery in one way to incorporate system-of-care principles

3=Agency has accepted and adapted the philosophy inherent in the system-of-care model and program theory

2=Agency has received information or training related to the wraparound approach or system-of-care principles

1=Agency has not received any information or training related to the wraparound approach or system-of-care principles

14. To what extent has (name of grant program) had an influence on the larger system across all child-serving sectors during the last year?

15. How well would you say (name of grant program) has worked to move toward the establishment of a jurisdiction-wide system of care?

16. How well would you say (name of grant program) has collaborated with other state efforts to transform mental health services during the last year?

17. What have been the major obstacles to expanding the system-of-care principles and philosophies beyond the scope of (name of grant program) ?

Sustainability

18. Please tell me about your community’s effort to sustain the system of care as CMHS grant funds decrease over time. [Probe to determine how they are responding to increased matching fund requirements.]

Have any aspects of the system/program been changed, reduced, or eliminated due to the decreases in CMHS funding? If so, what has occurred?

19. What aspects of (name of grant program) do you anticipate will be sustained over time? [Probe for family-driven, individualized/person-centered and youth-guided care, culturally competent, interagency involvement, accessible services, coordinated, community-based and least restrictive care, etc.]

20. What barriers exist that may hinder sustaining system change over time?

End of Instrument:

Thank you for participating in the System of Care Expansion Assessment portion of the National Systems of Care Expansion Evaluation.

Attachment 8c - Form C

OMB No. xxxx-xxxx

Expires: xx/xx/201x

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

SYSTEM OF CARE EXPANSION ASSESSMENT

FAMILY REPRESENTATIVE

INTRODUCTION

Thank you for your willingness to complete the system of care expansion assessment. The purpose of this interview is to identify the strategies and mechanisms implemented at the local service system level, and examine the extent to which these mechanisms are being implemented from family representative’s perspective.

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of the federal system of care expansion grant program.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 1 hour and 30 minutes.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

Your responses to these questions will be kept confidential and will not be shared outside of the evaluation team. In any of our reports, your responses will be combined with other people’s responses, so your answers will never be attributed to your name.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

The SOCEA interview focuses on the local-level management, structure, and procedures as well as direct service delivery level involving activities and practices in the delivery of direct services.

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

OMB No. xxxx-xxxx

Expires: xx/xx/201x

C. Family Representative/Representative of Family/Advocacy Organizations

Background

1. Do you represent a family advocacy organization?

1=No If no, skip to Question 4 on Page 2

2=Yes If yes, continue

a. What is the name of the organization you represent?

b. What is your role in that organization?

2. Please provide a brief history of the family advocacy organization you represent.

How long has it been active?

What are its funding sources?

What are its primary functions? [Probe for any indirect or direct services provided.]

[Data entry: code services]

What group of people does it represent or serve?

3. How would you characterize your organization’s relationship with the children’s mental health service system? [Probe for the formal and informal relationships, nature of the working relationship, etc.]

To what extent do you think that the goals of the service system fit with the mission or goals of your organization?

Management and Operations

4. Are there any mechanisms in place or have efforts been made to involve family members in program operations of (name of grant program) such as designing and implementing programs, providing staff training, working as staff, peer-support provision, resource specialists, helping with staff recruitment, developing the service array, attending management meetings, contributing to social marketing, etc.? (A.1.b)

If yes, please describe all of the different ways family members have been involved.

[Note: Do not count involvement in SOC evaluation.]

[Continue to probe for examples until the respondent reports no more.]

[Data entry: code ways]

5=Four or more examples of family involvement in program operations

4=Three examples of family involvement in program operations

3=Two examples of family involvement in program operations

2=One example of family involvement in program operations

1=No examples of family involvement in program operations

5. Are there any mechanisms in place or have efforts been made to involve youth in program operations such as designing and implementing programs, providing staff training, working as staff, helping with staff recruitment, attending management meetings, etc.? (A.2.a.)

If yes, please describe all of the different ways youth have been involved.

[Note: Do not count involvement in the SOC evaluation.]

[Continue to probe for examples until the respondent reports no more.]

[Data entry: code ways]

5=Four or more examples of youth involvement in program operations

4=Three examples of youth involvement in program operations

3=Two examples of youth involvement in program operations

2=One example of youth involvement in program operations

1=No examples of youth involvement in program operations

Entry into Services & Accessibility

6. Have there been any outreach activities or other efforts to inform the children, youth and families about (name of grant program) and available services? If yes, please describe. (D.8.a.)

[Data entry: code outreach activities]

How effective would you say these efforts have been, and in what ways?

To what extent have these efforts been sufficient to ensure that all children, youth, and families who need (name of grant program know that it’s here?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

7. Have there been any outreach activities to specific cultural groups or populations (e.g., language, religion, historical trauma, gender identity, alternative lifestyle, etc.)? If yes, please describe. (D.5.a.)

How effective would you say these activities have been, and in what ways?

To what extent have these efforts been sufficient to ensure that these specific groups or populations are aware of (name of grant program) ?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

8. What efforts have been made to ensure that the process for entry into services is easy and straightforward for families? [Probe for language level, family-friendly, person-centered etc.] (D.1.a.)

How effective would you say these efforts have been, and in what ways?

To what extent have they been sufficient?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made but have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Service Delivery

9. Since grant funds were received, what policies, structures or mechanisms have (name of grant program) implemented to promote family-driven service delivery? (For example, use of flexible funds to meet unique needs, development of specific services, utilization of natural supports, training on family-driven service delivery) (A.1.a.)

[Data entry: code mechanisms]

How effective would you say these efforts have been, and in what ways?

To what extent have these efforts been sufficient? What other things that could be done to maximize the provision of family-driven care?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

10. Since grant funds were received, what policies, structures or mechanisms has  (name of grant program) implemented to promote the provision of individualized, person-centered care (for example, use of flexible funds to meet unique needs, development of specific services and supports, training on the provision of individualized services and supports)? (A.3.a.)

[Data entry: code mechanisms]

Have these efforts been effective? If yes, in what ways?

To what extent have these efforts been sufficient? What other things could be done to maximize the provision of individualized, person centered services and supports?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

11. What efforts have been made to promote the provision of culturally and linguistically competent care? (A.5.a.)

Probe for:

• Training on community characteristics, traditions, culture, language, religion, historical trauma, gender identity, alternative lifestyles?

• Effort to meet language needs of clients. Are formal policies in place that address language needs?

• Training on how to build on family and youth culture in the delivery of services

How effective would you say these efforts have been, and in what ways?

Overall, to what extent have efforts been sufficient to ensure the provision of cultural and linguistically competent service delivery? What other things could be done to maximize culturally and linguistically competent service delivery?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

12. Does the system of care service array include services and supports for families? If yes, what are they? (B.1.a)

If yes, please describe them?

Are there services and supports that families should have access to that are not available in the array? If yes, what are they?

To what extent would you say that the service array includes all the types of services and supports that families need?

5=The service array includes all the services and supports that families need

4=The service array includes most of the services and supports that families need; a few are missing.

3=The service array includes some services and supports that families need, but several are missing.

2= The service array includes a few of the services and supports that families need, but many are missing

1= The service array includes none or very few of the services and supports that families need

12.a. How are families connected to family support services (e.g., respite, peer support, mentoring, flexible funds,)? (D.1.b.)

What efforts are in place to ensure that families receive information and access to support services as they enter (name of grant program)? [Probe for whether families have access to supports right away as opposed to having to wait until the formal service planning process].

How effective have these efforts been?

To what extent have these efforts been sufficient? Are there any additional efforts that should be made to ensure that families receive information and access to support services? [Probe for what more should be done.]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

12.b Are family advocacy or family peer mentorship (e.g., peer support providers, family resource specialists) services available through (name of grant program)?

If yes, please describe these services?

[Data entry: code types of services]

Please describe how and when families entering services learn about the availability of peer support services and are linked to them. (D.1.c.)

[Probe for whether families have access to family peer support right away as opposed to having to wait until the formal service planning process].

What proportion of families is linked to peer support services?

Do families sometimes experience problems trying access support services? If yes, please describe. [Probe for problems with insufficient capacity, and other access problems].

To what extent are family support services sufficiently available so that all families who want them can access them? Explain.

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

13. Are there any mechanisms in place or have efforts been made to facilitate the coordination of services and supports across agencies, organizations, and providers (for example, interagency team meetings, joint staff treatment team meetings, interagency case management meetings, etc.)? If yes, please describe. (A.7.b.)

[Data entry: code mechanisms]

How effective would you say these efforts have been, and in what ways?

To what extent have these efforts been sufficient? What else can be done to enhance coordination of services across agencies, organizations, and providers?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

14. Since grant funds were received, what has been done to ensure that services and supports in (name of grant program)’s service array have sufficient capacity such that all the children, youth, and families have access to them (for example, expanding network of contract providers, increasing contracts to increase capacity)? [Probe for capacity across all services in the array.] (B.8.a.)

[Data entry: code mechanisms]

How effective would you say these efforts have been, and in what ways?

To what extent have these efforts been sufficient? Are there any services and supports that some children, youth, and/or families cannot get because capacity is limited?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

15. Since grant funds were received, what has been done to make services more accessible in terms of scheduling of services or locations where services and supports are provided? [Probe for flexible hours including after hours and weekends; services offered in settings such as home, schools, community. Probe for applicability across all services in the array.] (B.8.b.)

[Data entry: code mechanisms]

How effective would you say these efforts have been, and in what ways?

To what extent have these efforts been sufficient? What else could be done to make services more accessible?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

16. Has the project put any mechanisms in place to minimize financial barriers to services and supports for children, youth, and families served by (name of grant program)? [Probe for sliding scale fees, scholarship funds, flexible funding, use of health insurance navigators, process for enrollment via HealthCare.gov or other health exchange, etc.] (B.8.c)

[Data entry: code mechanisms]

How effective have these efforts been, and in what ways?

Overall, have efforts been sufficient or has cost continued to be a barrier for some families to access the services they need?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

17. Please describe how flexible funds are used in (name of grant program)? (B.1.b. & B.3.b.)

What are the funds used for?

[Data entry: code uses]

What is the process for accessing flexible funds?

For what proportion of families are flexible funds used? Do all families who need flexible funds receive them?

To what extent are flexible funds used effectively to meet the needs of children and youth?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

To what extent are flexible funds used effectively to meet the needs of caregivers and families?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

18. How has the cultural background (e.g., race, traditions, culture, language, religion, historical trauma, gender identity, alternative lifestyles) of the intended population been considered in developing the services and supports in the service array? (B.5.a.)

Since grant funds were received, how have cultural organizations, community groups, etc. been involved in efforts such as providing services and supports, developing the service array, advising providers, etc.?

Has (name of grant program) added or modified any services or supports to address the cultural needs of specific groups? If yes, please describe.

How effective have these efforts been?

To what extent have efforts been adequate or sufficient to fully address the cultural needs of all children, youth, and families? What else can be done to meet the cultural needs of children, youth and families?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

19. What efforts are made to ensure that services provided through (name of grant program) have evidence of effectiveness?

What evidence-supported services or treatment models are provided through (name of grant program)? (B.4.a.)

-Family services and supports?

-Child/youth services and supports?

[Data entry: code evidence-supported models, treatments, approaches]

What proportion of services provided through (name of grant program) has evidence supporting their effectiveness?

5=All or almost all of the services and supports are evidence-supported

4=Most of the services and supports are evidence-supported

3=Some of the services and supports are evidence-supported

2=A few of the services and supports are evidence-supported

1=No, or almost no, services and supports are evidence-supported

What proportion of children, youth, and families receive evidence-supported services through (name of grant program)?

5=All or almost all children, youth, and families receive evidence-supported services and supports

4=Most children, youth, and families receive evidence-supported services and supports

3=Some children, youth, and families receive evidence-supported services and supports

2=A few children, youth, and families receive evidence-supported services and supports

1=No, or almost no, children, youth, and families receive evidence-supported services and supports

20. What efforts are made to ensure that evidence-supported services and treatments are provided with good fidelity and quality?

20.a. Are practitioners trained on evidence-supported services and treatments? (B.4.b.)

• If yes, who is trained?

• How are they trained (e.g., learning collaborative model)?

Have these efforts been effective to ensure practitioners have the skills to implement evidence-supported services and supports? If yes, explain.

Have these efforts been sufficient? Are there any additional efforts that should be made to ensure that practitioners provide evidence-supported models? [Probe for what more should be done.]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

20.b. How is fidelity to models and approaches maintained in practice (e.g., manuals provided to practitioners, booster training, ongoing professional development)? (B.4.c)

How effective have these efforts been to support faithful implementation of evidence-supported services and supports? If yes, explain.

Have these efforts been sufficient? Are there any additional efforts that should be made to ensure that practitioners faithfully implement evidence-supported models? [Probe for what more should be done.]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

20.c. Is there any effort to monitor fidelity to models and approaches across (name of grant program) and over time? If yes, please describe. (C.4.a.)

How effective have these efforts been to ensure that evidence-supported services and supports are implemented faithfully across the (name of grant program) and over time?

To what extent have these efforts been sufficient? Are there any additional efforts that should be made to ensure that evidence-supported services and supports are implemented faithfully across the (name of grant program) and over time? [Probe for what more should be done.]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Least Restrictive & Community Based

21. What efforts have been made to minimize the need for children, youth, and families to travel out of their home communities for services? [If response is “not a problem” (i.e., all services received in the community), probe for what allows them to serve all children, youth, and families in the community.] (A.9.a.)

[Data entry: code mechanisms]

How effective have these efforts been?

To what extent have these efforts been sufficient to ensure that all children, youth, and families are served in the community? What else can be done to ensure that all children, youth, and families are served in the community?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

22. What efforts have been made to reduce the number of children and youth who are served in settings that separate them from typically developing children and youth and/or their families? [If response is “not a problem” (e.g., children are never served in restrictive environments), probe for what allows them to serve children and youth in the least restrictive settings.] (A.10.a.)

[Data entry: code mechanisms]

How effective have these efforts been?

To what extent have these efforts been sufficient to eliminate the use of restrictive settings/service options? What else could be done?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

22.a. If a child or youth is placed outside of their community or in a restrictive setting What policies, structures or mechanisms have been implemented to ensure that children/youth and their families maintain regular contact with (name of grant program)? (A.10.b.)

[Data entry: code mechanisms]

How effective have these efforts been?

To what extent have these efforts been sufficient? What else could be done to maximize continuity of services with (name of grant program)?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

22.b. What policies, structures or mechanisms have been implemented to ensure that children/youth do not experience a disruption of services/supports upon return to their community following a restrictive or out-of community placement? (A.10.c.)

[Data entry: code mechanisms]

How effective have these efforts been?

To what extent have these efforts been sufficient? Are all children/youth immediately re-entered into (name of grant program) services/supports upon return to their community? Please describe.

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Evaluation and Quality Monitoring Activities

In the next few questions, I’m going to ask you about (name of grant program)’s overall system quality monitoring and improvement efforts. In particular, I’ll be asking whether data are being collected in certain areas and how these data are being used to improve the overall system.

23. Since grant funds were received, have there been any efforts to measure and track child and youth outcomes (for example, reduction of symptoms, improvement of social functioning, etc.)? (C.3.a.)

[If yes, ask the following questions.]

What areas (i.e., outcomes) have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND change has been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes that have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

24. What efforts have been made to measure and track family outcomes, that is, whether families served by the grant program are being strengthened by the services and supports they received (for example, reduced stress, improved communication and conflict resolution, better management of child or youth behavior, improved family dynamics)? (C.1.a.)

[If yes, ask the following questions.]

What areas (i.e., outcomes) have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND change has been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes that have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

25. Since grant funds were received, has there been any effort to measure family experiences with (name of grant program)’s service processes (e.g., wait time for services, ease of access, coordination of services, satisfaction with services, etc.)? (C.1.b.)

[If yes, ask the following questions.]

What areas or processes have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND change has been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes that have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

26. To what extent are family representatives involved in the quality monitoring process for (name of grant program) ? [Probe for membership on committees (e.g., family advisory to evaluation, program CQI committee), initiating special studies, participating in data collection, reporting findings to stakeholders, etc.] (C.1.c.)

Have there been any other quality monitoring activities that family members have participated in?

[Data entry: code roles]

[Continue to probe for examples until the respondent reports no more.]

5=Families involved in four different roles

4=Families involved in three different roles

3=Families involved in two different roles

2=Families involved in one role

1=Families not involved

27. To what extent are youth representatives involved in the quality monitoring process for (name of grant program) ? [Probe for membership on committees (e.g., youth advisory to evaluation, program CQI committee), initiating special studies, participating in data collection, reporting findings to stakeholders, etc.] (C.2.b.)

Have there been any other quality monitoring activities that youth have participated in?

[Data entry: code roles]

[Continue to probe for examples until the respondent reports no more.]

5=Youth involved in four different roles

4=Youth involved in three different roles

3=Youth involved in two different roles

2=Youth involved in one role

1=Youth not involved

28. How have evaluation and/or quality monitoring findings shared with families and youth?

Lessons Learned

29. Since grant funds were received, what have been the most effective strategies/activities undertaken to develop and implement the system/program?

30. Since grant funds were received, what have been the least effective strategies/activities undertaken to implement the system/program?

Achievement of Goals

31. Since grant funds were received, what have been the grant program’s most important achievements?

How has your community benefited from the system/program?

32. To what extent would you say (name of grant program) has had an influence on the larger system across all child-serving agencies?

33. To what extent would you say (name of grant program) has worked to move toward the establishment of a jurisdiction-wide system of care?

34. To what extent would you say (name of grant program) has collaborated with other state efforts to transform mental health services during the last year?

35. What have been the major obstacles to expanding the system-of-care principles and philosophies beyond the scope of (name of grant program) ?

Sustainability

36. Please tell me about the status of your community’s effort to sustain its system of care as CMHS grant funds decrease over time. [Probe to determine how they are responding to increased matching fund requirements.]

Have any aspects of the system/program been changed, reduced, or eliminated due to the decreases in CMHS funding? If so, what has occurred?

37. What aspects of the system/program do you anticipate will be sustained over time? [Probe for family-driven, individualized/person-centered and youth-guided care, culturally competent, interagency involvement, accessible services, coordinated, community-based and least restrictive care, etc.]

38. What barriers exist that may hinder sustaining system change?

End of Instrument:

Thank you for participating in the System of Care Expansion Assessment portion of the National Systems of Care Expansion Evaluation.

Attachment 8c - Form D

OMB No. xxxx-xxxx

Expires: xx/xx/201x

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

SYSTEM OF CARE EXPANSION ASSESSMENT

YOUTH REPRESENTATIVE

INTRODUCTION

Thank you for your willingness to complete the system of care expansion assessment. The purpose of this interview is to identify the strategies and mechanisms implemented at the local service system level, and examine the extent to which these mechanisms are being implemented from youth representative’s perspective.

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of the federal system of care expansion grant program.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 1 hour and 30 minutes.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

Your responses to these questions will be kept confidential and will not be shared outside of the evaluation team. In any of our reports, your responses will be combined with other people’s responses, so your answers will never be attributed to your name.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

The SOCEA interview focuses on the local-level management, structure, and procedures as well as direct service delivery level involving activities and practices in the delivery of direct services.

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

OMB No. xxxx-xxxx

Expires: xx/xx/201x

D. Youth Coordinator/Representative

Background Information

1. What is the structure of (name of grant program), and how do the youth-guided activities fit in it?

Describe briefly the type of clientele served by (name of grant program) and its key goals.

2. Since grant funds were received, what supports have the (name of grant program) provided to youth-guided efforts? [Probe for financial, materials, training, supervision, monitoring, etc.]

2.a. What has your role been?

Management and Operations

3. Since grant funds were received, what efforts have been made to involve youth in program operations such as providing staff training, serving as volunteer or paid program staff, peer mentors, youth group leaders, developing the service array, attending management meetings, governing board, evaluation advisory committee, contributing to social marketing and/or ourtreach, etc.? (A.2.a.)

[Note: Do not count involvement in evaluation or conducting outreach activities.]

[Continue to probe for examples until the respondent reports no more.]

[Data entry: code ways]

5=Four or more examples of youth involvement in program operations

4=Three examples of youth involvement in program operations

3=Two examples of youth involvement in program operations

2=One example of youth involvement in program operations

1=No examples of youth involvement in program operations

Peer Support Services

4. Since grant funds were received, have youth-guided support services or activities been included in the service array? Please describe. [Probe for advocacy, youth group, mentoring, informal peer-peer support, etc.] (B.2.a.)

If yes, please describe these services or activities?

[Data entry: code types of services]

Are there youth-guided support services or activities that youth should have access to that are not available in the array? If yes, what are they?

To what extent would you say that the service array includes all the types youth-guided support services or activities that youth need?

5=The service array includes all the youth-guided support services that youth need

4=The service array includes most of the youth-guided support services that youth need; a few are missing.

3=The service array includes some youth-guided support services that youth need, but several are missing.

2= The service array includes a few of the youth-guided support services that youth need, but many are missing

1= The service array includes none of the youth-guided support services that youth need

5.a Please describe how children and youth receiving services are informed of the availability of peer support services and are linked to them. (D.2.a.) [Probe for whether youth have access to family peer support right away as opposed to having to wait until the formal service planning process].

What proportion of youth is linked to peer support services?

To what extent is this sufficient to fill the need for these services? Should more be done to ensure that children and youth receive all the peer support services they need? Explain.

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

5.b. How long does it typically take for children and youth to begin receiving youth-guided and/or peer-to-peer support services?

To what extent does the process ensure that children and youth who want to participate in these programs will be able to in a timely way? (D.2.a)

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

6. To what extent do children and youth have access to youth support and peer mentoring programs throughout their service process? (F.2.a.)

5=Services are accessible through-out the service process

4= Services are accessible for most of the service process

3= Services are accessible for some of the service process

2= Services are accessible for a little of the service process

1= Services are accessible at no point or for very little of the service process

Evaluation and Quality Monitoring

In the next few questions, I’m going to ask you about (name of grant program)’s overall system quality monitoring and improvement efforts. In particular, I’ll be asking whether data are being collected in certain areas and how these data are being used to improve the overall system.

7. Since grant funds were received, have there been any efforts to measure and track child and youth outcomes (for example, reduction of symptoms, improvement of social functioning, etc.)? (C.3.a.)

[If yes, ask the following questions.]

What areas (i.e., outcomes) have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND change has been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes that have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

8. Since grant funds were received, has there been any effort to measure youth experiences with (name of grant program)’s service processes (e.g., wait time for services, ease of access, coordination of services, satisfaction with services, etc.)? (C.2.b.)

[If yes, ask the following questions.]

What areas or processes have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND change has been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes that have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

9. To what extent are youth representatives involved in the quality monitoring process for (name of grant program) ? [Probe for membership on committees (e.g., youth advisory to evaluation, program CQI committee), initiating special studies, participating in data collection, reporting findings to stakeholders, etc.] (C.2.a.)

What other quality monitoring activities have youth participated in?

[Data entry: code roles]

[Continue to probe for examples until the respondent reports no more.]

5=Youth involved in four or more different roles

4=Youth involved in three different roles

3=Youth involved in two different roles

2=Youth involved in one role

1=Youth not involved

Entry into Services

Now, let’s talk about when youth first enter (name of grant program). We are interested in their involvement in the enrollment process, as well as your perspective on the process as a youth coordinator.

10. Since grant funds were received, what kind of efforts have been made by (name of grant program) to reach out to youth in your community to inform them about the project and available services and supports? If yes, please describe. (D.8.a.)

[Data entry: code outreach activities]

How effective have these efforts been effective? If yes, how?

To what extent have these efforts been sufficient to ensure that all youth who need (name of grant program) know that it is here?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made but have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

10.a. As youth coordinator/representative, in what ways have you been you involved in this process?

8. What efforts have been made to ensure that the process for entry into services is easy and straightforward for children and youth? [Probe for language level, youth-friendly, person-centered, peer support, etc.] (D.3.a.)

How effective have these efforts been effective? If yes, how?

To what extent have they been sufficient?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made but have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Service Planning

12.a. Since grant funds were received in what ways has (name of grant program) involved children and youth in planning and developing their own plans of care?

[Probe for whether children/youth are commonly involved.]

12.b. Please provide specific examples of how children and youth have been involved in: (E.3.a.)

- identifying and prioritizing their problems or concerns

- developing goals and objectives

- requesting participants in the service planning process

- rejecting participants in the service planning process

- identifying and choosing service options

- rejecting service options

In general, has the process involved children and youth as much as it could have? If no, in what ways could it have been better?

[Note: If the situation has not come up but it would be possible, assign ½ point.]

5=Children/youth have been involved in service planning in at least 6 ways AND respondent reported that involvement has been sufficient

4=Children/youth have been involved in service planning in 5 ways OR involved in 6 ways but respondent reported it could have been better

3=Children/youth have been involved in service planning in 4 ways

2=Children/youth have been involved in service planning in 3 ways

1=Children/youth have involved in service planning in fewer than 3 ways

Summary

13. What other suggestions or recommendations do you have for how (name of grant program) could improve the way that it serves youth and their families?

Thank you for taking the time to answer my questions. Do you have any questions for me, or is there anything else you want to tell me about youth involvement that we did not discuss?

End of Instrument: Thank you for participating in the System of Care Expansion Assessment portion of the National Systems of Care Expansion Evaluation.

Attachment 8c – Form E

OMB No. xxxx-xxxx

Expires: xx/xx/201x

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

SYSTEM OF CARE EXPANSION ASSESSMENT

EVALUATION QUALITY MONITOR

INTRODUCTION

Thank you for your willingness to complete the system of care expansion assessment. The purpose of this interview is to identify the strategies and mechanisms implemented at the local service system level, and examine the extent to which these mechanisms are being implemented from an evaluation quality monitor’s perspective.

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of the federal system of care expansion grant program.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 1 hour.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

Your responses to these questions will be kept confidential and will not be shared outside of the evaluation team. In any of our reports, your responses will be combined with other people’s responses, so your answers will never be attributed to your name.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

The SOCEA interview focuses on the local-level management, structure, and procedures as well as direct service delivery level involving activities and practices in the delivery of direct services.

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

OMB No. xxxx-xxxx

Expires: xx/xx/201x

E. Evaluation and Quality Monitoring

1. What is the name of the agency that employs you?

What is your position/title?

Describe briefly the type of clientele served by (name of grant program) and its key goals.

2. Are you involved in any structured quality assurance, monitoring, or evaluation efforts?

If yes, what is your role?

If no, end interview.

3. Briefly describe the (name of grant program ’s efforts to monitor or assure quality. What types of monitoring activities have you engaged in during the past year? [Probe for peer review efforts, use of evaluation data, evaluation advisory committees, examination of consumer satisfaction, special studies, data collection, etc.]

4. How are these efforts structured and organized? [Probe for specialized office, standing versus rotating committee(s), types of committees.]

5. How are family representatives involved in the quality monitoring process for (name of grant program) ? [Probe for membership on committees, initiating special studies, participating in data collection, reporting findings to stakeholders, etc.] (C.1.c.)

Have there been any other quality monitoring activities that family members have participated in?

[Data entry: code roles]

[Continue to probe for examples until the respondent reports no more.]

5=Families involved in four or more different roles

4=Families involved in three different roles

3=Families involved in two different roles

2=Families involved in one role

1=Families involved in no roles

6. How are youth involved in the quality monitoring process for (name of grant program) ? [Probe for membership on committees, initiating special studies, participating in data collection, reporting findings to stakeholders, etc.] (C.2.a.)

Have there been any other quality monitoring activities that youth have participated in?

[Data entry: code roles]

[Continue to probe for examples until the respondent reports no more.]

5=Youth involved in four or more different roles

4=Youth involved in three different roles

3=Youth involved in two different roles

2=Youth involved in one role

1=Youth involved in no roles

7.a. Have there been any efforts to involve other child-serving agencies in the quality monitoring process? [Probe for membership on committees, initiating special studies, participating in data collection, reporting findings to stakeholders, etc.] (C.6.a.)

If yes, which agencies have been involved and in what ways?

1 = Mental Health 4 = Juvenile Justice 7 = Intellectual Disabilities

2 = Education 5 = Public Health 8 = Other (describe)

3 = Child Welfare 6 = Substance Abuse

[circle all that apply]

5=Five or more agencies involved

4=Four agencies involved

3=Three agencies involved

2=Two agencies involved

1=One agency involved

7. b. If yes to 7.a., in what roles have the agencies been involved? (C.6.a.)

[Data entry: code roles]

Have the agencies been involved in any other roles?

5=Agencies involved in four or more different roles

4=Agencies involved in three different roles

3=Agencies involved in two different roles

2=Agencies involved in one role

1=Agencies involved in no roles

8. What efforts have been made to ensure that the quality monitoring process is culturally competent? [Probe for efforts to have diversity on committees, to make the process comfortable for all, facilitate contributions, etc.] (C.5.b.)

To what extent have any of these efforts been effective in getting you closer to the goal of having a culturally competent quality monitoring process?

Do you think these efforts have been sufficient? What else could be done?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Now I’m going to ask you about (name of grant program)’s overall system quality monitoring and improvement efforts. In particular, I’ll be asking whether data are being collected in certain areas and how those data are being used to improve the overall system.

9. Since CMHS grant funds were received, what efforts have been made to measure and track family outcomes, that is, whether families served by the grant program are being strengthened by the services they receive (for example, reduced stress, improved communication and conflict resolution, better management of child behavior)? (C.1.a.)

[If yes, ask the following questions.]

What areas (i.e., outcomes) have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND changes have been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

10. Since grant funds were received, has there been any effort to measure family experiences with (name of grant program)’s service processes (e.g., wait time for services, ease of access, coordination of services, satisfaction with services, etc.)? (C.1.b.)

[If yes, ask the following questions.]

What areas or processes have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND changes have been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

11. Since grant funds were received, have efforts been made to monitor and improve how well services have been individualized for children and youth served by the grant program? (C.3.b.)

[If yes, ask the following questions.]

What areas have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND changes have been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

12. Since grant funds were received, have there been any efforts to measure and track child and youth outcomes (for example, reduction of symptoms, improvement of social functioning, etc.)? (C.3.a.)

[If yes, ask the following questions.]

What areas (i.e., outcomes) have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND changes have been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

13. Since grant funds were received, have there been any efforts to monitor youth experiences with (name of grant program), for example, how satisfied youth have been with services? (C.2.b.)

[If yes, ask the following questions.]

What areas have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND change has been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes that have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

14. a. Has there been any effort to examine the cultural competence of the services provided through (name of grant program) in particular? Please describe. (C.5.a.)

[If yes, ask the following questions.]

What areas have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND changes have been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

15. Has there been any effort to assess interagency involvement in the system and service delivery? (C.6.b.)

[If yes, ask the following questions.]

What areas have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND changes have been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

16. Has there been any effort to assess how well services are coordinated (for example, continuity of care, service transitions, information sharing among providers involved in the care of an individual family, etc.)? Can you give me an example? (C.7.a.)

[If yes, ask the following questions.]

What areas have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND changes have been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

17. What efforts have been made to examine the accessibility of services and of the system as a whole? Give me an example. (C.8.a.)

[If yes, ask the following questions.]

What areas have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND changes have been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

18. Have there been any attempts to measure and track how often, how many, or how long children and youth are served outside of their home communities? Give me some examples. (C.9.a.)

[If yes, ask the following questions.]

What areas have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND changes have been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

19. Since grant funds were received, have there been any efforts to measure and track whether children and youth were being served in settings that were more restrictive than was necessary? Can you give me some examples? (C.10.a.)

[If yes, ask the following questions.]

What areas have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND changes have been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

20. Since receiving grant funds, has there been any effort to monitor fidelity to evidence-supported models and approaches? If yes, please describe. (C.4.a.)

[If yes, ask the following questions.]

What evidence-supported models/approaches have been assessed?

Has that information been examined or analyzed?

[If information has been examined, ask:]

Have changes in services or in the system been made to respond to any problems identified? Please give me some examples.

Has there been a follow-up examination to find out whether the changes led to service or system improvements?

[If there was follow-up examination, ask:]

Were changes found? Please describe.

5=Data collected and examined in this area AND changes have been made to respond to problems found AND follow-up examination found improvement in problem OR examination of information collected found no need for improvement

4=Data collected and examined in this area AND changes have been made to respond to the problems found BUT follow-up examination of improvement in problem has not been conducted or is incomplete OR follow-up examination showed no improvements resulted from the change

3=Data collected in this area AND data examined BUT respondent reported NO changes have been made to respond to problems found

2=Data collected in this area but the information has not yet been examined

1=No data collected in this area; no effort to monitor quality in this area

888=Respondent unaware of quality monitoring or evaluation activities

21 Those are all the questions I have for you today. Is there anything that I didn’t cover that you would like me to know about (name of grant program)’s quality monitoring/assurance process?

End of Instrument:

Thank you for participating in the System of Care Expansion Assessment portion of the National Systems of Care Expansion Evaluation.

Attachment 8c – Form F

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

SYSTEM OF CARE EXPANSION ASSESSMENT

CARE COORDINATOR

INTRODUCTION

Thank you for your willingness to complete the system of care expansion assessment. The purpose of this interview is to identify the strategies and mechanisms implemented at the local service system level, and examine the extent to which these mechanisms are being implemented from a care coordinator’s perspective.

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of the federal system of care expansion grant program.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 1 hour and 42 minutes.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

Your responses to these questions will be kept confidential and will not be shared outside of the evaluation team. In any of our reports, your responses will be combined with other people’s responses, so your answers will never be attributed to your name.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

The SOCEA interview focuses on the local-level management, structure, and procedures as well as direct service delivery level involving activities and practices in the delivery of direct services.

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

OMB No. xxxx-xxxx

Expires: xx/xx/201x

F. Care Coordinator/Facilitator

These questions relate specifically to services provided to children, youth, and families served by (name of grant program) since receiving your CMHS grant funds.

Section I - Service Planning

1. What is your title?

Which agency/organization do you work for?

How long have you been employed by this agency?

Describe briefly the type of clientele served by (name of grant program) and its key goals.

2. Since grant funds were received, how have cases been assigned to you for care coordination (insert system-specific term)? [Probe for whether children, youth, or families are assigned (insert grant-specific term) based upon individual characteristics such as cultural preferences, gender, etc. or in other ways]

What has been your typical or average caseload size? Was this manageable? Why or why not?

3. Were all children, youth, and families served by (name of grant program) assigned care coordinators (insert grant-specific term )? If not, which children, youth, or families received (insert grant-specific term) services?

Are there different levels or types of care management/coordination? If yes, please describe

Do care management/coordination services differ based on the needs of a child or family? If so, how does care management/coordination differ? [Probe for levels and types of services, role of intensity of child/youth challenges and other factors]. (F.7.b.)

Are care coordination services sufficiently flexible and varied to meet children’s specific individual care coordination needs?

[Interviewer to rate the efforts made to vary the services based on the needs of a child or family]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

4. Since grant funds were received, what hours have you typically worked? (E.8.a.)

Have you been available to children, youth, and families evenings or weekends?

How frequently or in what percentage of cases have you worked with children, youth, and families after hours or on weekends?

To what extent have you been able to accommodate special scheduling requests?

5=Available at a wide range of times (including after-hours AND weekends), and there was also broad flexibility in scheduling

4=Available at a wide range of times (including after-hours OR weekends), and moderate flexibility in scheduling

3=Range of hours available but SET times for after-hours OR weekends; little flexibility to accommodate special requests

2=Business hours only; special requests accommodated in special (non-emergency) circumstances only

1=Business hours only; special requests not accommodated

5. Can children, youth, and families reach you in emergencies? If so, how?

If no, can children, youth, and families reach someone from (name of grant program)? If so, how are you reached?

6. In what locations and settings do you typically work? Probe for settings including schools, homes, community settings, etc. until list exhausted. (E.8.b.)

Have you been available to work with children, youth, and families in flexible settings (e.g., homes, schools, community settings, etc.)?

On average, what percent of your time is spent directly providing services in each of these settings or locations? [Probe for breakdown of time spent in each location.]

If other locations or settings are desired/needed, to what extent have you been able to accommodate special requests? Please provide examples.

5=Services offered in a wide range of locations (for example, homes, schools, in the community); in addition, there was also very broad flexibility in locations to meet family needs

4=Range of locations offered and moderately broad flexibility in locations to meet family needs

3=Range of locations offered but little flexibility to accommodate special requests

2=Agency offices only; special requests accommodated in special (non-emergency situations) circumstances only

1=Agency offices only; special requests not accommodated

In this section, I’ll be asking you about the service planning process.

7. Since grant funds were received, have service plans been developed for ALL children, youth, and families served by (name of grant program)? If no, in which cases were service plans developed? (E.3.b.)

Of the children enrolled in (name of grant program), what percentage of children, youth, and families have plans in place?

5=All children and youth (100%) have had individualized service plans

4=Almost all (90–99%) children and youth have had individualized service plans

3=Many (75–89%) children and youth have had individualized service plans

2=Some (50–74%) children and youth have had individualized service plans

1=Few (less than 50%) children and youth have had individualized service plans

8. a. What service planning model do you use? Is the same approach used by all (name of grant program) care coordinators (insert grant-specific term)? [Probe for a standardized person-centered approach such as Wraparound]

Is this approach evidence-supported? Describe. (E.4.a.)

1=No

2=Yes

8. b. Did you receive training on how to utilize this approach? (B.4.b.)

• If yes, are all (name of grant program) care coordinators trained in this model?

• How were you trained (e.g., traditional lectures, learning collaborative model, mentoring, supervision)?

How effective have these efforts been to ensure that care coordinators had the skills to implement evidence-supported services and supports? If yes, explain.

To what extent have these efforts been sufficient? Are there any additional efforts that should be made to ensure that care coordinators implement evidence-supported service planning models? [Probe for what more should be done.]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

8. c. Has there been any effort to make sure that care coordinators are practicing the approach as intended over time? (B.4.c)

If yes, please describe how this is done.

To what extent have these efforts been effective to support faithful implementation of evidence-supported service planning?

To what extent have these efforts been sufficient? What, if any, additional efforts should be made to ensure that practitioners faithfully implement evidence-supported service planning models? [Probe for what more should be done.]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

9. Please describe your process for assessing the needs of children and youth. (E.3.b.)

What areas are assessed? [Listen for assessment of multiple life domains.]

What, if any, standardized assessment tools are used in the assessment process?

How effective is the assessment approach used, and in what ways?

Overall, is this assessment process sufficient to obtain a full understanding of needs of children and youth? Are there changes that could be made to improve this process?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

10. a. Please describe your process for assessing the needs of the family as a whole, including parents, other caregivers, siblings, etc. during service planning. (E.1.b.)

What areas are assessed? [Listen for assessment of multiple life domains.]

What, if any, standardized assessment tools are used in the assessment process?

For approximately what percentage of families have you assessed needs?

5=Needs assessed for all (100%) families

4=Needs assessed for almost all (90–99%) families

3=Needs assessed for many (75–89%) families

2=Needs assessed for some (50–74%) families

1=Needs assessed for few (less than 50%) families

10. b. How effective is the assessment approach used, and in what ways?

Overall, is this assessment process sufficient to obtain a full understanding of families’ needs? Are there changes that could be made to improve this process? (E.1.b.)

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

10. c. Please give me examples of the kinds of services and supports that you have identified and arranged to meet the needs of families? (E.1.c.)

[Keep probing for more examples until it is clear that the respondent can think of no more examples or until four examples have been reported.]

Have these efforts been sufficient to assess needs and put services in place to meet the needs. What more could be done?

5=Four or more examples of needs incorporated into service plan AND respondent reported it could not have been better

4=Three examples of needs incorporated into service plan OR 4 or more examples but respondent reported it could have been better

3=Two examples of needs incorporated into service plan

2=One example of needs incorporated into service plan

1=No examples of needs incorporated into service plan

11. a. Since grant funds were received, has the service planning process included an assessment of the strengths of children and youth? (E.3.c.)

If yes, what kinds of questions are asked? [Probe for identification of child and youth strengths in life domains, identification of child and youth strengths using a form, etc.]

For approximately what percentage of children and youth you work with have you assessed strengths?

5=Strengths assessed for all (100%) children and youth

4=Strengths assessed for almost all (90–99%) children and youth

3=Strengths assessed for many (75–89%) children and youth

2=Strengths assessed for some (50–74%) children and youth

1=Strengths assessed for few (less than 50%) children and youth

11. b. Please give me examples of how you have incorporated children and youth strengths into the identification and planning of services. (E.3.c.) [Note: Examples should identify a strength, a service planned based on that strength, AND how the service benefited the child or youth]

[Keep probing for more examples until it is clear that the respondent can think of no more examples or until four examples have been reported.]

To what extent have these efforts been sufficient to assess, identify and incorporate children and youth’s strengths into the service plan. What more could be done?

5=Four or more examples of child or youth strengths incorporated into service plan AND respondent reported it could not have been better

4=Three examples of child or youth strengths incorporated into service plan OR 4 or more examples but respondent reported it could have been better

3=Two examples of child or youth strengths incorporated into service plan

2=One example of child or youth strengths incorporated into service plan

1=No examples of child or youth strengths incorporated into service plan

12.a. Has the service planning process typically included an assessment of the strengths of the family? (E.1.d.)

If yes, what kinds of questions are asked? (e.g., identify family strengths in life domains, identify family strengths using a checklist, etc.)

For approximately what percentage of families have you assessed strengths?

5=Strengths assessed for all (100%) families

4=Strengths assessed for almost all (90–99%) families

3=Strengths assessed for many (75–89%) families

2=Strengths assessed for some (50–74%) families

1=Strengths assessed for few (less than 50%) families

12.b. Please give me examples of how you have used family strengths to identify and plan services. (E.1.d.)

[Keep probing for more examples until it is clear that the respondent can think of no more examples or until four examples have been reported.]

To what extent have these efforts been sufficient to assess, identify and incorporate family strengths into the service plan? What more could be done?

5=Four or more examples of family strengths incorporated into service plan AND respondent reported it could not have been better

4=Three examples of family strengths incorporated into service plan OR 4 or more examples but respondent reported it could have been better

3=Two examples of family strengths incorporated into service plan

2=One example of family strengths incorporated into service plan

1=No examples of family strengths incorporated into service plan

13.a. Since grant funds were received, has the service planning process included a discussion of the culture of the child, youth, and family, for example, things that are important to them such as religion, race/ethnicity, family traditions, beliefs about health and illness, etc.? If so, please describe this process. (E.5.a.)

What kinds of questions do you ask, what kinds of things do you look for?

For approximately what percentage of children, youth, and families have you assessed culture?

5=Culture assessed for all (100%) children, youth, and families

4=Culture assessed for almost (90–99%) children, youth, and families

3=Culture assessed for many (75–89%) children, youth, and families

2=Culture assessed for some (50–74%) children, youth, and families

1=Culture assessed for few (less than 50%) children, youth, and families

13.b. Please give me examples of how you have used the culture of a child, youth, and family to identify or plan services. (E.5.a.)

[Keep probing for more examples until it is clear that the respondent can think of no more examples or until four examples have been reported.]

Have these efforts been sufficient, or could more work be done to utilize families’ culture in the planning of services?

5=Four or more examples of family culture incorporated into service plan AND respondent reported it could not have been better

4=Three examples of family culture incorporated into service plan OR 4 or more examples but respondent reported it could have been better

3=Two examples of family culture incorporated into service plan

2=One example of family culture incorporated into service plan

1=No examples of family culture incorporated into service plan

14. Do clients ever need for the service planning process to be conducted in a language other than English? If yes, how is that handled? (E.5.b.)

If no, is there a process in place should the need arise?

If the need has arisen, how effective have these efforts been?

To what extent is that process sufficient to meet the needs of all clients who do not speak English? How could it be improved?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

15.a. Since grant funds were received, have families typically been present at their service planning meetings? (E.1.a.)

Approximately, in what percentage of meetings have families been present?

5=Family member present in all or almost all (98–100%) meetings

4=Family member present in most (90–97%) meetings

3=Family member present in many (75–89%) meetings

2=Family member present in some (50–75%) meetings

1=Family member present in few (less than 50%) meetings

666=No planning meetings were held

15.b. Since grant funds were received, has the service planning process emphasized family involvement as partners? (E.1.a.)

Please provide specific examples of how families have participated and led in:

- identifying and prioritizing their problems or concerns

- developing goals and objectives

- requesting participants in the service planning process

- rejecting participants in the service planning process

- identifying and choosing service options

- rejecting service options

-other ways

In general, to what extent has the process involved and empowered families as much as it could have? If not entirely, in what ways could it have been better?

[Note: If the situation has not come up but it would be possible, assign ½ point.]

5=Families have been involved as partners in service planning in at least 6 ways AND respondent reported that family leadership has been sufficient

4=Families have been involved as partners in service planning in 5 ways OR involved in 6 ways but respondent reported it could have been better

3=Families have been involved as partners in service planning in 4 ways

2=Families have been involved as partners in service planning in 3 ways

1=Families have been involved as partners in service planning in fewer than 3 ways

16.a. Have you commonly involved children and youth in service planning? Under what circumstances do you think it is appropriate for children and youth to participate in service planning? (E.3.a.)

[If children/youth are commonly involved:]

16.b. Please provide specific examples of how children and youth have been involved in: (E.3.a.)

- identifying and prioritizing their problems or concerns

- developing goals and objectives

- requesting participants in the service planning process

- rejecting participants in the service planning process

- identifying and choosing service options

- rejecting service options

-other ways

In general, to what extent has the process involved children and youth as much as it could have? If not entirely, in what ways could it have been better?

[Note: If the situation has not come up but it would be possible, assign ½ point.]

5=Children/youth have been involved in service planning in at least 6 ways AND respondent reported that involvement has been sufficient

4=Children/youth have been involved in service planning in 5 ways OR involved in 6 ways but respondent reported it could have been better

3=Children/youth have been involved in service planning in 4 ways

2=Children/youth have been involved in service planning in 3 ways

1=Children/youth have involved in service planning in fewer than 3 ways

Immediate Access to Support and Peer Services

17. How are families connected to family support services (e.g., respite, flexible funds)? (D.1.b.)

What efforts are in place to ensure that families receive information and access to needed support services as they enter (name of grant program)? [Probe for whether families have access to supports right away as opposed to having to wait until the formal service planning process].

How effective are these efforts for connecting families to support services?

Is this sufficient to fill the need for these services? Explain.

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

17.a. Are family advocacy or family peer mentorship (e.g., peer support providers) services available through (name of grant program)?

If yes, what has been the role of the family advocate or family peer mentor?

Please describe how served families are informed of the availability of family peer support services and are linked to them. (D.1.c.) [Probe for whether families have access to family peer support right away as opposed to having to wait until the formal service planning process].

What proportion of families is linked to peer support services?

How effective is this process for connecting families to advocacy and peer mentoring?

Is this sufficient to fill the need for these services? Explain.

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

18. Are youth advocacy or peer mentorship (e.g., peer support providers) services available through (name of grant program)?

8. If yes, what has been the role of the youth advocate or youth peer mentor?

[Data entry: code roles]

18.a. Please describe how served youth are informed of the availability of peer support services and are linked to them. (D.2.a.) [Probe for whether youth have access to peer support right away as opposed to having to wait until the formal service planning process].

What proportion of youth is linked to peer support services?

How effective is this process for connecting youth to peer support services?

Is this sufficient to fill the need for these services? Explain.

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Coordination/Collaboration

19. Since grant funds were received, have any of the public child-serving agencies (for example, mental health, health, juvenile justice, education, child welfare) routinely participated in service planning? (E.6.a.) If yes, which ones?

1 = Mental Health 4 = Juvenile Justice 7 = Intellectual Disabilities

2 = Education 5 = Public Health 8 = Other (describe)

3 = Child Welfare 6 = Substance Abuse

[circle all that apply]

Have any agencies been difficult to engage in the planning process (for example, agencies that did not routinely participate, rarely responded to requests to attend planning meetings)? If so, which ones. Explain issues.

1 = Mental Health 4 = Juvenile Justice 7 = Intellectual Disabilities

2 = Education 5 = Public Health 8 = Other (describe)

3 = Child Welfare 6 = Substance Abuse

[circle all that apply]

Overall, how frequently would you say that all of the agencies involved with a child, youth, or family have participated in the service planning process?

5=Routine participation of most or all involved child-serving agencies such that they actively participated

4=Frequent but not routine participation of most or all involved child-serving agencies

3=Frequent participation was not routine for all agencies; some involved agencies routinely participated but others participated only sporadically

2=Few agencies routinely participated such that it was rare that all involved agencies participated in service planning

1=None of the involved agencies participated in service planning

20. Since grant funds were received, have others who provide support or services to a child, youth, or family, such as organizations in the community, direct service providers, or private providers, participated in service planning? (E.7.a) (Probe for primary health care providers and substance abuse treatment providers)

If yes, who tends to participate (generally)?

Overall, how frequently would you say that ALL of the other organizations and providers involved with a child, youth, or family have participated in service planning?

5=Routine participation of most or all involved organizations and providers such that they actively participated

4=Frequent but not routine participation of most or all involved organizations and providers

3=Frequent participation was not routine for all organizations and providers; some routinely participated but others participated only sporadically

2=Few organizations and providers routinely participated such that it was rare that all involved organizations and providers participated in service planning

1=None of the involved organizations and providers participated in service planning

21.a. What mechanisms has (name of grant program) put in in place, if any, to coordinate service planning processes across agencies, organizations, or providers? (E.7.b)

Is there a unified service planning meeting or process (for example, all agencies come together to develop cross-agency service plans together)? If yes, please describe.

Have staff attended service planning meetings across agencies? How frequently?

Have you ever attended a service planning meeting at another agency (for example, IEP at the schools, case planning meetings at child welfare)? If yes, please describe.

21.b. To what extent have efforts to coordinate service planning processes been effective? Please describe. (E.7.b.)

To what extent have efforts been sufficient? What else could be done to improve coordination across agencies, organizations, and providers?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Section II - Service Provision

I would like to now spend time talking about the services children, youth, and families have received.

22. Since grant funds were received, have there been occasions when children and youth who were enrolled in (name of grant program) have not received all of the services that were planned? (F.3.a.)

If yes, under what circumstances (for example, certain services or providers)? Why?

In what percentage of cases has this occurred?

What steps were taken to address this situation?

5=All (100%) children and youth received all services planned for them

4=Almost all (90–99%) children and youth received all services planned

3=Many (75–89% ) children and youth received all services planned

2=Some (50–74%) children and youth received all services planned

1=Few (less than 50%) children and youth received all services planned

23. Have there been occasions when families did not receive all of the services that were planned? (F.1.b.)

If yes, under what circumstances (e.g., certain services or providers)?

In what percentage of cases has this occurred?

What steps were taken to address this situation?

5=All (100%) families received all services planned for them

4=Almost all (90–99%) families received all services planned

3=Many (75–89% ) families received all services planned

2=Some (50–74%) families received all services planned

1=Few (less than 50%) families received all services planned

Service Array

Using the Services Card for Interviewing, ask:

24. Of the services you see listed on this card, which are present in (name of grant program)? [Refer to the Service Array Card for Interviewing. Clarify that this means that the service is available to children, youth, and families served by the grant program regardless of whether the service is in the community or has low capacity.]

[Be sure the respondent has a Service Array Card for Interviewing and read each service option listed. Have the respondent indicate whether each service has been present in the array since receiving grant funds. On the Service Array Card for Scoring, place a check for each service that has been present.]

24.a. Are there any other formal or support services for children, youth, and young adults (not family, sibling, or caregiver services) not listed on this card that are part of (name of grant program)’s service array? Please list.

[Determine whether the service can be coded elsewhere on the list. If not, write additional services on the Service Array Card for Scoring.]

Are there any services that children, youth, and young adults need that are not in the service array? If yes, what are they?

[Interviewer to rate adequacy of for children, youth, and young adults in the service array] (B.3.a.)

Use completed Service Array Card for Scoring to score this item

5=All required child/youth services are in the array AND additional services are also available

4=All required services child/youth are in the array BUT no additional service are present

3=Most required child/youth services are in the array (1–3 missing) (Presence of additional services is not relevant)

2=Some required child/youth services are in the array (4 missing) (Presence of additional services is not relevant)

1=Few required child/youth services are in the array (more than 4 missing) (Presence of additional services is not relevant)

24.b. Are there any other formal or support services specifically for families, siblings or caregivers not listed on this card that are part of (name of grant program)’s service array? Please list.

[Determine whether the service can be coded elsewhere on the list. If not, write additional services on the Service Array Card for Scoring.]

Are there any services that families, siblings or caregivers need that are not in the service array? If yes, what are they?

[Interviewer to rate adequacy of family services and supports in the service array] (B.1.a.)

Use completed Service Array Card for Scoring to score this item

5=All services needed by families are in the array

4=Many family services are in the array

3=Some family services are in the array

2=Few family services are in the array

1=No family services are in the array; all services are provided directly to children, youth, and young adults

24.c. Since grant funds were received, have youth-guided support services or activities been included in the service array? Please describe. [Probe for advocacy, youth group, mentoring, informal peer-peer support, etc.] (B.2.a.)

If yes, please describe these services or activities?

[Data entry: code types of services]

Please describe how and when youth entering services learn about the availability youth-guided support services or activities and are linked to them. (D.1.c.)

If yes, please describe them?

Are there youth-guided support services or activities that youth should have access to that are not available in the array? If yes, what are they?

To what extent would you say that the service array includes all the types youth-guided support services or activities that youth need?

5=The service array includes all the youth-guided support services that youth need

4=The service array includes most of the youth-guided support services that youth need; a few are missing.

3=The service array includes some youth-guided support services that youth need, but several are missing.

2= The service array includes a few of the youth-guided support services that youth need, but many are missing

1= The service array includes none of the youth-guided support services that youth need

25. Of the services you’ve identified as having been present in (name of grant program) ‘s service array, which have sufficient capacity to meet the needs of all the children, youth, and/or families who need them? [Circle whether or not each service has sufficient capacity to meet the need.] (F.8.a.)

[Clarify that insufficient capacity relates to services that some children, youth, and/or families cannot get at all, or that the respondent does not refer families to because of limited capacity.]

Use completed Service Array Card for Scoring to score this item

5=All services in the array have fully sufficient capacity such that all children, youth, and families who need them can get them

4=Most services in the array (75–95%) have fully sufficient capacity such that all children, youth, and families who need them can get them

3=Many services in the array (50–74%) have fully sufficient capacity such that all children, youth, and families who need them can get them

2=Some services in the array (25–49%) have fully sufficient capacity such that all children, youth, and families who need them can get them

1=Few services in the array (less than 25%) have fully sufficient capacity such that all children, youth, and families who need them can get them

26. Of the services in your array, how long (in days) have children, youth, and families you’ve served had to wait for non-emergency or non-crisis services? [Refer again to the Service Array Card for Interviewing.] (D.8.b.)

[On the Service Array Card for Scoring record the wait for each service (in days).]

Has (name of grant program) or its providers had to establish waiting lists for services? If yes, for which services?

5=All required services (except emergency/crisis) accessed within 14 days or less

4=Most required services (at least 75%) accessed within 14 days or less; all others accessed within 21 days

3=Many required services (at least 50%) accessed within 14 days or less; all others accessed within 21 days

2=Some required services (at least 25%) accessed within 14 days or less; all others accessed within 21 days

1=Few required services (less than 25%) accessed within 14 days or less

27. What has been a typical wait for emergency or crisis intervention services?

5=Crisis services received in less than 24 hours AND emergency services (e.g., in response to suicide attempt) received immediately

4=Crisis services received within 24 to 48 hours AND emergency services received immediately

3=Crisis services received within 48 to 72 hours AND emergency services received immediately

2=Crisis services received in more than 72 hours AND emergency services received immediately

1=Emergency services not received immediately regardless of timing of crisis services

28.a. Of the services in your array, which are provided in the community? [Refer again to the Service Array Card for Interviewing and circle whether or not each service is available in the community.] (F.9.a.)

Use completed Service Array Card for Scoring to score this item

5=All services in the array are provided in the community

4=Most services in the array (75–95%) are provided in the community

3=Many services in the array (50–74%) are provided in the community

2=Some services in the array (25–49%) are provided in the community

1=Few services in the array (less than 25%) are provided in the community

28.b. How many of the children, youth, and families you’ve worked with since grant funds were received have had to travel outside of their home communities to receive services? (F.9.a.)

How far have these children, youth, and families had to travel?

5=No children, youth, or families have had to leave the community for services

4=Few (1–2) children, youth, or families have had to leave the community for services

3=Some (3–4) children, youth, or families have had to leave the community for services

2=Many (5–6) children, youth, or families have had to leave the community for services

1=Very many (7 or more) children, youth, or families have had to leave the community for services

28.c. When children, youth and families need to leave the community for services, what efforts are made to transition the child back to services in the community? [Listen for transition plan that begins when the out-of-community services begin]. (F.9.b)

To what extent have these efforts been effective? If yes, in what ways?

To what extent have these efforts been sufficient? What else could be done to maximize continuity of contact?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Accessibility

29.a. Since grant funds were received, have children, youth, and families served through (name of grant program) had to pay out of pocket for any of the services they received? (F.8.e.)

If yes, for which services? [List services and how families paid for them (e.g., sliding scale fee, co-payment, full payment, etc.]

What proportion of children, youth, or families who needed these services were able to receive them despite the cost for the services? [For each paid service listed, indicate the proportion of families needing the service who could access it.]

5=Virtually all services were accessible to virtually all children, youth, and families such that it rarely occurred that needed services could not be accessed due to financial constraints

4=Most services were accessible to at least most children, youth, and families such that it rarely occurred that needed services could not be accessed due to financial constraints

3=Many services were accessible to at least many children, youth, and families but there were still key services that could not be accessed or there continue to be a few children, youth, or families who experienced difficulty accessing services due to financial constraints

2=Some services were financially accessible to at least some children, youth, and families

1=Few services were financially accessible to children, youth, and families

29.b. Are there any services that could have benefited the children, youth, and families you serve but have not been not provided because they were too expensive for the family? (F.8.e.)

5=No services were too expensive

4=One service was too expensive

3=Two services were too expensive

2=Three services were too expensive

1=Four or more services were too expensive

30. Since grant funds were received, has (name of grant program) provided transportation assistance to families? (F.8.d.)

If yes, what types of transportation assistance are provided and by whom?

Have there been any restrictions or limitations on transportation assistance, such as time of day, part of town, emergencies only, distances, etc.?

What percentage of children, youth, or families who needed transportation assistance have been able to receive it?

Overall, to what extent has the level of assistance provided by (name of grant program) been adequate?

5=Transportation assistance has been widely available such that all or almost all families could easily access it

4=Transportation assistance has been widely available but a few problems reported; transportation assistance has met the needs of most but not all families

3=Some transportation assistance has been available and sufficient to meet the needs of many families

2=Limited assistance has been available (e.g., assistance was limited to certain services, certain times of day, certain parts of town, certain distances, emergencies only, or certain income levels)

1=Very little or no transportation assistance has been available such that transportation needs for many families have not been met

Least Restrictive

31. What efforts have been made to reduce the number of children and youth who are served in settings that separate them from typically developing children and youth and/or their families? [If response is “not a problem” (e.g., children are never served in restrictive environments), probe for what allows them to serve children and youth in the least restrictive settings.] (F.10.a.)

[Data entry: code mechanisms]

To what extent have these efforts been effective? In what ways?

To what extent have these efforts been sufficient to eliminate the use of restrictive settings/service options?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

32. Thinking across all of the different children, youth, and families you’ve worked with since grant funds were received, has it ever been necessary for a child or youth to be served in a restrictive setting (for example, alternative school, hospital, group home, etc.)? If yes, please describe.

How frequently has this occurred?

33. When children and youth are placed in residential settings, how long do they typically remain there?

Generally speaking, how are residential services utilized in (name of grant program)? [Probe for whether they are intended to be short-term placements.] (F.10.c.)

5= Residential settings are typically utilized as short term placements

4= Residential settings are soften utilized as short-term placements and sometimes as long term placements

3= Residential settings are sometimes utilized as long term placements and sometimes used as short-term placements

2=Residential settings are soften utilized as long term placements and seldom used as short-term placements

1=Residential settings are typically utilized as long term placements

34. When restrictive or out-of-community settings are used, what efforts are made to transition or move the child or youth into less restrictive services or settings once appropriate? If so, please describe. (F.10.b.)

To what extent have these efforts been effective? In what ways?

To what extent have these efforts been sufficient (such that children and youth were transitioned to less restrictive services/settings as soon as possible)? If no, why not?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

34.a. When children and youth are placed in restrictive or out-of-community settings does he/she remain a client of (name of grant program)?

If yes, what policies, structures or mechanisms have been implemented to ensure that children/youth and their families maintain regular contact with (name of grant program)? (A.10.b.)

[Data entry: code mechanisms]

To what extent have these efforts been effective? In what ways?

To what extent have these efforts been sufficient? What else could be done to maximize continuity of contact?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

34.b. When children and youth return to their communities, what is the process for re-integrating them into services within their communities? Please describe the process. (F.9.d) [Probe for quick and smooth re-integration].

How effective have these efforts been, and in what ways?

To what extent have these efforts been sufficient? Are all children/youth immediately re-entered into services/supports upon return to their community?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Care Monitoring

35. Since grant funds were received, have you monitored how the individual children and youth you serve are doing in the service system (e.g., check in on service process and improvement)? (F.3.d.)

If yes, how do you do this?

What kinds of things have you routinely monitored? (e.g., progress, whether referred services were actually received, if not why not, experience with services, etc.)

How frequent have your monitoring efforts been?

Have you monitored service provision for ALL children and youth you work with?

35.a. Overall, to what extent have these efforts been effective? Do you feel that you have been able to monitor or keep up with service provision for all children and youth’s as well and as often as needed? (F.3.d.)

Have you been able to get the information you have needed to adequately monitor service provision?

Have there been any barriers to monitoring service provision for all child or youth’s (e.g., large caseloads)?

Overall, to what extent have these efforts been sufficient? What changes could be made to facilitate monitoring?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

36. How is the monitoring process the same or different for families (in comparison to children and youth)?

Coordination/Collaboration

37. Which agencies (e.g., mental health, education, juvenile justice, child welfare, etc.) provide services to the children, youth, and or families served through (name of grant program)?

1 = Mental Health 4 = Juvenile Justice 7 = Intellectual Disabilities

2 = Education 5 = Public Health 8 = Other (describe)

3 = Child Welfare 6 = Substance Abuse

[circle all that apply]

What types of services do they provide?

[Data entry: code service options]

What proportion of the children, youth, and families you serve receive services from more than one agency?

Are there some children, youth, and families who could benefit from services from multiple agencies who are not able to receive them? Please explain.

Are efforts sufficient to ensure that children, youth, and families receive the services they need from multiple agencies?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

38. Since grant funds were received, how have you worked with other agencies, organizations and providers to coordinate the various services that children youth, and families receive? (F.7.a.)

What kinds of information have you typically shared? What kinds of information have you typically received?

Have there been any obstacles or barriers that inhibit the coordination of services (for example, confidentiality regulations, agency policies, etc.)? Explain.

To what extent have efforts in this area been effective? In what ways?

To what extent have efforts in this area have been sufficient? explain

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

39. What happens when there are transitions or changes in a child, youth, or family’s service process, such as a change in providers or services? Are any efforts made to ensure coordinated transitions? (F.7.c.)

Who is responsible for ensuring that transitions/changes are smooth?

What types of information are exchanged? Between whom?

Is there any follow-up with the child, youth, and family and/or providers to make sure that the transition or change was made and is working?

If yes, how effective have these efforts been? If yes, in what ways?

To what extent have efforts in this area have been sufficient? Explain.

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Summary

40. What is your training/educational background?

1=Less than high school

2=High school

3=BA, BS in mental health, health, social sciences, BSW

4=BA, BS, in other non-health or social science area

5=MA, MS in mental health, health, social sciences, MSW

6=MA/MS in other non-health related area

7=Doctoral degree (what area) ______________________

8=Psychiatrist (MD)

9=Other doctoral degree

41. Since receiving your grant funds, have you received or participated in any training sessions?

If so, please describe the topics or content areas covered. [Probe on CASSP principles such as family involvement, cultural competency, individualized service provision, strengths-based service provision, evidence-supported services, etc.]

Where these training sessions provided as part of grant program activities?

42. Overall, how would you describe the impact of grant program activities has been on how your community serves children and youth with mental health needs?

What impact, if any, has (name of grant program) had on how you provide services?

End of Instrument:

Thank you for participating in the System of Care Expansion Assessment portion of the National Systems of Care Expansion Evaluation.

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

SYSTEM OF CARE EXPANSION ASSESSMENT

DIRECT SERVICES PROVIDER

INTRODUCTION

Thank you for your willingness to complete the system of care expansion assessment. The purpose of this interview is to identify the strategies and mechanisms implemented at the local service system level, and examine the extent to which these mechanisms are being implemented from a direct services provider’s perspective.

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of the federal system of care expansion grant program.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 1 hour.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

Your responses to these questions will be kept confidential and will not be shared outside of the evaluation team. In any of our reports, your responses will be combined with other people’s responses, so your answers will never be attributed to your name.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

The SOCEA interview focuses on the local-level management, structure, and procedures as well as direct service delivery level involving activities and practices in the delivery of direct services.

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

OMB No. xxxx-xxxx

Expires: xx/xx/201x

G. Direct Service Delivery Staff

Introduction

1. Briefly describe your agency/organization and its relationship with  (name of grant program) (e.g., partner, subcontractor, volunteer organization, etc.).

What kinds of services does your agency/organization provide—what does your organization do?

2. How long have you been working with this agency/organization?

3.a. What kinds of services or support has your agency provided to children, youth, and families served by (name of grant program) since their CMHS grant funds were received?

3.b. What is your function?

4. Of the children, youth, and families you have worked with since grant funds were received, how many were involved with (name of grant program) ?

5. How long does it typically take for a child, youth, or family to receive your services once they are referred to you? (F.8.f)

In your experience, what was the shortest time between referral and first service contact?

What was the longest time?

5=Service typically received in 7 days or fewer

4=Service typically received between 8 to 14 days

3=Service typically received between 15 and 21 days

2=Service typically received between 22 and 28 days

1=Service typically received in more than 28 days

Service Provision

Now I would like to talk about access to the services that you provide to children, youth, and families.

6. Since grant funds were received, how have children or youth and their families been assigned or referred to you for services?

Could any person, agency, or provider refer children, youth, and families to you for services? If no, why not?

Have any attempts been made to match children, youth, and families with providers based on race, language needs, gender, clinical specialty, etc.?

7. Since grant funds were received, what hours have you typically worked? (F.8.b.)

To what extent do you work with children, youth, and families in the evenings or weekends?

If so, when and how frequently do you work in the evening or on weekends?

What happens when a child, youth, or family can’t meet during the hours you work?

To what extent are you able to make special arrangements to work with them when they are available? Please provide examples.

How frequently have you done this?

5=Available at a wide range of times (including after-hours AND weekends), and there was also broad flexibility in scheduling

4=Available at a wide range of times (including after-hours OR weekends), and moderate flexibility in scheduling

3=Range of hours available but SET times for after-hours OR weekends; little flexibility to accommodate special requests

2=Business hours only; special requests accommodated in special (non-emergency) circumstances only

1=Business hours only; special requests not accommodated

8. Can children, youth, and families reach you in emergency situations? If yes, how?

If not, can they reach someone from (name of grant program) who can help? If yes, how?

9. Since grant funds were received, in what locations have you typically worked? (F.8.c.)

Have you ever worked with children, youth, and families in their homes, at the school, or other places outside of your office or building? If yes, where?

How much of your time (directly providing services) have you spent working with children, youth, and families in each of these locations or settings? [Probe for breakdown of time spent in each location.]

If a child, youth, or family can’t meet you in your office or at the places you normally work—have you been able to make special arrangements? How frequently have you done this?

5=Services offered in a wide range of locations (for example, homes, schools, in the community); in addition, there was also very broad flexibility in locations to meet family needs

4=Range of locations offered and moderately broad flexibility in locations to meet family needs

3=Range of locations offered but little flexibility to accommodate special requests

2=Agency offices only; special requests accommodated in special (non-emergency situations) circumstances only

1=Agency offices only; special requests not accommodated

10.a. Do you routinely incorporate children and youth strengths into your provision of services? (F.3.c.)

Do you do this with all children and youth or just certain children?

If not all (100%), for approximately what percentage do you incorporate children and youth strengths?

5=Strengths incorporated for all (100%) children and youth

4=Strengths incorporated for almost all (90-99%) children and youth

3=Strengths incorporated for many (75-89%) children and youth

2=Strengths incorporated for some (50-74%) children and youth

1=Strengths incorporated for few (less than 50%) children and youth

10.b. Please provide examples of how you have incorporated children and youth strengths when you are working with them. (F.3.c.) [Note: Examples should identify a strength, a service planned based on that strength, and how this benefited the child or youth]

[Continue probing for more examples until it is clear that the respondent can think of no more examples.]

Overall, to what extent do you think that you have been able to use children and youth strengths as well as you could have? What could be done to better use children and youth strengths?

5=Four or more examples of child/youth strengths incorporated into service provision AND respondent reported it could not have been better

4=Three examples of child/youth strengths incorporated into service provision OR 4 or more examples but respondent reported it could have been better

3=Two examples of child/youth strengths incorporated into service provision

2=One example of child/youth strengths incorporated into service provision

1=No examples of child/youth strengths incorporated into service provision

11.a. Do you routinely incorporate family strengths into your provision of services? (F.1.c.)

For approximately what percentage of families do you do this?

5=Strengths incorporated for all (100%) families

4=Strengths incorporated for almost all (90–99%) families

3=Strengths incorporated for many (75–89%) families

2=Strengths incorporated for some (50–74%) families

1=Strengths incorporated for few (less than 50%) families

11.b. Please provide examples of how you have incorporated family strengths into service provision for the families with whom you have worked. (F.1.c.)

[Continue probing for more examples until it is clear that the respondent can think of no more examples.]

Overall, to what extent do you think that you have been able to use family strengths as well as you could have? What could be done to better use family strengths?

5=Four or more examples of family strengths incorporated into service provision AND respondent reported it could not have been better

4=Three examples of family strengths incorporated into service provision OR 4 or more examples but respondent reported it could have been better

3=Two examples of family strengths incorporated into service provision

2=One example of family strengths incorporated into service provision

1=No examples of family strengths incorporated into service provision

12.a. Do you routinely discuss child, youth, and family cultural background? (F.5.a.)

If yes, for approximately what percentage of children, youth, and families?

5=Culture assessed for all (100%) children, youth, and families

4=Culture assessed for almost all (90–99%) children, youth, and families

3=Culture assessed for many (75–89%) children, youth, and families

2=Culture assessed for some (50–74%) children, youth, and families

1=Culture assessed for few (less than 50%) children, youth, and families

12.b. Please give me examples of how you have incorporated child, youth, and family culture into the provision of services or made changes in how you work with a child, youth, or family to be more sensitive to their culture. (F.5.a.)

[Continue probing for more examples until it is clear that the respondent can think of no more examples or until four examples have been reported.]

To what extent have these efforts been sufficient to incorporate culture. What more could be done to better incorporate culture into services provided?

5=4 or more examples of child, youth, and family culture incorporated into service provision AND respondent reported that it could not have been better

4=3 examples of child, youth, and family culture incorporated into service provision OR 4 or more examples but respondent reported it could have been better

3=2 examples of child, youth, and family culture incorporated into service provision

2=1 example of child, youth, and family culture incorporated into service provision

1=No examples of child, youth, and family culture incorporated into service provision

13. Do clients ever need to receive the services you provide in a language other than English? If yes, how is that handled? (F.5.b.)

If no, is there a process in place should the need arise?

How effective are these efforts? Describe.

To what extent is that process sufficient to meet the needs of all clients who do not speak English?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

14. Do you involve families in their child’s, youth’s, or family’s services or treatment? (F.1.a.)

Do you keep caregivers informed about what is going on in services/treatment with their child, youth, or family and their child’s, youth’s, or family’s progress? Please describe.

Do you encouraged caregivers to express their opinion or offer advice about what you should be doing with their child or youth, what kinds of things you should be working on, etc.? Probe for examples.

Can you give me any examples of when you have altered the way you have worked with a child, youth, or family due to concerns or feedback you received from families?

(IF APPLICABLE) Have families ever been involved or participated in services to their child youth, or other family members, for example attending a session with you and the child or youth, going with you and the child or youth or other family members on outings, etc.? If yes, please provide examples.

[Continue probing for more examples until it is clear that the respondent can think of no more examples.]

Overall, have you involved families as much as you could have in their child, youth, or family’s services/treatment or are there things that you think could be done to help them become more involved?

5=Families have been involved in service provision in at least 4 ways AND respondent reported that involvement has been sufficient

4=Families have been involved in service provision in 3 ways OR families have been involved in 4 or more ways but respondent reports that more involvement needed

3=Families have been involved in service provision in 2 ways

2=Families have been involved in service provision in 1 way

1=Families have not been involved in service provision

15. Since grant funds were received, how have agencies, organizations and providers worked to coordinate the various services that children, youth, and families receive? (F.7.a.)

What kinds of information have you typically shared? What kinds of information have you typically received?

Which agencies, organizations, or providers typically work well together to coordinate services? Which do not

Do primary health providers and substance abuse treatment providers participate and collaborate together with other providers?

Is there anything that inhibits or compromises the coordination of services (e.g., confidentiality regulations, agency policies, etc.)?

To what extent have efforts in this area been effective? In what ways?

To what extent do you think that efforts in this area have been sufficient so that all providers, organizations, and agencies know their roles, and what is going on with the child, youth, and family? Why or why not?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

16. Since grant funds were received, how have transitions or changes in a child, youth, and family’s services, such as a change in providers or services, been coordinated? (F.7.c.)

Who is responsible for ensuring smooth transitions/changes?

What types of information are exchanged? Among whom?

Has anyone (e.g., therapist, case manager, facilitator, teachers, etc.) ever called you or let you know that changes were being made to a child, youth, or family’s services? Please describe.

What efforts are made to follow up with the child, youth, or family and with providers to make sure that the transition or change was made and was working?

To what extent have efforts been effective? In what ways?

To what extent do you think that efforts in this area have been sufficient? What could improve coordination of service transitions or changes?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Evidence-supported Care

17a. In your typical practice, do you use evidence-supported treatment models? If yes, describe them. (F.4.b.)

If yes, what models do you use?

Do you use these evidence-supported models with all children you serve from (name of grant program? If not, with whom do you use these models?

1=No

2=Yes

[Code types of clients who receive evidence-supported treatment models.]

17.b. Did you receive training on how to implement evidence-supported treatment models? (B.4.b.)

• If yes, what models have you been trained in?

• Are all (name of grant program) practitioners trained in this (these) model(s)?

• How were you trained (e.g., traditional lectures, learning collaborative model)?

To what extent have these efforts been effective to ensure that you had the skills to implement evidence-supported services and supports? If yes, explain.

To what extent have these efforts been sufficient? Are there any additional efforts that should be made to ensure that practitioners implement evidence-supported service planning models? [Probe for what more should be done.]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

17.c. Have there been any efforts to make sure that practitioners are implementing the approach as intended (with fidelity) over time? (B.4.c)

If yes, please describe these efforts.

To what extent have these efforts been effective to support faithful implementation of evidence-supported models? If yes, explain.

To what extent have these efforts been sufficient? Are there any additional efforts that should be made to ensure that practitioners faithfully implement evidence-supported models? [Probe for what more should be done.]

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Monitoring progress

18. Since grant funds were received, have you monitored how the individual children and youth you serve are doing in the service system (e.g., check in on service process and improvement)?

How do you do this? [Probe for formal assessment/evaluation of needs and progress]

What kinds of things have you routinely monitored? (e.g., progress, whether child/youth is improving.)

How frequent have your monitoring efforts been?

Have you monitored service provision for ALL children and youth you work with? If not all, for whom do you monitor the service process?

18.a Overall, to what extent have these efforts been effective? Do you feel that you have been able to monitor improvement for all children and youth’s as well and as often as needed? (F.3.d.)

Overall, to what extent have these efforts been sufficient? What changes could be made to facilitate monitoring?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

18.b Have changes been made to child and family services based on the monitoring results? (F.4.a)

If yes, please describe.

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

I have just a couple of final questions.

19. Since grant funds were received, have you received or participated in any training sessions?

If yes, please describe the topics or content areas covered. [Probe on CASSP principles such as family involvement, cultural competency, individualized/person-centered services, strengths-based care, etc.]

Were these trainings provided as part of (name of grant program) activities?

20. Overall, how has (name of grant program)’s activities impacted how your community serves children and youth with behavioral health challenges?

Overall, how much has (name of grant program) impacted how you provide service

End of Instrument:

Thank you for participating in the System of Care Expansion Assessment portion of the National Systems of Care Expansion Evaluation.

Attachment 8c – Form H

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

SYSTEM OF CARE EXPANSION ASSESSMENT

CAREGIVERS

INTRODUCTION

Thank you for your willingness to complete the system of care expansion assessment. The purpose of this interview is to identify the strategies and mechanisms implemented at the local service system level, and examine the extent to which these mechanisms are being implemented from a caregiver’s perspective.

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of the federal system of care expansion grant program.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 45minutes.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

Your responses to these questions will be kept confidential and will not be shared outside of the evaluation team. In any of our reports, your responses will be combined with other people’s responses, so your answers will never be attributed to your name.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

The SOCEA interview focuses on the local-level management, structure, and procedures as well as direct service delivery level involving activities and practices in the delivery of direct services.

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

Systems of Care Expansion Assessment (SOCEA)

Caregiver Consent Form

Purpose

The Substance Abuse and Mental Health Services Administration (SAMHSA) in the United States Department of Health and Human Services is sponsoring a national evaluation of children’s mental health services and systems of care. A System of Care is a coordinated network of community-based services and supports that are organized to meet the challenges that children and youth with serious mental health needs and their families face. You have been invited to participate in this evaluation because (system of care name) has received funding to improve services for children and families. The purpose of this interview is to see how well the services provided by the (system of care name) work and find out the different ways in which children and youth are involved in their system of care. We want to find out if the services you receive help you.

Description of Participation

Participation in this survey is voluntary. The decision to participate in this interview is completely your own. You will be asked to participate in one interview that will take approximately 45 minutes. There is no right or wrong answer. This interview will be conducted on the phone or computer with a trained interviewer who will ask you to answer a set of questions about the services you received. You can talk about what you think about the program and the services.

Risks

You may feel uncomfortable about answering some questions about your experiences. You can stop, take a break, or skip any questions you do not want to answer. You may discontinue participation at any time.

Benefits

You will not get any direct benefit from being interviewed. However, the information you provide may help improve the services offered to children, youth, and their families.

Confidentiality

The information you share with us will only be used for the purposes of this study. We will keep your information private to the extent permitted by law. However, if you report any intent to harm yourself or someone else, we will report it to the proper authorities.

Your healthcare services or insurance coverage will not be affected by anything you say during the interview. We will record the interview to help us with our notes. The session recordings will be stored on a password protected computer and will be destroyed at the end of the project. The recording will be reviewed only by a few project staff. Your name will not be used in any reports we write. This signed consent form and any forms and records from the study will be kept in a secure place that only project staff will be able to access.

Contact Information

If you have any questions about this project, you may call SAMHSA’s Project Officer for this evaluation, Ms. Kaitlyn Harrington at 240-276-1928.

If you have questions about your rights and welfare as a research participant, please call the Westat Human Subjects Protections office at 1-888-920-7631. Please leave a message with your full name, the name of the research study that you are calling about (CMHI Evaluation), and a phone number beginning with the area code.

OMB No. xxxx-xxxx

Expires: xx/xx/201x

H. Caregiver of Child Served by the System/Program

Introduction

1. First, I’d like to hear a little bit about your child and family?

Is your child a girl or a boy? How old is s/he?

Who else is in your family?

2. Why did your family become involved with (name of grant program) ?

[Probe for reasons the child or youth entered treatment.]

How did you learn about (name of grant program) ? How were you referred?

How long has your child and family been receiving services through (name of grant program)?

3. What services do your child and family currently receive through (name of grant program) now?

Has your family ever received any other services through (name of grant program)?

4. In addition to the services received through (name of grant program) , has your child and family received services from other providers, organizations, programs or agencies? If yes, what were they?

Now I’d like for you to think back to when you first brought your child and family to (name of grant program) for services.

5. When you first tried to get services, how complicated or difficult was it to get into (name of grant program) ? (D.1.a)

When you first tried to get services from (name of grant program) what were the steps in the process for getting services? Interviewer: Probe for details of steps such as filling out forms, if they talked to one person or multiple people, etc.

Thinking back at all you had to do, rate on a scale of 1 to 5 (with 5 being the least complicated), how complicated or difficult it was to enter/enroll in (name of grant program) ?

Respondent’s rating

5=Entry process was not at all complicated/difficult. Very few steps were involved.

4=Entry process was slightly complicated/difficult.

3=Entry process was somewhat complicated/difficult. Several steps were involved.

2=Entry process was moderately. Many steps involved.

1=Entry process was extremely complicated/difficult. Very many steps involved.

Interviewer’s rating

5=Entry process was not at all complicated/difficult. Very few steps were involved.

4=Entry process was slightly complicated/difficult.

3=Entry process was somewhat complicated/difficult. Several steps were involved.

2=Entry process was moderately. Many steps involved.

1=Entry process was extremely complicated/difficult. Very many steps involved.

6. When you first brought your child to (name of grant program) were your immediate needs identified and were you given information about available supports?

[Probe for whether families have access to supports right away as opposed to having to wait until the formal service planning process].

6.a How well would you say (name of grant program) met your family’s immediate support needs when you first brought your child for services? D.1.b.)

Respondent’s rating

5=Extremely well

4=Very well

3=Moderately well

2=Somewhat well

1=Not at all well

Interviewer’s rating

5=Extremely well

4=Very well

3=Moderately well

2=Somewhat well

1=Not at all well

When you started services, were you given information about a family organization working with (name of grant program) and what they offer to families?

Were you connected to family advocacy and/or peer support services? If yes, describe the process. [Probe for how long it took and how smooth the process was to be connected to these services].

6.b. How well would you say (name of grant program) met your family’s needs for advocacy and/or peer support services needs when you first brought your child for services in a timely manner? (D.1.c.)

Respondent’s rating

5=Extremely well

4=Very well

3=Moderately well

2=Somewhat well

1=Not at all well

Interviewer’s rating

5=Extremely well

4=Very well

3=Moderately well

2=Somewhat well

1=Not at all well

7. How much time passed between when your child and family first tried to get services from (name of grant program) until you actually started receiving services? Was this a good timeframe for you, or did you need to receive services sooner? (D.8.b.)

On a scale of 1 to 5, with 5 being the best, how would you rate the length of time it took for your child or family’s first service to begin?

Respondent’s rating

5=Timeframe was perfect, no changes needed

4=Timeframe was very fast, could use minor improvement

3=Timeframe pretty fast, could use some improvement

2=Timeframe pretty slow, could use quite a bit of improvement

1=Timeframe entirely too slow, needs a great deal of improvement

Interviewer’s rating

5=Timeframe was perfect, no changes needed

4=Timeframe was very fast, could use minor improvement

3=Timeframe pretty fast, could use some improvement

2=Timeframe pretty slow, could use quite a bit of improvement

1=Timeframe entirely too slow, needs a great deal of improvement

8. Were you and your family treated with respect and made to feel comfortable throughout the enrollment process? (D.1.a.)

How well do you think the staff paid attention to and respected what you had to say?

On a scale from 1 to 5, with 5 being the best, how respected and comfortable did you feel during the process for entering (name of grant program) ?

Respondent’s rating

5=Family felt extremely respected and comfortable

4=Family felt very respected and comfortable

3=Family felt moderately respected and comfortable

2=Family felt somewhat respected and comfortable

1=Family felt extremely disrespected and uncomfortable

Interviewer’s rating

5=Family felt extremely respected and comfortable

4=Family felt very respected and comfortable

3=Family felt moderately respected and comfortable

2=Family felt somewhat respected and comfortable

1=Family felt extremely disrespected and uncomfortable

Now I’d like to ask you some questions about what happened when you and the staff at (name of grant program) were deciding what services would be best for your child and family. We call this the service planning process. We think of service planning as different from treatment planning, it covers not just what should happen in therapy or counseling, but ALL the services that your child and family needs. Service planning can happen when families first come for help but can also happen later on when changes in services are needed and when the plan needs to be updated. In these questions, we are talking about when you were first planning services, as well as any updates that were made.

Some of the questions I’ll be asking are about your child’s services, others are about your family’s services, and others are about both your child and family. I’ll try to be clear, but if you’re not sure what I’m asking about, please ask me to clarify.

9. When you first came to (name of grant program) , was there any type of service planning that was different than or covered more than treatment planning, did anyone help to decide what services your child and family should receive? (E.3.b.)

[Probe as necessary to distinguish between service planning and treatment planning.]

1=Treatment plan

2=Service plan

If yes for either service planning or treatment planning, continue.

Who was the main person you worked with to plan services? [Probe for the first name and function (e.g., case manager or therapist) who worked with the family to plan services. Use that name where you see (name of case manager/therapist) .]

10. Since entering (name of grant program) , has your child or family been involved with staff from other agencies that work with children such as child welfare, juvenile justice, education, health, etc.? If yes, which agencies? [Probe for agencies that might be relevant for a child of this age, needs, and background]. (E.6.a.)

If yes, did anyone from any of these agencies work with you and (name of case manager/therapist) to plan services for your child and family? If so, who?

Was there any other agency that you thought should have helped to plan services but did not?

5=All involved agencies were present

4=Most involved agencies were present

3=Some involved agencies were present

2=Few of the involved agencies were present

1=One involved agency was present (but family involved with more than one)

666=Family involved with only one agency

11. Considering all of the people who have worked with your child and family since entering (name of grant program) , including the staff at (name of grant program) , the agencies you just mentioned (if any), and other providers or organizations, how well do you think that all of these different people worked with each other to plan services for your child and family? (E.7.b.)

Do you think they could have done a better job working together so that the service planning process would have been better coordinated?

On a scale from 1 to 5, with 5 being the best, how well do you think they all coordinated the service planning process?

Respondent’s rating

5=Extremely coordinated

4=Very coordinated

3=Moderately coordinated

2=Somewhat coordinated

1=Not at all coordinated

666=Only one party involved

Interviewer’s rating

5=Extremely coordinated

4=Very coordinated

3=Moderately coordinated

2=Somewhat coordinated

1=Not at all coordinated

666=Only one party involved

12. How well did the people who were working with your child and family involve you in the service planning process? (E.1.a)

Were you encouraged to bring someone to the meeting with you, perhaps for support?

Were you asked whether there was anyone you did not want to be present in the meeting?

Were you asked to talk about what you thought were the most important concerns for your child and family?

Were you encouraged to help develop your child’s and family’s goals and objectives?

Were you given a choice of services that you thought would be most helpful for your child and family?

Were you able to turn down services that you did not want your child and family to receive?

Are there other ways that you were involved in service planning? If yes, please describe.

Overall, were you as involved in the service planning process as you think you should have been?

5=Caregiver was involved in service planning in at least 6 ways AND respondent reported that involvement has been sufficient

4=Caregiver was involved in service planning in 5 ways OR involved in 6 ways but respondent reported that it could have been better

3=Caregiver was involved in service planning in 4 ways

2=Caregiver was involved in service planning in 3 ways

1=Caregiver was involved in service planning in fewer than 3 ways

13. What about your child? How have the people working with your child and family involved your child in planning his/her services? (E.3.a)

If your child was not involved, do you your child should have been more involved? [Probe for whether it was appropriate for the child to be involved given his/her challenges, age, or caregiver’s preference.]

If yes, how was your child involved?

Was your child encouraged to bring someone to the meeting with you, perhaps for support?

Was your child asked whether there was anyone he/she did not want to be present in the meeting?

Was your child asked about what he/she thought were the most important concerns?

Was your child encouraged to participate in developing his/her goals and objectives?

Was your child given a choice of which services he/she wanted?

Was your child able to turn down any services he/she didn’t want?

Overall, was your child as involved as you would have liked, or could there have been more involvement?

5=Child was involved in service planning in at least 6 ways AND respondent reported that involvement has been sufficient

4=Child was involved in service planning in 5 ways OR involved in 6 ways but respondent reported that it could have been better

3=Child was involved in service planning in 4 ways

2=Child was involved in service planning in 3 ways

1=Child was involved in service planning in fewer than 3 ways

666=Caregiver reported it was not appropriate for child/youth to participate

14. When you were working with (name of case manager/therapist) to plan services, did she/he talk with you about your child’s strengths (the things he/she is interested in and/or good at, his/her hopes and goals for the future)? (E.3.c)

If yes, would you mind giving me some examples of things you talked about? [Probe for examples until respondent can think of no more.]

How were your child’s strengths used in planning his/her services? What could have been done better?

5=Strengths explicitly discussed and at least three examples given of how strengths were incorporated into the service plan AND respondent reported it could not have been better

4=Strengths explicitly discussed and two examples given of how strengths were incorporated into the service plan but respondent reported it could have been better

3=Strengths explicitly discussed and one example given of how strengths were incorporated into the service plan

2=Strengths explicitly discussed but not (or very, very minimally) incorporated into the service plan

1=No discussion of strengths

15. What services were planned for your child during the planning process? [List all services planned.]

16. How well did the services planned for your child meet his/her individual or specific needs? (E.3.b)

Were there any services that you thought your child needed but were not included in the service plan? If yes, what were they?

Were there services included in your child’s service plan that you think he/she didn’t really need? If yes, please explain.

On a scale from 1 to 5, with 5 being the best, how well would you say the service plan developed for your child matched what your child really needed?

Respondent’s rating

5=Child’s needs matched extremely well

4=Child’s needs matched very well

3=Child’s needs matched moderately well

2=Child’s needs matched somewhat well

1=Child’s needs not matched well

Interviewer’s rating

5=Child’s needs matched extremely well

4=Child’s needs matched very well

3=Child’s needs matched moderately well

2=Child’s needs matched somewhat well

1=Child’s needs not matched well

17.a. Did (name of case manager/therapist) talk with you about your family’s strengths (for example, the good things about your family, qualities that make your family strong, your hopes, etc.)? (E.1.d.)

If yes, would you mind giving me some examples of things you talked about? [Probe for examples until respondent can think of no more.]

How did he/she use your family’s strengths in planning services for your family? What could have been done better?

5=Strengths explicitly discussed and at least three examples given of how strengths were incorporated into the service plan AND respondent reported it could not have been better

4=Strengths explicitly discussed and two examples given of how strengths were incorporated into the service plan but respondent reported it could have been better

3=Strengths explicitly discussed and one example given of how strengths were incorporated into the service plan

2=Strengths explicitly discussed but not (or very, very minimally) incorporated into the service plan

1=No discussion of strengths

17.b. How were your family’s strengths used in service delivery, in the services that you or your child received?. (F.1.c)

If yes, please describe some examples. [Probe for multiple examples until respondent can think of no more.]

How did he/she use your family’s strengths in planning services for your family? What could have been done better?

5=Strengths explicitly discussed and at least three examples given of how strengths were incorporated into the service plan AND respondent reported it could not have been better

4=Strengths explicitly discussed and two examples given of how strengths were incorporated into the service plan but respondent reported it could have been better

3=Strengths explicitly discussed and one example given of how strengths were incorporated into the service plan

2=Strengths explicitly discussed but not (or very, very minimally) incorporated into the service plan

1=No discussion of strengths

18. When you were working with (name of case manager/therapist) to plan services, did he/she ever talk with you about your family’s needs and the services or other kinds of supports that you or other people in your family could use (for example, respite care, support groups, family advocacy, sibling support, behavior management training)? (E.1.b.)

If yes, did (name of case manager/therapist) try to identify or put into place any services to meet those needs? Please describe.

[List all services planned for the family.]

Were there any other services or supports that you and your family needed but were not a part of your service plan?

5=Family needs were explicitly considered and services were planned that would fully meet their needs

4=Family needs were explicitly considered and services were planned that would meet most of their needs

3=Family needs were explicitly considered and services were planned that would meet some of their needs

2=Family needs considered, but no services were planned AND family reporting having need(s)

1=Family needs were not discussed (regardless of whether family reported having needs)

666=Family reporting having had no needs

19. Did (name of case manager/therapist) ever ask you about your family’s culture and background, for example, your family’s beliefs, (beliefs related to your heritage/community), traditions, values, religious preferences, ideas about parenting, recreational activities, lifestyle, etc.? (E.5.a.)

[If yes,] what are would you mind giving me examples of some of the kinds of things you talked about?

How did (name of case manager/therapist) use this information to plan services for your child and family?

Overall, do you think that he/she did a good job planning services that fit with your family’s background and culture? If not, what could have been done better?

5=Culture explicitly discussed and at least three examples given of how culture was incorporated into the service plan AND respondent reported it could not have been better

4=Culture explicitly discussed and two examples given of how culture was incorporated into the service plan but respondent reported it could have been better

3=Culture explicitly discussed and one example given of how culture was incorporated into the service plan

2=Culture explicitly discussed but not (or very, very minimally) incorporated into the service plan

1=No discussion of culture

20.a. Does your family speak a language other than English? If yes, is this the language you are most comfortable speaking? (E.5.b.)

If yes, was (name of case manager/therapist) able to speak to you in this language when you were working with him/her to plan services?

If no, was someone there who could speak your language and interpret? Who?

5=Service planning process was conducted entirely in family’s preferred language

4=Service planning process was conducted in English but formal interpretation was provided by the grant

3=Service planning process was conducted in English but informal interpretation services provided by grant

2=Service planning process was conducted in English and family brought interpreter because no other option was available

1=Service planning process was conducted in English and no interpretation was conducted or any interpretation had to be done by a child in the family

666=Not applicable. Family’s primary language is English

20.b. [If family’s language preference is not English, ask:] What about when your child and family were actually receiving the services? Were you able to get any services in the language you are most comfortable with? (F.5.b)

If yes, which services were provided in your language and which were not? Were there any services that you would have preferred were available in your language?

Were interpretation services provided to you?

If no interpretation services were provided, how was this handled (e.g., did a family member or friend have to provide the translation)?

5=Key services were provided entirely in family’s preferred language

4=Key services were provided in English but formal interpretation was provided by the program

3=Key services were provided in English but informal interpretation services provided by program

2=Key services were provided conducted in English and family brought interpreter because no other option was available

1=Key services were provided in English and no interpretation was conducted or any interpretation had to be done by a child in the family

666=Not applicable. Family’s primary language is English

21. Were any of the things you talked about with (name of case manager/therapist) when you were planning services, ever written down in a service plan (such as the services you wanted, goals, strengths, needs, etc.)? (E.3.b.)

1=No, 2=Yes

If yes, did you get a copy of the service plan? Did you have to ask for it or did you automatically receive it?

If no copy received, were you able to see a copy of the service plan?

Now I want to spend the rest of our time together talking specifically about the services your child and family have received since entering (name of grant program) .

22. Has your child received all of the services that you and (name of case manager/therapist) decided that s/he should have or that were written in the service plan? (F.3.a.)

[Probe by referring back to Question 14 and checking whether all services listed there were received.]

If not, what services were planned that you thought were important for your child to have that he/she never received? Why do you think he/she has not received those services?

5=Child received all of the services that were planned

4=Child received most of the services that were planned including those that the caregiver said were most important

3=Child received many of the services that were planned OR received most of the planned services but has not yet received services the caregiver identified as important

2=Child received a few of the services that were planned (importance of services not important here)

1=Child received no services outlined in the plan

23. What about services planned for you or for other members of your family — have you received all these services? (F.1.b)

[Refer back to question 17 and check whether all services listed there were received.]

If not, what services were planned that you thought were important for your family to have that were never received? Why do you think your family has not received those services?

5=Family received all of the services that were planned

4=Family received most of the services that were planned including all those that the caregiver said were most important

3=Family received many of the services that were planned OR received most of the planned services but has not yet received services the caregiver identified as important

2=Family received a few of the services that were planned (importance of services not relevant)

1=Family received no services outlined in the plan

666=No services were planned for the family

24. Were services scheduled at convenient times for your child and family? If no, please explain. (F.8.b)

What times would have worked better for you? Have you been able to get any services in the evenings or on weekends if that were more convenient for you?

On a scale from 1 to 5, with 5 being the best, how convenient and/or flexible would you say the scheduling of services has been?

Respondent’s rating

5=Extremely convenient

4=Very convenient

3=Moderately convenient

2=Somewhat convenient

1=Not at all convenient

Interviewer’s rating

5=Extremely convenient

4=Very convenient

3=Moderately convenient

2=Somewhat convenient

1=Not at all convenient

25. What about the places or locations where your child and family received services — how have these worked for you and your child? (F.8.c.)

If not good for one or both, what would have worked better? Have you ever had a choice or say in where you wanted to receive services?

On a scale from 1 to 5, with 5 being the best, how convenient and/or flexible would you say the location of services has been?

Respondent’s rating

5=Extremely convenient

4=Very convenient

3=Moderately convenient

2=Somewhat convenient

1=Not at all convenient

Interviewer’s rating

5=Extremely convenient

4=Very convenient

3=Moderately convenient

2=Somewhat convenient

1=Not at all convenient

26. Have you ever had to pay out of pocket for any of the services that your child and family received through (name of grant program) ? (F.8.e.)

If yes, which services? Has part of the cost of the services ever been covered by private insurance, Medicaid, or other healthcare program?

Have there been any services that you thought were important for your child and family but that you did not receive because of the cost? Explain.

On a scale from 1 to 5, (with 5 indicating that cost is not at all a barrier), how much of a barrier to services would you say cost has been?

Respondent’s rating

5=Not at all a barrier

4=Slightly a barrier

3=Somewhat a barrier

2=Moderately of a barrier

1=Largely a barrier

Interviewer’s rating

5=Not at all a barrier

4=Slightly a barrier

3=Somewhat a barrier

2=Moderately of a barrier

1=Largely a barrier

27.a. After deciding which services your child and family would receive, how long has it usually taken (in calendar days) to actually receive those services? [Probe for shortest, longest, and average wait for services received since entering (name of grant program .] (F.8.f.)

What about the services that you considered most important? How long have you had to wait for those services?

In general, what have you thought about the length of time it took to get the services once they were planned?

5=No or almost no wait for non-emergency services. Services received in 7 days or fewer.

4=Some wait for non-emergency services. Services received between 8 to 14 days.

3=Moderate wait for non-emergency services. Services received between 15 to 21 days.

2=Long wait for non-emergency services. Services received between 22 and 28 days

1=Very long wait for non-emergency services. Services received in more than 28 days.

27 b. Has your child or family received any emergency or crisis services since entering (name of grant program) ? If yes, how long did you have to wait to get those services? (F.8.f.)

5=Crisis services received in less than 24 hours AND emergency services (e.g., in response to suicide attempt) received immediately

4=Crisis services received within 24 to 48 hours AND emergency services received immediately

3=Crisis services received within 48 to 72 hours AND emergency services received immediately

2=Crisis services received in more than 72 hours AND emergency services received immediately

1=Emergency services not received immediately regardless of timing of crisis services

666=Not applicable, no emergency or crisis services received

28. Have you ever received any kind of transportation assistance (cab fare, bus tokens, shuttle bus, etc.) from (name of grant program) ? Who helped you with this? (F.8.d.)

Have you been able to get help with transportation when you needed it? How easy or hard was it to ask for transportation support and get it?

What, if any, problems have you had getting transportation assistance?

5=Transportation assistance was always or almost always available when the family needed it. Family had no trouble accessing this service and reported little or no problems.

4=Transportation assistance was most often available when the family needed it. Family reported having some trouble with transportation assistance.

3=Family used transportation assistance, but reported having moderate trouble with transportation assistance.

2=Family used transportation assistance, but reported considerable difficulties accessing the service.

1=Family reported needing transportation assistance, but could not access it.

666=Family reported never needing or wanting to use transportation services.

29. How well do you think that all the staff and people serving your child and family have worked with each other to make sure that your child’s and family’s services have been coordinated? (F.7.a.)

Do you think that they all have known who has been involved with your child and family and what their different roles/jobs have been?

Have they worked together to make sure there have been no scheduling problems with your child and family’s appointments?

Have they shared information with each other or did you feel you had to repeat everything to everybody because the providers and staff were not communicating well with each other?

Overall, do you feel that your child’s and family’s services have been coordinated well, or could they have done a better job? Can you give me some examples of problems?

5=Extremely well coordinated. No problems reported.

4=Very well coordinated. Minor problems reported.

3=Moderately well coordinated. Some problems reported.

2=Somewhat coordinated. Considerable problems reported.

1=Poorly coordinated. Major problems reported.

666=Only one party involved.

30. Have there been any changes in the staff or providers you or your child have worked with, for example, a new therapist, case manager, respite provider etc.? Explain. (F.7.c)

Have there been any other kinds of changes in your services, for example getting a new service or going to a new place for services? Explain.

If yes, when you switched services or providers, did things go smoothly, or did you feel that you were starting all over with the new service or provider? Or did you feel you were picking up where the previous one left off?

5=Transitions went very smoothly; no disruption of care; family didn’t feel they were starting all over again

4=Transitions went pretty smoothly, a few or minor problems still reported; could have been done a little better

3=Transitions went moderately smoothly but quite a few problems or moderate problems still reported; could have been done somewhat better

2=Transitions went somewhat smoothly; several or substantial problems still reported

1=Transitions went not at all smoothly; many or major problems reported; family felt they were starting all over again

666=No transitions or changes reported

31. Have any of the providers working with your child showed you assessment or evaluation results to discuss if your child is doing? If yes, please describe how information about your child is shared with you. (F.4.a)

If yes, have changes been made to your child’s services based on assessments of your child’s progress?

5=Assessment results were shown to the caregiver and changes were made to services OR results indicated no change in services were needed

4= Assessment results were shown to the caregiver and their role in informing service was discussed in general, but not in relation to child needs

3= Assessment results were shown to the caregiver and explained, but their role in informing service was not discussed

2= Assessment results were shown to the caregiver but they were explained

1=Assessment results not shown to caregiver

32. What have the different providers who work with your child and family done to include you in your child’s and family’s services? (F.1.a.)

For example, were you encouraged you to offer your ideas about your child’s services?

Were you considered you to be the primary decision-maker about your child’s services?

Were you encouraged to let them know when something was not working well for your child and family?

Were you for suggestions about changes that could be made to improve your child’s or family’s care?

Were you asked to participate in services for your child and family? If yes, please describe.

Were you kept you informed about what was going on in services and keep you updated about your child’s and family’s progress, such as how things were going, what was working, what wasn’t?

Overall, could providers have done a better job keeping you involved in your child’s and family’s services?

5=Caregiver was involved in service provision in at least 4 ways AND respondent reported that involvement has been sufficient

4=Caregiver was involved in service provision in 3 ways OR involved in 4 ways but respondent reported that it could have been better

3=Caregiver was involved in service provision in 2 ways

2=Caregiver was involved in service provision in 1 way

1=Caregiver was not involved in service provision

Thank you for taking the time to answer my questions. Do you have any questions for me, or is there anything else you want to tell me about youth involvement that we did not discuss?

End of Instrument:

Thank you for participating in the System of Care Expansion Assessment portion of the National Systems of Care Expansion Evaluation.

Attachment 8c – Form I

OMB No. xxxx-xxxx

Expiration Date: xx/xx/201x

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

SYSTEM OF CARE EXPANSION ASSESSMENT

YOUTH RECEIVING SERVICES

INTRODUCTION

Thank you for your willingness to complete the system of care expansion assessment. The purpose of this interview is to identify the strategies and mechanisms implemented at the local service system level, and examine the extent to which these mechanisms are being implemented from a youth’s perspective.

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of the federal system of care expansion grant program.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 30 minutes.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

Your responses to these questions will be kept confidential and will not be shared outside of the evaluation team. In any of our reports, your responses will be combined with other people’s responses, so your answers will never be attributed to your name.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

The SOCEA interview focuses on the local-level management, structure, and procedures as well as direct service delivery level involving activities and practices in the delivery of direct services.

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

Systems of Care Expansion Assessment (SOCEA)

Parental Permission Form

Purpose

The Substance Abuse and Mental Health Services Administration (SAMHSA) in the United States Department of Health and Human Services is sponsoring a national evaluation of children’s mental health services and systems of care. A System of Care is a coordinated network of community-based services and supports that are organized to meet the challenges that children and youth with serious mental health needs and their families face. We are asking for your permission to have your child participate in an interview with a trained interviewer who will ask a set of questions about his/her involvement in (system of care name). The purpose of the interview is to find out the different ways in which children and youth are involved in their system of care. Your child was asked to participate because s/he currently receives services in a system of care community. Your child’s input is important in helping us understand how systems of care serve children and young adults, and what works best. The results of the study will be used to help improve services for children, young adults and their families

Description of Participation

Participation in this survey is voluntary and your child’s participation is completely his/her choice. Your child will be asked to participate in one interview that will take approximately 30 minutes. This interview will be conducted on the phone or computer with a trained interviewer who will ask your child to respond to a set of questions about the children’s mental health system of care in your community. Your child will be able to share what s/he thinks about the program and the services s/he has received.

Risks

Your child may feel uncomfortable about answering some questions about his/her experiences. Your child can stop or take a break at any time, and skip any questions s/he does not want to answer. Your child may discontinue participation at any time.

Benefits

Your child will not get any direct benefit from being interviewed. However, the information your child provides may help improve the services offered to children, youth, and their families.

Confidentiality

The information your child shares with us will only be used for the purposes of this study. Your child’s answers will not be shared with you or anyone else outside of this project. We will keep your child’s information private to the extent permitted by law. However, if your child says anything about hurting themselves or others, we have to report it to the proper authorities.

Your child’s healthcare services or insurance coverage will not be affected by anything s/he says during the interview. We will record the interview to help us with our notes. The session recordings will be stored on a password protected computer and will be destroyed at the end of the project. The recording will be reviewed only by a few project staff. Your name or your child’s name will not be used in any reports we write. This signed permission form and any forms and records from the study will be kept in a secure place that only project staff will be able to access.

Contact Information

If you or your child has any questions about this project, you may call SAMHSA’s Project Officer for this study, Ms. Kaitlyn Harrington at 240-276-1928.

If you or your child have questions about their rights and welfare as a research participant, please call the Westat Human Subjects Protections office at 1-888-920-7631. Please leave a message with your full name, the name of the research study that you are calling about (CMHI Evaluation), and a phone number beginning with the area code.

Systems of Care Expansion Assessment (SOCEA)

Youth Agreement to Participate Form (ages 14-17)

Purpose

The Substance Abuse and Mental Health Services Administration (SAMHSA) in the United States Department of Health and Human Services is sponsoring a national evaluation of children’s mental health services and systems of care. We want to find out if the services you receive help you. The information you give us will help to improve the lives of children, youth and families that use these services, as well as improve the services children and their families receive.

Description of Participation

Participation in this survey is voluntary. The decision to participate in this interview is completely your own. Your parent already gave us permission to talk with you. You will be asked to participate in one interview that will take about 30 minutes. There is no right or wrong answer. This interview will be done on the phone or the computer with a trained interviewer. You will be asked a set of questions about the services you receive and you can talk about what you think about them.

Risks

You may feel uncomfortable about answering some questions about your experiences. You can stop, take a break, or skip any questions you do not want to answer. If you become upset, there are people close by we can get to help you. You may stop participating at any time.

Benefits

You will not get any direct benefit from being interviewed. However, the information you provide may help improve the services offered to children, youth, and their families.

Confidentiality

The information you share with us will only be used for the purposes of this study. We will not share it with your parents or anyone else outside of this project. We will keep your information private to the extent permitted by law. However, if you say anything about hurting yourself or someone else, we have to report it.

The care you get when you come to this office will not be affected by anything you say. We would like to record the interview to help us with our notes. The session recordings will be stored on a password protected computer and will be destroyed at the end of the project. The recording will be reviewed only by a few project staff. Your name will not be used in any reports we write. This signed consent form and any forms and records from the study will be kept in a secure place that only project staff will be able to access.

Contact Information

If you have any questions about this project, you may call SAMHSA’s Project Officer for this evaluation, Ms. Kaitlyn Harrington at 240-276-1928.

If you have questions about your rights and welfare as a research participant, please call the Westat Human Subjects Protections office at 1-888-920-7631. Please leave a message with your full name, the name of the research study that you are calling about (CMHI Evaluation), and a phone number beginning with the area code.

Systems of Care Expansion Assessment (SOCEA)

Young Adult Consent Form

Purpose

The Substance Abuse and Mental Health Services Administration (SAMHSA) in the United States Department of Health and Human Services is sponsoring a national evaluation of children’s mental health services and systems of care. A System of Care is a coordinated network of community-based services and supports that are organized to meet the challenges that children and youth with serious mental health needs and their families face. You have been invited to participate in this evaluation because (system of care name) has received funding to improve services for children and families. The purpose of this interview is to see how well the services provided by the (system of care name) work and find out the different ways in which children and youth are involved in their system of care. We want to find out if the services you receive help you.

Description of Participation

Participation in this survey is voluntary. The decision to participate in this interview is completely your own. You will be asked to participate in one interview that will take approximately 30 minutes. There is no right or wrong answer. This interview will be conducted on the phone or computer with a trained interviewer who will ask you to answer a set of questions about the services you received. You can talk about what you think about the program and the services.

Risks

You may feel uncomfortable about answering some questions about your experiences. You can stop, take a break, or skip any questions you do not want to answer. If you become upset, there are people close by we can get to help you. You may discontinue participation at any time.

Benefits

You will not get any direct benefit from being interviewed. However, the information you provide may help improve the services offered to children, youth, and their families.

Confidentiality

The information you share with us will only be used for the purposes of this study. We will keep your information private to the extent permitted by law. However, if you report any intent to harm yourself or someone else, we will report it to the proper authorities.

Your healthcare services or insurance coverage will not be affected by anything you say during the interview. We will record the interview to help us with our notes. The session recordings will be stored on a password protected computer and will be destroyed at the end of the project. The recording will be reviewed only by a few project staff. Your name will not be used in any reports we write. This signed consent form and any forms and records from the study will be kept in a secure place that only project staff will be able to access.

Contact Information

If you have any questions about this project, you may call SAMHSA’s Project Officer for this evaluation, Ms. Kaitlyn Harrington at 240-276-1928.

If you have questions about your rights and welfare as a research participant, please call the Westat Human Subjects Protections office at 1-888-920-7631. Please leave a message with your full name, the name of the research study that you are calling about (CMHI Evaluation), and a phone number beginning with the area code.

OMB. xxxx-xxxx

Expires: xx/xx/201x

1. Youth Served by the System of Care

Background Information

1. Can you tell me a little bit about yourself and your family?

Do you attend school? What grade are you in?

What do you like to do?

Who else is in your family?

2. Why did you become involved with (name of grant program) ? [Probe for reasons the youth is receiving services.]

How did you learn about (name of grant program) ? Who referred you?

How long have you and your family been receiving services through (name of grant program)?

3. What services do you and your family currently receive through (name of grant program) ?

In addition to these, have or your family ever received any other services through (name of grant program)? [Probe for services].

4. In addition to the services received through (name of grant program) , have you and your family received services from other providers, organizations, programs or agencies? If yes, what were they? [Probe for organizations and services]

Now I’d like for you to think back to when you first came to (name of grant program).

5. How did you become involved with the (name of grant program)? [Probe for details about the entry process].

Was it easy or difficult to get into (name of grant program) ? (D.3.a.)

On a scale of 1 to 5, with 5 being the best, how would you rate how complicated or difficult it was for you or your family’s first service to begin?

Respondent’s rating

5=Entry process was not at all complicated/difficult. Very few steps were involved.

4=Entry process was slightly complicated/difficult.

3=Entry process was somewhat complicated/difficult. Several steps were involved.

2=Entry process was moderately complicated/difficult. Many steps involved.

1=Entry process was extremely complicated/difficult. Very many steps involved.

Interviewer’s rating

5=Entry process was not at all complicated/difficult. Very few steps were involved.

4=Entry process was slightly complicated/difficult.

3=Entry process was somewhat complicated/difficult. Several steps were involved.

2=Entry process was moderately complicated/difficult. Many steps involved.

1=Entry process was extremely complicated/difficult. Very many steps involved.

5. How much time passed between when you and your family first tried to get into (name of grant program) until you actually started receiving services.? Was this a good timeframe for you, or did you need to receive services sooner? (D.8.b.)

On a scale of 1 to 5, with 5 being the best, how would you rate the length of time it took for you or your family’s first service to begin?

Respondent’s rating

5=Timeframe was perfect, no changes needed

4=Timeframe was very fast, could use minor improvement

3=Timeframe pretty fast, could use some improvement

2=Timeframe pretty slow, could use quite a bit of improvement

1=Timeframe entirely too slow, needs a great deal of improvement

Interviewer’s rating

5=Timeframe was perfect, no changes needed

4=Timeframe was very fast, could use minor improvement

3=Timeframe pretty fast, could use some improvement

2=Timeframe pretty slow, could use quite a bit of improvement

1=Timeframe entirely too slow, needs a great deal of improvement

7. Were you treated with respect and made to feel comfortable throughout the enrollment process? (D.3.a.)

Did the staff pay attention to and respect what you had to say?

On a scale from 1 to 5, with 5 being the best, how respected and comfortable did you feel during the process for entering (name of grant program) ?

Respondent’s rating

5=Youth felt extremely respected and comfortable

4= Youth felt very respected and comfortable

3= Youth felt moderately respected and comfortable

2= Youth felt somewhat respected and comfortable

1= Youth felt extremely disrespected and uncomfortable

Interviewer’s rating

5= Youth felt extremely respected and comfortable

4= Youth felt very respected and comfortable

3= Youth felt moderately respected and comfortable

2= Youth felt somewhat respected and comfortable

1= Youth felt extremely disrespected and uncomfortable

8.a. When you first came to (name of grant program) were you given information about available peer support services (e.g., advocacy, youth group, peer-to-peer mentors? Were you given information about the (name of youth organization or program) and what they offer? (D.2.a.)

[Probe for whether youth have access to peer supports right away as opposed to having to wait until the formal service planning process].

How well would you say (name of grant program) met your needs for advocacy and/or peer support services needs in a timely manner?

Respondent’s rating

5=Extremely well

4=Very well

3=Moderately well

2=Somewhat well

1=Not at all well

Interviewer’s rating

5=Extremely well

4=Very well

3=Moderately well

2=Somewhat well

1=Not at all well

8.b. Since entering services at (name of grant program, have you been connected to any peer support services (e.g., youth group, peer mentor)? (F.2.a.)

If not, would you have liked to be connected to peer support services?

If yes, were you able to continue utilizing those peer support services as long as you wanted?

5=Services were accessible through-out the service process

4= Services were accessible for most of the service process

3= Services were accessible for some of the service process

2= Services were accessible for a little of the service process

1= Services were accessible at no point or for very little of the service process

Now I’d like to ask you some questions about what happened when you and the staff at (name of grant program) were deciding what services would be best for you and your family. We call this the service planning process.

9. When you first came to (name of grant program), did anyone help to decide what services you and your family should receive?

If yes, who was the main person you worked with to plan services? [Probe for the first name and function (e.g., case manager or therapist) who worked with the family to plan services. Use that name where you see (name of case manager/therapist) .]

10. Since entering (name of grant program) , have you or your family been involved with staff from other agencies such as child welfare, juvenile justice, education, health, etc.? If yes, which agencies? Probe for all relevant agencies? (E.6.a.)

If yes, did anyone from any of these agencies work with you and (name of case manager/therapist) to plan services for you and your family? If so, who?

Were there any other agencies that you thought should have helped to plan services but did not? If yes, which ones?

5=All involved agencies were present

4=Most involved agencies were present

3=Some involved agencies were present

2=Few of the involved agencies were present

1=One involved agency was present (but family involved with more than one)

666=Family involved with only one agency

11.a. How well did the people who were working with you and your family involve you in the service planning process? (E.3.a.)

Were you encouraged to bring someone to the meeting with you, perhaps for support?

Were you asked whether there was anyone you did not want to be present in the meeting?

Were you asked to talk about what you thought were the most important concerns for yourself?

Were you encouraged to help develop you own goals and objectives?

Were you given a choice of services that you thought would be most helpful for yourself?

Were you able to turn down services that you did not want to receive?

Were there other ways that you were involved in planning your services? If yes, in what ways?

Overall, were you as involved in the service planning process as you think you should have been?

5=Youth was involved in service planning in at least 6 ways AND respondent reported that involvement has been sufficient

4= Youth was involved in service planning in 5 ways OR involved in 6 ways but respondent reported that it could have been better

3= Youth was involved in service planning in 4 ways

2= Youth was involved in service planning in 3 ways

1= Youth was involved in service planning in fewer than 3 ways

11.b. Was the service plan (plan of care) written down? Were you asked to review and sign it? Did you receive a copy of it? (E.3.b.)

1=No

2=Yes

12. When you were working with (name of case manager/therapist to plan services, did she/he talk with you about your strengths (things you like to do, your interests, things you are good at, your hopes)? Would you mind giving me some examples of things you talked about? [Probe for multiple examples] (E.3.c.)

How were your strengths used in planning your services? What could have been done better?

5=Strengths explicitly discussed and at least three examples given of how strengths were incorporated into the service plan AND respondent reported it could not have been better

4=Strengths explicitly discussed and two examples given of how strengths were incorporated into the service plan but respondent reported it could have been better

3=Strengths explicitly discussed and one example given of how strengths were incorporated into the service plan

2=Strengths explicitly discussed but not (or very, very minimally) incorporated into the service plan

1=No discussion of strengths

Now let’s talk about the services you are receiving or have received since you entered (name of grant program).

13. What services were planned for you and your family?

[List the services planned].

Have you received all the services that were in your service plan? (F.3.a.)

If no, what planned services have you or your family not received? [Probe for planned service listed earlier].

If no, why did you not receive some of the services that were planned for you? If yes, [Probe for reasons: lack of funds, no openings, waiting list, not in community, etc.]

5=Child/youth received all of the services that were planned

4=Child/youth received most of the services that were planned

3=Child/youth received many of the services that were planned

2=Child/youth received a few of the services that were planned

1=Child/youth received no services outlined in the plan

14. What have the different service providers you have worked with done to include you in decisions about your services? (F.3.b.)

For example, were you encouraged to offer your ideas about services?

Were your ideas or input about your services considered?

Were you encouraged to let them know when something was not working well?

Were you asked for suggestions about changes that could be made to improve your services?

In what other ways, were you included in decisions about your services?

Overall, how well did your service providers get or use your suggestions about the services they were providing to you?

5=Youth was involved in service provision in 4 ways AND respondent reported that involvement has been sufficient

4= Youth was involved in service provision in 3 ways OR involved in 4 ways but respondent reported that it could have been better

3= Youth was involved in service provision in 2 ways

2= Youth was involved in service provision in 1 way

1= Youth was not involved in service provision

Summary

15. On a scale from 1 to 5, with 5 being the best, how much would you say  (name of grant program) has helped young people?

5=Very much

4=A lot

3=Moderately

2=Somewhat

1=Not at all

16. What has been the best thing about receiving services through (name of grant program)?

17. Do you have any suggestions or recommendations for how (name of grant program) could improve the way that it serves children, youth and families?

18. On a scale from 1 to 5, with 5 being the best, how well do you think (name of grant program) is meeting the needs of children, youth and families?

5=Extremely well

4=Very well

3=Moderately well

2=Somewhat well

1=Not well at all

Thank you for taking the time to answer my questions. Do you have any questions for me?

End of Instrument:

Thank you for participating in the System of Care Expansion Assessment portion of the National Systems of Care Expansion Evaluation.

Attachment 8c – Form J

OMB No. 0930-03xx

Expiration Date: xx/xx/201x

CHILDREN’S MENTAL HEALTH INITIATIVE

NATIONAL SYSTEM OF CARE EXPANSION EVALUATION

SYSTEM OF CARE EXPANSION ASSESSMENT

SERVICE PROVIDERS FROM OTHER AGENCIES

INTRODUCTION

Thank you for your willingness to complete the system of care expansion assessment. The purpose of this interview is to identify the strategies and mechanisms implemented at the local service system level, and examine the extent to which these mechanisms are being implemented from a service provider’s perspective.

CONFIDENTIALITY/INFORMED CONSENT

The National Evaluation team is conducting an evaluation of the federal system of care expansion grant program.

We will be asking you to share your opinions and ideas about various topics related to system of care implementation and expansion.

This session will last approximately 1 hour.

Your participation is completely voluntary, and you have the right to stop at any time or to refuse to answer any question.

Your responses to these questions will be kept confidential and will not be shared outside of the evaluation team. In any of our reports, your responses will be combined with other people’s responses, so your answers will never be attributed to your name.

[OBTAIN INFORMED CONSENT]

We would like to record this interview so that we can be sure to accurately capture your responses. A recording would only be reviewed by a small number of National Evaluation staff members.

[OBTAIN CONSENT TO RECORD SESSION]

INSTRUCTIONS

The SOCEA interview focuses on the local-level management, structure, and procedures as well as direct service delivery level involving activities and practices in the delivery of direct services.

As I ask you the interview questions, please ask for any clarification as needed and provide as accurate information as possible.

OMB No. xxxx-xxxx

Expires: xx/xx/201x

J. Direct Service Staff from Other Public Child-Serving Agencies

Introduction

1. Please briefly describe your agency and its relationship with the (name of grant program) .

[Data entry: Code agency]

What kinds of services does your agency/organization provide — what does your organization do?

2. How long have you been working with this agency/organization?

What is your function?

[Data entry: Code function]

3. Since grant funds were received, what types of services or support has your agency provided to children, youth, and families served by (name of grant program) ? What services do you specifically provide?

4. Since grant funds were received, what percentage of the children, youth, and families you served have also been involved with (name of grant program) ?

5. Since grant funds were received, how has (name of grant program) informed you about program operations such as changes in eligibility criteria, referral processes, and service/program components? (A.7.a.)

[Data entry: code mechanisms]

On a scale of 1–5 (with 5 being best) how effective have these efforts been in ensuring that all parties have been aware of program procedures and operations?

5=very effective

4=moderately effective

3=somewhat effective

2=minimally effective

1=not effective

Entry into Grant Services

6. How complicated is it for families to enter into (name of grant program) services? (D.1.a.)

Please describe the steps involved in the entry process after families are referred to (name of grant program) ? [Probe for what occurs between referral and the first service contact.]

On a scale of 1 to 5 (with 5 being the least complicated) how efficient is it for children, youth, and families to enter (name of grant program) ?

Respondent’s rating

5=Entry process was not at all complicated/difficult. Very few steps were involved.

4=Entry process was slightly complicated/difficult.

3=Entry process was somewhat complicated/difficult. Several steps were involved.

2=Entry process was moderately. Many steps involved.

1=Entry process was extremely complicated/difficult. Very many steps involved.

7. How much time typically passes between referral to (name of grant program) and the first service contact after the intake and screening assessment are completed (not including emergencies or crises)? (D.8.b.)

In your experience, what was the shortest time between referral and first service contact?

What was the longest time?

5=Service typically received in less than one week

4=Service typically received in more than one but less than two weeks

3=Service typically received in more than two but less than three weeks

2=Service typically received in more than three but less than four weeks

1=Service typically received in more than four weeks

Service Planning

8. Since grant funds were received, have you had any involvement in service planning for children and youth served by the (name of grant program) (e.g., attending child and family team service planning meetings, providing input, etc.)?

1=No If no, skip to question 15

2=Yes If yes, continue

9. Have staff from any of the other public agencies routinely participated in service planning (for example, staff from mental health, health, juvenile justice, education, child welfare)? (E.6.a.) If yes, which ones?

1 = Mental Health 4 = Juvenile Justice 7 = Intellectual Disabilities

2 = Education 5 = Public Health 8 = Other (describe)

3 = Child Welfare 6 = Substance Abuse

[circle all that apply]

To what extent have ALL partner agencies participated, or only those involved with the child, youth, or family?

Have any agencies been difficult to engage in the planning process? (e.g., agencies that did not routinely participate, rarely responded to requests to attend planning meetings, etc.) If yes, which ones?

1 = Mental Health 4 = Juvenile Justice 7 = Intellectual Disabilities

2 = Education 5 = Public Health 8 = Other (describe)

3 = Child Welfare 6 = Substance Abuse

[circle all that apply]

Overall, how frequently would you say that all of the agencies involved with a child, youth, or family participated in the child and family team service planning process?

5=Routine participation of most or all involved child-serving agencies such that they routinely and actively participated

4=Frequent but not routine participation of most or all involved child-serving agencies

3=Frequent participation was not routine for all agencies; some involved agencies routinely participated but others participated only sporadically

2=Few agencies routinely participated such that it was rare that all involved agencies participated in service planning

1=None of the involved agencies participated in service planning

10. Have others who provided support or services to a child, youth, or family, such as organizations in the community, direct service providers, or private providers, participated in service planning? (E.7.a.) (Probe for primary health care providers and substance abuse treatment providers)

If yes, who tends to participate (generally)?

Overall, how frequently would you say that ALL of the other organizations and providers involved with a child, youth, or family participated in service planning?

5=Routine participation of most or all involved organizations and providers such that they routinely and actively participated

4=Frequent but not routine participation of most or all involved organizations and providers

3=Frequent participation was not routine for all organizations and providers; some routinely participated but others participated only sporadically

2=Few organizations and providers routinely participated such that it was rare that all involved organizations and providers participated in service planning

1=None of the involved organizations and providers participated in service planning

11.a. Since grant funds were received, what efforts been made to coordinate service planning processes across agencies, organizations, or providers? (E.7.b.)

Is there a unified service planning meeting or process? (e.g., all agencies come together to develop unified or complementary service plans together)

To what extent have staff attended service planning meetings across agencies? What is typical?

11.b. To what extent have efforts to coordinate service planning processes been effective? Please describe? (E.7.b.)

To what extent have efforts been sufficient? What else could be done to improve coordination of the service planning process across agencies, organizations, and providers?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

12. Since grant funds were received, to what extent have service plans been developed for ALL children, youth, and families served by (name of grant program) ? If not all, in which cases have service plans been developed? (E.3.b.)

Of the children enrolled in (name of grant program), what percentage of children, youth, and families have had plans in place?

5=All children and youth (100%) have had individualized service plans

4=Almost all (90–99%) children and youth have had individualized service plans

3=Many (75–89%) children and youth have had individualized service plans

2=Some (50–74%) children and youth have had service plans

1=Few (less than 50%) children and youth have had service plans

13.a. Since grant funds were received, to what extent have family members been present at their service planning meetings? (E.1.a.)

Approximately what percentage of the meetings have families attended?

5=Family member present in all (100%) meetings

4=Family member present in almost all (90–99%) meetings

3=Family member present in many (75–89%) meetings

2=Family member present in some (50–75%) meetings

1=Family member present in few (less than 50%) meetings

666=No planning meetings have been held

13.b. Since grant funds were received, to what extent has the service planning process emphasized family involvement as partners? (E.1.a.)

If families have been commonly involved, please provide specific examples of how families have participated in:

- identifying and prioritizing their problems or concerns

- developing goals and objectives

- requesting participants in the service planning process

- rejecting participants in the service planning process

- identifying and choosing service options

- rejecting service options

-other ways

In general, to what extent has the process involved and empowered families as much as it could have? Could it have been better?

[Note: If the situation has not come up but it would be possible, assign ½ point.]

5=Families have been involved as partners in service planning in at least 6 ways AND respondent reported that family leadership has been sufficient

4=Families have been involved as partners in service planning in 5 ways OR involved in 6 ways but respondent reported that it could have been better

3=Families have been involved as partners in service planning in 4 ways

2=Families have been involved as partners in service planning in 3 ways

1=Families have been involved as partners in service planning in fewer than 3 ways

14.a. To what extent have children and youth been involved in service planning? Under what circumstances do you think it is appropriate for children and youth to participate in service planning?

14.b. If children/youth have been commonly involved, please provide specific examples of how children and youth have been involved in: (E.3.a.)

- identifying and prioritizing their problems or concerns

- developing goals and objectives

- requesting participants in the service planning process

- rejecting participants in the service planning process

- identifying and choosing service options

- rejecting service options

- other ways

In general, did the process involve children and youth as much as it could have? Could it have been better?

[Note: If the situation has not come up but it would be possible, assign ½ point.]

5=Children/youth have been involved in service planning in at least 6 ways AND respondent reported that involvement has been sufficient

4=Children/youth have been involved in service planning in 5 ways OR involved in 6 ways but respondent reported it could have been better

3=Children/youth have been involved in service planning in 4 ways

2=Children/youth have been involved in service planning in 3 ways

1=Children/youth have involved in service planning in fewer than 3 ways

Service Provision

Using the Services Card for Interviewing, ask:

15. Of the services you see listed on this card, which are present in (name of grant program)? [Refer to the Service Array Card for Interviewing. Clarify that this means that the service is available to children, youth, and families served by the grant program regardless of whether the service is in the community or has low capacity.]

[Be sure the respondent has a Service Array Card for Interviewing and read each service option listed. Have the respondent indicate whether each service has been present in the array since receiving grant funds. On the Service Array Card for Scoring, place a check for each service that has been present.]

15.a. Are there any other formal or support services for children, youth, and young adults (not family, sibling, or caregiver services) not listed on this card that are part of (name of grant program)’s service array? Please list.

[Determine whether the service can be coded elsewhere on the list. If not, write additional services on the Service Array Card for Scoring.]

Are there any services that children, youth, and young adults need that are not in the service array? If yes, what are they?

[Interviewer to rate adequacy of for children, youth, and young adults in the service array] (B.3.a.)

Use completed Service Array Card for Scoring to score this item

5=All required child/youth services are in the array AND additional services are also available

4=All required services child/youth are in the array BUT no additional service are present

3=Most required child/youth services are in the array (1–3 missing) (Presence of additional services is not relevant)

2=Some required child/youth services are in the array (4 missing) (Presence of additional services is not relevant)

1=Few required child/youth services are in the array (more than 4 missing) (Presence of additional services is not relevant)

15.b. Are there any other formal or support services specifically for families, siblings or caregivers not listed on this card that are part of (name of grant program)’s service array? Please list.

[Determine whether the service can be coded elsewhere on the list. If not, write additional services on the Service Array Card for Scoring.]

Are there any services that families, siblings or caregivers need that are not in the service array? If yes, what are they?

[Interviewer to rate adequacy of family services and supports in the service array] (B.1.a.)

Use completed Service Array Card for Scoring to score this item

5=All services needed by families are in the array

4=Many family services are in the array

3=Some family services are in the array

2=Few family services are in the array

1=No family services are in the array; all services are provided directly to children, youth, and young adults

16. Of the services you’ve identified as having been present in (name of grant program) ’s service array, which have sufficient capacity to meet the needs of all the children, youth, and/or families who need them? [Circle whether or not each service has sufficient capacity to meet the need.] (B.8.a.)

[Clarify that insufficient capacity relates to services that some children, youth, and/or families cannot get at all even though they need them, or that the respondent does not refer to because of the limited capacity.]

Use completed Service Array Card for Scoring to score this item

5=All services in the array have fully sufficient capacity. All children, youth, and families can get all the services they need

4=Most services in the array (75–95%) have fully sufficient capacity such that all children, youth, and families who need them can get them

3=Many services in the array (50–74%) have fully sufficient capacity such that all children, youth, and families who need them can get them

2=Some services in the array (25–49%) have fully sufficient capacity such that all children, youth, and families who need them can get them

1=Few services in the array (less than 25%) have fully sufficient capacity

17.a. Of the services in your array, which are provided in the community? [Refer again to the Service Array Card for Interviewing and circle whether or not each service is available in the community.] (F.9.a.)

Use completed Service Array Card for Scoring to score this item

5=All services in the array are provided in the community

4=Most services in the array (75–95%) are provided in the community

3=Many services in the array (50–74%) are provided in the community

2=Some services in the array (25–49%) are provided in the community

1=Few services in the array (less than 25%) are provided in the community

17.b. Thinking across all of the different children, youth, and families you’ve worked with since grant funds were received, have any had to travel outside of their home communities to receive services? (F.9.a.)

How far have these children, youth, and families had to travel?

5=No children, youth, or families had to leave the community for services

4=Few (1–2) children, youth, or families had to leave the community for services

3=Some (3–4) children, youth, or families had to leave the community for services

2=Many (5–6) children, youth, or families had to leave the community for services

1=Very many (7 or more) children, youth, or families had to leave the community for services

18.a. Thinking across all of the different children, youth, and families you’ve worked with since grant funds were received, has it ever been necessary for a child or youth to be served in a restrictive setting (for example, alternative school, hospital, group home, etc.)? If yes, please describe.

How frequently has this occurred?

18.b. In these situations, were any efforts made to transition or move the child or youth into less restrictive services or settings once appropriate? If so, please describe. (F.10.b.)

To what extent have these efforts been effective?

To what extent have these efforts been sufficient (such that children and youth were transitioned to less restrictive services/settings as soon as possible)? If not, why not?

5=Efforts made have been very effective and sufficient to accomplish larger goals; no or only minor additional efforts needed

4=Efforts made have been moderately effective but not sufficient to accomplish larger goal; some additional efforts needed

3=Efforts made have been somewhat effective but not sufficient to accomplish larger goal; considerable additional efforts needed

2=Efforts have been made BUT have not been effective or have been minimally effective

1=No or almost no effort has been made toward accomplishing larger goal

Summary

19. Since grant funds were received, have you received or participated in any training sessions provided as part of (name of grant program) activities?

If yes, please describe the topics or content areas covered. [Probe on CASSP principles such as family involvement, cultural competency, individualized care, strengths-based care, evidence-based treatment, care coordination, etc.]

20. Since grant funds were received, what have been (name of grant program) ’s most important achievements?

21. What have been the biggest barriers or obstacles faced by (name of grant program) ?

22. How has your agency benefited from (name of grant program) ?

23. How has your community benefited from (name of grant program) ?

OMB No. xxxx-xxxx

Expires: xx/xx/201x

System/Program_______________________ Interviewer___________________________________

Respondent Data Entry #_________________ Date_________________________________________

Service Array Card for Scoring

	Data Entry		Service Options		Present in the array ✔ [Q. 26]	Sufficient capacity? [Q. 27]			Wait (in days) [Q. 28]	Provided in the community? [Q. 29]
						No	Yes			No		Yes
1	40.		Diagnostic and evaluation services			1	2			1		2
2	41.		Neurological and/or neuro-psychological assessment			1	2			1		2
3	42.		Outpatient individual counseling			1	2			1		2
4	43.		Outpatient group counseling			1	2			1		2
5	44.		Outpatient family counseling			1	2			1		2
6	45.		Medication management			1	2			1		2
7	46.		Care management/coordination			1	2			1		2
8	47.		Wraparound service planning model			1	2			1		2
9	48.		Respite care			1	2			1		2
10	49.		Professional consultation			1	2			1		2
11	50.		24-hour, 7-day-a-week emergency services, including mobile crisis outreach and crisis intervention			1	2			1		2
12	51.		Intensive day treatment services			1	2			1		2
13	52.		Therapeutic foster care			1	2			1		2
14	53.		Therapeutic group home			1	2			1		2
15	54.		Intensive home-based services (e.g., family preservation services)			1	2			1		2
16	55.		Transition-to-adult services			1	2			1		2
17	56.		Residential treatment			1	2			1		2
18	57.		Inpatient hospitalization			1	2			1		2
19	58.		Alcohol and drug treatment			1	2			1		2
20	59.		Alcohol and drug prevention			1	2			1		2
21	60.		Family advocacy and peer support			1	2			1		2
22		61.	Youth advocacy and peer support			1	2			1		2
Evidence-based Treatments (EBTs)
23		62.	Multi-systemic Therapy (MST)			1	2			1		2
24		63.	Functional Family Therapy (FFT)			1	2			1		2
25		64.	Multi-dimensional Treatment Foster Care			1	2			1		2
26 27 28		65. 66. 67.	Behavioral parent training (e.g.,Triple P, Incredible Years, Parent Child Interaction Therapy) If yes, specify EBT(s): ­­­­­­­­­­­­­­­­­­­­­____________________ _______________________________________ _______________________________________			1 1 1 1	2 2 2 2			1 1 1 1		2 2 2 2
29		68.	Brief Strategic Family Therapy			1	2			1		2
30		69.	Family Team Decision Making			1	2			1		2
31 32 33		70. 71. 72.	Cognitive Behavioral Therapy (CBT) If yes, specify types: ____________________ _______________________________________ _______________________________________			1	2			1		2
34		73.	Trauma-focused Cognitive Behavioral Therapy (TF-CBT)			1	2			1		2
Other Trauma-related Evidence-based Treatments (specify)
35		74.				1	2			1		2
36		75.				1	2			1		2
37		76.				1	2			1		2
Other Evidence-based Treatments (specify)
38		77.				1	2			1		2
39		78.				1	2			1		2
40		79.				1	2			1		2
41		80.				1	2			1		2
Other Formal or Support Services (specify)
42		81.				1		2			1		2
43		82.				1		2			1		2
44		83.				1		2			1		2
45		84.				1		2			1		2

Note: When scoring, if responses are missing (e.g., don’t know) for more than 3 services in a given column, do not score the item that relates to that column.

Required services are shaded in grey.

Service Array Card for Interviewing

	Service Options
1	Diagnostic and evaluation services
2	Neurological and/or neuro-psychological assessment
3	Outpatient individual counseling
4	Outpatient group counseling
5	Outpatient family counseling
6	Medication management
7	Care management/coordination
8	Wraparound service planning model
9	Respite care
10	Professional consultation
11	24-hour, 7-day-a-week emergency services, including mobile crisis outreach and crisis intervention
12	Intensive day treatment services
13	Therapeutic foster care
14	Therapeutic group home
15	Intensive home-based services (e.g., family preservation services)
16	Transition-to-adult services
17	Residential treatment
18	Inpatient hospitalization
19	Alcohol and drug treatment
20	Alcohol and drug prevention
21	Family advocacy and peer support
22	Youth advocacy and peer support
Evidence-based Treatments (EBTs)
23	Multi-systemic Therapy (MST)
24	Functional Family Therapy (FFT)
25	Multi-dimensional Treatment Foster Care
26 27 28	Behavioral parent training (e.g.,Triple P, Incredible Years, Parent Child Interaction Therapy) If yes, specify EBT:
29	Brief Strategic Family Therapy
30	Family Team Decision Making
31 32 33	Cognitive Behavioral Therapy (CBT) If yes, specify types:
34	Trauma-focused Cognitive Behavioral Therapy (TF-CBT)
	Other Trauma-related Evidence-based Treatments (specify)
35
36
37
	Other Evidence-based Treatments (specify)
38
39
40
41
	Other Formal or Support Services (specify)
42
43
44
45

End of Instrument:

Thank you for participating in the System of Care Expansion Assessment portion of the National Systems of Care Expansion Evaluation.

Attachment # 8 Systems of Care Expansion Assessment Page 91