0930-0339_OMB_SS_PartA 1.24.17

Evaluation OF the Substance Abuse and

Mental Health Services administration’s (samhsa’S) cooperative agreements to benefit homeless individuals (CABHI) program

Supporting Statement

A. JUSTIFICATION

1. Circumstances of Information Collection

The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Mental Health Services (CMHS) and Center for Substance Abuse Treatment (CSAT) are requesting a revision from the Office of Management and Budget (OMB) for data collection activities under OMB No. 0930-0339, which expires on 1/31/2017. These data collection activities will be conducted for SAMHSA’s evaluation of the CABHI services grant program, which is scheduled through September 2020. The data collection activities described in this package include telephone interviews, site visits with guided interviews, and web-based assessments, as follows:

• Project Director (PD) Telephone Interview & Web Survey

• Site Visit Guides

• Evidence-Based Practice (EBP) Self-Assessment, Parts 1 & 2

• Permanent Supportive Housing (PSH) Self-Assessment

The CABHI grant program is authorized under Sections 506, 509, and 520A of the Public Health Service Act, as amended. The program also aligns with SAMHSA’s Recovery Support strategic initiative and addresses Healthy People 2020 Objectives: Mental Health and Mental Disorders (Topic Area HP 2020-MHMD) and Substance Abuse (Topic Area HP 2020-SA).

Background of the CABHI Program

SAMHSA’s mission is to reduce the impact of substance abuse and mental illness on America’s communities; one important element of that is meeting the treatment, housing, and support service needs of people who are homeless and have substance use disorders, mental disorders, or both. From 2001 through 2010, SAMHSA’s CSAT funded projects through the Grants for the Benefit of Homeless Individuals (GBHI) grant program, which focused on expanding and strengthening treatment services for people who are homeless or at risk of homelessness and have substance use disorders, mental health disorders, or both. From 2007 through 2010, SAMHSA’s CMHS also funded projects with a focus on homeless populations through the Services in Supportive Housing (SSH) grant program, which provided services to individuals and families experiencing chronic homelessness in coordination with existing PSH programs and resources. In 2011, CMHS and CSAT began jointly funding the CABHI grant program to build upon the success of the GBHI and SSH programs.

CABHI emphasizes the value of providing access to permanent housing and supportive services for people who are chronically homeless and have mental disorders, substance use disorders, or both. It enhances and extends the GBHI and SSH efforts by focusing on the development and expansion of local infrastructures that integrate treatment and services for mental and substance use disorders, permanent housing, and other critical services for individuals experiencing chronic homelessness.

Over the past 5 years, the CABHI program has continued to grow and evolve. CABHI grants were originally awarded to community-based entities (hereafter referred to as “Communities”) through 2012. Beginning in 2013 through 2015, CABHI grants were awarded to State agencies (i.e., State Mental Health Authorities or Single State Agencies for Substance Abuse) with an additional focus on statewide planning for integrated services. In 2016, the cohort of focus for the current evaluation, SAMHSA funded CABHI grants at three levels:

• State (up to $1.5 million per year),

• Local Government (up to $800,000 per year), and

• Community (up to $400,000 per year).

The varying levels of 2016 CABHI grantees are united by the goal of enhancing and expanding infra­structure and capacity for mental health and substance abuse treatment and related support services for individuals experi­encing chronic homelessness or veterans, families, or youth experiencing homelessness as a result of these conditions. This is accomplished through the provision of PSH, behavioral health treatment, and recovery support services, and enrollment in health insurance, Medicaid, or other mainstream benefit programs.

To ensure that project clients receive comprehensive and coordinated services, grantees must devote at least 70% of grants funds to providing the following types of treatment and support services: outreach and engagement, case management, behavioral health treatment services, trauma-informed services, peer support, family-driven and youth-guided frameworks, collaboration amongst providers, and support aimed at service and treatment retention. Although CABHI funds cannot be used to directly fund housing, grantees must connect their clients to permanent housing.

The grantees must also engage in several activities to address the CABHI program’s focus on infrastructure development (e.g., developing a statewide plan to sustain partnerships across public health and housing systems), using up to 20% of grant funds for States and 10% for Local Governments and Communities. In line with prior CABHI cohorts, grantees must also have a governing body composed of stakeholders across service systems that supports the goal of improving infrastructure and is charged with monitoring project implementation and progress (i.e., steering committees, State Interagency Councils on Homelessness).

Precursor to the CABHI Evaluation

The data collection activities for which SAMHSA is seeking revised approval were developed for an evaluation of the 2009 through 2012 GBHI, SSH, and CABHI grant cohorts, which included 127 grantees. The PD Phone Interview/Web Survey, Site Visits Guides, and EBP and PSH Self-Assessments were successfully implemented, providing data critical to the achievement of the evaluation’s objectives. This evaluation was recently concluded with key results showing that clients needed many types of services and grant projects were successful in meeting those needs, key client outcomes (substance abuse, mental health, homelessness, and arrests) improved significantly, and clients who received more services tended to report greater improvements in outcomes. The evaluation also found that, to a large extent, grantees successfully implemented services in accord with grant requirements, numerous types of organizations successfully implemented the grants, and partnerships and collaboration were important elements of these grants.

Overview of the CABHI Evaluation

In 2016, SAMHSA funded a 4-year cross-site evaluation of the 30 CABHI grant projects initially funded in 2016. The primary task of the CABHI evaluation is to conduct a comprehensive process and outcome evaluation, addressing questions related to the implementation of the CABHI grant projects and the extent to which they were able to meet the program’s goals.

Process evaluation primarily represents what is done to and for the client; this aspect of the evaluation will also include a focus on structure, or the resources available in the service delivery system, which represent the capacity to deliver quality care, but not the care itself. The CABHI evaluation process measures include characteristics of the grantee organization and its partnerships; the system within which the project is embedded; relationships with stakeholders; characteristics of the target population; services received, including implementation of EBPs; staffing patterns; costs of services; barriers and facilitators of project implementation; and project sustainability efforts.

The outcome evaluation will focus on outputs, which are the most immediate or proximal results of project activities (e.g., changes in partner collaboration, the number of clients enrolled in mainstream benefits), and client outcomes, particularly those related to behavioral health and homelessness and housing instability. Statistical analyses (described in Section A.16) will be used to examine changes in client outcomes from baseline to follow-up and to explore whether changes in outcomes were associated with client characteristics, receipt of services, service models, or other project or grantee characteristics.

The evaluation questions (EQs) that will be addressed by the CABHI evaluation are listed in Table 1.

Table 1. CABHI Evaluation Questions by Domain

Systems

EQ1. What organizations, agencies, and individuals (federal, state, and local) are involved in state or community Interagency Councils on Homelessness? What is the nature of the collaborations? (Process)

EQ2. What are the barriers to or facilitators of state- and community-level collaboration and partnership development? How are they addressed? (Process)

EQ3. To what extent do CABHI activities lead to enhanced coordination and collaboration across mental and physical health providers, housing providers, and other organizations and agencies? (Outcome)

EQ4. Does the type of partner dyad (e.g., mental health treatment providers, housing providers) influence the degree to which coordination and collaboration are enhanced?

EQ5. What is the impact of collaboration across federal, state, and local agencies on CABHI grant activities, processes, and outcomes? (Outcome)

EQ6. How do state- and local-level systems change in response to CABHI activities? (Outcome)

Expansion and Access to Services

EQ7. Who provided (project staff), what services (modality, type, intensity, duration), to whom (individual characteristics), in what context (system, community), and at what cost (facilities, personnel, dollars)? (Process)

EQ8. What strategies were used to maintain fidelity to any EBP chosen or intervention across providers over time? (Process) Was greater fidelity reported for certain strategies or EBPs?

EQ9. How many individuals were reached through the program? (Process) What were their characteristics (demographics, illnesses, housing status, and history)? How did the number of individuals reached and their characteristics compare with grantee targets? Was success in meeting or exceeding enrollment targets associated with grantee configurations or strategies?

EQ10. To what extent were services based on the needs of specific service recipients (e.g., trauma-focused services)? (Process)

EQ11. To what extent do CABHI projects lead to increased access to stable, permanent housing? (Outcome) Were certain grantee configurations, models or strategies, services or service bundles (including the use of EBPs), or target populations associated with greater increases?

EQ12. To what extent do CABHI projects lead to increased access and enrollment in mental health services and supports? (Outcome) Were certain grantee configurations, models or strategies, services or service bundles (including the use of EBPs), or target populations associated with greater increases?

EQ13. To what extent do CABHI projects lead to increased access and enrollment in private insurance, Medicaid, or other benefits (e.g., SSI/SSDI, TANF, SNAP)? (Outcome)

EQ14. What funding strategies did state or community Interagency Councils on Homelessness develop or adopt for sustainability of mental health, substance use, housing, and other supports and services? (Process)

EQ15. How did the grantees use evaluation findings and performance measurement to support improved collaboration with stakeholders and support sustainability? (Process)

Participant Outcomes

EQ16. To what extent do CABHI projects lead to improved housing stability? (Outcome)

EQ17. Where comparisons are possible and appropriate, how do CABHI grant participants compare with similar nonparticipants on HMIS measures (e.g., housing tenure)? (Outcome)

EQ18. To what extent do CABHI projects lead to improved behavioral health, daily life and functioning (e.g., social connectedness, relationships, employment/education, criminal justice involvement), enrollment in benefits, and decreased substance use? (Outcome)

EQ19. What project/contextual factors were associated with outcomes? (Outcome) Were certain grantee configurations, models or strategies, services or service bundles (including the use of EBPs), or target populations associated with greater increases?

EQ20. What individual factors were associated with outcomes, including race/ethnicity and sexual identity (sexual orientation/gender identity)? (Outcome)

To address these evaluation questions, it is critical to gather accurate data on the many varied strategies and services that grantees and their partners are likely to use. This will be done, in part, by drawing directly from the previous evaluation and using the PD Phone Interview/Web Survey and EBP and PSH Self-Assessments that were developed to collect complementary and comprehensive data on a wide array of grant activities. To improve upon the approach in the previous evaluation, process data will be collected at multiple time points, with careful coordination of the timing of different data collections so as not to overburden the grantees. Data gathered with these instruments have proven valuable in describing how grantees implemented their grants, and in helping to account for changes in client outcomes. While it is important that common information is collected across the grantees and their partners, it is also important to understand the distinctions and nuances they present. Relying entirely on standard quantitative data collection instruments can obscure important details. Therefore, both quantitative (e.g., surveys) and qualitative (e.g., site visits) data collection methodologies will be used; moreover, analytically the resulting data will not be used in isolation but rather will be blended to achieve the optimal degree of cross-site standardization and deep understanding of each site.

Client-level data collection is not included in this request to OMB because the evaluation will make use of the client-level data (hereafter referred to as CSAT GPRA data) that CABHI projects collect under OMB No. 0930-0208 as a requirement of their grant. Additionally, OMB approval (No. 0930-0320) was received separately for a supplemental client interview and stakeholder survey that will also be used for the CABHI evaluation; a request to OMB for revision will be submitted separately prior to the expiration date.

As previously described, the 2016 CABHI grantees were funded at three levels: States, Local Governments, and Communities. Within the States (n=3), sub-recipients are contracted to implement the project and provide services at the local level; for two States that is the county level and for one state that is the Department of Housing and Urban Development (HUD) Continuum of Care (CoC) level. The State sub-recipients will be the entities that will enroll and provide services to clients, while the States themselves will act in oversight and coordinating roles. Therefore, the State sub-recipients will be viewed by the evaluation similarly to Local Government and Community grantees as they share much the same function. The number of States and sub-recipients, Local Governments, and Communities funded by CABHI in 2016 is listed in Table 2.

Table 2. The 2016 CABHI Grantees

Funding Level	Number
State	3
State sub-recipient	9
Local Government	12
Community	15

2. Purpose and Use of Information

The following section describes each of the data collection activities that are the focus of this request to OMB.

1. Project Director Telephone Interview & Web Survey

The PD Phone Interview/Web Survey (Attachment 1) is designed to systematically collect key grantee project characteristics which will directly inform the process evaluation component and will also provide essential data by documenting the partnerships and services each grantee includes in their project. The PD Interview includes two components, a semi-structured telephone interview and a Web survey. The telephone interview focuses on areas that are more nuanced and may require discussion to fully understand the details (e.g., target population, housing) while the Web survey provides complementary information to the phone interview that is better captured through Web administration (e.g., grantee characteristics, services provided).

The PD Phone Interview/Web Survey is composed of the following sections: Grantee Agency and Project Characteristics, Target Population, Stakeholders/Partners, Services, EBPs/Best Practices, Housing, Project Organization and Implementation, Sustainability, Local Evaluation, Technical Assistance, and Lessons Learned. Table 2 provides a brief description of each section and the EQs they address.

Table 3. PD Phone Interview/Web Survey Sections and Evaluation Questions

Grantee Agency and Project Characteristics (EQ7, EQ9, EQ19)

Asks for general information on a grantee’s agency and project setup, including funding sources used, client count targets, geographic areas targeted and general information on the type of staff used.

Target Population (EQ7, EQ9, EQ19)

Asks for information on the grantee’s recruitment process, any demographic based admission criteria and any targeted populations by race/ethnicity, age, sex, health and treatment status, mental health and substance abuse severity, homeless status and other special population categories.

Stakeholders and Partners (EQ1, EQ3, EQ4, EQ5, EQ6, EQ7)

Asks the grantee to identify the type of partners and partnerships used to implement their project and their integration into the project.

Services (EQ7, EQ10, EQ11, EQ12, EQ13, EQ19)

Identifies the clinical and wraparound services available to project clients and the structure in which they are available—how many clients receive the service, who provides the service, where is the service provided, how is the service paid for and the length of time that clients can receive the service.

Evidence Based Practices/Best Practices (EQ7, EQ8, EQ11, EQ12, EQ19)

Identifies the primary clinical and non-clinical Evidence Based Practices (EBPs). It outlines which EBPs were proposed by grantee projects and which EBPs were ultimately implemented, how many clients receive the EBPs, who provides the EBPs and where the EBPs are provided. Questions also cover the phase of implementation, fidelity monitoring, and identification of barriers and facilitators to implementation.

Housing (EQ7, EQ11, EQ12, EQ19)

Asks what types of housing are included in the project. For each type of housing implemented by the grantee, this section asks about the project’s focus in terms of housing clients, the type of housing sites available, type of funding used, degree of client choice, the type of housing support services provided, restrictions on placing clients in housing, the degree of separation between housing and services, and housing philosophy adopted by the project.

Project Organization and Implementation (EQ2, EQ3, EQ7, EQ19)

Asks the grantee respondent to rate staff experience, partner support during implementation, implementation and operation effectiveness, and identify barriers to project implementation.

Sustainability (EQ15, EQ16)

Establishes a baseline understanding of how grantees plan to sustain their projects after SAMHSA funding ends by identifying sustainability plans, activities and the partners involved in sustainability efforts.

Technical Assistance (EQ varies, depending on the type of TA received)

Asks if the grantee has made any technical assistance (TA) requests through SAMHSA and, if yes, asks for the type of TA provided and if it impacted implementation and/or ongoing program implementation.

Local Evaluation (EQ15)

Identifies the type of evaluation conducted by the grantee project, the type of data collected and from whom, and its planned use.

Lessons Learned (EQ varies, depending on the topic the respondent provides a lesson learned)

Asks grantees to briefly describe one lesson learned for each of the following categories: serving their target population, project implementation, implementing EBPs, partner collaboration, and sustainability.

2. Site Visit Guides

Site visits are an important component of program evaluation, providing the opportunity to see what is really occurring “on the ground,” which cannot always be conveyed as precisely through other data collection modes. The purpose of the grantee Site Visits is to collect detailed qualitative information and economic data on project activities conducted by the grantees and their partners, which will directly inform the process evaluation. The qualitative data will also provide essential information for the outcome evaluation component by documenting the interventions provided to clients and the implementation, barriers, facilitators, challenges and successes for each grant project visited.

Site Visit Guides (Attachment 2) consist of semi-structured discussions with six types of respondents: (1) Opening Session/Project Director and Management Staff (e.g., grantee project directors, project managers/coordinators); (2) Case Managers, Treatment and Housing Staff/Providers (e.g., clinical treatment staff, support services staff, case managers, housing providers, etc.); (3) Stakeholders (e.g., primary partners and other key stakeholders); (4) Evaluators; (5) Clients (project participants); and (6) Financial Staff. This approach allows information to be collected from multiple perspectives giving a fuller picture of the grant project; the interviews are complementary but not redundant.

The Site Visit Guides will be customized to each grantee as some questions may not be relevant to all grantees. The guides are structured as discussions, with written questions to be used as a general guide but adjusted depending on the interviewees’ experience and understanding of the grantee project. The topics covered in each discussion guide are reviewed below.

Opening Session/Project Director Discussion Guide. The purpose of this session is several fold: (1) To ensure the site visitors understand the grantee agency and its relationship to the program and community homeless services; (2) to gain a solid overview of the project by reviewing (and revising as needed) the organizational chart, project logic model, and client flow chart; (3) to understand the treatment and other services implemented by the project, barriers and facilitators to project implementation, and lessons learned; and (4) to obtain an overview of project staffing and sustainability.

As outlined in Tables 2 and 3, the PD Phone Interview/Web Survey and the Opening Session/Project Director Discussion Guide (PD Discussion Guide) share common topic areas which are complementary in design and purpose. The PD Interview typically collects quantitative data which provides the basic characteristics of the grantee’s program in a standardized format across all grantee programs. These data as described will be used to develop typologies across all sites. The PD Discussion Guide allows grantees to discuss particular characteristics, how they came to be, why they are important, and the challenges and/or successes associated with that characteristic. For example, the PD Interview identifies the number and type of agencies the grantee partners with, while the PD Discussion Guide discusses why a partner was selected, what they bring to the project and the challenges and/or successes in working with that partner. Similarly, the PD Interview allows for systematic identification of each grantee’s primary EBPs while the PD Discussion Guide questions identify how an EBP was implemented, the modifications needed and its fidelity to core components. In general, the site visit questions, unlike the PD Interview questions, are discussion prompts and guides which are asked as needed and as relevant to a particular site. The data from the PD Interview questions will be used both for the process component, including development of typologies, and for the outcome component. The data from the site visit protocol interviews will be used to develop case studies to inform SAMHSA of unique lessons learned with regard to implementing particular services models for particular subpopulations and settings.

Table 4 provides a brief description of each section of the discussion and the EQs they address.

Table 4. Opening Session/Project Director Discussion Guide: Sections and Evaluation Questions

Overview of Grantee and Partner Agencies (EQ1)

Provides qualitative data on the grantee’s mission, role in the local treatment system, how the grant program fits into the grantee’s other work and whether the grantee has any other SAMHSA grants.

Community Context (EQ1, EQ2, EQ3, EQ5, EQ14, EQ19)

Identifies key characteristics of the grantee’s local treatment and service system including the resources available, the services (clinical and recovery support) typically available, and the gaps in services. The local resources available for housing services are also identified along with the barriers to obtaining housing and gaps in housing services. To better understand how the grantee operates within the treatment and service system, a set of questions ask about the grantee’s relationship with system, and whether the grantee participates in local efforts to end homelessness, is aware of 10 year plans to end homelessness, and is involved in the homeless continuum of care.

Brief Project Overview (EQ1, EQ7)

Focuses on how the grant project and grantee are organized to provide services to clients, including a review of the grantee’s project logic model. The section also includes questions on how the grantee relates to its partners and stakeholders in the context of providing grant services, whether there have been any challenges and how partners/stakeholders have collaborated with the grantee.

Target Population (EQ7, EQ9)

Identifies the grantee’s target population and criteria for enrollment, which will be used to categorize the grantees and better understand any changes made to the target population.

Client Flow (EQ7, EQ3, EQ6)

Provides a detailed, step-by-step schematic of the grantee project from the client perspective. It is critical in helping to establish a solid understanding of the day-to-day workings of the project, how services are delivered and how clients move through the project over time. Questions cover how the grantee identifies, recruits and screens clients, how and what services are typically provided to clients and how housing services are integrated into the project.

Systems and Client Outcomes (EQ5, EQ6, EQ11, EQ12, EQ13, EQ16, EQ17, EQ18)

Asks the grantee how project implementation impacts the grantee’s agency and treatment system. The section also asks for grantee input on tracking client outcomes, which outcomes should be tracked and if the CSAT GPRA questions are useful measures for the client outcomes of interest.

Barriers, Facilitators and Innovations (EQ2, EQ4, EQ19, EQ20)

Asks the grantee to discuss barriers and facilitators faced by their project, how barriers were or are being addressed, and any innovations developed. Grantees are asked specifically about barriers and facilitators to implementation, service delivery and innovations developed in response to challenges.

Lessons Learned (EQ2, EQ8, EQ10, EQ15, EQ19, EQ20)

Allows the evaluation to collect data on the various grantee strategies used to successfully implement their project which will provide valuable information to SAMHSA for performance monitoring and future grantees. Grantees are specifically asked about lessons learned regarding project organization, target population, client outcomes, overall system outcomes and any changes they would make if they started the project over again.

Sustainability (EQ14, EQ15)

Asks about the grantee’s sustainability plans including whether they have a written plan, the project elements they would seek to sustain, how grant project sustainability fits within the agency’s strategic goals, how sustainability fits within community and service system goals including HUD Consolidated Plans or Continuum of Care, how involved partners are in sustainability plans, and plans to use evaluation and/or data to promote sustainability.

Project Staffing (EQ7, EQ8)

Explores whether and how the grant project was impacted by staffing issues such as hiring delays, staff turnover, staff alignment with the target population and staff training.

SAMHSA Training (EQ2, EQ8, EQ15)

Asks whether the grantee has requested and received any training from SAMHSA and, if so, its impact on the project.

Evidence Based Practices (EBPs)/Best Practices and Training (EQ2, EQ8)

Focuses on the main treatment EBPs used by the grantee, asking interviewees to describe the main components, any modifications, additional funding and staff requirements to implement the EBP/Best Practice. Additional questions ask grantees to describe the types of training and technical assistance received around EBPs. Identifying the EBPs implemented by grantees and better understanding what adaptations were made to fit the grantee’s local context is central to documenting how well grantees are meeting SAMHSA’s expectations in implementing EBPs.

Permanent Supportive Housing Questions (EQ11, EQ12, EQ16)

The questions in this section ask these grantees to describe the components of Permanent Supportive Housing (choice, services, payment, special needs, legal rights, readiness requirements, client control over unit, and client input) within the context of their project, which will help provide qualitative confirmatory information on the degree to which grantees implement PSH.

Case Managers, Treatment, and Housing Staff/Provider Discussion Guide. The purpose of this session is to collect detailed information on the program services and housing provided to clients from the staff delivering services. The discussion guide focuses on service implementation, alignment of services with client needs, barriers and facilitators, and lessons learned related to housing, treatment, and case management/wraparound services. These questions cover all of the types of services that may be provided under the SAMHSA funding, but sections will be administered only as relevant to the grant project. Table 5 provides a brief description of each section in the discussion and the EQs they address.

Table 5. Case Managers, Treatment, and Housing Staff/Provider Discussion Guide: Sections and Evaluation Questions

Overview of Treatment, Case Management, & Housing Providers (EQ1, EQ4, EQ7, EQ8)

As many grantees use partner case management, treatment and housing agencies, understanding how each of these providers fit into the grantee project is essential. If the grantee provides case management, treatment or housing services directly (i.e. not via partners), this section will be covered in the Opening Session/Project Director Interview. Questions focus on the nature of the collaboration between the grantee and the partner(s), the partner(s) role in delivering services, integration of treatment services with housing and wraparound services, and whether and how the grant has impacted the partner agency.

Client Flow (EQ7, EQ3, EQ6)

Provides a detailed, step-by-step schematic of the case management, treatment and housing services from the client perspective. It is critical in helping to establish a solid understanding of the day-to-day workings of the project, how services are delivered and how clients move through the project over time. Questions cover how the grantee identifies, recruits and screens clients, how and what services are typically provided to clients and how housing services are integrated into the project.

Here, the client flow section complements the client flow section in the Opening Session/Project Director discussion by obtaining the provider’s perspective, which will provide additional important details on how clients enter, receive and exit case management, treatment and housing.

Alignment of Services with Client Needs (EQ7, EQ9, EQ10)

Focuses on the degree to which services align with client needs and asks how service planning occurs, whether client strengths are identified, and whether and how clients are given choices in their treatment/services received.

Client Outcomes (EQ11, EQ12, EQ13, EQ16, EQ17, EQ18)

Asks for provider input on tracking client outcomes, which outcomes should be tracked, and if the CSAT GPRA questions are useful measures for the client outcomes of interest.

Barriers, Facilitators and Innovations (EQ2, EQ4, EQ19, EQ20)

Allows for providers to discuss the various barriers and facilitators faced by their project, how barriers were or are being addressed and any innovations developed. Providers are asked specifically about barriers and facilitators to implementation, service delivery and innovations developed in response to challenges.

Lessons Learned (EQ2, EQ8, EQ10, EQ15, EQ19, EQ20)

Allows the evaluation to collect data on the various grantee strategies used to successfully implement their project which will provide valuable information to SAMHSA for performance monitoring and future grantees. Providers are specifically asked about lessons learned regarding client flow/resource use, use of EBPs, alignment with client needs, client outcomes, system outcomes and housing accessibility.

Stakeholder Discussion Guide. The purpose of the stakeholder discussion is to learn about projects from the perspective of the associated partner providers, key stakeholders and local funders. The discussion aims to learn about the agencies/providers involved in the project, the ways in which they are involved, and their perspectives on how the project has been implemented, its impact on and contribution to the community, and efforts made toward sustainability. Some grantees may not have external providers or stakeholders but instead may be working with other departments within their agency (their internal partners) for treatment, wraparound services and/or housing. In these cases, the internal partners would participate in an abbreviated stakeholder discussion. Table 6 provides a brief description of each section in the discussion and the EQs they address.

Table 6. Stakeholder Discussion Guide: Sections and Evaluation Questions

Overview of Associated Providers Involved with the Project (EQ1, EQ4, EQ7, EQ8)

Allows the interviewer to better understand how each partner fits into the overall service system and their specific role in the grantee’s project. Questions ask about the services provided, client population typically served, geographic area targeted, and experience with SAMHSA grants and the grantee agency.

Relationship between Associated Providers/Key Stakeholders/Local funders & the Project (EQ1, EQ2, EQ3, EQ5, EQ14, EQ19)

Collects information on how the grantee and its partners collaborate and how partners may collaborate with each other. Questions help identify the mechanisms for collaboration, such as stakeholder committees, community consortiums, or other formal meetings.

Associated Providers/Key Stakeholders/Local Funders Perspective on Services and Client Outcomes (EQ3, EQ5, EQ6)

Provides the partner’s perspective on services provided through the grantee project and the impact the project has on clients. Questions ask whether the project is serving the targeted population as intended, whether there are similar services available to clients besides the grantee’s project, and the project’s effect on client outcomes.

Systems Change (EQ5, EQ6, EQ9, EQ13)

Identifies any change to the service system, such as policies, housing markets, and funding streams, due to the grantee’s project.

Barriers, Facilitators & Innovations (EQ2, EQ4, EQ19, EQ20)

Asks stakeholders to discuss barriers and facilitators faced by the project, how barriers were or are being addressed, and any innovations developed. Stakeholders are asked specifically about barriers and facilitators to implementation, service delivery and innovations developed in response to challenges.

Project Sustainability Activities (EQ14, EQ15)

Asks questions about the partner’s involvement in sustainability plans, including whether they have participated in any meetings, reviewed evaluation reports and/or data, and the partner’s overall perspective on sustainability planning. Partners are also asked how sustaining the grantee’s project would fit into the overall system service, which components are the most important to sustain, and what impact not sustaining the project would have on clients and the community.

Evaluator Discussion Guide. The purpose of the evaluator discussion guide is to understand the grantee project’s local evaluation and quality assurance activities; how the evaluation is incorporated into sustainability activities, project implementation, and EBP/PSH fidelity; and lessons learned. Table 7 provides a brief description of each section in the discussion and the EQs they address.

Table 7. Evaluator Discussion Guide: Sections and Evaluation Questions

Evaluation Overview & Integration Into Project (EQ1, EQ4, EQ7, EQ8)

Asks the grantee evaluator to describe the local evaluation and how it is integrated with the project’s planning, management, clinical meetings, sustainability planning, quality assurance and feedback to clients.

Process Evaluation (EQ1, EQ4, EQ7, EQ8)

Asks about the process evaluation component of the local evaluation, if a process evaluation is being conducted. The questions focus on the overall aim of the process evaluation, how client participation is measured, how housing received by clients is tracked, how data are collected on project sustainability and how the project uses data to improve.

CSAT GPRA Data and Outcome Evaluation (EQ15)

Asks how CSAT GPRA data are collected, what is most useful about the CSAT GPRA data, and whether CSAT GPRA data are matched with other locally collected outcome data. If additional outcome data are collected, the measures and plans to use the data are discussed.

Evaluation Analysis and Reporting (EQ15)

Collects information about the local evaluator’s role in reporting findings, how the evaluation is managed and what, if any, main findings are available so far.

Fidelity Assessment (EQ8)

Asks the evaluator whether and how fidelity assessments are conducted for the primary EBPs being implemented in the project.

Client Focus Group Discussion Guide. The purpose of this session is to learn about the grantee project from the client perspective. The questions for the group begin with basic information about the participating clients, such as length of involvement with the project, history of homelessness, and prior participation in services similar to those provided by the project. The remaining questions focus on the types of services clients have received including housing, their satisfaction with these services compared to previous experiences, other services available in the community that are similar, and policy or program recommendations for other projects focused on reducing homelessness. Table 8 provides a brief description of each section in the discussion and the EQs they address.

Table 8. Client Focus Group Discussion Guide: Sections and Evaluation Questions

Descriptive Client Information (EQ7)

Collects basic information on the clients participating in the group and includes questions on how long a client has been in the project, experience with other projects, and past incidents of homelessness and experiences with treatment/housing services.

Services Clients Receive through the Project (EQ7, EQ9, EQ11, EQ12)

Asks clients how they became involved in the project, what type of services they have received, various requirements/restrictions to participate in the project, services they feel they need but have not received, whether they have had to pay for services and whether they received assistance in accessing benefits.

Housing for Clients (EQ11, EQ12, EQ16)

Asks clients about the assistance they have received in obtaining housing, the process they had to go through, the type of housing they currently have and their housing plans once they finish the grantee project.

Client Satisfaction and Recommendations for Change (EQ10, EQ20)

Asks clients what they liked or did not like about the project, what they would change about the project, how satisfied they are with the housing services, and their opinion about the project staff. The section concludes by asking what outcomes the clients have experienced and whether they have any recommendations to help address barriers or challenges they have experienced.

Financial Staff Cost Questionnaire. The questionnaire is designed to collect resource use and economic information from the grantee and includes costs incurred during the fiscal year in which the site visit falls, labor hours in the past month, and partner funding for services provided to clients in the grantee project. Questions ask for estimates of staff labor in a typical week; while estimates may be less precise than time diaries or data extraction from a staff time reporting system, the method allows for reliable labor data to be uniformly collected across all grantees with minimal staff burden. The questionnaire will be completed by the project director or other designated staff with the assistance of a financial officer, as needed. To help reduce grantee burden, sections will be pre-populated using information collected during a document review process and updated as needed by the site visitor during the cost interview. The costs and labor allocation data are used to calculate project and service level costs. Combined with partner funding information, it provides a fuller picture of the full cost of implementing grant projects which helps inform sustainability and future funding decisions made by SAMHSA. Table 9 provides a brief description of each section in the discussion and the EQs they address.

Table 9. Financial Staff Cost Questionnaire: Sections and Evaluation Questions

Ongoing Costs (EQ7, EQ14)

Collects economic data at a grantee project level, focusing on costs incurred in service delivery. Questions ask about the costs associated with labor (e.g., employees, contracted), building space costs, depreciation, supplies and materials, any miscellaneous costs, and administrative overhead.

Labor Allocation (EQ7)

Collects economic data at the service level, which allows for the costs of specific services to be calculated. Estimates are provided for the number of hours per week grantee staff typically work on each unique service category defined in the “Labor Allocation”. For each staff type (e.g., Counselors, Case Managers), the average wage is also collected.

Partner Services (EQ14)

Collects basic funding information on the services provided by the grantee’s partners and other service system stakeholders by asking the grantee to identify the funding sources for services not funded with CABHI funds.

3. Evidence-Based Practice (EBP) Self-Assessment

The EBP Self-Assessment (Attachment 3) is a web-based survey completed by the grantees with the purpose of collecting information on the services implemented in grantee projects that have a demonstrable evidence base and are appropriate for the target population. All grant projects are required to implement at least one EBP, though many choose to implement more. Data from the self-assessment will provide a description of the interventions received by project clients providing the ability to aggregate by practice, assess the relationship of specific services to project effects, and inform the interpretation of results. This data collection tool supports the process and outcome evaluations by identifying which EBPs are implemented and how grantees achieve EBP implementation.

The EBP Self-Assessment tool is divided into two parts. Part 1 collects qualitative information on general implementation of the projects’ primary EBPs. This part will be administered to all of the CABHI projects. Part 2 collects implementation data on a selected group of EBPs and will be administered only to projects using the selected EBPs.

The EBP Self-Assessment Part 1 was developed from three instruments specifically designed and used to examine factors that influence EBP implementation: the General Organizational Index ([GOI]; Bond et al., 2009); the State Health Authority Yardstick ([SHAY]; Finnerty et al., 2009); and the Installation Stage Assessment ([ISA]; Fixsen & Blase, 2010). Table 10 provides an overview of the type of information that is collected within each section of the survey; this data addresses EQ5–8, EQ12, and EQ19.

Table 10. EBP Self-Assessment Part 1 Sections

Readiness to Implement & Leadership

Asks about EBP selection, experience implementing the EBP, priority placed on the EBP, formal implementation plans, and support from the implementing agency’s leadership.

Funding

Asks how the EBP has been funded and how it will continue to be funded if the EBP is implemented after the project ends.

Hiring, Training & Supervision (8 items)

Asks about staff recruitment and expert consultation, training, and supervision/oversight received to support implementation of the EBP.

Fidelity/Outcomes Monitoring & Performance Improvement

Asks about screening, client reach, and fidelity assessment, including frequency and use of fidelity performance data and the degree to which the EBP has been implemented to fidelity.

Overall Barriers/Facilitators

Asks which factors have served as barriers or facilitators to implementation of the EBP.

The EBP Self-Assessment Part 2 focuses on a selected group of EBPs that includes Assertive Community Treatment (ACT), Integrated Dual Disorders Treatment (IDDT), Illness Management and Recovery (IMR), Supported Employment (SE), Critical Time Intervention (CTI), and SSI/SSDI Outreach, Access, and Recovery (SOAR). These EBPs were selected because they have a SAMHSA EBP Fidelity Toolkit, are well-defined and measurable project models (instead of practice strategies), and historically are commonly used by SAMHSA’s homeless services grantee projects. Limiting the scope helps balance respondent burden with the need to collect information on EBPs that is comparable across a number of grantee projects. Information provided during the PD Phone Interview/Web Survey will be used to identify the grantees who are implementing the selected group of EBPs and this subset of grantees will be invited to complete Part 2 of the EBP Self-Assessment.

Each EBP has its own module with questions designed to collect both quantitative and qualitative data. These modules are based on standardized fidelity checklists that have tested scoring systems to be able to determine the degree of implementation fidelity for each EBP assessed (McHugo et al., 2007). The selected practices have a SAMHSA EBP Fidelity Tool Kit (ACT: DHHS Publication No. SMA-08-4344; IDDT: DHHS Publication No. SMA-08-4366; IMR: DHHS Publication No. SMA-09-4462; and SE: DHHS Publication No. SMA-08-4364) or a well-tested fidelity scale (CTI) documented in the National Registry of Evidence-based Programs and Practices from which the Part 2 self-assessment questions were developed, retaining the key fidelity dimensions/components of each EBP. At the end of each module is a question about dimensions of the EBP that the grantee may have found difficult to implement and two questions about specific modifications that may have been made to the EBP by the local grantee project. This information will be used to produce recommendations for SAMHSA and the field regarding needed future research and practice improvements. Table 11 details the type of information collected within each Part 2 module; this data addresses EQ7–8, EQ12–13, and EQ19.

Table 11. EBP Self-Assessment Part 2 Modules

Assertive Community Treatment (ACT)/Intensive Case Management (ICM) Module

Asks about structure and composition, organizational boundaries, the nature of services, and adaptations and challenges.

Integrated Dual Disorders Treatment (IDDT) Module

Asks about staffing, service delivery, and adaptions and challenges.

Illness Management and Recovery (IMR) Module

Asks about staffing, programming, assignment and services, and adaptations and challenges.

Supported Employment (SE) Module

Asks about staffing, organization, services, and adaptations and challenges.

Critical Time Intervention (CTI) Module

Asks about program structure and staffing, timing of engagement, assessment and treatment planning, outreach, services, and adaptations and challenges.

SSI/SSDI Outreach, Access, and Recovery (SOAR) Module

Asks about staffing, training, client documentation and the application process, data tracking, and adaptations and challenges.

4. Permanent Supportive Housing (PSH) Self-Assessment

The PSH Self-Assessment (Attachment 4) is a web-based survey completed by the grantee to understand the extent to which grantees were implementing key dimensions of PSH, capture the variability of the PSH model among the grant projects, and provide valuable contextual information at the service level through which to help interpret participant-level outcome data. All grant projects are expected to connect clients to permanent housing and provide supportive services.

The PSH Self-Assessment instrument was developed using the SAMHSA PSH toolkit (DHHS Publication No. SMA-10-4509) as a primary resource and the Pathways Housing First Fidelity Scale-ACT version (Tsemberis, 2010) and the Full Service Partnership (FSP) Practices Scale (Gilmer, Stefancic, Ettner, Manning, & Tsemberis, 2010). These resources were used to construct a comprehensive self-assessment instrument for PSH, which includes seven dimensions from the SAMHSA PSH toolkit and a service philosophy module and ACT team module from the Pathways Housing First Fidelity Scale and FSP Practices Scale. Information is collected on the following dimensions: choice of housing, separation of housing and services; decent, safe, and affordable housing; housing integration; tenancy rights; access to housing; flexible, voluntary services; service philosophy; and team-based behavioral health. This data will allow the evaluation to address EQ5–7, EQ11–12, EQ14, and EQ19

5. Changes

The PD Phone Interview/Web Survey was originally developed to be conducted as a telephone interview. However, after review of the instrument it became clear that some sections were better suited for Web administration and did not need to be walked through by an interviewer and the PD. For instance, PDs are very familiar with the services that are provided by their projects and can more easily review a checklist of services and select those that are provided to project clients, rather than having an interviewer ask if each service was provided. The PD Interview has therefore been separated into two components, a telephone interview and a Web-based survey. The content of the collected information has not changed, only the mode of collection. No other changes have been made to the instruments.

3. Use of Information Technology

Web-based Surveys

The EBP Self-Assessment, PSH Self-Assessment, and the Web component of the PD Phone Interview/Web Survey are self-administered, Web-based surveys to be completed through an online data collection system. Before any Web-based data collection begins, SAMHSA will secure a system authorization to operate, which includes a security assessment and privacy impact assessment.

Using a web instrument allows for automated data checks as well as for skip procedures which will reduce the burden among respondents and possibility of data entry error, thereby increasing the efficiency of data entry and improving data quality. The automated data checks will help respondents give valid responses (e.g. restricting the range of responses to 0 to 365 when asked about “number of days events occurred in the past year”), and also ensure that responses follow the expected format (e.g. numbers or dates where those are expected). Responses will generate skip patterns for later questions in the instrument, where the respondents only complete relevant sets of questions and do not see others (e.g., in the PD Phone Interview/Web Survey, respondents are not administered follow-up questions about services they do not provide). Using a web-based system also provides the capability to send automatic email reminders to grantees if and when surveys have not been completed.

Telephone and In-person Interviews

The telephone component of the PD Phone Interview/Web Survey and the Site Visit Guides are telephone and in-person interviews; respondents will be read questions by an evaluation interviewer while a note taker records each response. With respondent consent, interviews will be recorded as a back-up to the note taker. After the interview, the interviewer and note taker will review the completed interview notes for accuracy; any areas of discrepancy will be validated with the recording. Once the interview responses are considered final, the recording will be deleted. Until they are deleted, the recordings will be kept by the note taker on a secure, password protected computer.

3. Effort to Identify Duplication

This evaluation is collecting information unique to the CABHI program that is otherwise not available for these grantees because of the scale and breadth of the grantee projects’ implementation: nationwide, across a spectrum of provider settings, and across a broad cross-section of populations. Where possible, the evaluation team will make use of secondary data from other sources like HUD’s homeless management information system. However, the information collected through the data collection tools included in this request to OMB are not duplicative of any ongoing data collection efforts.

5. Involvement of Small Entities

Participation in this evaluation will not impose a significant impact on small entities. CABHI grantees may include state agencies, tribal organizations and other jurisdictions, local governments, and community service providers. Some of the community service providers may be small entities; however, the CABHI data collection instruments are designed to include only the most pertinent information needed to be able to carry out the evaluation effectively, and their impact will not be significant.

6. Consequences If Information Collected Less Frequently

The EBP and PSH Self-Assessments will be conducted once during the evaluation.

The multiple data collection points for the PD Phone Interview/Web Survey and Site Visit Guides are necessary to measure the grant projects’ start-up activities early (i.e., while they are happening or soon after) and full project implementation later. It is important to adequately track and evaluate the grantee projects’ progress and change over time. Over the course of the grants, the CABHI projects may modify aspects of their design for a variety of reasons (e.g., a key partner leaves the project, the project is able to add a primary service, etc.). It is important to accurately capture these modifications, which necessitates data collection at multiple time points. The evaluation has made every effort to ensure that data are collected only when necessary and that extraneous collection will not be conducted.

7. Consistency with the Guidelines in 5 CFR 1320.5(d)(2)

This information collection fully complies with the guidelines in 5 CFR 1320.5(d)(2).

8. Consultation Outside the Agency

The notice required by 5 CFR1320.8(d) was published in the Federal Register on November 23, 2016 (81 FR 84601). No coments were received.

The previous evaluation of the 2009-2012 GBHI, SSH, and CABHI grant cohorts was advised by an expert panel that reviewed each of the data collection instruments. The experts’ relevant areas of expertise include homelessness and housing, behavioral health, program implementation, EBPs (including PSH), and cultural competence; some are also previous recipients of GBHI, SSH, or CABHI grants. The list of experts is provided in Table 12.

Table 12. Expert Panel Members

Expert	Affiliation	Contact Information
Margarita Alegría, Ph.D.	Harvard Medical School Director and Professor of Psychiatry Center for Multicultural Mental Health Research, Cambridge Health Alliance 120 Beacon Street, 4th Floor Somerville, MA 02143	Phone: (617) 503-8447 E-mail: malegria@charesearch.org
Gary Bond, Ph.D.	Westat, Senior Research Associate, Heath Studies Sector 1600 Research Boulevard Rockville, MD 20850	Phone: (603) 676-7577 E-mail: GaryBond@Westat.com
Brian Dates, M.A.*	Southwest Counseling Solutions Director of Evaluation and Research 1700 Waterman Detroit, MI 48209	Phone: (313) 841-7442 E-mail: bdates@swsol.org
Louis Kurtz, M. Ed.	Division of Behavioral Health Kentucky Department for Behavioral Health, Development and Intellectual Disabilities Acting Division Director 100 Fair Oaks Lane 4E-D Frankfort, KY 40621	Phone: (502) 564-4456 E-mail: louis.kurtz@ky.gov
William McAllister, Ph.D.	Institute for Social and Economic Research and Policy Senior Research Fellow Center for Homelessness Prevention Studies, Associate Director Columbia University 420 West 118th St, MC 3355 New York, NY 10027	Phone: (212) 854-5781 E-mail: wm134@columbia.edu
Stephen Metraux, Ph.D.*	Department of Health Policy & Public Health, Associate Professor University of the Sciences in Philadelphia 600 South 43rd Street Philadelphia, PA 19104-4495	Phone: (215) 596-7612 E-mail: s.metraux@usp.edu
Roger H. Peters, Ph.D.	Department of Mental Health Law and Policy, Professor Louis de la Parte Florida Mental Health Institute, University of South Florida 13301 North Bruce B. Downs Boulevard Tampa, FL 33612-3807	Phone: (813) 974-9299 E-mail: peters@fmhi.usf.edu
Alan Rabideau	Consumer and Family Member Consultant 112 Kincheloe Drive Kincheloe, MI 49788	Phone: (906) 495-7158 E-mail: jawenodee_inini@yahoo.com
Michael Rowe, Ph.D.	Program for Recovery & Community Health, Associate Clinical Professor of Psychiatry & Co-Director Yale University, School of Medicine 319 Peck Street, Building 1 New Haven, CT 06513	Phone: (203) 764-8690 E-mail: michael.rowe@yale.edu
David Smelson, Psy.D.*	University of Massachusetts Medical Center, Professor of Psychiatry Translational Research Edith Nourse Rogers Memorial Veterans Hospital, Director 55 Lake Avenue North Worcester, MA 01655	Phone: (508) 856-3768 x 15122 E-mail: david.smelson@umassmed.edu
Sally J. Stevens, Ph.D.*	Southwest Institute for Research on Women Executive Director Department of Gender and Women’s Studies, Professor College of Social and Behavioral Sciences, University of Arizona 925 North Tyndall Avenue Tucson, AZ 85721	Phone: (520) 626-9558 E-mail: sstevens@email.arizona.edu
Sam Tsemberis, Ph.D.*	Pathways to Housing, Inc. Founder and CEO Columbia University Medical Center, Clinical Assistant Professor of Psychiatry 55 West 125th Street, 10th Floor New York, NY 10027	Phone: (212) 289-0000 x1101 E-mail: stsemberis@pathwaystohousing.org
*Also prior GBHI, SSH, and CABHI grantees

9. Payment to Respondents

No incentives or gifts will be given to respondents.

10. Assurance of Confidentiality

Concern for privacy and protection of respondents’ rights will play a central part in the implementation of all evaluation components. CABHI evaluation team members have extensive experience protecting and maintaining the privacy of respondent data. All data will be securely stored on a protected server. As stated previously, a privacy impact assessment will be completed before data collection begins.

Sensitive respondent information, such as birthdates and social security numbers, will not be collected. The focus of this collection is on programmatic data (i.e., information about the organizations and implemented interventions) at the grantee project levels along with aggregated, non-identifying respondent-level data (e.g., estimated percent of individuals receiving specific types of intervention approaches). All information collected for the CABHI evaluation will be reported in aggregate; individual responses will not be identified. Procedures for data collection consent are described below.

Site Visits and PD Telephone Interview

Project-level Site Visit Interviews. Prior to beginning a site visit interview, the respondents will be read, asked to sign, and provided a copy of the consent form. The consent form informs respondents of their rights, including the right to not answer any question; respondents must provide written consent to participate in the site visit discussion before the interview begins. If a respondent does not provide written consent to participate then the interview will not take place.

Site Visit Client Focus Group. Prior to beginning the focus group, respondents will be read, asked to sign, and provided a copy of a consent form. They will be instructed to provide only their first name and first initial of their last name on the consent form, in an additional effort to protect their privacy. The consent form informs respondents of their rights, including the right to not answer any question or participate in any discussion; respondents must provide written consent to participate before the focus group discussion begins. If a respondent does not provide written consent to participate then that individual will be excused from the group.

PD Telephone Interview. Prior to the telephone component of the PD Interview, the respondent will be sent a consent form for their review. Before the interview begins, the interviewer will read a standardized script for consent that informs them of their rights, including the right to not answer any question; respondents must provide verbal consent to participate in the interview before it begins. If a respondent does not provide verbal consent to participate then the interview will not take place.

Respondents will also be asked for their consent to record the site visit discussions and PD telephone interview. Recordings will be used to ensure that information is correctly captured across the multiple site visit interviews, information has been consistently captured across interviewers, and to correct and clarify written notes as needed and as part of data quality assurance procedures. Recordings will only be accessible to the contractor and will be stored on password-protected secure servers and destroyed once de-identified notes are completed.

PD Web Survey, EBP Self-Assessment, and PSH Self-Assessment

Prior to beginning a Web-based survey, the respondent will review a brief statement of consent that informs them of their rights, including the right to not answer any question, and that completing the self-assessment is voluntary. The respondent will not be able to proceed with the survey unless they indicate that they agree to participate.

The CABHI systems development team takes responsibility for ensuring that the Web and data system is properly maintained and monitored. Server staff will follow standard procedures for applying security patches and conducting routine maintenance for system updates. Data will be stored on a password-protected server, and access to data in the system will be handled by a hierarchy of user roles, with each role conferring only the minimum access to system data needed to perform the necessary functions of the role.

For all data collection activities, the contractor will use passwords to safeguard all project directories and analysis files containing completed survey data to ensure that there is no inadvertent disclosure of study data. Contractor staff has been trained on handling sensitive data and the importance of privacy. In addition, interviews and focus group guides and all consent forms will be reviewed by the contractor’s Institutional Review Board ([IRB]; Federal Wide Assurance #3331). In keeping with 45 CFR 46, Protection of Human Subjects, the CABHI procedures for data collection, consent, and data maintenance are formulated to protect respondents’ rights and the privacy of information collected. Strict procedures will be followed for protecting the privacy of respondents’ information and for obtaining their informed consent.

11. Questions of a Sensitive Nature

Most of the information reported by respondents during the PD Phone Interview/Web Survey, Site Visits and EBP and PSH Self-Assessments is not sensitive personal information as interviews focus only on programmatic details of the grant projects.

The data collection tools will be reviewed by the contractor’s IRB (FWA #3331) and data collection for each tool will not begin until it is approved or exempted.

Site Visit Guides do include a client focus group during which sensitive topics may be discussed. Informed consent will be obtained for participation in the client focus group during which the participants will be informed that all questions asked during the focus group are voluntary. Participants will be assured that they may stop participation in the focus group at any time without penalty from the grantee project. If a participant is caused any distress, the focus group facilitator will connect them, with permission from the participant, with someone from the grant project who they can speak with. Participants are also asked to only provide their first name in an effort to maintain their privacy and not to disclose any information that is shared during the focus group.

12. Estimates of Annualized Hour Burden

The estimated number of respondents, responses per respondent, and burdens hours are described for each CABHI data collection activity below. There are no direct costs to respondents other than their time to complete the instrument. Table 13 provides annual respondent, number of responses, burden, and cost details for each instrument. Respondent costs are calculated as total burden hours × the average hourly wage for associated job categories (as reported in the 2015 Occupational Employment Statistics by the Bureau of Labor Statistics). The test procedures used to determine burden hours are provided in Section B.4.

PD Phone Interview/Web Survey

A total of 39 respondents are expected to complete the PD Phone Interview/Web Survey; this includes a respondent from all of the CABHI grantees (n=30) and the State sub-recipients (n=9). Each interview is expected to take 2.1 hours; this includes time to complete the telephone component (1 hour) and the Web survey (1.1 hours). Respondents will be asked to complete the PD Phone Interview/Web Survey one time during Year 1 and one time during Year 3 of the evaluation.

Site Visit Guides

All 30 of the CABHI grantees are expected to participate in the site visits, which will be conducted once in Year 2 and once in Year 3 of the evaluation. As previously described, participants will include grantee project directors and project management staff, financial staff, evaluators, behavioral health treatment staff, support services staff, case managers, housing providers, primary partners and other key stakeholders, and project participants. The number of respondents per discussion will vary across projects. Based on the contractor’s site visit experience during the previous evaluation, the following average number of respondents will participate in each discussion: 10 respondents will in participate in the Opening Session/Project Director Interview (2.5 hours), 18 respondents will participate in the Case Manager, Treatment, Housing Staff/Provider Interview (2 hours), 9 respondents will participate in the Stakeholder Interview (1.5 hours), 2 respondents will participate in the Evaluator Interview (1 hour), 15 respondents will participate in the Client Focus Group (1.5 hours), and 2 respondents will participate in the Cost Interview (2 hours).

EBP Self-Assessment, Parts 1 & 2

Thirty-six respondents (9 State sub-recipients, 12 Local Governments, 15 Communities) are expected to complete the EBP Self-Assessment – Part 1. The Part 1 self-assessment may be completed up to 3 times (based on the number of primary EBPs being implemented by the projects), for a maximum of 108 responses. The average time to complete the Part 1 self-assessment is 35 minutes.

Thirty-six respondents (9 State sub-recipients, 12 Local Governments, 15 Communities) are expected to complete the EBP Self-Assessment – Part 2. The Part 2 self-assessment may be completed up to 3 times (based on the number of Part 2 EBPs being implemented by the projects), for a maximum of 108 responses. The average time to complete the Part 2 self-assessment is 15 minutes.

The EBP Self-Assessment Part 1 & 2 will be administered in Year 2 of the evaluation.

PSH Self-Assessment

Thirty-six respondents (9 State sub-recipients, 12 Local Governments, 15 Communities) are expected to complete the PSH Self-Assessment one time; the self-assessment will be administered in Year 2 of the evaluation. The average time to complete the self-assessment is 40 minutes.

Table 13. Annualized Data Collection Burden

Instrument/Activity	Number of Respondents	Responses per Respondent	Total Number of Responses	Hours per Response	Total Burden Hours	Average Hourly Wage	Total Respondent Costa
PD Phone Interview/Web Survey	39	1	39	2.1	82	$33.38	$2,734
Site Visit Guides:
Opening Session/Project Director Interview	300b	1	300	2.5	750	$33.38	$25,035
Case Manager, Treatment, Housing Staff/Provider Interview	540c	1	540	2	1,080	$18.52	$20,002
Stakeholder Interview	270d	1	270	1.5	405	$22.19	$8,987
Evaluator Interview	60e	1	60	1	60	$33.38	$2,003
Client Focus Group	450f	1	450	1.5	675	$7.25	$4,894
Cost Interview	60g	1	60	2	120	$33.38	$4,006
Subtotal	1,650		1,680		3,090		$64,927
EBP Self-Assessment Part 1	36	3	108	0.58	63	$33.38	$2,091
EBP Self-Assessment Part 2	36	3	108	0.25	27	$33.38	$901
PSH Self-Assessment	36	1	36	0.67	24	$33.38	$805
TOTAL	1,650h		1,971		3,286		$71,458

^aTotal respondent cost is calculated as hourly wage × hours per response × total number of responses.

^b10 respondents x 30 site visits = 300 respondents

^c18 respondents x 30 site visits = 540 respondents

^d9 respondents x 30 site visits = 270 respondents

^e2 respondents x 30 site visits = 60 respondents

^f15 respondents x 30 site visits = 450 respondents

^g2 respondents x 30 site visits = 60 respondents

^hEstimated number of total unique respondents; some respondents, such as project directors, will overlap across the data collection activities.

13. Estimates of Annualized Cost Burden to Respondents

There are no respondent costs for capital or start-up or for operation or maintenance.

14. Estimates of Annualized Cost to the Government

The annualized cost to the government is approximately $1,450,552. The estimated 4-year total cost to the government for the data collection is $5,802,208. This includes approximately $1,438,574 per year (or $5,754,295 total) for developing the instruments; programming and maintaining the online data collection system; providing data collection training to grantees and learning laboratories; contractor labor for managing data collection; processing, cleaning, and housing data; and analyzing and reporting data. Approximately $11,978 per year (or $47,912 total) represents SAMHSA costs to manage/administer the data collection and analysis for 10% of one employee (GS-14-4, $119,776 annual salary).

15. Changes in Burden

Currently, there are 2,835 burden hours in the OMB inventory. SAMHSA is now requesting 3,286 hours. The program change of an increase of 451 hours is mostly due to an increase in the number of site visits to be conducted annually. In the previous evaluation, 25 site visits were conducted, whereas for the CABHI evaluation, 30 site visits will be conducted. Additionally, for some of the site visit interviews (e.g., the Case Management, Treatment, and Housing Provider Interview), the 9 State sub-recipients will be treated as their own site given that they closely mirror the function of a CABHI Local Government or Community grantee, further increasing the estimated number of respondents for those interviews. These increases in burden are offset by the reduction in the number of respondents for the PD Phone Interview/Web Survey, EBP Self-Assessment, and the PSH Self-Assessment. Table 14 provides an overview of the current OMB inventory of respondents and burden hours for each instrument, the requested change in respondents and burden hours for each instrument, and an explanation of the change.

Table 14. Changes in Burden

Instrument	Previous Total of:		Requested Total of:
	Respondents	Hours	Respondents	Hours
Project Director Phone Interview/Web Survey	158	331.8	39	82
Change in Burden: Decrease in number of respondents.
Site Visit: Opening Session/PD Interview	250	675	300	750
Change in Burden: Increase in number of respondents. Estimated time was modified from 2.7 to 2.5 hours based on experience in the previous evaluation.
Site Visit: Case Manager/Treatment/ Housing Provider Staff Interview	375	750	540	1080
Change in Burden: Increase in number of respondents.
Site Visit: Stakeholder Interview	175	262.5	270	405
Change in Burden: Increase in number of respondents.
Site Visit: Evaluator Interview	60	60	60	60
Change in Burden: No change (increase in number of site visits is offset by the decrease in number of respondents per interview, based on experience in the previous evaluation).
Site Visit: Client Focus Group	300	450	450	675
Change in Burden: Increase in number of respondents.
Site Visit: Cost Interview	60	120	60	120
Change in Burden: No change (increase in number of site visits is offset by the decrease in number of respondents per interview, based on experience in the previous evaluation).
EBP Self-Assessment Part 1	127	73.7	36	63
Change in Burden: Decrease in number of respondents. Number of responses per respondent increased from 1 to 3.
EBP Self-Assessment Part 2	87	43.5	36	27
Change in Burden: Decrease in number of respondents. Estimated hours per response was modified from .5 to .25. The average time to complete one Part 2 survey is .25 hours. The previous estimate of .5 was a cumulative total assuming that some sites would complete the survey for more than one EBP, which was based on the number of Part 2 EBPs each project said they were implementing. That information is not yet available so instead, SAMHSA is decreasing the hours per response (from .5 to .25) and increasing the number of possible responses per respondent (from 1 to 3).
PSH Self-Assessment	100	67	36	24
Change in Burden: Decrease in number of respondents.

16. Time Schedule, Publications, and Analysis Plan

Time Schedule

Table 15 outlines the time points for the collection of information during the CABHI grant project years.

Table 15. Time Schedule for Data Collection

Activity	Year 1 (10/2016–9/2017)	Year 2 (10/2017–9/2018)	Year 3 (10/2018–9/2019)
Project Director Interview	X		X
Site Visits		X	X
EBP Self-Assessment Part 1		X
EBP Self-Assessment Part 2		X
PSH Self-Assessment		X

Publications

The CABHI evaluation will help SAMHSA reach its diverse stakeholders through targeted products and innovative dissemination venues. The evaluation’s objective for all reports and dissemination products is to provide user-friendly documents and presentations that help SAMHSA successfully disseminate and explain the findings. The dissemination plan includes products in a variety of formats for a variety of target audiences. Audiences for these reports will include SAMHSA Centers, the evaluation’s SAMHSA Contracting Officer’s Representatives (CORs), CABHI grantees and participants, and the broader mental health, substance abuse, and housing and homelessness fields (e.g., academia, researchers, policy-makers, providers). The CABHI evaluation recognizes that different audiences are best reached by different types of report formats. For example, reports created for SAMHSA Centers and the CORs will require in-depth information, such as substantive background and discussion sections, to supplement the analytic approach. Reports created for CABHI grantees will be concise handouts with helpful and easy-to-read graphics on performance data rather than lengthy text. The CABHI evaluation will develop an assortment of dissemination products, including short and long analytic reports, annual evaluation reports, research and policy briefs, ad hoc analytic reports, journal articles, best practice summaries, and conference or other presentations.

Analysis

The CABHI evaluation will use a combination of qualitative and quantitative analysis to assess project structure and process. Assessing structure and process is a key element of evaluating any behavioral healthcare program or system. Structure represents resources, institutions and their interrelationships, and legal and policy consideration in a system and can apply to individual practitioners, groups of practitioners, organizations and agencies and programs. Process represents the development of a project as well as the services that are provided to the client. Economic data is also collected to specifically document the resources needed to implement a project’s process and services, which are used in cost benefit and effectiveness calculations. Combined, structure and process data provide the context to interpret client- and system-level outcome data.

Qualitative Data. The PD Phone Interview/Web Survey and the EBP and PSH Self-Assessments will provide valuable qualitative data, but the majority will come from the Site Visit Guides. Qualitative analysis will focus on describing the characteristics of the grantee organization and their partnerships, the system within which the project is embedded, relationships with stakeholders, characteristics of the target population, project planning, services provided including implementation of EBPs, the types and models of housing integrated into the project, and project outcomes including sustainability. Descriptive analyses of these measures provide information on implementation of the CABHI projects. Qualitative analysis helps identify common trends and themes across grantees and will especially focus on identifying barriers and facilitators to implementing project activities and the solutions grantees found to common challenges. Of key importance is the impact of CABHI projects on the system in which they operate.

Qualitative narrative data (from Site Visit Guide summary notes) will be subject to content analysis. Discussion guides will also include structured questions with close-ended responses that can be quantified and analyzed accordingly; for narrative information that does not lend itself easily to quantitative coding, data will be transcribed and uploaded into ATLAS.ti, a software package used for coding qualitative data. We will use a grounded theory approach to guide our coding process. First, all lines of text will be subject to open-coding, where codes are expressed in the present progressive tense. Second, open codes will be reduced into a set of axial codes. Finally, theoretical codes will be used to structure the presentation of the qualitative findings. Such analyses will reveal common themes across data. We expect to see the following themes emerge from the qualitative analyses: changes to grantee plans, the types of barriers, challenges and responses encountered during implementation; facilitators to implementation and operation; and collaboration among grantees, their partners, and other community agencies and organizations. If supported by the data, common themes, including barriers and facilitators, will be explored more deeply to identify possible mediators and moderators. Results will be presented in narrative text. Commonalities identified across grantees may be incorporated into analyses, testing the association between structural and process variables and client and project outcomes through Hierarchical Linear Modeling (HLM), which is discussed below.

Quantitative Data. The PD Phone Interview/Web Survey and the EBP and PSH Self-Assessments primarily collect quantitative data which is used for both descriptive statistics and statistical analysis. Descriptive analysis and tables will report key statistics, such as means, sample size, standard errors, and t- and χ²- test results, where appropriate. The basic approach will pool data across grantees and programs. When appropriate, findings will be presented separately for key project characteristics (e.g., type of housing integrated into the project; population targeted; type, number and combination of EBPs offered). Statistical analysis, especially HLM, can identify associations between measures of structure and process and individual client access to core services, individual client outcomes, client perceptions of care, and project sustainability. Importantly, within this framework we can test the extent to which programmatic and contextual characteristics moderate or mediate changes in client outcomes. Change in client outcomes will be measured through the longitudinal CSAT GPRA baseline and follow-up data, and is central to the outcome component.

Statistical analysis, such as HLM, will be used to estimate the association between grantee characteristics collected with the PD Phone Interview/Web Survey, Site Visit Guides (including cost data), and the EBP and PSH Self-Assessments and mean change in client-level outcomes between baseline and follow-up. HLM is appropriate for these analyses because this modeling approach accounts for the clustering of clients within grantees and allows the model to be adjusted for client characteristics and contextual factors. Adjusted mean changes in client outcomes will provide easy-to-understand estimates of possible project effects. Although these estimates are not intended to be causally interpreted, we do intend to explore how project approaches and characteristics alter service delivery and outcomes. In this way, variation among the grantees will be used in analyzing ‘key ingredients’ of models for achieving different outcomes. Our analyses will not be as definitive as true comparative effectiveness analyses that are based on randomized control trials, and we will temper our conclusions accordingly. In addition, to improve the accuracy of our inferences we may use quasi-experimental approaches such as propensity score matching, instrumental variable methods and regression discontinuity designs. As appropriate, subgroup analyses will be conducted in which the data will be stratified (e.g. by client type) to assess whether outcomes differ among subgroups.

Economic Data. The Site Visit Guides include a cost questionnaire to collect data to estimate the costs of the CABHI grant projects at the client, grantee, and system levels. For this evaluation, analysis focuses on estimating the cost of CABHI so that it can be directly compared with other models of treatment delivery and compared with the cost bands specified by the National Outcome Measures performance measurement initiative. To do this, the costs of implementing CABHI services are separated from the costs of developing and revising the CABHI protocols and from the costs of administering the CABHI grant project. The cost analysis will provide both dollar estimates and estimates of the amount of resources used so that the results can be applied to different circumstances and prices. The evaluation will identify the key drivers of cost, allowing decision makers to identify critical cost components of the intervention. The detailed economic study will also facilitate sensitivity analysis, which assesses the degree to which conclusions are robust to changes in key assumptions.

17. Display of Expiration Date

OMB approval expiration dates will be displayed.

18. Exceptions to Certification for Statement

There are no exceptions to the certification statement. The certifications are included in this submission.

23