0930-0339_OMB_SS_PartB 1.24.17

Evaluation OF the Substance Abuse and

Mental Health Services administration’s (samhsa’S) cooperative agreements to benefit homeless individuals (CABHI) program

Supporting Statement

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

The following sections provide a description of the respondents and sampling methods for each data collection tool included in this request to OMB for the CABHI Evaluation. As a reminder, the 2016 CABHI grantees include 3 States, 12 Local Governments, and 15 Communities. The States also include 9 State sub-recipients whose role is to provide services, including permanent supportive housing (PSH) and other evidence-based practices (EBPs), to CABHI clients.

Project Director (PD) Interview and Site Visits

Using a census approach, the target universe for the PD Phone Interview/Web Survey is all of the 2016 CABHI grant project directors (n=30) and the State sub-recipient coordinators (n=9). All 30 grant projects are also expected to participate in the Site Visits. Respondents vary across the site visit interviews, as follows:

• Opening Session/Project Director Interview: project directors, project coordinators, project managers, State sub-recipient coordinators, and other administrative-level staff as selected by the grant project director.

• Case Management, Treatment, and Housing Staff Interview: staff providing services to clients, such as outreach specialists, case managers, behavioral health counselors, housing specialists, benefits specialists, peer support specialists, etc.

• Stakeholder Interview: project partner staff and other key project stakeholders.

• Client Focus Group: project participants.

• Evaluator Interview: project evaluators.

• Cost Interview: project directors and staff who are knowledgeable about project finances, such as finance directors, financial managers, etc.

EBP Self-Assessment

The EBP Self-Assessment – Part 1 target universe is all 2016 CABHI grant projects that are implementing at least one EBP. As the provision of CABHI services through an EBP is a grant requirement, it is expected that all State sub-recipients (n=9), Local Governments (n=12), and Communities (n=15) will be recruited to complete the Part 1 survey.

The target universe for the EBP Self-Assessment – Part 2 is all grant projects that are implementing at least one of the 6 EBPs selected for an in-depth assessment (i.e., Assertive Community Treatment, Integrated Dual Disorders Treatment, Illness Management and Recovery, Supported Employment, Critical Time Intervention, and SSI/SSDI Outreach, Access, and Recovery). The selected EBPs that are the focus of the Part 2 survey are commonly used with homeless populations and historically have high usage rates among SAMHSA’s homeless grant program grantees. Therefore, there is a high likelihood that each State sub-recipient (n=9), Local Government (n=12), and Community (n=15) will implement at least one of the selected EBPs and will be recruited to complete the Part 2 survey.

Respondents to both parts of the EBP Self-Assessment will include grant project directors, State sub-recipient coordinators, or a key staff member knowledgeable about the EBPs being implemented by the project, as designated by the grant project director.

PSH Self-Assessment

The target universe for the PSH Self-Assessment is all of the 2016 CABHI grant projects that are using the PSH model. All grant projects are expected to connect clients to permanent housing and provide supportive services; therefore, it is expected that all State sub-recipients (n=9), Local Governments (n=12), and Communities (n=15) will be recruited to complete the PSH survey. Respondents to the PSH Self-Assessment will include grant project directors, State sub-recipient coordinators, or a key staff member knowledgeable about the PSH model being implemented by the project, as designated by the grant project director.

Using a census approach for these data collection activities is necessary due to the heterogeneous nature of the CABHI projects. They encompass a wide variety of organizational types and structures that are implementing a range of interventions with various outcome goals. The variety between the projects makes it critical to the evaluation to capture the details of each project to be able to answer the evaluation questions and assess which project characteristics and mix of interventions are associated with better outcomes for the targeted populations. Additionally, SAMHSA will use these data to monitor project performance and provide annual updates on progress.

2. Information Collection Procedures

The CABHI Evaluation data collection will be centralized and coordinated through the data collection team leader and an evaluation training and technical assistance (ETTA) liaison team assigned to each grant project. This ETTA liaison team will be the primary point of contact to each grantee concerning questions about data collection, reporting, and feedback. Furthermore, these liaisons will help collect data. For example, all PD telephone interviews will be conducted by the ETTA liaison assigned to that individual grant project.

PD Telephone Interview

Grant project directors and State sub-recipient coordinators will be contacted by their ETTA liaison team via email with telephone follow-up to set up a mutually convenient time for the PD telephone interview during regularly scheduled business hours. Before conducting the telephone interviews, the evaluation team will review grant applications (submitted to SAMHSA by each grantee and given to the evaluation team by SAMHSA) and other documents (e.g., progress reports) that detail the characteristics of the project. The liaison staff will abstract information relevant to the evaluation (e.g., project structure, interventions) to become familiar with the grant project prior to conducting the interviews. This pre-abstracted information will be used to prepopulate some interview questions to reduce respondent burden. For instance, a list of the project partners and target population will be prepopulated and confirmed or updated with the respondent, as opposed to asking the respondent to generate the list while on the telephone.

Once the interview is scheduled, the ETTA liaison team will provide the respondent with an electronic version of the consent form and the instrument and a toll-free, passcode-protected telephone conference number. Before beginning the telephone interview, consent will be requested to record the interview to confirm, if needed, the accuracy of noted responses. The senior evaluator from the project’s ETTA liaison team will lead the respondent through the interview while a junior evaluator will record responses and take notes. After the interview, the interviewer and note taker will review the responses for accuracy. Any areas of discrepancy will be validated with the recording (if consented by the respondent); once the responses are considered final, the recording will be deleted. An electronic version of the telephone interview will be maintained on a password protected, secure server accessible only to the CABHI evaluation team. After the interview, the interviewer will send an email thanking the respondent for his or her participation.

A procedures manual has been developed for the administration of the telephone interviews and training will be provided to all interviewers and note takers to walk through interview procedures and questions.

Site Visit Guides

To schedule a site visit, the ETTA liaison team will send an email to the grant project director to arrange a pre-site visit call. Other project staff (e.g., the project evaluator) will be included in the call at the grant project director’s discretion. To provide sufficient information prior to the call, the ETTA liaison team will send the project director the Site Visit Guides topics list and potential respondents for each interview. During the call, a date for the site visit will be set and an agenda will be developed. If needed, site visit logistics will be finalized through additional calls and via email. Prior to the site visit, the ETTA liaison team will send the grant project director the following additional documents: 1) the finalized agenda and logistics information (e.g., site visitors’ hotel and contact information, contact information for the primary grantee site contact), and 2) a draft Client Flow Chart developed by the evaluation team from the review of the grant application to be reviewed and discussed during the Opening Session of the site visit.

Each participant will be provided with a written consent form prior to beginning their participation and procedures, including audio recording, will be verbally explained in addition to the participants’ own review of the consent. Each individual and group discussion will be digitally recorded (if the interviewee consents) and written notes will be taken on a laptop computer. The recordings will be summarized and the recording will be deleted once the summary notes are finalized. Summary notes will be entered into an Atlas.ti database and used for qualitative analysis and will also be used to write a brief summary of the site visit that will be submitted to SAMHSA.

A procedures manual has been developed for the site visits and training will be provided to all interviewers and note takers to ensure consistent use of standard interview procedures and questions.

Web-Based Surveys

The administration of the PD Interview web component, EBP Self-Assessment, and PSH Self-Assessment will adhere to the following procedures. Emails will be sent to the designated respondents to inform them that data collection has started. A link to enter the Web-based survey will be included in the email, as well as information on the background, purpose, types of questions, and length of the survey. When the survey is first accessed, the consent form will appear and the respondent will not be able proceed until the consent form is completed. If the respondent declines, a log off message will appear; if consent is provided, the survey will begin. When respondents complete the survey a “thank you” email will be sent automatically. Nonresponders will be sent predefined, automated weekly emails as needed, to remind them to complete their survey. If the survey is not completed within 3 weeks, the respondent will be contacted via telephone by the assigned liaison.

3. Methods to Maximize Response Rates

Grant projects are required to participate in all CABHI evaluation activities by the terms and conditions of the CABHI grant award. The CABHI evaluation team will employ a number of strategies to help ensure grant projects participate with a 100% response rate.

Grant projects will be assigned an ETTA liaison team that will serve as the primary point of contact for the evaluation. These teams will provide TA not only to assist the projects with their participation in the CABHI evaluation, but also with their own local evaluations, if desired. The CABHI ETTA process will include monthly check-in calls with the grant projects to answer any questions about evaluation activities, problem solve issues, or connect projects to additional resources. These communications will allow the CABHI evaluation team to build a strong relationship with the grant projects and provide an opportunity to notify the projects when a data collection activity is starting, explain the purpose, answer questions, and provide reminders to nonresponders. Additionally, the CABHI evaluation team will conduct webinars to introduce the evaluation and provided detailed overviews of each data collection activity as they are rolled out.

The CABHI evaluation team will work with the grant projects to schedule telephone interviews and site visits at the time most convenient for the respondents. Respondents will be provided the interview topics in advance of the call or the visit so they will be knowledgeable about the type of information to be collected. For all data collection activities, reminder emails will be sent to non-responders on a weekly basis following the initial email invitation to participate. After three email reminders, the ETTA liaison teams will follow up with the grant project by telephone. If needed—although the evaluation team does not anticipate that it will be necessary—the follow-up reminder emails will include the grant project’s government project officer.

The CABHI evaluation team will do all that is possible to support the grant projects in their participation in the evaluation, such as staggering timing of different elements of data collection so that grant projects do not feel as though they are being asked to do too much all at once, or providing an alternative way to complete the Web-based surveys.

4. Test of Procedures

As previously described, the PD Phone Interview/Web Survey, Site Visit Guides, EBP Self-Assessment, and PSH Self-Assessment were developed and successfully implemented for SAMHSA’s evaluation of the 2009-2012 GBHI, SSH, and CABHI cohorts. The evaluation team identified that parts of the PD Interview are better suited for self-administration through a Web survey. The PD instrument has been revised to reflect this change; no other procedural changes are needed. Additionally, the burden estimates from the previous evaluation were accurate, except for the Opening Session/Project Director Interview held during the site visits, which has been reduced slightly from 2.7 hours to 2.5 hours.

5. Statistical Consultants

As noted in Section A.8, SAMHSA consulted extensively throughout the previous evaluation with an expert panel, many of whom will continue to provide consultation on an expert panel for the CABHI Evaluation. In addition, the contractor team is comprised of several experts who will be directly involved in data collection and statistical analysis. Also, SAMHSA advisors will be consulted throughout the evaluation on various statistical aspects of the design, methodological issues, and data analysis. Table 1 provides details of these team members and advisors.

Table 1. Statistical Consultants for the CABHI Evaluation

Expert	Affiliation	Contact Information
CABHI Evaluation Staff
James Trudeau, Ph.D. Project Director	Principal Scientist Center for Justice, Safety, and Resilience RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709	Phone: 919-485-7751 E-mail: trudeau@rti.org
Arnie Aldridge, Ph.D. Deputy Project Director	Research Economist Behavioral Health Economics Program RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709	Phone: 919-990-8389 E-mail: aaldridge@rti.org
Antonio Morgan-Lopez, Ph.D. Data Analysis Task Lead	Senior Research Quantitative Psychologist Risk Behavior and Family Research RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709	Phone: 919-316-3436 E-mail: amorganlopez@rti.org
Aileen Rothbard, Sc.D. Statistician	Research Professor School of Social Policy and Practice & Department of Psychiatry 3535 Market Street, Room 3014 Philadelphia, PA 19104-2648	Phone: 215-573-7770 E-mail: rothbard@mail.med.upenn.edu
SAMHSA Advisors
Kirstin Painter, Ph.D. Contracting Officer’s Representative	Public Health Advisor CMHS, SAMHSA 5600 Fishers Lane, Room 14E89D Rockville, MD 20857	Phone: 240-276-1932 Email: Kirstin.Painter@samhsa.hhs.gov
Sarah Ndiangui, M.P.H. Alternate Contracting Officer’s Representative	Public Health Advisor CSAT, SAMHSA 5600 Fishers Lane, Room 13E85B Rockville, MD 20857	Phone: 240-276-2918 Email: Sarah.Ndiangui@samhsa.hhs.gov

REFERENCES

Bond, G. R., Drake, R. E., Rapp, C. A., McHugo, G. J., & Xie, H. (2009). Individualization and quality improvement: Two new scales to complement measurement of program fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 36: 349–357.

Finnerty, M.T., Rapp, C.A., Bond, G.R., Lynde, D.W., Ganju, V., Goldman, H.H. (2009). The State Health Authority Yardstick (SHAY). Community Mental Health Journal, 45(3):228-36.

Fixsen, D.L., and Blase, K.A. (2010). Stage-Based Measures of Implementation Components: Installation Stage Assessment. Chapel Hill, NC: University of North Carolina Chapel Hill, National Implementation Research Network.

Gilmer, TP, Stefancic, A, Ettner, SL, Manning WG, & Tsemberis S. (2010). Effect of full-service partnerships on homelessness, use and costs of mental health services, and quality of life among adults with serious mental illness. Archives of General Psychiatry 67(6): 645-652.

McHugo, G. J., Drake, R. E., Whitley, R., Bond, G. R., Campbell, K., Rapp, C. A., Goldman, H. H., Lutz, W. J., & Finnerty, M. T. (2007). Fidelity outcomes in the National Implementing Evidence-Based Practices Project. Psychiatric Services, 58, 1279-1284.

Tsemberis, S. (2010). Housing First: The Pathways model to end homelessness for people with mental illness and addiction. Hazelden: Minnesota.

LIST OF ATTACHMENTS

1: Project Director Telephone Interview & Web Survey

2: Site Visit Guides & Cost Questionnaire

3: Evidence-based Practice (EBP) Self-Assessment Part 1 & Part 2

4: Permanent Supportive Housing (PSH) Self-Assessment

5