HomelessProgramsEval OMB SS-B_1.13.14

The NATIONAL Evaluation OF the Substance Abuse and

Mental Health Services administration’s (samhsa’S) homeless programs

Supporting Statement

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

The following sections provide a detailed description of the respondents and sampling methods for each data collection tool, while Table 13 provides an overview of the grantees involved in each data collection activity.

Table 13. Grantees Included in Data Collection

		National Evaluation of SAMHSA’s Homeless Programs Grantees Included in Data Collection
	Homeless Program	GBHI	CABHI	SSH	PATH	TOTAL
	Cohort Year* (# of grantees)	2009 (25) 2010 (23)	2011 (23) 2012 (8)	2009 (43) 2010 (5)	56 grantees (renewed annually)	183
DATA COLLECTION ACTIVITIES	PD Interview	GBHI 2010 (23)	All grantees (31)	All grantees (48)	All grantees (56)	158
	Site Visits	4 to 5 grantees per year. Approximately 14 grantees will be selected through a stratified random sample over 3 years.	Approximately 6 grantees per year. Approximately 18 grantees will be selected through a stratified random sample over 3 years.	Approximately 9 grantees per year. Approximately 28 grantees will be selected through a stratified random sample over 3 years.	5 grantees per year. 15 grantees will be selected through a stratified random sample over 3 years.	25 site visits per year for 3 years. 75 site visits total.
	EBP Self-Assessment Part 1	All grantees (48)	All grantees (31)	All grantees (48)	NA	127
	EBP Self-Assessment Part 2	All grantees implementing at least one of the 5 EBPs selected for an in-depth assessment. Approximately 22 grantees.	All grantees implementing at least one of the 5 EBPs selected for an in-depth assessment. Approximately 25 grantees.	All grantees implementing at least one of the 5 EBPs selected for an in-depth assessment. Approximately 40 grantees.	NA	87
	PSH Self-Assessment	All grantees implementing a PSH model. Approximately 21 grantees.	All grantees implementing a PSH model. Approximately all grantees (31)	All grantees implementing a PSH model. Approximately all grantees (48)	NA	100

*Year listed is the year the cohort was first funded, most cohorts are funded for 3 to 5 years.

PD Interview

The PD Interview target universe is all grantees in the GBHI 2010, SSH 2009-2010, and CABHI 2011-2012 cohorts (n=102) and the 56 PATH grantees. GBHI 2009 grantees are not included as these grantees were included in a prior data collection. The sampling method is a census of all grantee projects in the cohorts listed and all 158 grantee projects are expected to complete the interview. Respondents will include grantee project directors.

Site Visit Guides

The Site Visit Guides target universe is grantees in the GBHI 2010, SSH 2009-2010, CABHI 2011-2012 and PATH 2010-2012 cohorts. GBHI 2009 grantees are not included as these grantees were included in a prior cross-site evaluation. Pending OMB approval, 20 site visits per year, over a three year period, will be held with a stratified randomized sample of GBHI 2010, SSH 2009-2010, CABHI 2011-2012 grantee projects. The following sampling method was reviewed with and endorsed by the Evaluation’s Technical Panel. Using SAMHSA’s knowledge of the programs and the contractor’s analysis of GPRA (OMB No. 0930-0208) and NOMS (OMB No. 0930–0285) data related to recruitment and retention, grantees that have not been able to implement their programs will be eliminated from sample selections. While studying these sites might provide some information on the challenges of implementation, on balance we believe that evaluation resources should be focused on sites with some degree of implementation and prospects for achieving intended outcomes. Setting these sites aside will help eliminate what are sometimes called “Type 3” errors, in which it is decided that a program or intervention is not effective when in fact it was not adequately implemented. The remaining sites will be stratified on the following dimensions: Homeless Program (GBHI, SSH, CABHI); implementing PSH/Housing First versus not; primary EBP(s); special population(s) served; type of grantee organization; urban setting versus not; and geographical region (4 regions). This stratification scheme would result in a matrix of potential selection criteria. We will randomly order the sites within each cell and tentatively select the first site on the list; if there is some reason to not include that site (e.g., SAMHSA has information indicating the site would not be fruitful for close study), we will tentatively select the next site on the list and explore whether there is reason to not select it. If any of the cells have no eligible sites, we will confer with SAMHSA on which cell(s) should be used as a replacement; any replacement cells would contain more than one site. If replacement cells are used, we will select the additional sites using the same randomly-ordered list as for the primary site selection. We have selected this approach because it (1) minimizes “Type 3” error, (2) ensures a variety of sites will be visited, and (3) leaves to random chance the selection of specific sites, which strengthens the generalizability of findings to other sites that are not visited.

Pending OMB approval, 5 site visits per year over a three year period will be held with a stratified randomized sample of PATH grantees. The selection of 15 of 56 PATH grantee project sites will be through a similar stratified randomized sampling method as described for the other Homeless Programs grantee projects above. However, the number and type of factors determining stratification somewhat differ from the selection of the GBHI, CABHI and SSH grantee project sites to accommodate differences in funding levels and services and the smaller number of sites to be visited. The stratification will result in a matrix of potential selection criteria using geographical region, urban setting versus not, funding level, and special populations served. The 15 sites will be chosen randomly from the completed sampling matrix to provide good representation of the breadth of PATH grantees. If there is some reason to not include a site selected (e.g., SAMHSA has information indicating the site would not be fruitful for close study), we will tentatively select the next site on the list and explore whether there is reason to not select it.  

All 75 grantee projects selected are expected to complete the site visit per prior participation rates in similar SAMHSA cross-program evaluations.

EBP Self-Assessment

The EBP Self-Assessment – Part 1 target universe is all grantees in the following cohorts: GBHI 2009-2010, SSH 2009-2010, and CABHI 2011-2012. The sampling method is a census of all grantee projects in the cohorts listed and all 127 grantee projects are expected to complete the assessment. Respondents will include grantee project directors or a key staff member knowledgeable about the EBPs being implemented by the project, as designated by the grantee project director. There will be one respondent per grantee project.

The EBP Self-Assessment – Part 2 target universe is the grantees from the GBHI 2009-2010, SSH 2009-2010, and CABHI 2011-2012 cohorts who are implementing at least one of the 5 EBPs selected for an in-depth assessment, per information collected during the PD Interview. Part 2 includes the following EBPs: ACT, IDDT, IMR, Supported Employment, and CTI. From this subsample, the sampling method is a census of all grantee projects meeting the above criteria. A preliminary review of all Homeless Programs grant applications indicate that approximately 87 grantee projects are implementing one or more of the core EBPS and therefore are eligible to complete the second part of the EBP Self-Assessment. There will be one respondent per grantee project.

PSH Self-Assessment

The PSH Self-Assessment target universe is the grantees from the GBHI 2009-2010, SSH 2009-2010, and CABHI 2011-2012 cohorts who are implementing a PSH model, per information collected during the PD Interview. From this subsample, the sampling method is a census of all grantee projects meeting the above criteria. A preliminary review of all Homeless Programs grant applications revealed that approximately 100 grantees projects are implementing PSH and therefore are eligible to complete the PSH Self-Assessment. Respondents will include grantee project directors or a key staff member knowledgeable about the PSH model being implemented by the project, as designated by the grantee project director. There will be one respondent per grantee project.

2. Information Collection Procedures

PD Interview

As noted above, respondents to the PD Interview telephone interview are grantee project directors. Publicly available contact information (e.g., phone number, email address) for the grantee project directors will be provided by SAMHSA to the Evaluation contractor. The contractor’s evaluation team will contact grantee project directors via email (with telephone follow-up) to setup a mutually convenient time during regularly scheduled business hours. Prior to conducting the PD Interview, the evaluation team will review grant applications that detail the proposed characteristics of the project (submitted to SAMHSA by each grantee and given to the evaluation team by SAMHSA) and extract information relevant to the evaluation (e.g., project structure, proposed services to be provided, proposed EBPs and housing models to be implemented, targeted populations). This information will be recorded onto a computerized template that will pre-populate about 51% of the PD Interview questions for grantee and project characteristics, services, partners and key stakeholders, target population to be served, planned sustainability activities and local evaluation plans. This will allow the grantees to confirm or update information during the PD Interview about what is actually being implemented, which not only helps reduce grantee burden, but also directly helps identify changes and adjustments grantees make to their programs once they are implemented.

Once the interview is scheduled, the contractor will provide the grantee project director with an electronic version of the consent form, the partially pre-populated PD Interview and a toll-free, passcode protected telephone conference number. Prior to beginning the PD Interview, the respondent will be read a script for consent that informs the respondent of their rights, including the right to not answer any question, and asks for their verbal consent to participate in the interview. If consent is provided, a senior evaluator from the contractor’s evaluation team will lead the respondent through the interview. The estimated time for the interview is 3.5 hours, which includes time for the project director to review and gather material prior to the telephone interview. If needed, respondents will be offered to take at least one short break during the interview. Responses will be entered by a junior team member directly into the computerized version of the PD Interview during the interview, which will be reviewed by the senior and junior evaluation team members and corrected for any input errors. The electronic version of the PD Interview will be maintained on a password protected, Point Sec-encrypted secure server accessible only to the contractor’s evaluation team. The grantee agency and project director will be assigned ID numbers so that the respondent name and grantee agency will not appear on the PD Interview computerized form along with responses. Following the interview, the interviewer will send an email thanking the grantee project director for his or her participation.

Site Visit Guides

As noted above, site visit participants include the Project Director and Management staff; Case Managers, Treatment, and Housing staff/providers; Stakeholders; Evaluators; Clients and Financial staff. Once site visit selection is finalized using the sampling method outlined above, an email will be sent to the project director and local evaluator to arrange a pre-site visit call. Once the pre-site visit call is scheduled, the contractor will provide the grantee with a toll-free, passcode protected telephone conference number; at the grantee’s project director’s discretion, he or she can include project staff on the pre-site call. To provide sufficient information prior to the call, the evaluation will also send the project director the Site Visit Guides topics and potential respondents for each interview. During the call, a date for the site visit will be set and an agenda will be developed. If needed, site visit logistics will be finalized through additional calls and via email. Prior to the site visit, the evaluation team will send three additional documents: 1) the finalized agenda and logistics information (e.g., site visitors’ hotel and contact information, contact information for the primary grantee site contact), 2) the full Site Visit Guides, and 3) a draft Client Flow Chart developed by the evaluation team from the review of the grant application to be reviewed and discussed during the Opening Session of the site visit. The Site Visit Guides will be customized to each grantee as some questions may not be relevant to all grantees, depending on housing and service models, and select sections will be pre-populated based on the information obtained from review of the grant application; during the site visits, these pre-populated items will be reviewed by grantees and updated as needed by the site visitor interviewer.

Site visits to the selected GBHI 2010, SSH 2009-2010, and CABHI 2011-2012 projects will be three days in duration and include three evaluation site visitors. Site visits to the selected PATH project sites will be two days in duration and include one evaluation site visitor. The evaluation site visit team will travel to the grantee’s project site for the mutually scheduled site visit date. Site visits will be conducted in-person, typically at the grantee’s location or where the project implements its services to participants. The site visit team will speak with different staff members, partner representatives and stakeholders throughout the day with concurrent meetings held to best utilize both grantee staff, partners and clients and the evaluation team. Each participant will be provided a written consent form prior to beginning their participation and procedures, including audio recording, will be verbally explained in addition to the participants’ own review of the consent. Each individual and group discussion will be digitally recorded (if the interviewee consents) and written notes will be taken on a laptop computer. The recordings will be transcribed and the recording will be deleted once the transcript is finalized. Transcripts will be entered into an Atlas.ti database and used for qualitative analysis.Written notes will be used to clean transcripts, including to clarify questions resulting from the transcribed audio recordings. Written notes will also be used to write a brief summary of the site visit and will be submitted to SAMHSA.

EBP Self-Assessment and PSH Self-Assessment

During the PD Interview, grantees will be informed about the EBP and PSH Self-Assessment and that they will be contacted by the evaluation team to complete the EBP Self-Assessment and may be contacted to complete the PSH Self-Assessment, if they meet the selection criteria.

All grantees will be sent an email by the evaluation team with a link to complete the EBP Self-Assessment – Part 1 on practice implementation. Information from the PD Interview will be used to identify grantees that are implementing one or more of the five sets of EBP assessment questions (ACT, IDDT, IMR, SE, CTI) and are eligible to complete the EBP Self-Assessment – Part 2 and to identify grantees that are implementing PSH and are eligibile to complete the PSH Self-Assessment. Emails to complete these surveys will include the unique grantee participant ID and site ID and secure log-in information to the web-based surveys, which will be hosted on the contractor’s secure servers through the evaluation website. The respondent and grantee agencys’ names will not appear with any of the responses and will be kept separately on a secure password protected, Point Sec-encrypted computer only available to the evaluation Project Director and Project Manager. Following a welcome page, the consent form will appear and the respondent will not be able proceed until the consent form is completed. If the respondent declines, a log off message will appear; if the participant provides consent, the survey will begin. There are separate consent forms for the EBP Self-Assessment and the PSH Self-Assessment. The assessments can be completed in one sitting or progress can be saved and continued at a later time until the grantee completes all questions and selects to submit the self-assessment. All data submitted by grantees will be saved securely on the contractor’s secure, password protected and encrypted servers.  The resulting datasets will only be available to evaluation team members actively involved in analyzing the EBP Self-Assessment and PSH Self-Assessment data.

3. Methods to Maximize Response Rates

While grantees in select programs (GBHI and CABHI) are required to participate in the Homeless Evaluation data collection activities, the PD Interview, Site Visit Guides and EBP and PSH Self-Assessments are designed to help ensure that each grantee project responds. Efforts to reach a 100% response rate begins at recruitment and focuses on reducing grantee burden and communicating information about the data collection efforts before its implementation. The contractor has already begun the engagement process by hosting webinars to introduce the evaluation to the grantees and presenting at the grantee’s annual conference, held by SAMHSA. The contractor’s evaluation team will aim to identify the most convenient time for grantee project directors to complete the PD Interview, Site Visit Guides and EBP and PSH Self-Assessments. Prior to site visits, participants will also be provided the interview topics so they will be knowledable about the type of information to be collected. The contractor will use the following strategies to further achieve sufficient response rates:

• Send grantees an initial email invitation that explains the study and its importance, why they are being asked to participate, and how they can contact the contractor for additional information.

• Send reminder emails to non-respondents and, if approved by SAMHSA, ask grantee project directors to also encourage non-respondents to participate.

• If needed, for the EBP and PSH Self-Assessments, allow respondents some other way to participate other than over the web (e.g. mailed hard copy or conducted over the telephone).

4. Test of Procedures

The PD Interview and Site Visit Guides were first developed and implemented in the GBHI 2009 cross-site evaluation effort. For the Homeless Programs Evaluation, the PD Interview and Site Visit Guides were revised based on prior experience (e.g., removing redundant questions, clarifying question wording and instructions, shifting questions to probes or probes to questions) and to incorporate the Evaluation’s wider scope and the additional grantee programs. With these changes, the PD Interview and Site Visit Guides still have the same topics, similar format, and number of questions. As such, the experience from the GBHI FY2009 evaluation is relevant to the current versions. The procedure for the PD Interview was previously piloted during the GBHI cross-site evaluation to good effect and with 100% participation; grantees appreciated the ability to pre-review both the consent form and the interview questions so they were prepared for the focus of the evaluation and to review and discuss the questions during the telephone interview. The Site Visit Guides and procedures were piloted during the GBHI cross-site evaluation with 100% grantee participation and grantees found the Site Visit process to be straightforward, well-designed and flexible to the uniqueness of the individual grantee projects. Based on feedback from this piloting, the Project Director/Opening Session and the Treatment/Case Management/Housing Staff Discussion Guides were consolidated to more effectively use staff time. The cost questionnaire was also streamedlined by using broader service categories to better align with the categories used across all grantees.

The EBP Self-Assessment and PSH Self-Assessment are developed from standardized SAMHSA EBP Fidelity Tool Kits (ACT: DHHS Publication No. SMA-08-4344; IDDT: DHHS Publication No. SMA-08-4366; IMR: DHHS Publication No. SMA-09-4462; SE: DHHS Publication No. SMA-08-4364; and PSH: DHHS Publication No. SMA-10-4509), which have been tested with SAMHSA grantees (McHugo et al., 2007), and other validated assessment tools (the GOI [Bond et al., 2009]; the SHAY [Finnerty et al., 2009]; the ISA [Fixsen & Blase, 2010]; Pathways Housing First Fidelity Scale-ACT version [Tsemberis, 2010]; the FSP Practices Scale [Gilmer et al. 2010]) that are used regularly by organizations similar to the Homeless Program grantees (Gilmer & Katz, 2012; Macnaughton, Goering, & Nelson, 2012; Stergiopoulos et al., 2012). A majority of the EBP Self-Assessment Part 1 and Part 2 fidelity component questions and the SAMHSA PSH fidelity checklist that comprises the majority of the PSH Self-Assessment were given during the GBHI cross-site evaluation site visits to good response and yielded reliable useful information about practice implementation (Broner et al., 2011a; Broner et al., 2011b; Stainbrook et al., 2011). As such, these tools have been well tested with the targeted respondents.

Additionally, mock pencil and paper PD Interviews were performed with contractor staff who were a part of the previous GBHI evaluation and included past local evaluators of Homeless Programs projects; “interviewees” answered the questions from the point of view of GBHI grantee sites they had site visited through the previous evaluation and thus, knew the project well. The interview, including time to pre-review the questions and informed consent, is estimated to take 3.5 hours. The practice tests were timed using a variety of answer patterns as the time required to complete the interview will vary depending on the grantees’ service models, housing models, etc. Mock pencil and paper EBP Self-Assessments and PSH Self-Assessments were also completed by contractor staff who have experience implementing or evaluating projects that have implemented EBPs and PSH models in populations similar to those found in the Homeless Programs (e.g., homeless, history of/current substance abuse disorders, mental health disorders, etc.). The EBP Self-Assessment – Part 1, including time for informed consent, is estimated to take 35 minutes. The EBP Self-Assessment – Part 2 is estimated to take 30 minutes. The PSH Self-Assessment, including time for informed consent, is estimated to take 40 minutes.

5. Statistical Consultants

As noted in Section A.8, SAMHSA has consulted extensively with an expert panel who will continue to provide expert advice throughout the course of the evaluation. In addition, the contractor team is comprised of several experts who will be directly involved in the data collection and statistical analysis. Also, contractor in-house experts will be consulted throughout the program on various statistical aspects of the design, methodological issues, economic analysis, database management, and data analysis. Exhibit 4 provides details of these team members and advisors.

Exhibit 4. Data Collection and Analysis Team Members and Advisors

Expert	Affiliation	Contact Information
Mike Anastario, Ph.D. Statistician	Public Health Research Analyst Crime, Violence and Justice Research Program RTI International 121 W. 27th Street, Suite 1001 New York, NY 10001	Phone: 212-367-8084 Email: manastario@rti.org
Nahama Broner, Ph.D. Cross-Program Evaluation Project Director	Senior Research Psychologist Crime, Violence and Justice Research Program RTI International 121 West 27th Street, Suite 1001 New York, NY 10001	Phone: 212-367-1990 E-mail: nbroner@rti.org
Alexander Cowell, Ph.D. Cross-Program Evaluation Cost Analysis Team Leader	Economist Behavioral Health Economics Program RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709	Phone: 919-541-8754 E-mail: cowell@rti.org
Antonio Morgan-Lopez, Ph.D. Senior Statistician	Senior Research Quantitative Psychologist Risk Behavior and Family Research RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709	Phone: 919-316-3436 E-mail: amorganlopez@rti.org
Pamela Lattimore, Ph.D. Cross-Program Evaluation Data Analysis Team Leader	Principal Scientist Crime, Violence and Justice Research Program RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709	Phone: 919-485-7759 E-mail: lattimore@rti.org
Aileen Rothbard, Sc.D. Statistician	Research Professor School of Social Policy and Practice & Department of Psychiatry 3535 Market Street, Room 3014 Philadelphia, PA 19104-2648	Phone: 215-573-7770 E-mail: rothbard@mail.med.upenn.edu
James Trudeau, Ph.D. Cross-Program Evaluation Data Collection Team Leader	Director Crime, Violence and Justice Research Program RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709	Phone: 919-485-7751 E-mail: trudeau@rti.org
Gary A. Zarkin, Ph.D. Senior Advisor	Director Behavioral Health Research Division RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709	Phone: 919-541-5858 E-mail: gaz@rti.org

REFERENCES

Aarons, G.A., Hurlburt, M., Horowitz, S.M. (2011). Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors. Administration and Policy in Mental Health, 38: 4-23.

Ball, S. A., Martino, S., Nich, C., Frankforter, T. L., van Horn, D., Crits-Christoph, P., et al. (2007). Site matters: Multisite randomized trial of motivational enhancement therapy in community drug abuse clinics. Journal of Consulting and Clinical Psychology, 75, 556–567.

Basu A., Kee R., Buchanan D., and Sadowski L. 2012. Comparative Cost Analysis of Housing and Case Management Program for Chronically Ill Homeless Adults Compared to Usual Care. Health Services Research. 47 (1pt2): 523–543

Bond, G. R., Drake, R. E., Rapp, C. A., McHugo, G. J., & Xie, H. (2009). Individualization and quality improvement: Two new scales to complement measurement of program fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 36: 349–357.

Borsari, B., & Carey, K. B. (2000). Effects of a brief motivational intervention with college student drinkers. Journal of Consulting and Clinical Psychology, 68, 728–733.

Broner, N., Trudeau, J., Barrick, K., Tibaduiza, E., Maxwell, K. Aldridge, A, & Cowell, A. (2011a, November). Service and housing model approaches to ameliorate substance use disorders, reduce homelessness and increase housing stability: Initial process and outcome findings. Presented at the annual meeting of the American Public Health Association, Washington, DC.

Broner, N., Trudeau, J., Lattimore, P. K., Barrick, K., Williams, J., & Maxwell, K. D. (2011b, November). Services and housing interventions for criminal justice-involved homeless populations with substance use and co-occurring mental disorders. Presented at the annual meeting for the American Society of Criminology, Washington, DC.

Brown, J. M., & Miller, W. R. (1993). Impact of motivational interviewing on participation and outcome in residential alcoholism treatment. Psychology of Addictive Behaviors, 7, 211–218.

Burt, M. R., Aron, L. Y., Douglas, T., Valente, J., Lee, E., & Iwen, B. (1999). Homelessness: Programs and the people they serve. Washington, DC: Interagency Council on the Homeless.

Caton, C., Wilkins, C., & Anderson, J. (2007). People who experience long-term homelessness: Characteristics and interventions. In D. Dennis, G. Locke, & J. Khadduri (Eds.), Toward understanding homelessness: The 2007 National Symposium on Homelessness Research (pp. 4-1 through 4-44). Washington, DC: U.S. Department of Housing and Urban Development and U.S. Department of Health and Human Services. Retrieved October 26, 2009, from http://aspe.hhs.gov/hsp/homelessness/symposium07/

Cheng, A. L., & Kelly, P. J. (2008). Impact of an integrated service system on client outcomes by gender in a national sample of a mentally ill homeless population. Gender Medicine, 5, 395–404.

Conrad, K. J., Hultman, C. I., Pope, A. R., Lyons, J. S., Baxter, W. C., Daghestani, A. N., et al. (1997). Case managed residential care for homeless addicted veterans: Results of a true experiment. Medical Care, 36, 40–53.

Cowell, A. J., Hinde, J., Orme, S., Aldridge, A., & Broner, N. (2011, May). The role of federal grant funding in strengthening the system of care for homeless people with substance use and co-occurring mental health problems: A first look. SAMHSA GBHI National Cross-Site Evaluation in Brief Series. New York, NY: RTI International. Contract #: HHSS283200700002I/ HHSS28342002T.

Drake, R. E., McHugo, G. J. & Noordsy, D. L. (1993). Treatment of alcoholism among schizophrenic patients: Four-year-outcomes. American Journal of Psychiatry, 150, 328–329.

Drake, R. E., Yovetich, N. A., Bebout, R. R., Harris, M., & McHugo, G. J. (1997). Integrated treatment for dually diagnosed homeless adults. Journal of Nervous and Mental Disease, 185, 298–305.

Finnerty, M.T., Rapp, C.A., Bond, G.R., Lynde, D.W., Ganju, V., Goldman, H.H. (2009). The State Health Authority Yardstick (SHAY). Community Mental Health Journal, 45(3):228-36.

Fixsen, D.L., and Blase, K.A. (2010). Stage-Based Measures of Implementation Components: Installation Stage Assessment. Chapel Hill, NC: University of North Carolina Chapel Hill, National Implementation Research Network.

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation Research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231), http://nirn.fmhi.usf.edu/resources/detail.cfm?resourceID=31.

Gilmer, TP, Stefancic, A, Ettner, SL, Manning WG, & Tsemberis S. (2010). Effect of full-service partnerships on homelessness, use and costs of mental health services, and quality of life among adults with serious mental illness. Archives of General Psychiatry 67(6): 645-652.

Gilmer, T. & Katz, M. (2012). Combining fidelity and research: Lessons from a study of California's full service partnerships. Presentation at the Housing First Partners' Conference.

Greenberg, G. A., & Rosenheck, R. A. (2008). Jail incarceration, homelessness and mental health: A national study. Psychiatric Services, 59, 170–177.

Hiday, V., Swartz, M., Swanson, J., Borum, R., & Wagner, H. (1999). Criminal victimization of persons with severe mental illness. Psychiatric Services, 50, 62–68.

Kertesz, S. G., Crouch, K., Milby, J. B., Cusimano, R. E., & Schumacher, J. E. (2009). Housing First for homeless persons with active addiction: Are we overreaching? The Milbank Quarterly, 87, 495–534.

Joseph, H., & Langrod, J. G. (2004). The homeless. In J. H. Lowinson, P. Ruiz, R. B. Millman, & J. G. Langrod (Eds.), Substance abuse: A comprehensive textbook (4th ed.; pp. 1141–1168). Philadelphia: Lippincott, Williams, & Wilkins.

Kline, A., Callahan, L., Butler, M., St. Hill, L., Losonczy, M. F., & Smelson, D. A. (2009). The relationship between military service era and psychosocial treatment needs among homeless veterans with a co-occurring substance abuse and mental health disorder. Journal of Dual Diagnosis, 5, 357–374.

Mallett, S., Rosenthal, D., & Keys, D. (2005). Young people, drug use, and family conflict: Pathways into homelessness. Journal of Adolescence, 28, 185–199.

Martell, D. A., Rosner, R., & Harmon, R. B. (1995). Base-rate estimates of criminal behavior by homeless mentally ill persons in New York City. Psychiatric Services, 46, 596–601.

Macnaughton, E.L., Goering, PN, & Nelson, G.B. (2012). Exploring the value of mixed methods within the at Home/Chez Soi Housing First Project: A strategy to evaluate the implementation of a complex population health intervention for people with mental illness who have been homeless. Canadian Journal of Public Health, 103(Suppl. 1): S57-S62.

McGovern, M. P., McHugo, G. J., Drake, R. E., Bond, G. R., & Merrens, M. R. (in press). Implementing evidence-based practices: a practical guide for behavioral health professionals. Center City, MN: Hazelden (Chapter 3).

McHugo, G. J., Drake, R. E., Whitley, R., Bond, G. R., Campbell, K., Rapp, C. A., Goldman, H. H., Lutz, W. J., & Finnerty, M. T. (2007). Fidelity outcomes in the National Implementing Evidence-Based Practices Project. Psychiatric Services, 58, 1279-1284.

McNiel, D.E., Binder, R.L., Robinson, J.C. (2005). Incarceration Associated With Homelessness, Mental Disorder, and Co-occurring Substance Abuse. Psychiatric Services, 56, 840-846.

Miller, W. R., Benefield, G., & Tonigan, J. S. (1993). Enhancing motivation for change in problem drinking: A controlled comparison of two therapist styles. Journal of Consulting and Clinical Psychology, 61, 455–461.

Moore, G., Gerdtzl, M., Manias, E., Hepworth, G., and Dent, A. (2007). Socio-demographic and clinical characteristics of re-presentation to an Australian inner-city emergency department: implications for service delivery. BMC Public Health, 7.

Mueser, K. T., Drake, R. E., & Miles, K. M. (1997). The course and treatment of substance use disorder in persons with severe mental illness. In L. S. Onken, J. D. Blaine, S. Genser, & A. M. Horton (Eds.), Treatment of drug-dependent individuals with comorbid mental disorders (pp. 86–109). Rockville, MD: National Institute on Drug Abuse.

National Law Center on Homelessness and Poverty; National Coalition for the Homeless. (2009). Homes not handcuffs: The criminalization of homelessness in U.S. cities. Washington, DC: Authors. Retrieved November 10, 2009, from http://www.nationalhomeless.org/publications/crimreport/crimreport_2009.pdf

National Coalition for the Homeless. (2009). Substance abuse and homelessness. Retrieved October 17, 2009, from http://www.nationalhomeless.org/factsheets/addiction.html

Nelson, G., Aubry, T., & Lafrance, A. (2007). A review of the literature on the effectiveness of housing and support, Assertive Community Treatment, and intensive case management interventions for persons with mental illness who have been homeless. American Journal of Orthopsychiatry, 77, 350–361.

North, C. S., Eyrich, K. M., Pollio, D. E., & Spitznagel, E. L. (2004). Are rates of psychiatric disorders in the homeless population changing? American Journal of Public Health, 94, 103–108.

Project MATCH Research Group. (1997). Matching alcoholism treatments to client heterogeneity: Project MATCH posttreatment drinking outcomes. Journal of Studies on Alcohol, 58, 7–29.

Rukmana, D. Where the homeless children and youth come from: A study of the residential origins of the homeless in Miami-Dade County, Florida. (2008). Children and Youth Services Review, 30, 1009–1021.

Rosenblum, A., Magura, S., Kayman, D. J., & Fong, C. (2005). Motivationally enhanced group counseling for substance users in a soup kitchen: A randomized clinical trial. Drug and Alcohol Dependence, 80, 91–103.

Shelton, K., Taylor, P., Bonner, A., & van den Bree, M. (2009). Risk factors for homelessness: Evidence from a population-based study. Psychiatric Services, 60, 465–472. doi:10.1176/appi.ps.60.4.465

Stainbrook, K., Hanna, J., Trudeau, J., & Broner, N. (2011, November). Strategic collaborations among GBHI grantees to expand and strengthen treatment and housing for homeless populations. Presented at the annual meeting of the American Public Health Association, Washington, DC.

Stephens, R. S., Roffman, R. A., & Curtin, L. (2000). Comparison of extended versus brief treatments for marijuana use. Journal of Consulting and Clinical Psychology, 68, 898–908.

Stergiopoulos, V., O’Campo P., Gozdzik, A., Jeyaratnam, J., Corneau, S., Sarang, A. & Hwang, S.W. (2012). Moving from rhetoric to reality: Adapting Housing First for homeless individuals with mental illness from ethno-racial groups. BMC Health Services Research, 12:345.

Tsemberis, S. (2010). Housing First: The Pathways model to end homelessness for people with mental illness and addiction. Hazelden: Minnesota.

U.S. Department of Housing and Urban Development, Office of Community Planning and Development. (2009). The 2008 annual homeless assessment report to Congress. Washington, DC: Author.

U.S. Department of Veterans Affairs, Veterans Administration. (2009). Homeless veterans: Overview of homelessness. Washington, DC: Author. Retrieved on November 17, 2009, from http://www1.va. gov/homeless/page. cfm?pg=1

van den Bree, M. B. M., Shelton, K., Bonner, A., Moss, S., Thomas, H., & Taylor, P. J. (2009). A longitudinal population-based study of factors in adolescence predicting homelessness in young adulthood. Journal of Adolescent Health, 45, 571–578.

Vangeest, J. B., & Johnson, T. P. (2002). Substance abuse and homelessness: Direct or indirect effects? Annals of Epidemiology, 12, 455–461.

LIST OF ATTACHMENTS

1: Project Director Telephone Interview

2: Site Visit Guides & Cost Questionnaire

3: Evidence-based Practice (EBP) Self-Assessment Part 1 & Part 2

4: Permanent Supportive Housing (PSH) Self-Assessment

5: Project Director Telephone Interview Consent

6: Staff Site Visit Interview Consent

7: Client Site Visit Focus Group Consent

8: Evidence-based Practice (EBP) Web Consent

9: Permanent Supportive Housing (PSH) Web Consent

10: Analysis Table Shells

9