TCE-HIV_OMB_Supporting_Statement_07-14-2010_FINAL

TARGETED CAPACITY EXPANSION PROGRAM FOR SUBSTANCE ABUSE TREATMENT AND HIV/AIDS (TCE-HIV) SERVICES

MULTI-SITE EVALUATION

SUPPORTING STATEMENT

A. JUSTIFICATION

1. Circumstances of Information Collection

The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Treatment (CSAT) is requesting approval from the Office of Management and Budget (OMB) for the data collection activities for the Targeted Capacity Expansion Program for Substance Abuse Treatment and HIV/AIDS (TCE-HIV) Services Multi-Site Evaluation (hereafter referred to as the Multi-Site Evaluation). There are a total of 8 data collection instruments for this evaluation (i.e., administrative staff semi-structured interview guide, grantee direct services staff semi-structured interview guide, partner/collaborator semi-structured interview guide, client focus group guide, baseline client level survey, discharge client level survey, six month follow up client level survey and client dosage form). The evaluation activities will be conducted with 48 Grantees, 40 of which are treatment Grantees and 8 of which are outreach/pretreatment Grantees. Evaluation activities will include the administration of baseline, discharge, and follow-up surveys of clients receiving TCE-HIV treatment services; conducting focus groups with clients receiving substance abuse treatment; and conducting semi-structured interviews with TCE-HIV program staff and program collaborators/partners who are overseeing or delivering TCE-HIV services. The evaluation is intended to provide insight about the impact of outreach/pretreatment and treatment programs offered through SAMHSA TCE-HIV grant funding, and to better understand the mechanisms by which TCE-HIV programs affect positive change in the reduction of substance abuse and HIV risk behaviors in racial and ethnic minority communities.

The TCE-HIV grants are authorized under Section 509 of the Public Health Service Act (42 U.S.C.290bb-2), as amended. The program also addresses Healthy People 2020 Focus Area 2020-6 (Substance Abuse) (U.S. Department of Health and Human Services).

SAMHSA/CSAT recognized the association between substance abuse and HIV risk and developed the TCE-HIV services grant program to improve the delivery of services to racial and ethnic minority communities disproportionately impacted by substance abuse and HIV across the United States (Centers for Disease Control and Prevention, 2006). The TCE-HIV Grantees are tasked with enhancing and expanding substance abuse treatment and/or outreach and pretreatment services in conjunction with HIV/AIDS services in high-risk, underserved communities highly affected by both substance abuse and HIV/AIDS. SAMHSA/CSAT’s TCE-HIV program is funded by Minority AIDS Initiative (MAI, 1998) appropriated dollars. The MAI was established in October 1998 in response to the growing concern about the impact of HIV/AIDS on racial and ethnic minorities in the United States. The goals of the MAI are to improve HIV-related health outcomes for racial and ethnic minorities in communities disproportionately affected by HIV/AIDS, and to reduce HIV/AIDS related health disparities.

SAMHSA/CSAT provides the MAI funds for grants to expand treatment through increasing access and availability of services to a larger number of clients and/or to enhance services by improving the quality and/or the intensity of services in conjunction with HIV/AIDS services in African-American, Hispanic, and/or other racial and ethnic communities affected by the twin epidemics of substance abuse and HIV/AIDS. Enhanced and expanded services will, in turn, improve outcomes related to substance use and substance abuse prevention, HIV/AIDS prevention, criminal activities and criminal justice involvement, and risk behaviors for HIV infection. The primary goal of SAMHSA/CSAT’s TCE-HIV program is to improve access to substance abuse treatment services through increasing capacity and outreach to racial and ethnic minority populations. The current cohort includes 48 Grantees, which are located in 19 states. As part of their SAMHSA/CSAT TCE-HIV initiative grant requirements, Grantees have agreed to provide data for outreach/pretreatment and clinical treatment as part of their participation in an ongoing Multi-Site Evaluation. It is SAMHSA/CSAT’s intention to use the successful conceptualization, planning, and implementation of a multi-site evaluation to independently evaluate the TCE-HIV program in a manner that is comprehensive (i.e., process, outcome), will establish baseline and long-term performance goals, and will describe the effects of the TCE-HIV program. Further, this first-of-a-kind multi-site evaluation will allow SAMHSA/CSAT to investigate the influence of the grant awards on the providers, the clients they serve, and the community, as well as the efficacy of differing treatment modalities (e.g., outpatient treatment, residential treatment) on changes in substance abuse, HIV risk behaviors, and health-related outcomes.

Studies demonstrate the effectiveness of substance abuse treatment in reducing drug use and HIV risk behaviors (General Accounting Office, 1998). Treatment of drug dependence has been associated with decreases in HIV risk behavior and transmission (Metzger et al., 1993; Booth, Crowley, & Zhang, 1996; Prendergast, Urada, & Podus, 2001). Consistently, studies demonstrate that substance abuse treatment decreases sexual behaviors associated with increased risk of HIV infection (Broome, Joe, & Simpson, 1999; Magura, Rosenblum, & Rodriguez, 1998; Camacho, Bartholomew, Joe, Cloud, & Simpson, 1996; Banks, Brown, & Ajuluchukwu, 1991; Bastos, et al., 2000; Somlai, Kelly, McAuliffe, Ksobiech, & Hackel, 2003; Hartel & Schoenbaum, 1998; Sullivan, Metzger, Fudala, & Fiellin, 2005; Murphy, et al., 2008). These and other studies led the Institute of Medicine (IOM) to identify substance abuse treatment as a critical component in preventing the spread of HIV infection (IOM, 2001). In addition to research findings of the effectiveness of substance abuse treatment in reducing drug use and HIV risky behavior, research also demonstrates that substance abuse treatment for persons who are HIV infected is associated with positive behaviors (e.g., increased antiretroviral adherence, decreased repeated emergency department visits, decreased hospitalizations, and increased dental care) (Turner, Laine, Cosler, & Hauck, 2003; Turner, Laine, Yang, & Hauck, 2003; Palepu et al., 2001).

The existing literature on substance abuse and HIV risk provides a baseline of research evidence, upon which the Multi-Site Evaluation is built. The impact and success of pretreatment/outreach and substance abuse treatment approaches can be greatly affected by community, program, and client variables, such as community involvement/support, implementation of evidence-based practices (EBPs), and staff experience and training, which can serve to support a program’s success or impede it. The Multi-Site Evaluation provides an opportunity to address and more thoroughly examine a number of areas about which there is comparatively little research (e.g., interaction of treatment modality, dosage, and length of stay at substance abuse organizations), and will thereby improve understanding of the mechanisms underlying effective substance abuse treatment and reduced HIV risk behaviors.

The Multi-Site Evaluation is grounded theoretically in an ecological systems approach, based on the presumption that positive treatment outcomes are determined by multiple tiers within a hierarchy of influence, and that a given client’s treatment success will be a function of treatment program operations, individual characteristics, and broader community contextual and socio-cultural environmental influences. Our approach combines an ecological theory with a risk and resiliency framework. The ecological model highlights the interaction between the clients served and their environments such as peers, community, and culture. The resiliency framework focuses on risk and protective processes for each ecological system.

According to the ecological model (Bronfenbrenner, 1979), four primary subsystems influence client behavior and behavioral change:

• Macrosystem: Larger societal factors affecting clients, such as culture and local policies

• Exosystem: Community and the TCE-HIV funded programs located within the community context

• Mesosystem: Families, peers, and other relational contexts

• Microsystem: Individuals/clients

Bronfenbrenner’s (1979, 1986) ecological model integrated with a risk and resiliency framework (Dekovic, 1999; Rutter, 1987) is appropriate for examining the influence of community-, Grantee-, and client-level factors on clients’ behavior.

Within the ecological subsystems exists constructs that help define and operationalize the Multi-Site Evaluation. Based on the results of our comprehensive literature review and the previous studies summarized above, nine constructs have been identified within the ecological framework as critical influences on TCE-HIV program services and client behavior. These constructs are presented in Exhibit 1.

Exhibit 1: Ecological Model and Multi-Site Evaluation Constructs

Ecological Subsystem	Construct
Macrosystem (Community Level)	Community/Contextual Environment Collaborations/Partnerships
Exosystem (Grantee Level)	Evidence-Based Practices (EBPs) Staffing/Training HIV Testing
Mesosystem/Microsystem (Client Level)	Substance Abuse and HIV Risk Behaviors Social System Quality of Life Client Satisfaction

We propose a rigorous evaluation approach examining the full ecological hierarchy in which clients live to facilitate a comprehensive analysis of client outcomes. Through the evaluation of the fiscal year 2008 (FY08) TCE-HIV Grantees, the Multi-Site Evaluation Team can identify associations between treatment approaches and client outcomes.

The Multi-Site Evaluation will be able to address several overarching questions and provide SAMHSA/CSAT with valuable knowledge regarding how the TCE-HIV program is meeting the overall goals of the MAI. Specifically, questions to be addressed in this evaluation include:

• What are the community and contextual conditions within which the TCE-HIV programs exist and provide services?

• How do TCE-HIV Grantee characteristics facilitate or impede program activities, such as comprehensive services, service coordination/integration, and expanded organizational/ programmatic capacity?

• What impact do Grantee characteristics have on client-level outcomes?

• How do client characteristics (e.g., education, employment, income, family and living conditions, social support, satisfaction) lead to behavioral changes?

• What impact does the TCE-HIV program have on client-level outcomes?

This Multi-Site Evaluation is the first TCE-HIV evaluation and represents the most comprehensive assessment of SAMHSA/CSAT’s TCE-HIV program ever undertaken. This complete evaluation will allow SAMHSA/CSAT to determine the extent to which the TCE-HIV program has met its objective of expanding and enhancing substance abuse outreach/pretreatment and treatment services in conjunction with HIV/AIDS services to racial and ethnic minority communities; and the resultant effects of the TCE-HIV program on clients served.

2. Purpose and Use of Information

The purpose of this evaluation is to provide insight about the effects of substance abuse outreach/pretreatment and treatment programs offered through SAMHSA TCE-HIV grant funding, and to better understand the mechanisms by which TCE-HIV programs affect positive change in the reduction of substance abuse and HIV risk behaviors in racial and ethnic minority communities. These data collected will be utilized to address the purpose of the Multi-Site Evaluation, which is theoretically grounded in an ecological systems approach, and based on the presumption that positive treatment outcomes are determined by multiple tiers within a hierarchy of influence, and that a given client’s treatment success will be a function of treatment program operations (i.e., Grantee level), individual/relationship characteristics (i.e., Client level), and broader community contextual and socio-cultural environmental influences (i.e., Community level). Based on this theoretical framework, two interrelated evaluation efforts are supported: a process evaluation and an outcome evaluation. The process evaluation will serve the critical role of establishing the overall evaluation’s context and consequently will aid in the interpretation of its findings. The process evaluation will also describe the content of Grantees’ interventions and their underlying evidence base. Moreover, the process evaluation will provide information about the TCE-HIV services delivered, who delivered them, how they were delivered, to whom they were delivered, and whether these services modified the Grantee’s overall substance abuse treatment service delivery system.

While the process evaluation will collect data from all 48 Grantees, the outcome evaluation component is focused on client-level treatment outcomes and will, therefore, focus only on the 40 Grantees providing substance abuse treatment services. The outcome evaluation will provide evidence on the effects of the TCE-HIV treatment interventions on Grantees and clients. In a Multi-Site Evaluation such as this TCE-HIV evaluation, the outcome evaluation provides critical information on the Grantee characteristics that are believed to affect TCE-HIV client outcomes. Consolidating the information on program components across Grantees provides the basis for identifying commonalities among the diverse programs and populations being served. Using the variation across Grantees, the Multi-Site Evaluation can assess the effect of specific variables on clients’ outcomes in ways that the individual Grantees cannot. The outcome evaluation will also provide evidence on how specific programmatic characteristics relate to effectiveness in reducing substance abuse and HIV risk behaviors.

Process and outcome data collection call for different measurement and collection strategies. To address the different data needs of the process and outcome evaluations, process data collection will be implemented at two distinct points in time, via site visits; outcome data collection will be implemented continuously via counselor-administered client surveys and counselor-completed dosage forms upon client discharge. The purpose of the site visits are to observe program processes, to conduct semi-structured interviews with project staff and partners/collaborators, to conduct client focus groups, and to provide any technical assistance needed for client survey administration and dosage form completion.

Process measures will be collected through a site visit during the first data collection point (Year 2). The process measures are designed to capture information on the specific activities associated with implementation, the staff involved in program implementation activities, and the amount of time required for a client to complete treatment. The second process data collection point (Year 4) will consist of a follow-up site visit to collect a repeat measure of all data collected during Year 2. In addition to data collection, the Multi-Site evaluation team will provide technical assistance to Grantees on client survey administration and dosage form completion. To determine the effect of TCE-HIV services, baseline measures will be compared with the same indicators approximately 24 months after the initial measurement. Comparisons between these time points will yield information on community and program changes as measured by site visit instruments. Simultaneously, client outcome measures and client dosage data will be collected continuously via a counselor-administered client survey and review of client charts to document service exposure, respectively. The client surveys are designed to assess whether and to what extent TCE-HIV services are associated with changes in the key constructs of interest (i.e., HIV risk behaviors, social connectedness, quality of life) for clients. TCE-HIV counselors will facilitate the completion of the client survey for each treatment client at baseline, discharge, and 6 months post baseline. Program staff will complete the dosage form for each client upon treatment discharge.

Process Evaluation. The process evaluation of the 48 TCE-HIV Grantees will focus on the implementation evaluation questions. Primarily, questions related to the service delivery process will be addressed; these will determine activities required of Grantees to implement and sustain TCE-HIV services. Specifically, process data will assess:

• Community/contextual conditions

• Community partnerships

• Program characteristics

• System resources

• Grantee activities and operations

• Staffing and training

• Client conditions (e.g., barriers to receiving services)

The semi-structured interviews with project directors, Grantee staff, and community collaborators/partners, along with client focus groups will provide the data necessary to conduct the process evaluation. The following sections present a description of the process data collection methodologies.

Administrative Staff Semi-Structured Interviews: (See Attachment 3 - Document 1.) The goal of this interview is to collect information regarding the development and changes in TCE-HIV program operations, staffing, training, and programming (e.g., outreach-pretreatment and/or treatment activities); to provide improved understanding of program, agency, and community capacity changes that result from TCE-HIV activities; and to collect documentation of changes in the number or nature of partnerships and collaborations both internal and external to the TCE-HIV program agency.

Individuals who perform administrative tasks related to the TCE-HIV program (e.g., Project Director, Program Manager, and Executive Director) are eligible to be interviewed. It is estimated that one administrative staff member per Grantee site will be interviewed during a given interview session. The administrator interviews will be conducted in two parts. The first part will be conducted with an executive staff member of the agency; and the second part with the Project Director or Coordinator of the TCE-HIV project. It is likely that one person from the agency may fulfill both roles/positions. If this is the case, the full interview will be conducted with that person.

Direct Services Staff Semi-Structured Interviews: (See Attachment 5 - Document 1.) The goal of this interview will be to collect information regarding the development of outreach/pretreatment and treatment operations/activities and to improve the understanding of program changes that result from TCE-HIV activities. Individuals from the Grantee organization who have direct contact with clients to perform outreach/pretreatment and treatment-related tasks will be eligible to be interviewed. Administrative staff members and direct services staff members (e.g., outreach workers, counselors) perform different functions in grantee organizations and can each provide a different vantage point on treatment operations/activities within an organization. To obtain a balanced set of data, information will be collected from both. Examples of those performing direct service tasks include outreach workers, treatment counselors, and case managers. It is estimated that more than one direct services staff member will participate in a given interview session.

Partner/Collaborator Semi-Structured Interviews: (See Attachment 7 - Document 1.) The goal of this interview will be to collect information regarding the development of the relationship between partners and the TCE-HIV Grantee agency; to document the types of activities and services the partners provide in collaboration with the Grantee; and to assess partners’ perceptions of improved client outcomes related to collaboration between partner and Grantee agency. “Partner/collaborator” refers to agencies or organizations that provide services and activities related to the TCE-HIV program. It is estimated that one to two individuals per community partner agency will participate in a given interview session.

Client Focus Groups: (See Attachment 1a - Document 1.) The goal of the client focus group conducted during TCE-HIV Multi-Site Evaluation site visits will be to collect data from a subsample of TCE-HIV treatment clients regarding their satisfaction with the treatment program, barriers and facilitators of treatment services, and client-level outcomes (i.e., substance use/abuse, risk behavior, quality of life). The Multi-Site Evaluation staff will work collaboratively with Grantee staff to identify client focus group participants. Only those clients who have been administered the GPRA and who are receiving TCE-HIV funded treatment services will be eligible to participate in the client focus group. Several inclusion and exclusion criteria will be used to ensure client participants are representative of the target audience served by the Grantee. For example, clients who have been in treatment for fewer than 14 days may be unable to share information based on their limited time in treatment. Additionally, the recruitment of clients for the focus groups will strive for a balance in participants with regard to gender and age. A specified list of focus group inclusion and exclusion criteria will be developed and shared with the Grantees to aid in client identification. It is estimated that up to nine clients will participate in each focus group session.

Outcome Evaluation. While the process evaluation will produce critically useful descriptive information at a community level and grantee level regarding the TCE-HIV Grantees, how their programs are implemented, and changes through the course of the 5-year funding cycle, the outcome evaluation of the 40 treatment Grantees will provide evidence on the effects of the TCE-HIV interventions on clients.

Specifically the outcome evaluation will provide data regarding:

• Client characteristics

• HIV testing and status

• Criminal activity

• Education and housing

• Client satisfaction

• Substance abuse (the sole measure is the GPRA questionnaire)

• HIV risk behavior

• Exposure to services (i.e., dosage)

The client surveys will provide the data necessary to conduct a complete outcome evaluation. The following paragraphs present a description of the client survey.

Client Survey: (See Attachment 1a - Documents 2 & 3, Attachment 1b - Document 1.) The goals of the client survey are to collect information that will document client HIV risk behavior and client conditions. This includes measuring variables that impact a client’s situation. For example, social connectedness measures a client’s family and living arrangements, as well as use of recovery services. In addition, clients can be impacted by their quality of life (e.g., educational status, employment status, criminal justice involvement, trauma exposure), which will also be assessed.

The targeted audience for the TCE-HIV evaluation client surveys is clients in a treatment program administered by the 40 Grantee treatment programs. The client surveys will be administered by Grantee program counselors. All clients who received the initial baseline survey (see Attachment 1a - Document 2) will be asked to complete a follow-up survey at discharge (see Attachment 1a -Document 3), and at 6 months post baseline (see Attachment 1b - Document 1). Data collection at the two follow-up points is necessary to measure the short- and longer-term client outcomes associated with Grantee treatment approaches, which is one of the primary objectives of the TCE-HIV Multi-Site Evaluation.

The client survey was developed from a combination of new questions (i.e., Sections A–C, Section E13–E25) and from subscales of existing measures (i.e., Sections D, E1–E12 and Section F). Each subscale was carefully selected to assess the construct of interest yet maintain its ability to be a stand-alone subscale that can be scored separately from the entire measure.

Section D (items D1–D9) on the client survey is from the Social Support subscale questions from the Texas Christian University Client Evaluation of Self and Treatment (TCU CEST). Questions for Section E1–E12 are the depression and anxiety symptom dimensions from the Brief Symptom Inventory (BSI). Section F contains the Taking Steps subscale questions from the Stages of Change Readiness and Treatment Eagerness Scale (SOCRATES). A general description of these existing measures and the reliability and validity information for each subscale can be found below.

• Texas Christian University Client Evaluation of Self and Treatment (TCU CEST) (Texas Christian University, 2005). The Client Evaluation of Self and Treatment (TCU CEST) was developed as part of NIDA Grant R37 DA13093, Transferring Drug Abuse Treatment and Assessment Resources. The TCU CEST includes a set of assessments that “target” specific needs and status of clients in different stages of change during treatment. The coefficient alpha for the Social Support Scale is .84.

• Brief Symptom Inventory (BSI) (Derogatis, 1993). The purpose of this measure is to identify self-reported, clinically relevant psychological symptoms in adults. Internal consistency estimates for the two subscales are .85 (depression) and .81 (anxiety). Good internal consistency reliability is supported by several other independent studies (Croog, Levine, Testa, & Brown, 1986; Derogatis & Coons, 1993).

• Stages of Change Readiness and Treatment Eagerness Scale (SOCRATES) (Miller & Tonigan, 1997). Psychometric analyses reveal the following psychometric characteristics of the Taking Steps scale questions:

• Cronbach Alpha = .83 – .96

• Intraclass Test-Retest Reliability = .91

• Pearson Test-Retest Reliability = .93

Client Dosage Form: (See Attachment 1b - Document 2.) The goal of this form is to collect data regarding the type and amount of contact that a client has with the program. Individual clients in the TCE-HIV program will likely have very different types and amounts of contact (due to absences, participation in different components, or dropping out), thus it is critical to have dosage information to accurately assess program effects. The client dosage form will be completed by the Grantee counselor at client discharge.

Data collected via semi-structured interviews, dosage forms, client focus groups, and surveys will be used to enable SAMHSA/CSAT’s TCE-HIV program to increase its effectiveness in providing substance abuse treatment and HIV services for minority populations. Additionally, the Multi-Site Evaluation will help SAMHSA/CSAT achieve the goals of its TCE-HIV program and the Minority AIDS Initiative (MAI). Specifically findings of the evaluation will:

• Determine the effect of the TCE-HIV services programs

• Identify the extent to which the programs have achieved the goals of the MAI

• Document TCE-HIV models and ascertain which models contribute to significant reduction in risk behaviors among drug-using populations.

3. Use of Information Technology

Data collection for the outcome evaluation component requires the use of information technology to develop an online data collection system. This online system will provide the three client surveys (i.e., baseline, discharge, 6 months post baseline) for data entry by Grantee staff. Additionally, the client dosage form, which is completed through a review of clients’ records by Grantee staff at client discharge, will also be available through the online data collection system. Technology will be used to manage, secure, and store the data to ensure data management control.

Client Surveys: Grantee staff will administer the client survey during a face-to-face encounter at baseline, at discharge, and at 6 months post baseline. Client survey data will be entered into an online system at the Grantee site that will be password protected. Each day, the TCE-HIV evaluation staff will upload the data over a secure network connection directly to a server at JBS headquarters where the data will also be encrypted and password protected. Details on JBS’ network security procedures and security protocols are presented in Attachment 2a - Document 1. Using protected electronic data is the most secure form of data management because it eliminates the possibility of paper documents being lost by the survey staff or of data being lost in transit or delivered to an incorrect location. However, not all the Grantees may be equipped to enter this data into the online system, in which case paper copies of the completed surveys from these Grantees will be submitted to JBS only with unique alphanumeric identifiers and the data will be entered into the online system at JBS. Paper copies will be stored in a locked file cabinet, with no name/identifying information attached.

Client Dosage Form: Grantee staff will review client treatment records and complete the client dosage form for each client upon his or her discharge from the program. Similar to the client survey, password-protected dosage data will be entered into an online system at the Grantee site. On a daily basis, the TCE-HIV evaluation staff will upload these data over a secure network connection directly to a server at JBS headquarters where they will also be encrypted and password protected. In the event that Grantees are not equipped to enter these data into the online system, paper copies of the completed dosage forms from these Grantees will be submitted to JBS with only unique alphanumeric identification and the data will be entered at JBS. All paper copies will be stored in a locked file cabinet, with no name/identifying information attached.

4. Effort to Identify Duplication

The Multi-Site Evaluation Team has identified items answered through GPRA data collection that are of interest to the TCE-HIV evaluation. To ensure minimal duplication, the Multi-Site Evaluation team has eliminated these items (e.g., demographic information, substance use information) from our Multi-Site Evaluation instruments. This action was taken to reduce Grantees’ burden. This action, however, was taken with the understanding that SAMHSA/CSAT will provide the contractor for the Multi-Site Evaluation with access to all raw TCE-HIV Grantee GPRA data at the three data collection points (i.e., baseline, discharge, 6 months post baseline). It is critical to the TCE-HIV evaluation to have access to raw individual GPRA data as these items include:

• Client demographics

• Client substance use/abuse

• Family living conditions

• Education, employment, and income

• Crime and criminal justice status

• Mental and physical health

• Social connectedness.

Without access to the raw GPRA data for these measures, the contractor for the Multi-Site Evaluation will be unable to provide SAMHSA/CSAT with answers to TCE-HIV evaluation questions.

The GPRA data will be made available for the purposes of this evaluation, as a key and sole measure of client substance use/abuse as well as a supplement to other measures of the client survey data. Because the GPRA data represents a critical client data source for substance use/abuse data, the client survey has been created to supplement the GPRA questionnaire by including additional measures (i.e., not including substance use/abuse items) necessary for evaluating the full spectrum of client outcomes and the moderators and mediators of those outcomes. Grantees will be asked to use the client GPRA identification number on the client survey at the three data collection points, so that the client surveys can be linked to the GPRA data, which contains demographics, 30-day substance use, and other constructs (e.g., social connectedness, family living conditions, employment) that are needed to answer the evaluation questions. Due to the availability of the GPRA data, the client surveys were designed to avoid any duplication. To aid in this process, discussions with Grantee staff were conducted to verify the kind of data being collected.

5. Involvement of Small Entities

It is likely that many of the project’s respondents will be from relatively small treatment programs. Information collection for this study is not anticipated to have a significant impact on the individuals or on the programs or practices with which respondents may be affiliated.

The information to be obtained from respondents is the minimum necessary to achieve the objectives of the evaluation; however, completion of survey instruments, participation in focus groups, and/or participation in semi-structured interviews will likely induce burden. To reduce this burden, every attempt will be made to move respondents quickly through questions. For example, a screener question will be used in the client survey that asks about previous testing with HIV positive results. If the respondent indicates that he/she has not previously tested HIV positive, the interviewer will skip to the next category.

6. Consequences if Information Is Collected Less Frequently

During this 5-year multi-site evaluation, the frequency of data collection from the Grantees and the clients they serve is held to the minimum necessary to meet the needs of the evaluation objectives. The data collection points for the process evaluation component are generally acceptable intervals (i.e., in year 2 and year 4) for assessing changes in the community- and program-level constructs that will be assessed. Similarly, the outcome data collection points for the client survey are at acceptable intervals (i.e., baseline, discharge, 6 months post baseline) to coincide with GPRA data collection. The single data collection point for the client dosage form at discharge is ideal for assessing client exposure to services while in treatment.

Client Surveys: Client surveys will be administered to each client receiving treatment from the 40 treatment Grantees. All clients completing the survey at baseline will also complete the survey at discharge from treatment, and 6 months following baseline to mirror the GPRA questionnaire administration. Data collection at discharge and 6 months following baseline is necessary to measure the short- and longer-term outcomes of the TCE-HIV program.

Data collection at discharge is necessary to measure short-term outcomes related to treatment services provided by TCE-HIV Grantees. These outcomes include: changes in HIV risk behaviors, social support, and quality of life. Following up at 6 months post baseline is optimal for producing useful outcome data. Waiting until 6 months after baseline from treatment services allows enough time for any longer-term effects of TCE-HIV services including changes in risk behaviors and/or secondary outcomes, such as education and housing status, criminal involvement, arrests, and other health care utilization. Alternatively, waiting more than 6 months jeopardizes the validity of the data collected. As time passes, self-reported data become less accurate. Moreover, follow-up response rates, especially among much of the population to which TCE-HIV is being delivered, decrease over time.

Semi-Structured Interviews: The semi-structured interviews will be administered to staff and partners/collaborators from each of the 48 TCE-HIV Grantees during site visits conducted in Year 2. To determine the effect of TCE-HIV services, evaluation staff will compare baseline measures with the same indicators approximately 24 months (Year 4) after the initial measurement. Comparisons between these time points will yield information on community, program, and staff changes as measured at Year 2 and Year 4. Collecting follow-up data at 24 months after the Year 2 site visit is optimal for producing useful comparison data, as community and programmatic level changes are likely to be manifested within the 24 month cycle. Alternatively, if the information is collected after more than 24 months, the resultant shortened period for data analysis may result in a lack of important information for SAMHSA/CSAT on how best to understand the community and contextual conditions in which the TCE-HIV programs exist and provide services.

Client Focus Groups: Focus groups will be conducted during site visits in Year 2 and Year 4 with a subsample of treatment clients at each of the 40 treatment Grantees. The goals of the client focus groups are to gather information about client substance abuse treatment history, client behavior (i.e., substance use/abuse, risk behavior, quality of life), client perception of social support, and client satisfaction with the TCE-HIV program.

No technical or legal barriers to reduce burden exist if information is collected less frequently.

7. Consistency with the Guidelines in 5 CFR 1320.5(d)(2)

This information collection fully complies with the guidelines in 5 CFR 1320.5(d)(2).

8. Consultation Outside the Agency

The notice required by 5 CFR1320.8(d) was published in the Federal Register on October 23, 2009 (74 FR 54830-54831). No comments were received in response to this notice.

SAMHSA has made extensive use of experts in the area of substance abuse and HIV/AIDS research to provide guidance on the design of the Multi-Site Evaluation. An expert panel meeting was held in November 17–18, 2008 to review various aspects of the Multi-Site Evaluation, including the evaluation plan, data collection procedures, methods, and literature review. The list of experts who participated is provided in Exhibit 2.

Exhibit 2: Expert Panel Members

Expert	Affiliation	Contact Information
Faye Belgrave, PhD	Director, Center for Cultural Experiences in Prevention Virginia Commonwealth University College of Humanities and Sciences Department of Psychology 806 West Franklin Street Richmond, VA 23284-2018	Phone: 804-827-3908 E-mail: fzbelgra@vcu.edu
Benjamin Bowser, PhD	Interim Dean, California State University, East Bay 3103 Meiklejohn Hall Hayward, CA 94542	Phone: 510-885-3173 Fax: 510-885-2390 E-mail: benjamin.bowser@csueastbay.edu
Karen Corsi, ScD	University of Colorado Medical School––Project Safe 1741 Vine St. Denver, CO 80206	Phone: 303-315-0951 Fax: 303-316-7697 E-mail: karen.corsi@uchsc.edu
Chinazo Cunningham, MD	Clinical Faculty, Residency Program in Social Internal Medicine and Primary Care Montefiore Medical Center 111 East 210th Street Bronx, NY 10467	E-mail: ccunning@montefiore.org
Agatha Eke, PhD	Centers for Disease Control and Prevention 1600 Clifton Rd., NE Mailstop E-37 Atlanta, GA 30333	Phone: 404-639-1906 Fax: 404-639-1950 E-mail: aeke@cdc.gov
Jeffrey Friedman, MA	Long Island Association for AIDS Care 60 Adams Ave. Hauppauge, NY 11788	Phone: 631-385-2451 E-mail: jfriedman@liaac.com
Robert Fullilove, EdD	Associate Dean, Community and Minority Affairs Professor, Clinical Sociomedical Sciences Co-Director, Community Research Group. Columbia University Mailman School of Public Health 722 West 168th Street New York, NY 10032	Phone: 212-305-4734 E-mail: ref5@columbia.edu
Vincent Guilamo-Ramos, PhD	Columbia University, Associate Professor, School of Social Work 1255 Amsterdam Avenue New York, NY 10027	Phone: 212-851-1659 Fax: 212-851-2206 E-mail: rg650@columbia.edu
Lena Lundgren, PhD	Boston University Center for Addictions Research and Services 232 Bay State Road, 4th Floor Boston, Massachusetts 02215	Phone: 617-353-7222 Fax: 617-358-2368 E-mail: llundgre@bu.edu
Jacques Normand, PhD	Director AIDS Research Program National Institute on Drug Abuse (NIDA) 6001 Executive Boulevard Bethesda, MD 20892	Phone: 301-402-1919 E-mail: jnormand@nida.nih.gov
Sybil Ward, AA	Jefferson Comprehensive Care System 2020 W. 3rd Street, Suite 201 Little Rock, AR 72205	Phone: 501-663-7166 E-mail: sybilwardjccsi@comcast.net
Chyvette Williams, PhD	Assistant Professor Health and Policy Administration School of Public Health (MC 923) University of Illinois at Chicago 1603 W. Taylor Street Chicago, IL 60612	Phone: 312-355-5299 E-mail: chevy@uic.edu
Nickolas Zaller, PhD	The Miriam Hospital Immunology Center 164 Summit Ave RISE/CFAR Building Providence, RI 02096	Phone: 401-793-4875 E-mail: nzaller@lifespan.org

Additional consultation was sought after substantive work was completed on the TCE-HIV Multi-Site Evaluation design. A smaller Project Working Group (PWG) was convened to advise and provide recommendations to SAMHSA/CSAT and the JBS Evaluation Team on all phases of the TCE-HIV Multi-Site Evaluation including the design, instrumentation development, implementation, and analysis. As a result of PWG consultation and feedback, data collection instruments have been revised. See Attachment 2a - Document 2 for summary of PWG feedback and revisions made to instruments. The scope and purpose of the PWG will evolve in response to the changing needs of the project as the evaluation is implemented and the JBS Team engages in data collection, management, and analysis. The PWG will meet frequently (i.e., quarterly) during the initial phase of the evaluation project and will meet as needed during the implementation, analysis, and reporting phases.

The PWG is an interdisciplinary group of Federal staff, the JBS Team, and external consultants (i.e., evaluators and researchers) who can provide a broad perspective on the various evaluation-related tasks. The PWG members have expertise in substance abuse treatment and prevention issues, HIV testing and prevention, program implementation, minority populations, and program evaluation. The list of PWG members is provided in Exhibit 3.

Exhibit 3: Project Working Group Members

Member	Affiliation	Contact Information
Ravinia Hayes-Cozier	National Minority AIDS Council Director of Government Relations and Public Policy 1931 13th St. NW Washington, DC 20009-4432	Phone: 202-483-6622 ext. 308 Fax: 202-483-1135 E-mail: rhayescozier@nmac.org
Jim Derzon, PhD	Battelle Memorial Institute Task Lead for Quantitative Analysis 2101 Wilson Boulevard Suite 800 Arlington, VA 22201-3008	Phone: 703-248-1640 Fax: 703-527-5640 E-mail: Derzonj@battelle.org
Colin Flynn	Department of Health and Mental Hygiene Chief, Center for HIV Surveillance and Epidemiology 500 North Calvert St. Shillman Building, 5th Floor Baltimore, MD 21202	Phone: 410-767-5050 Fax: 410-767-6489 E-mail: flynnc@dhmh.state.md.us
Vincent Guilamo-Ramos, PhD	Columbia University, Associate Professor, School of Social Work 1255 Amsterdam Avenue New York, NY 10027	Phone: 212-851-1659 Fax: 212-851-2206 E-mail: rg650@columbia.edu
Susan Hayashi, PhD	JBS International, Inc. Vice President 5515 Security Lane, Suite 800 North Bethesda, MD 20852-5007	Phone: 301-495-1080 ext. 4588 Fax: 301-587-4352 E-mail: shayashi@jbsinternational.com
Warren Hewitt, MS	SAMHSA/CSAT AIDS Coordinator 1 Choke Cherry Rd Rockville, MD 20857	Phone: 240-276-1616 Fax: 240-276-1670 E-mail: Warren.hewitt@samhsa.hhs.gov
Kevin Hylton, PhD	Alliances for Quality Education, Inc. TCE-HIV Multi-Site Evaluation Deputy Project Director 8181 Professional Place, Suite 110 Landover, Maryland 20785	Phone: 301-583-8424 Fax: 301-583-8422 E-mail: khylton@aqe-inc.com
Jennifer Kasten, PhD	JBS International, Inc. TCE-HIV Multi-Site Evaluation Lead 5515 Security Lane, Suite 800 North Bethesda, MD 20852-5007	Phone: 240-645-4145 Fax: 301-495-1080 E-mail: jkasten@jbsinternational.com
Resa Matthew, PhD	JBS International, Inc. TCE-HIV Multi-Site Evaluation Project Director 5515 Security Lane, Suite 800 North Bethesda, MD 20852-5007	Phone: 240-645-4608 Fax: 301-495-1080 E-mail: rmatthew@jbsinternational.com
Naomi Tomoyasu, PhD	SAMHSA/ CSAT Branch Chief 1 Choke Cherry Rd Rockville, MD 20857	Phone: 240-276-1613 E-mail: Naomi.tomoyasu@samhsa.hhs.gov
Willie Tompkins, PhD	SAMHSA/CSAT Government Project Officer 1 Choke Cherry Rd Rockville, MD 20857	Phone: 240-276-2899 E-mail: willie.tompkins@samhsa.hhs.gov
Ping Yu, PhD	Battelle Memorial Institute TCE-HIV Multi-Site Evaluation Subcontract Director/Co-PI 2101 Wilson Boulevard Suite 800 Arlington, VA 22201-3008	Phone: 703-875-2981 Fax: 703-527-5640 E-mail: Yup@battelle.org

Additional consultation was sought from TCE-HIV Grantees and representative respondents during pilot site visits conducted October–December 2009. The visits provided the Multi-Site Evaluation team with feedback and comments regarding data collection instruments and processes. As a result of the consultation and feedback, data collection instruments have been revised. See Attachment 4 - Document 1 for a summary of pilot site visit feedback and revisions made to instruments. The list of TCE-HIV Grantee consultants is provided in Exhibit 4.

Exhibit 4: TCE-HIV Grantee Consultants

Consultant	Affiliation	Contact Information
Carla Hewitt	Safe Haven 1140 North Capital St, NW Washington, DC 20018	Phone: 202-589-1505 E-mail: chewitt@safehaven.org
Basha Silverman, MA	Brandywine Counseling 2713 Lancaster Avenue Wilmington, DE 19805	Phone: 302-656-2348 Fax: 302-656-0746 E-mail: bsilverman@gmail.com
Jim May, PhD	Richmond Behavioral Health Authority 107 South Street Richmond, VA 23219	Phone: 804-819-4202 Fax: 804-819-8783 E-mail: jmay@rbha.org

9. Payment to Respondents

Client Surveys: Cash equivalent incentives will be offered in lieu of cash payments. Gift cards from major stores (e.g., Walmart™, Target™) will be used as an incentive for treatment clients to complete the client-level survey at baseline, discharge, and 6 months post baseline. Survey research literature suggests that monetary incentives increase response rate, with no known adverse effect on reliability (Dillman, 2000). The use of incentives to increase response rate is particularly important when collecting data from hard to reach populations who are providing information of a sensitive nature (e.g., sexual risk behaviors). Incentives will be provided to clients in a tiered structure with a $10 gift card provided at baseline, $15 gift card at discharge, and $20 gift card at 6 months post baseline.

Client Focus Groups: Gift cards ($20) from major stores (e.g., Walmart™, Target™) will be used as an incentive for treatment clients to participate in the focus groups. Although participation in the focus groups is voluntary, respondents are likely to perceive a time cost and burden associated with their participation. Clients will be offered a $20 gift card for their participation in a focus group. Pilot tests were conducted with representative participants to determine the appropriateness of the incentive.

10. Assurance of Confidentiality

JBS’ Institutional Review Board (IRB) application has been approved which ensures the Multi-Site Evaluation Team meet corporate, industry, and society standards to protect study participants. This approval ensures compliance with the spirit and the letter of regulations from the Department of Health and Human Services (DHHS) governing such projects. JBS’ systems and procedures for collecting and processing data are designed to help ensure the privacy, to the extent of the law, of study participants and the data they provide. Documents with data about Grantees or individual clients will be identified by an assigned study alpha numeric identification number.

Grantees have been approved to administer the GPRA, thus informed consent is in place for Grantee staff to collect GPRA questionnaire data. For this evaluation, Grantee staff will provide an Information Sheet which they will read/present to the client as part of the survey administration. The Information Sheet provides pertinent information about the administration, risks, and benefits of the client survey. However, it is advised that Grantees also follow local and/or state requirements related to client informed consent. After completing their own data collection (i.e., GPRA), Grantee staff who deliver TCE-HIV services will then briefly explain to the client the reason for an additional survey, describe the survey length, and explain the process. The process for administering the client survey is designed to protect privacy, reduce client discomfort and burden, and ensure the collection of quality data.

The client survey will include the OMB approval expiration dates, the statement of survey burden, and the statement that the evaluation is federally sponsored. The Grantee staff will administer the survey to the client in a private location (e.g., an office) to ensure privacy. Staff will read each question and the list of responses for each question to clients and record their answers.

Client survey data will be entered at the Grantee site into a password-protected online system. Each day, the TCE-HIV evaluation staff will upload the data over a secure network connection directly to a server at JBS headquarters where the data will also be encrypted and password protected. However, not all the Grantees may be equipped to enter this data into the online system, in which case paper copies of the surveys with only alpha numeric identifiers (i.e., GPRA identification numbers) from these Grantees will be provided and entered onsite at JBS. Any paper copies will be stored in a locked file cabinet, with no name/identifying information attached.

The TCE-HIV Multi-Site Evaluation staff will use passwords to safeguard project directories and analysis files containing completed survey data to ensure that there is no inadvertent disclosure of study data. The Multi-Site Evaluation staff also will be trained on handling sensitive data and the importance of privacy. As a further precautionary measure, the data being collected will have no identifying information that can be linked back to the client. In keeping with 45 CFR 46, Protection of Human Subjects (Attachment 2a - Document 3), the TCE-HIV procedures for data collection, consent, and data maintenance are formulated to protect respondents’ rights and the privacy of information collected.

Data from the TCE-HIV client surveys will be kept strictly private to the extent of the law in compliance with the Privacy Act of 1974 (5 U.S.C. 552a) (Attachment 2b - Document 1.) The privacy of data records will be explained to all respondents during the consent process and in the consent forms. Limits to privacy (e.g., notifying authorities to protect study participants or someone else from serious harm, including child abuse/neglect) will be explained and included in the consent form.

For clients providing information in the client focus groups, the responses will be kept private; that is, no identifying information will be linked with the information provided and the information will be reported in aggregate. This de-identification process will extend to semi-structured interviews completed by Project Directors, Grantee staff, and community/partner collaborators. Any direct quotes that are used in reporting will not be attributed directly to the speaker, but will be credited only as a focus group participant comment.

11. Questions of a Sensitive Nature

Client Survey and Client Focus Group: The TCE-HIV multi-site client surveys and focus groups, by necessity, will collect sensitive information (e.g., substance use, mental health, and other health and social risk factors) because these are all outcomes of interest to SAMHSA/CSAT. Sensitive information of this nature is always regarded as highly private and clients’ privacy in federally assisted treatment programs is assured through strict adherence to the statute (42 U.S.C. §290dd.2) and regulation (42 C.F.R. Part 2) regarding confidentiality of Alcohol and Drug Abuse Patient Records. The application for a Certificate of Confidentiality is in progress and will be submitted to the appropriate DHHS agency. This certificate issued by SAMHSA and other DHHS agencies is designed to protect identifiable research information from forced or compelled exposure. The Certificate of Confidentiality protects the investigators from being forced, even under a court order or subpoena, to release information that could identify survey or focus group participants. However, the evaluation team may release identifying information in some circumstances. For example, the team may disclose medical information in cases of medical necessity, or take steps (including notifying authorities) to protect participants or someone else from serious harm, including child abuse/neglect. The Multi-Site Evaluation team and Grantee IRBs will also ensure that additional appropriate mechanisms and procedures are in place to protect the privacy of the identifiable information to be obtained in the evaluation.

Respondents will be informed about the purpose of the data collection and that responding to all questions is voluntary. They will be assured that they may stop taking the survey at any time. In addition, specific assurances will be provided to respondents concerning the safety and protection of data collected from them.

Administrative Staff Semi-Structured Interviews: No sensitive information will be collected from those individuals who perform administrative tasks related to the TCE-HIV program (e.g., Project Director, Program Manager, and Executive Director). The interview staff of the Multi-Site Evaluation team will obtain signed consent for participation in the interview data collection. Respondents will be informed about the purpose of the data collection and that responding to all interview questions is voluntary. In addition, specific assurances will be provided to respondents concerning the safety and protection of data collected from them.

Direct Services Staff Semi-Structured Interviews: No sensitive information will be collected from direct services staff members (e.g., outreach workers, counselors). The interview staff of the Multi-Site Evaluation team will obtain signed consent for participation in the interview data collection. Respondents will be informed about the purpose of the data collection and that responding to all interview questions is voluntary. In addition, specific assurances will be provided to respondents concerning the safety and protection of data collected from them.

Partner/Collaborator Semi-Structured Interviews: No sensitive information will be collected from partner/collaborator (e.g., agencies or organizations that provide services and activities related to the TCE-HIV program). The interview staff of the Multi-Site Evaluation team will obtain signed consent for participation in the interview data collection. Respondents will be informed about the purpose of the data collection and that responding to all interview questions is voluntary. In addition, specific assurances will be provided to respondents concerning the safety and protection of data collected from them.

12. Estimates of Total Hour Burden

Estimate of the Total Hour Burden of the Collection of Information from Clients. The total client sample size for the TCE Multi-site data collection effort is estimated to be a maximum of 4,800 adult respondents (e.g., aged 18 and over). The baseline survey is expected to have a response rate of 100 percent, resulting in 4,800 respondents completing the baseline survey. The discharge survey is expected to have a response rate of 100 percent resulting in 4,800 respondents completing the discharge survey. The set goal response rate for the 6-month post-baseline survey is 80 percent of the baseline sample, resulting in 3,840 respondents completing the 6 month post baseline survey. The Year 2 and Year 4 treatment focus groups are each expected to have a response rate of 80 percent (i.e., not every Grantee will be able to provide nine clients who are willing to participate), resulting in a total of 720 respondents participating in focus groups. Based on these response rates, it is expected that client total responses will be 14,160. Exhibit 5 presents estimates of total burden and Exhibit 5a presents estimates of annualized burden based on pilot testing.

The hour burden for the client surveys (i.e., baseline, discharge, 6 month post baseline) is calculated using the average completion time based on survey pilot testing (see Attachment 4 - Document 1). The time required to complete the surveys varies with client characteristics, in particular, substance use behaviors. Based on pilot testing, the total time to complete the client survey––including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information––was found to be 25 minutes. Exhibit 5 presents estimates of total burden based on pilot testing.

Exhibit 5. Multi-site Data Collection Burden for Clients, Grantee Staff, and Collaborators

Instrument/Activity	Number of Respondents	Responses per Respondent	Total Responses	Hours per Response	Total Burden Hours	Hourly Wage	Total Respondent Costa
Baseline data collection (clients)	4,800	1	4,800	.42	2,016	$20.32	$40,965.12
Discharge data (clients)		1	4,800	.42	2,016	$20.32	$40,965.12
6 months post baseline data collection (clients)		1	3,840	.42	1,613	$20.32	$32,776.16
Treatment focus group Year 2 (Client)		1	360	1.0	360	$20.32	$7,315.21
Treatment Focus Group Year 4 (client)		1	360	1.0	360	$20.32	$7,315.21
Client Subtotal	4,800		14,160		6,365		$129,336.82
Annualized Client Total	1,600	--	4,720	--	2,122
Project Director/Program Manager (Semi-Structured Interviews)	96	2	192	.75	144	$29.12	$4,193.28
Annualized PD/PM Total	32	--	64	--	48
Grantee Direct Services Staff (Semi-Structured Interviews)	432	2	864	1.0	864	$19.05	$16,459.20
Annualized Service Staff Total	144	--	288	--	288
Treatment Dosage Form (Completed by program staff)	4,800	1	4,800	.25	1,200	$19.05	$22,860.00
Annualized Dosage Total	1,600	--	1,600	--	400
Community Collaborators (Semi-Structured Interviews)	240	2	480	1.0	480	$19.21	$9,220.80
Annualized Collaborators Total	80	--	160	--	160
TOTAL	10,368		20,496		9,053		$182,070.10
Annualized Totals (3-year clearance for project)	3,456	--	6,832	--	3,018	--	--

^aTotal respondent cost is calculated as hourly wage × time spent on survey × number of respondents.

Estimate of the Total Hour Burden of the Collection of Information from Project Directors/Program Managers. The total project director/program manager sample size for the TCE multi-site data collection effort is estimated to be a maximum of 96 respondents (48 sites, 2 respondents at each site). Exhibit 5 presents estimates of total burden and Exhibit 5a presents estimates of annualized burden based on pilot testing.

Estimate of the Total hour Burden of the Collection of Information from Grantee Direct Services Staff. The total grantee staff sample size for the TCE multi-site qualitative data collection effort is estimated to be a maximum of 432 respondents (48 sites, 9 respondents in each site). Exhibit 5 presents estimates of total burden and Exhibit 5a presents estimates of annualized burden based on pilot testing.

The client treatment dosage form which is completed by Grantee staff is expected to have a response rate of 100 percent, resulting in the treatment dosage forms being completed at discharge for 4,800 treatment clients with the total time to complete the dosage form being 15 minutes. Exhibit 5 presents estimates of total burden and Exhibit 5a presents estimates of annualized burden based on pilot testing.

Estimate of the Total Hour Burden of the Collection of Information from Community Collaborators. The total community collaborator sample size for the TCE-HIV multi-site data collection effort is estimated to be a maximum of 240 respondents (48 sites, 5 respondents in each site). Exhibit 5 presents estimates of total burden and Exhibit 5a presents estimates of annualized burden based on pilot testing.

Estimate of the Total Cost Burden to the Respondents for the Collection of Information from Clients. There are no direct costs to respondents other than their time to participate in the study. The total cost for the evaluation of the time respondents spend completing the surveys and participating in focus groups is $129,336.82 (number of total baseline client respondent hours, plus discharge and 6-month post baseline hours, plus treatment focus group client respondent hours × $20.32), the estimated average hourly wages for adults as published by the U. S. Bureau of Labor Statistics, 2008).

Estimate of the Total Cost Burden to the Respondents for the Collection of Information from Project Directors/Program Managers. There are no direct costs to respondents other than their time to participate in the study. The total cost of the time respondents spend completing these interviews is $4,193.28 (number of project director respondent hours × $29.12, the estimated average hourly wages for individuals working in health-related occupations as published by the U.S. Bureau of Labor Statistics, 2008).

Estimate the Total Cost Burden to the Respondents for the Collection of Information from Grantee Direct Services Staff. There are no direct costs to respondents other than their time to participate in the study. The total cost of the time respondents spend participating in the semi-structured interviews and completing client treatment dosage forms is $39,319.20 (number of grantee staff respondent hours × $19.05, the estimated average hourly wages for individuals working in health-related staff occupations as published by the U. S. Bureau of Labor Statistics, 2008).

Estimate the Total Cost Burden to the Respondents for the Collection of Information from Community Collaborators. There are no direct costs to respondents other than their time to participate in the study. The total cost of the time respondents spend participating in these interviews is $9,240.00 (number of practitioner respondent hours × $19.21, the estimated average hourly wages for individuals working in health services-related occupations as published by the U. S. Bureau of Labor Statistics, 2008).

Exhibit 5a. Annualized Summary Table

Respondents	Number of Respondents	Responses/Respondent	Total Responses	Hours per Response	Total Burden Hours
CLIENT DATA COLLECTION INSTRUMENTS
Clients-Baseline, discharge, 6-Months data collection	4,800	3	13,440^	.42	5,645
Annualized Client Survey Total	1,600	--	4,480	--	1882
Client Focus Groups Year 2 and Year 4	720*	1	720	1.0	720
Annualized Client FG Total	240	--	240	--	240
ADMINISTRATOR INTERVIEW INSTRUMENTS
Project Director/Program Manager (Semi-Structured Interviews)	96	2	192	.75	144
Annualized PD/PM Total	32	--	64	--	48
DIRECT SERVICE STAFF INSTRUMENTS
Grantee Direct Services Staff (Semi-Structured Interviews)	432	2	864	1.0	864
Annualized Service Staff Total	144	--	288	--	288
Treatment Dosage Form (Completed by program staff)	4,800	1	4,800	.25	1,200
Annualized Dosage Total	1,600	--	1,600	--	400
COLLABORATORS/PARTNERS INTERVIEW INSTRUMENTS
Community Collaborators (Semi-Structured Interviews)	240	2	480	1.0	480
Annualized Collaborators Total	80	--	160	--	160
Annualized Totals (3-year clearance for project)	3,456	--	6,832	--	3,018

^ This number is derived from 4800+4800+3840 = 13,440 for 100% response rate at two data collection time points and 80% at the third data collection time point.

*These respondents are a subset of the 4,800 clients so they are not included in the total number of respondents.

13. Estimates of Annualized Cost Burden to Respondents

There are no respondent costs for capital or start-up or for operation or maintenance.

14. Estimates of Annualized Cost to the Government

The annualized cost for the project per year is $2,506,000. These costs cover all aspects of meetings and logistics, evaluation design, testing, data collection, site visits, information technology, analysis, and reporting. In addition, it is estimated that one full-time SAMHSA/CSAT staff member will spend 25 percent of his or her time (520 hours) to manage and administer the project. Assuming an annual salary of $100,000, government personnel costs will be $125,000 over a 5-year period. Thus, total annualized project costs are $2,531,000.

15. Changes in Burden

This is a new data collection.

16. Time Schedule, Publications, and Analysis Plan

Time Schedule. Exhibit 6 outlines the key time points for the study and for the collection of information. The requested period also allows for training and start-up activities associated with the preparation required for data collection.

Exhibit 6. Time Schedule for Entire Project

Activity	Time Schedule
Obtaining OMB approval for data collection	October 2010
Site visit data collection (Year 2)	3 months post OMB approval for 4 months
Baseline client-level survey data collection	3 months post OMB approval for 28 months
Discharge client-level survey data collection (includes client dosage form)	4–8 months after OMB approval at client discharge (discharge will vary based on scheduled treatment duration)
Six-month post baseline client-level survey data collection	10–14 months after OMB approval
Site visit data collection (Year 4)	27 months after OMB approval for 4 months
Data analysis	Beginning one year post OMB approval
Dissemination of findings Interim reports, manuscripts, final report	Beginning 18 months post OMB approval through 2013

Publications. The TCE-HIV Multi-Site Evaluation is designed to determine the process and outcome of TCE-HIV services’ programs, identify the extent to which the programs have achieved the goals of the MAI, and document TCE-HIV models and ascertain which models contribute to significant reduction in risk behaviors among drug-using populations. It is therefore important to prepare and disseminate reports, concept papers, journal articles, and oral presentations that clearly and concisely present project results so that they can be appreciated by both technical and nontechnical audiences. The TCE-HIV Multi-Site Evaluation Team will:

• Produce rapid-turnaround analysis papers, briefs, and reports

• Prepare and submit monthly progress reports and a final TCE-HIV Multi-site Evaluation report

• Prepare a final multi-site findings report, including an executive summary

• Deliver presentations at professional and federally sponsored conventions and meetings

• Prepare and submit articles for publication in peer-reviewed journals

• Disseminate reports and materials to entities inside and outside SAMHSA/CSAT.

Analysis Plan. Process and outcome data collection require different measurement, collection, and analytical strategies. Both quantitative and qualitative data will be analyzed to assess outcomes as well as processes of the TCE-HIV programs. The planned approach is to use state-of-the-art statistical methods whenever appropriate to analyze the qualitative and quantitative data collected. Qualitative data analysis will utilize a thematic approach to uncover underlying themes. The three types of quantitative analyses that will be used are descriptive statistics, multivariate analysis, and multilevel analysis. Finally, triangulation of methods (e.g., qualitative and quantitative data), when feasible, will be conducted to examine additional aspects of results that cannot be accomplished with individual methods. The qualitative and quantitative data analysis plan is explained in more detail in the remainder of this section.

Qualitative Data Analysis: Qualitative data for this project will come from semi-structured interviews, focus groups, and responses to open-ended survey questions. The information will be utilized to address specific process evaluation questions. To facilitate the systematic analysis of the interview data, which will be collected during Grantee and community interviews, ATLAS.ti (Version 6.0), a computer-assisted qualitative data analysis software package, will be utilized. Before the analyses begin, Multi-Site Evaluation staff will take part in a training workshop to improve their facility with the capabilities and updated features of the software program.

Semi-Structured Interviews: Interview data will be analyzed to identify themes extracted from material collected from Grantee administrators, direct service staff, and partners/collaborators about their respective programs and services. As a first step in the data analysis process, audiotapes of the interviews will be transcribed and cleaned to remove any respondent identifying information and all transcription mistakes; these transcriptions will be the data used for the study.

Client Focus Groups: Focus group data will be analyzed to determine common themes expressed by clients about their substance abuse treatment history, behavior (e.g., substance use/abuse, risk behavior, quality of life), perception of social support, and satisfaction with the TCE-HIV program. As noted for the semi-structured interview data analysis plan, audiotapes of the focus groups will be transcribed and cleaned to remove any respondent identifying information and all transcription mistakes. These transcriptions will be the data used for the study. After the data is prepared for analysis, an inductive content analysis will be conducted on qualitative data.

The initial step in the analysis process will be reading the raw data (i.e., original text) to discover underlying raw data themes. Raw data themes will then be grouped according to salient responses that correspond to the main questions of the study:

• What are the community and contextual conditions in which the TCE-HIV programs exist and provide services?

• How do TCE-HIV Grantee characteristics facilitate or impede program activities, such as comprehensive services, service coordination/integration, and expanded organizational/ programmatic capacity?

• What impact do Grantee characteristics have on client-level outcomes?

• How do client characteristics (e.g., education, employment, income, family and living conditions, social support, satisfaction) lead to behavioral changes?

• What impact does the TCE-HIV program have on client-level outcomes?

The raw data themes will then be grouped into lower order themes based on common topics. Next, following the same coding procedures for grouping raw data themes, lower order themes will be grouped into higher order themes. Finally, higher order themes will be grouped into major categories. Consensus among researchers conducting the analyses will be reached at each step of the analytical process (i.e., raw data themes, lower order themes, higher order themes, and major categories) before proceeding to the next step to achieve inter-coder reliability. This process ensures a consistent understanding and interpretation of the data.

Quantitative Data Analysis: TCE-HIV is a complex, multilevel, and multi-component program. It therefore calls for multivariate and multilevel analyses to understand the complex relationships among treatment program operations, individual characteristics, and broader community contextual and socio-cultural environmental influences and client outcomes. These analyses address the overall evaluation question of what impact TCE-HIV services have on client-level outcomes related to substance abuse, risky behaviors, HIV testing, social support, quality of life, attitudes toward treatment, and satisfaction with services.

Depending on the availability of the data and the number of waves of data available (data points) for each data element, the approach will be to apply the most appropriate multivariate or multilevel analyses to answer the overall outcome evaluation question and associated sub questions. These multivariate or multilevel analyses are referred to as “cross-site” or “main effect” analyses, as they may be based on pooled data from a portion of 40 Grantees providing treatment services or evidence appropriate for meta-analysis.

The Multi-Site Evaluation Team will use quantitative data analysis to examine both Grantee and client-level outcomes. The analysis of both of these levels is explained in more detail below.

Descriptive Statistics: The Multi-Site Evaluation Team will use descriptive statistics to describe the main features of Grantees, their clients, and the communities in which they provide services. Descriptive statistics will allow the reporting of community, Grantee, and client characteristics that will provide a detailed picture of the TCE-HIV program and its participants. A utility of descriptive analysis will be to understand the distribution of variables of interest. Frequencies will be run or the means and standard deviations of each variable will be calculated to examine the central tendency and deviation from the mean, in order to examine the skewness of distribution of the data and to correct the distribution of the data prior to conducting further analyses. Descriptive statistics will also be used to conduct cross tabulations to examine the relationship between the variables. The degree of association between variables will be examined for statistical significance. Finally, descriptive statistics will be used to support higher-level analyses.

Grantee Outcomes: Grantee characteristics and their effect on program service administration and service delivery can be examined through multilevel analyses. The Grantee outcome evaluation will use information developed in the process evaluation to identify the services most likely to be affected by Grantee characteristics. Quantitative analyses of the Grantee characteristics will provide critical information about the services being provided and how these services change over time. Qualitative Grantee data will come from the administrative staff semi-structured interview (including the Grantee profile update), direct services staff semi-structured interviews, and partners/collaborators semi-structured interviews.

Client Outcomes: Client-level quantitative analyses will be primarily model-based and will be performed for both individual Grantee populations and the total population pooled across Grantees as appropriate. At the simplest level, regressions will be applied to assess reductions in client substance abuse outcomes over time for different number of sessions of substance abuse education (e.g., no sessions to two to three times per month), controlling for different client characteristics (e.g., age, sex, race, substance abuse treatment history, and HIV status). Specifically, a general linear regression model (GLM) can be developed to regress each of the 16 client substance abuse outcomes collected through the client-level survey on social support, while controlling for the effects of number of sessions of substance abuse education recorded on the client dosage form and client characteristics. In this case, if the estimate for the social support variable is negative and statistically significant, this finding can be interpreted as indicating that client substance use reduces over time as the client’s perceived social support increases, even after the effects of number of sessions of substance abuse education as well as client age, sex, race, substance abuse treatment history, and HIV status are considered.

A basic GLM could be specified as the following general equation:

Y_ijt = f ( + βPOST_t + TIME_it + CONTROL_ijt) + ε_ijt

where

Y_ijt is the outcome for individual i from Grantee j at time t;

f (∙) represents the linking function;

, β, , and  are coefficients to be estimated;

POST_t is an indicator variable that equals 1 if the data collection occurs at follow-up;

TIME_it is a vector of calendar time-related control variables (e.g., calendar year and indicators for quarters);

CONTROL_ijt is a vector of Grantee and client characteristics and a set of specific indicators that may affect the outcome, including client demographics and program characteristics; and

ε_ijt is the residual error term.

The primary coefficient of interest (β) measures the pre to post change in the outcome. That is, it quantifies the association between the treatment and the outcome.

17. Display of Expiration Date

OMB approval expiration dates will be displayed on the client surveys, participant data sheets, and consent forms.

18. Exceptions to Certification for Statement

There are no exceptions to the certification statement. The certifications are included in this submission.

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

This section presents information regarding the collection of data for the Multi-Site Evaluation. As noted previously, the Multi-Site Evaluation is theoretically grounded in an ecological systems approach which supports two interrelated evaluations: a process evaluation and an outcome evaluation. Data collected during the process evaluation will serve the critical role of establishing the overall evaluation’s context and consequently will aid in the interpretation of its findings. Process data will also provide information about the TCE-HIV services delivered, who delivered them, how they were delivered, to whom they were delivered, and how these services came to modify the Grantee’s overall substance abuse treatment service delivery system.

While process data will be collected from all 48 Grantees, outcome data will be collected from the 40 Grantees providing treatment services. Data collected during the outcome evaluation provide information regarding the effects of the TCE-HIV treatment interventions on Grantees and clients

1. Respondent Universe and Sampling Methods

Attachment 4 - Document 2 presents the subset and estimated universe of respondents for both evaluation data collection activities. For the process data collection, the semi-structured interviews will consist of a subset of program staff (e.g., program administrator, treatment counselor) and a subset of collaborators/partners. Client focus groups will consist of a representative subset of clients that are targeted by each of the 40 treatment programs.

The outcome data collection activities using the client survey will occur with the estimated universe of respondents (i.e., individuals entering substance abuse treatment programs) across the 40 substance abuse treatment programs, which is estimated to be 4,800 at baseline, 4,800 at discharge, and 3,840 at 6 months post baseline. It is expected that the recruitment methods coupled with incentives and intensive tracking mechanisms will result in response rates of 100 percent at baseline and discharge, and 80 percent at 6 months post baseline for the client survey administration. Additional outcome data employing the client dosage form will be collected by program staff from clients’ treatment records upon client discharge from treatment. The following presents a description of both process and outcome data collection methods.

Administrative Staff Semi-Structured Interviews: Individuals from each of the 48 Grantee sites who perform administrative tasks related to the TCE-HIV program (e.g., Project Director, Program Manager, and Executive Director) are eligible to be interviewed and their input will help document community and contextual factors. It is estimated that two administrative staff members per Grantee site will be interviewed in Year 2 and Year 4 of the TCE-HIV Multi-Site Evaluation. (See Attachment 3 - Document 1).

It is estimated that two administrative staff members per Grantee site will be interviewed in a two-part interview session. The first part of the interview will be conducted with an executive staff member of the agency; and the second part with the Project Director, Program Manager, or Program Coordinator of the TCE-HIV project. The determination was made to conduct a two-part interview with these two categories of staff because in some instances, these staff members have different perspectives on some of the key issues to be explored. In some instances, one person from the agency may fulfill both roles/positions. In this situation, the full interview will be conducted with that one person.

Direct Services Staff Semi-Structured Interviews: Individuals from the 48 Grantee organizations who have direct contact with clients to perform outreach/pretreatment and treatment-related tasks will be eligible to be interviewed. Examples of people performing direct service tasks include outreach workers, counselors, and case managers. It is estimated that up to nine direct service staff per Grantee site will participate in the interview session conducted during site visits in Year 2 and Year 4 (See Attachment 5 - Document 1).

Community Partners/Collaborators Semi-Structured Interviews: Grantee community partners/collaborators are those agencies or organizations that provide services and activities related to the TCE-HIV program. It is estimated that up to five community partners/collaborators per Grantee may participate in the interview sessions conducted during site visits in Year 2 and Year 4. Grantees may have more than five community partners, thus selection criteria has been developed to prioritize partners recruited to be interviewed. Selection criteria include: treatment source, housing, employment, re-entry criminal justice services, and medical services. These priorities may shift based on the nature of the Grantee program (e.g., for programs targeting HIV positive clients, interviews with partners providing medical care would be a main priority). (See Attachment 7 - Document 1).

Client Focus Groups: The targeted universe for the client focus groups are clients who have been in substance abuse treatment for at least 14 days. The Multi-Site Evaluation staff will work collaboratively with Grantee staff to identify client focus group participants. A member of the Grantee staff assigned to help with focus group recruitment will be asked for assistance in the distribution of flyers advertising the focus group, announcing the focus group to clients when appropriate, and serving as a primary point of contact regarding the client focus group. See Attachment 2b - Document 2 for an example of a focus group recruitment flyer and Attachment 2b – Document 3 for the client focus group sign-up roster. The focus groups will strive for a balance in participants with regard to gender, age, length and number of times in treatment, and serostatus. It is estimated that up to nine clients from each of the 40 treatment Grantees will participate in focus groups conducted during site visits in Year 2 and Year 4. (See Attachment 1a - Document 1).

Client Surveys: The targeted universe for the TCE-HIV Multi-Site Evaluation client surveys are clients from the 40 treatment Grantee programs. The clients who enter treatment following OMB clearance will be eligible to be surveyed. The first 120 clients from each of the 40 treatment Grantees who are administered the GPRA will be surveyed. The client-level surveys will be administered by trained Grantee program counselors when clients enter the treatment program. All of those clients receiving the baseline survey will complete a follow-up survey at discharge, and at 6 months post baseline. It is expected that the use of incentives and tracking mechanisms will result in an 80 percent response rate at 6 months post baseline. (See Attachment 1a - Documents 2 & 3, Attachment 1b - Document 1).

Client Dosage Forms: The targeted universe for the TCE-HIV Multi-Site Evaluation client dosage forms are the 120 clients from each of the 40 treatment Grantee programs who, following OMB clearance, have been administered the GPRA and client survey. The client dosage forms will be completed by trained Grantee program counselors when clients are discharged from treatment. It is expected that a 100 percent dosage form completion rate will be achieved because all clients will be discharged from a treatment program. (See Attachment 1b - Document 2).

2. Information Collection Procedures

In order to conduct the Multi-Site Evaluation, data will be collected using different methods for each of the two process data collection points (i.e., during Year 2 and Year 4 site visits), as well as the ongoing client survey and dosage form outcome data collection. Each data collection method proposed supports the type of evaluation questions being asked and the targeted respondents. The data collection process will be a careful and systematic mixed-method data collection approach, in order to gather high quality data from each of the 48 TCE-HIV Grantees. Attachment 4 - Document 2 presents the proposed estimated universe and selection methods for the project. Semi-structured interviews, focus groups, client surveys, and dosage forms will be used to collect data from the target population (e.g., Grantee staff, partners/collaborators, and clients). Data collection will occur across both process and outcome evaluation activities. A general description of key data collection procedures is provided below. A description of the site visit and data collection instruments is also provided.

Site Visits. In order to initiate the Year 2 site visits, a Multi-Site Evaluation team member will call the Grantee Project Director. See Attachment 4 - Document 3 for the site visit call script. The initial phone contact will be followed by an e-mail confirmation letter which provides additional details about the site visit. See Attachment 4-Document 4 for a sample site visit confirmation letter.

The Multi-Site Evaluation Team will conduct site visits at two data collection time points with each of the 48 Grantees. For the first round of site visits, a Multi-Site Evaluation team member will contact the Grantees to schedule the site visit. The second site visit will be conducted approximately 2 years following the first site visit (see Exhibit 6, A.16). Each site visit will be conducted by experienced members of the Multi-Site Evaluation Team. The site visits are anticipated to last 2 to 3 days, in order to gather all required data. The purpose of the site visits are to observe program processes, to conduct semi-structured interviews with project staff and community partners, to conduct client focus groups, and to provide technical assistance to Grantees on client survey administration and dosage form completion.

Semi-Structured Interviews: During each site visit, semi-structured interviews will be conducted with the following respondents: 1) Program Administrator, 2) Grantee direct services staff, and 3) community collaborators/partners. The semi-structured interview process will be led by a senior evaluation team member. The interviews may take place with a single person or a group of persons (e.g., more than one treatment staff member may be interviewed in the same session). The lead interviewer will describe the purpose of the interview, have respondent(s) review and sign the informed consent (Attachment 4 - Document 5, Attachments 6 & 8). The interviewer will use the appropriate interview guide (e.g., administrator, direct services staff, partners/collaborators) for conducting the interviews (see Attachment 3 - Document 1, Attachment 5 - Document 1, Attachment 7 - Document 1). The lead interviewer will guide the discussion, and then respondents will complete a participant demographic information sheet (Attachment 3 - Document 2, Attachment 5 – Document 2, Attachment 7 - Document 2). A note taker will record in detail the respondents’ statements. To provide additional documentation, and for quality assurance, interviews will also be digitally recorded with the permission of the interviewees. At the conclusion of the interview session, the lead interviewer will summarize the discussion. Debriefing sessions will be conducted with Grantee personnel at the conclusion of the site visits to summarize the visit and address any questions posed by Grantee staff about the visit. Follow-up telephone calls to Grantee staff may be conducted when necessary to further clarify information obtained during the visit.

Focus Groups: Clients from the 40 treatment Grantee sites will be recruited to participate in the in-person focus groups. A small group of clients (up to nine clients per focus group) will be invited to the focus groups discussion and guided by a moderator to address specific questions and discuss their experience in treatment. The Multi-Site Evaluation staff will work collaboratively with the Grantee staff to identify and recruit client participants based on criteria including age, gender, length and number of times in treatment, and serostatus.

The client focus groups will be led by a moderator who is a senior evaluation team member. An additional evaluation staff person will assist by attending to recording and note taking. The moderator will describe the purpose of the focus group, identify SAMHSA/CSAT as the sponsoring agency, explain all focus group procedures, assure privacy to the client participants, and request participation. In an effort to ensure that clients understand what is being asked of them, Grantee staff will read aloud the consent form to clients (see Attachment 2b - Document 4). After clients have agreed to participate, they will be asked to sign the consent form and will be given a copy for their records.

A scheduled time and place for the focus group will be established prior to the site visit. Focus groups will be scheduled at a time when a licensed clinician is on duty, in the event that disclosure of sensitive information by focus group participants causes them discomfort. After respondents agree to participate, the focus group will begin. The moderator will use a written guide (see Attachment 1a - Document 1) to conduct the focus group discussion. With the permission of the respondents, all focus groups will be audiotaped for later transcription and analysis. Following the discussion, the moderator will read aloud the client participant demographic sheet and brief program satisfaction questionnaire and will assist clients as they complete the sheet (see Attachment 1b - Document 3). Finally, the moderator will summarize the activities and comments at the end of the discussion. Incentives will be distributed after the focus group is completed.

Client Surveys and Client Dosage: A client survey and a client dosage form will be completed as part of the project evaluation activities. Upon OMB approval, the Multi-Site Evaluation Team will train Grantee staff via web/teleconference how to administer client surveys and complete client dosage forms. The team will develop a comprehensive question-by-question training session for Grantee staff with information regarding how to complete the survey and form along with the context for the collection of these data. The training will be developed as a PowerPoint presentation and will provide a step-by-step process for administration and completion of the survey and form (Attachments 2c & 2d).

• Client Survey: As described in Section A.6, the client survey will collect data from individuals at baseline, discharge, and 6 months post baseline. Data collection at the follow-up point is necessary to measure the short- and longer-term outcomes of the TCE-HIV programs implemented by the Grantees. Because measuring these outcomes is one of the primary objectives of the TCE-HIV evaluation, less frequent data collection would greatly compromise the integrity of the TCE-HIV evaluation. Grantee staff at each of the 40 Grantee treatment sites will administer the client survey to 120 clients admitted into treatment post OMB approval. The client survey will be administered in a private location (e.g., an office) in order to ensure privacy. It is expected that the client survey will be administered to 4,800 clients at baseline. It is then anticipated that the client survey will be administered to 4,800 clients at discharge, and to 3,840 at 6 months post baseline.

• Client Dosage Form: The client dosage form will be completed for treatment clients by a trained Grantee counselor at client discharge. Dosage is a measure of the type and amount of contact that a client has with the program. Individual clients in the TCE-HIV program will likely have very different types and amounts of contact (due to absences, participation in different components, or dropping out), thus it is critical to have dosage information to accurately assess program effects. It is expected that the client dosage form will be completed for 4,800 clients at discharge (120 clients at 40 Grantee treatment sites).

Client survey and dosage data will be entered into a password protected online system at the Grantee sites. Each day, the TCE-HIV evaluation staff will upload the data over a secure network connection directly to a server at the contractor’s headquarters where the data will also be encrypted and password protected. Details on the network security procedures and security protocols are presented in Attachment 2a – Document 1. Using protected electronic data is the most secure form of data management because it eliminates the possibility of paper documents being lost by the survey staff or of data being lost in transit or delivered to an incorrect location. However, not all the Grantees may be equipped to enter this data into the online system, in which case paper copies of the completed surveys from these Grantees will be submitted and these data entered at JBS by the Multi-Site Evaluation staff. No name/identifying information will be collected at any time and any paper copies will be stored in a locked file cabinet.

3. Methods to Maximize Response Rates

The ability to gain the cooperation of potential respondents is important to the success of this Multi-Site Evaluation.

Semi-Structured Interviews: The TCE-HIV Multi-Site Evaluation Team anticipates that the semi-structured interviews will be completed for 100 percent of the Grantee staff and community partners who are requested to provide information. To maximize participation rates, the TCE-HIV Evaluation Team interview staff will follow protocols that will reduce the burden on Grantee staff and community partners. Planning and preparation in advance of the interview is crucial for these protocols. The protocols include proper timing, scheduling, and location of the interviews to accommodate the Grantee staff and their partners. The Grantee staff and their community partners will be informed, in advance, of the purpose and significance of the interview.

Focus Groups: The TCE-HIV Multi-Site Evaluation Team anticipates over recruiting (12 clients) for each focus group. It is estimated that no more than 75% (n=9 clients) will participate in each focus group. Clients will be recruited based on specific criteria such as clients who have been administered the GPRA, have been in substance abuse treatment for at least 14 days and are willing to participate in the focus group.

The TCE-HIV Evaluation Team will work collaboratively with Grantee staff to recruit a representative group of clients for the focus group utilizing inclusion and exclusion criteria, to accomplish a balance of gender, age, length and number of times in treatment, and serostatus across focus groups. Each focus group will consist of seven to nine treatment clients who have been administered the GPRA and who are receiving TCE-HIV services. The TCE-HIV Evaluation Team will then work with project staff to provide them with information they can post and/or distribute to potential participants about the focus groups. As an added incentive to maximize response rates, gift cards ($20) will be provided and will be distributed after the focus group is completed.

Client Survey: The TCE-HIV Multi-Site Evaluation Team anticipates a 100 percent response rate for the baseline and discharge surveys, and 80 percent response rate for the 6 months post baseline survey. In order to achieve this rate, trained Grantee staff will administer the survey to the first 120 clients, admitted into treatment after OMB approval, who have been administered the GPRA and have an assigned GPRA identification number. As an additional measure, cash equivalent incentives will be offered in lieu of cash payments. Gift cards from Walmart™ or Target™ will be used as incentive for treatment clients to complete the client-level survey at baseline, discharge, and 6 months post baseline. Incentives will be provided to clients in a tiered structure with a $10 gift card provided at baseline, $15 gift card at discharge, and $20 gift card at 6-month post baseline. The evaluation team and the Grantee staff will employ several strategies to maintain high response rates:

• Stress the importance of the project as well as the evaluation team’s commitment to respondent privacy;

• Train survey staff for handling sensitive information collection in a respectful manner;

• Develop surveys in English and Spanish; and

• Ensure that Grantees have comprehensive tracking forms and procedures in place.

To improve follow-up response rates, the Grantee staff will collect detailed contact information, including alternate addresses and phone numbers and contact information of secondary sources who may know the respondents’ contact information at follow-up.

Client Dosage Form: The TCE-HIV Multi-Site Evaluation Team anticipates that the client dosage form will be completed for 100 percent of the 120 clients who are administered the GPRA and the client survey at baseline. Trained Grantee counselors will complete the client dosage form for clients at discharge from the treatment program. The client dosage form will be completed for 4,800 clients at discharge (120 clients at 40 Grantee treatment sites).

4. Tests of Procedures

Pilot tests of semi-structured interviews, client focus groups, client survey measurement instruments, and data collection procedures to be used in the process and outcome evaluation were conducted with representative subsamples of the target populations. All pilot tests were conducted with nine or fewer individuals. Attachment 4 - Document 1 provides a summary of pilot test feedback for each of the measurement instruments tested and outlines the changes that were made to the data collection instruments based on this feedback.

Semi-Structured Interviews: The pilot study participants for the semi-structured interview guides and procedures consisted of representatives from prior cohorts of the TCE-HIV initiative. The decision was made not to pilot any current cohort of Grantee staff and partners as this may have primed them for actual data collection. Grantee staff and partners who were interviewed for the pilot test were asked to respond to the interview questions and to provide feedback on the clarity of the questions and guidelines. They were also asked to comment on the appropriateness of the questions and procedures for the intended audience.

Client Focus Groups: Pilot tests of the focus group guides were conducted with substance abuse treatment clients from prior TCE-HIV grantees. The pilot study participants took part in a focus group discussion. After participating in the focus group, participants were then asked to comment on the clarity of the questions and identify problems or concerns with the questions. Participants were also asked to provide feedback on the appropriateness of the questions.

Grantee staff from prior TCE-HIV cohorts assisted with the recruitment of focus group participants for the pilot test. Case managers and counselors shared with their clients that participation in the focus group would give them an opportunity to talk about their experiences in the TCE-HIV program. Interestingly, during a focus group discussion, a client stated that participation in the discussion was an opportunity to “give back” (i.e., an opportunity to share experiences that might help others). Grantee staff provided recommendations for how to recruit focus group participants for the Multi-Site Evaluation. Grantee staff suggested placing flyers and sign-up sheets on walls and they also suggested that case managers be allowed to recommend clients for participation in the focus group.

Client Surveys: Program staff members from the current cohort of TCE-HIV Grantees were trained in how to administer the client survey. Pilot tests of the client survey were conducted with a subsample of nine clients from the target population. Specifically, trained Grantee staff administered the client-level survey to 3 clients at baseline, 3 clients at discharge, and 3 clients at 6 months post baseline. Grantee staff was then asked to comment on the clarity of the questions and identify problems or concerns with the questions and format of the surveys. Grantee staff provided feedback on the appropriateness of the questions for the intended audience. Finally, Grantee staff also provided client feedback regarding the client-level survey.

Dosage Forms: Grantee staff from the target population was trained in how complete the dosage data form. Pilot tests of the dosage data forms were conducted using records from a subsample of nine clients from the target population. Trained staff completed the client dosage form using clients’ treatment records and commented on their experience completing the client dosage form (e.g., challenges encountered in completing the form, time to complete the form).

5. Statistical Consultants

As noted in Section A.8, the TCE-HIV Multi-Site Evaluation Team has consulted with an expert panel and a project working group that have reviewed all data collection and analysis methodologies outlined in this package. They will also continue to provide expert advice throughout the course of the project. In addition, several in-house experts will be consulted throughout the 5-year project on various statistical aspects of the design, methodological issues, implementation issues, database management, and data analysis. Exhibit 7 provides information about these advisors.

Exhibit 7. Senior Advisors

Jim Derzon, PhD	Battelle Memorial Institute TCE-HIV Multi-Site Evaluation Task Lead for Quantitative Analysis 2101 Wilson Boulevard Suite 800 Arlington, VA 22201-3008	Phone: 703-248-1640 Fax: 703-527-5640 E-mail: Derzonj@battelle.org
Susan Hayashi, PhD	JBS International, Inc. Vice President 5515 Security Lane, Suite 800 North Bethesda, MD 20852-5007	Phone: 301-495-1080 ext. 4588 Fax: 301-587-4352 E-mail: shayashi@jbsinternational.com
Kevin Hylton, PhD	Alliances for Quality Education, Inc. TCE-HIV Multi-Site Evaluation Deputy Project Director TCE-HIV Evaluation Project 8181 Professional Place, Suite 110 Landover, Maryland 20785	Phone: 301-583-8424 Fax: 301-583-8422 E-mail: khylton@aqe-inc.com
Jennifer Kasten, PhD	JBS International, Inc. TCE-HIV Multi-Site Evaluation Lead 5515 Security Lane, Suite 800 North Bethesda, MD 20852-5007	Phone: 240-645-4145 Fax: 301-587-4352 E-mail: jkasten@jbsinternational.com
Resa Matthew, PhD	JBS International, Inc. TCE-HIV Multi-Site Evaluation Project Director 5515 Security Lane, Suite 800 North Bethesda, MD 20852-5007	Phone: 240-645-4608 Fax: 301-587-4352 E-mail: rmatthew@jbsinternational.com
Ping Yu, PhD	Battelle Memorial Institute TCE-HIV Multi-Site Evaluation Subcontract Director/Co-PI 2101 Wilson Boulevard Suite 800 Arlington, VA 22201-3008	Phone: 703-875-2981 Fax: 703-527-5640 E-mail: Yup@battelle.org

REFERENCES

Banks, S. E., Brown, L. S., & Ajuluchukwu, D. (1991). Sexual behaviors and HIV infection in intravenous drug users in New York City. Journal of Addictive Diseases, 10, 15–23.

Bastos, F. I., Lowndes, C. M., Derrico, M., Castello-Branco, L. R., Linhares-De-Carvalho, M. I., & Oelemann, W. (2000). Sexual behaviour and infection rates for HIV, blood-borne and sexually transmitted infections among patients attending drug treatment centres in Rio de Janeiro, Brazil. International Journal of STD and AIDS, 11, 383–392.

Booth, R. E., Crowley, T. J., & Zhang, Y. (1996). Substance abuse treatment entry, retention and effectiveness: out-of-treatment opiate injection drug users. Drug and Alcohol Dependence, 42, 11–20.

Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and design. Cambridge, MA: Harvard University Press.

Bronfenbrenner, U. (1986). Ecology of the family as a context for human development: Research perspectives. Developmental Psychology, 22, 723–742.

Broome, K. M., Joe, G. W., & Simpson, D. D. (1999). HIV risk reduction in outpatient drug abuse treatment: individual and geographic differences. AIDS Education and Prevention, 11(4), 293–306.

Camacho, L. M., Bartholomew, N. G., Joe, G. W., Cloud, M. A., & Simpson D. D. (1996). Gender, cocaine and during-treatment HIV risk reduction among injection opioid users in methadone maintenance. Drug and Alcohol Dependence, 41(1), 1–7.

Centers for Disease Control and Prevention. (2006). Racial/ethnic disparities in diagnoses of HIV/AIDS--33 states, 2001–2004. Morbidity and Mortality Weekly Report, 55(5), 121–125.

Croog, S. H., Levine, S., Testa, M. A., & Brown, B. (1986). The effect of antihypertensive therapy on the quality of life. The New England Journal of Medicine, 314(26), 1657–1664.

Dekovic, M. (1999). Risk and protective factors in the development of problem behavior during adolescence. Journal of Youth and Adolescence, 28, 667–685.

Derogatis, L. R. (1993). SCL-90-R: Symptom checklist-90-R: administration, scoring & procedures manual (3rd ed.). Minneapolis, MN: National Computer Systems, Inc.

Derogatis, L. R. & Coons, H., L. (1993). Self-report measures of stress, In L. Goldberger, S. Breznitz (Eds.), Handbook of Stress: Theoretical and Clinical Aspects (2nd ed.). New York, NY: Simon & Shuster.

Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method (2nd ed.). New

York, NY: John Wiley and Sons.

General Accounting Office. (1998). Drug abuse: Research shows treatment is effective but benefits may be overstated. Washington, DC: General Accounting Office.

Hartel, D. M. & Schoenbaum, E. E. (1998). Methadone treatment protects against HIV infection: two decades of experience in the Bronx, New York City. Public Health Reports, 113, 107–115.

Institute of Medicine (2001). No time to lose: getting more from HIV prevention. Washington, DC: National Academy Press.

Magura, S., Rosenblum, A., & Rodriguez, E. M. (1998). Changes in HIV risk behaviors among cocaine-using methadone patients. Journal of Addictive Diseases, 17(4), 71–90.

Metzger, D. S., Woody G. E., McLellan, A. T., O'Brien, C. P., Druley, P., Navaline, H., DePhilippis, D., Stolley, P., & Abrutyn, E. (1993). Human immunodeficiency virus seroconversion among intravenous drug users in- and out-of-treatment: an 18-month prospective follow-up. Journal of Acquired Immune Deficiency Syndromes, 6(9), 1049–1056.

Miller, W. R. & Tonigan, J. S. (1997). Assessing the drinkers’ motivation for change: the Stages of Change Readiness and Treatment Eagerness Scale (SOCRATES), In A. Marlatt, G. VandenBos (Eds.), Addictive behaviors: Readings on etiology, prevention, and treatment. Washington, DC: American Psychological Association.

Minority Aids Initiative. (1998). U.S. Congress, Omnibus Consolidated and Emergency Supplemental Appropriations Act, 1999 (PL 105-277), October 21, 1998. Retrieved on February 12, 2010 at http://minorityhealth.hhs.gov/templates/browse.aspx?lvl=2&lvlID=36.

Murphy, D. A., Brecht, M. L., Herbeck, D., Evans, E., Huang, D., & Hser, Y. I. (2008).

Longitudinal HIV risk behavior among the Drug Abuse Treatment Outcome Studies (DATOS) adult sample. Evaluation Review, 32(1), 83–112.

Palepu, A., Tyndall, H. Leon, H., Muller, J., O'Shaughnessy M. V., Schechter, M. T., & Anis, A. H. (2001). Hospital utilization and costs in a cohort of injection drug users. Canadian Medical Association Journal, 165(4), 415–20.

Prendergast, M. L., Urada, D., & Podus, D. (2001). Meta-analysis of HIV risk-reductions within drug abuse treatment programs. Journal of Consulting and Clinical Psychology, 69(3), 389–405.

Rutter, M. (1987). Psychosocial resilience and protective mechanisms. American Journal of Orthopsychiatry, 57(3), 316–331.

Somlai, A. M., Kelly J. A., McAuliffe, T. L., Ksobiech, K., & Hackl, K. L. (2003). Predictors of HIV sexual risk behaviors in a community sample of injection drug-using men and women. AIDS and Behavior, 7(4), 383–93.

Sullivan, L. E., Metzger, D. S., Fudala, P. J., & Fiellin, D. A. (2005). Decreasing international HIV transmission: The role of expanding access to opioid agonist therapies for injection drug users. Addiction, 100(2), 150–158.

Texas Christian University. (2005). Client Evaluation of Self and Treatment, Intake Version (TCU CEST-Intake). Retrieved February 12, 2010 from http://www.ibr.tcu.edu/pubs/datacoll/commtrt.html#Form-CEST.

Turner, B. J., Laine, C., Cosler, L., & Hauck, W. W. (2003). Relationship of gender, depression, and health care delivery with antiretroviral adherence in HIV-infected drug users. Journal of General Internal Medicine, 18(4), 248–257.

Turner, B. J., Laine, C., Yang, C. P., Hauck, W. W. (2003). Effects of long-term, medically supervised, drug-free treatment and methadone maintenance treatment on drug users' emergency department use and hospitalization. Clinical Infectious Diseases, 37, Suppl 5, S457–S463.

U.S. Bureau of Labor Statistics (2008). Retrieved on February 12, 2010 at http://www.bls.gov/oes/2008/may/oes_nat.htm#b21-0000).

U.S. Department of Health and Human Services. Office of Disease Prevention and Health

Promotion. Healthy People 2020: The Road Ahead. Retrieved on February 9, 2010, at http://www.healthypeople.gov/HP2020/objectives/ViewObjective.aspx?Id=595&TopicArea=Substance+Abuse&Objective=SA+HP2020%e2%80%936&TopicAreaId=46.

List of Attachments

Attachment 1a – Client Instruments

• Document 1 - Client Focus Group Discussion Guide

• Document 2 - Intake/Baseline Client-Level Survey

• Document 3 - Discharge Client-Level Survey

Attachment 1b – Client Instruments Continued

• Document 1 – Six Month Follow-Up Client-Level Survey

• Document 2 - Client Dosage Form

• Document 3 - Client Data Sheet

Attachment 2a – Client Other

• Document 1 - JBS International Network Security

• Document 2 - Project Working Group Feedback Results

• Document 3 - 45 CFR 46

Attachment 2b – Client Other Continued

• Document 1 - The Privacy Act of 1974

• Document 2 - Client Focus Group Recruitment Flyer

• Document 3 - Client Focus Group Sign Up Roster

• Document 4 - Client Consent Form

Attachment 2c – Client Other Continued

• Document 1 - Client-Level Survey Grantee Training Presentation

Attachment 2d – Client Other Continued

• Document 2 - Client Dosage Form Grantee Training Presentation

Attachment 3 - Administrative Staff Instruments

• Document 1 - Administrative Staff Semi-Structured Interview Guide

• Document 2 - Administrative Staff Data Sheet

Attachment 4 – Administrative Staff Other

• Document 1 - Pilot Test Results

• Document 2 - Proposed Sample and Methods by Stage

• Document 3 - Site Visit Call Script

• Document 4 - Site Visit Confirmation Letter

• Document 5 - Administrative Staff Consent Form

Attachment 5 – Direct Services Staff Instruments

• Document 1 - Direct Services Staff Semi-Structured Interview Guide

• Document 2 - Direct Services Staff Data Sheet

Attachment 6 – Direct Services Other

• Document 1 - Direct Services Staff Consent Form

Attachment 7 – Partner/Collaborator Instruments

• Document 1 - Partner/Collaborator Semi-Structured Interview Guide

• Document 2 - Partner/Collaborator Data Sheet

Attachment 8 – Partner/Collaborator Other

• Document 1 - Partner/Collaborator Consent Form