SUPPORTING STATEMENT FOR
VIOLENCE INTERVENTION TO ENHANCE LIVES (VITEL) SUPPLEMENTAL GRANT EVALUATION
The Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) Center for Substance Abuse Treatment (CSAT) is requesting approval from the Office of Management and Budget (OMB) for the data collection activities for the Evaluation of CSAT’s Violence Intervention To Enhance Lives (VITEL) Supplemental Grant Program (hereafter referred to as the VITEL program). This Evaluation (hereafter referred to as the Evaluation) consists of a multi-faceted approach to understanding the impact of intimate partner violence (IPV) screening and referral to trauma-informed care as well as the mechanisms that facilitate positive change in the reduction of IPV, substance use disorder (SUD) and HIV risk behaviors among racial/ethnic women offered through SAMHSA’s Targeted Capacity Expansion: Substance Abuse Treatment for Racial/Ethnic Minority Women at High Risk for HIV/AIDS (TCE-HIV: Minority Women) grant program. These activities will be conducted with five grantees and include: 1) administration of baseline, discharge and 6-month post-baseline surveys of clients receiving IPV screening and referral services, 2) focus groups with clients receiving IPV and SUD services, 3) documentation of IPV service and other referral service(s) engagement, and 4) semi-structured interviews with VITEL program staff and partner/collaborating staff supporting IPV services.
The VITEL grants are authorized under Section 509 of the Public Health Service Act, as amended. This program addresses Healthy People 2020 Substance Abuse Topic Area HP 2020-SA.
Racial/ethnic minority women are disproportionately impacted by SUD, HIV, and IPV across the United States. As such, VITEL was developed to engage the intersection of IPV, trauma, SUD, and HIV for women with and/or at-risk for HIV. Thus VITEL Grantees are tasked with implementing IPV screening within the existing TCE-HIV: Minority Women grant program, which funded 35 grantees in FY 2013 to enhance and expand SUD treatment services in conjunction with HIV/AIDS services for African American, Hispanic/Latina and other racial/ethnic minority women (ages 18 years and older), including heterosexual, lesbian, and bisexual persons, women who were previously incarcerated, and their significant others, who have substance use disorders and/or co-occurring substance use and mental disorders and are living with or at risk for HIV/AIDS. CSAT’s TCE-HIV program is funded by the Minority AIDS Initiative (MAI; 1998) appropriated dollars. The VITEL program is funded by the Department of Health and Human Services’ (HHS) Secretary’s Minority AIDS Initiative Fund (SMAIF). These funds are a subset of the full MAI resources appropriated by Congress. SMAIF resources are distributed on a competitive basis to HHS agencies and staff offices to support the goals of the National HIV/AIDS Strategy (NHAS) within racial/ethnic minority communities in the U.S.
The goals of the VITEL program are: 1 increase access to care and improve health outcomes for people living with HIV and AIDS, 2) reduce HIV-related health disparities resultant from IPV screening tool implementation, and 3) determine the feasibility of integrating IPV screening in behavioral health settings.
It is CSAT’s intention as a formal process, through the successful conceptualization, planning and implementation of an Evaluation, to independently evaluate the VITEL program in a manner that is comprehensive (i.e., process, short and long term outcome), that will establish baseline and long-term performance goals, and that will describe the impact of the VITEL program. This novel Evaluation will allow CSAT to investigate the feasibility of integrating IPV screening in behavioral health settings and changes in IPV incidence, substance use, HIV risk behaviors, and health-related outcomes.
The link between substance abuse and IPV is well-documented (Gilbert et al., 2012) by the victim (El-Bassel, Witte, Wada, Gilbert, & Wallace, 2001; Martin, Clark, Lynch & Kupper, 1999; Wingood, Diclemente, & Raj, 2000); by the perpetrator (Kub, Campbell, Rose, & Soeken, 1999; Sharps, Campbell, Campbell, Gary, & Webster, 2001) and by both the victim and abuser (Gilbert, El-Bassel, Rajah, Foleno & Frye, 2001; Muhajarine, 1999). Fals-Stewart (2003) and Schumacher et al. (2001) specifically recognized that IPV increases during periods of alcohol and substance use. Additionally, trauma and violence inhibits an individual’s advancement along the HIV Care Continuum (Schafer et al., 2012; Cohen et al., 2004; Mugavero et al., 2009). Highly Active Antiretroviral Therapy (HAART) adherence and discontinuation, and HIV health outcomes are significantly impacted by trauma, stress, and other psychosocial risk factors (Leserman, 2008). Lastly, women who experience all of the factors of the substance abuse, violence, and AIDS (SAVA) syndemic were 6.77 times more likely to have depressive symptoms (Illangasekare, 2013). A September 2013 White House report reaffirms this attention on trauma and its reduction as a means to reducing new HIV infections and improving the lives of people (women) living with HIV/AIDS (PLWHA).
Evidence-based IPV screening tools are rarely and inconsistently used in clinical settings. IPV-related injuries account for about 25% of all injuries that are treated in emergency departments and represent the leading cause of nonfatal injury to women (Gilbert et al., 2013; CDC, 2001; Cunradi et al., 2012; Foran & O’Leary, 2008). Less than 10% of HIV providers routinely screen for IPV (Zink et al., 2004), yet incidence is highly disproportionate among populations at risk for HIV (HRSA, 2009). Furthermore, several studies (Siemieniuk et al., 2013; Illangasekare et al., 2012) found that HIV-positive women in HIV treatment experienced significantly higher prevalence of IPV. The disproportionate burdens of HIV and IPV are most acutely felt by Black/African American and Hispanic/Latina women who account for 80% of new HIV diagnoses among women in 2011 and approximately 40.4% of reported lifetime IPV experiences. Other racial/multiracial and ethnic women have comparable IPV rates, but rates of HIV infection are considerably lower (Morales-Aleman et. al, 2014; Chen et al., 2013). As such, IPV is frequently not detected or under-detected in racial and ethnic minority women who may be at increased risk of experiencing emotional, physical, and/or sexual violence from their intimate partner.
There are numerous barriers to IPV screening from both patient and clinician perspectives. Health professionals may lack training, office protocols, and time, awareness of IPV and screening tools, and/or resources for referrals. Patients may experience judgment from providers (Dean, 2013; Ard & Makadon, 2011), low self-esteem, abuse escalation, fear of children being removed, or sense a provider’s reluctance as significant barriers for disclosure (Chen et al., 2013; Wingood et al., 2013). These barriers support the need for healthcare workers and those at community-based agencies delivering substance use treatment and recovery support services to have incentives and technical support to assess the possibility of IPV.
The proposed evaluation of the VITEL project will be incorporated into SAMHSA’s HIV Consolidated Evaluation contract that has a primary task of conducting a comprehensive process and outcome evaluation of selected SAMHSA HIV programs which includes the TCE-HIV: Minority Women grant program. This existing contract will be modified to include SMAIF funds to support the implementation of VITEL evaluation activities. Theoretically grounded in the ecological framework (Bronfenbrenner, 1979), which suggests positive health outcomes are determined by interaction of many factors at four levels (individual, relationship, community and societal) and the risk and resiliency frameworks, which focuses on risk and protective processes for each ecological system, the VITEL process and outcome evaluation seeks to:
Identify the extent to which the implementation of IPV screening and referral supports the TCE-HIV: Minority Women program, MAI, and NHAS goals and objectives; and
Document implementation of IPV screening and ascertain which implementation strategy contributes to significant reduction in risk for IPV among substance using racial and ethnic minority women.
The VITEL process evaluation establishes the overall evaluation’s context and aids in the interpretation of its findings. The outcome evaluation provides information about the impact VITEL intervention had on clients, grantees, and their communities/partnerships. The outcome evaluation also provides evidence about how specific programmatic characteristics relate to effectiveness in reducing IPV, substance use, and risky HIV behavior.
The Evaluation will address several overarching questions and provide SAMHSA with valuable knowledge regarding how the VITEL program is meeting the overall SMAIF goals. Specifically, questions to be addressed in this evaluation include:
What are the community and contextual conditions in which the VITEL programs exist and provide services?
How do VITEL Grantee characteristics facilitate or impede program activities, such as comprehensive services, service coordination/integration, and expanded organizational/ programmatic capacity?
What impact do Grantee characteristics have on client-level outcomes?
How do client characteristics (e.g., education, employment, income, family and living conditions, social support, satisfaction) lead to behavioral changes?
What impact does the VITEL program have on client-level outcomes?
This Evaluation will allow CSAT to determine the extent to which the VITEL program has met its objective of expanding and enhancing SUD treatment services in conjunction with IPV screening and referral and HIV/AIDS services to racial and ethnic minority women; and the resultant impact of VITEL on clients served.
This Evaluation is based on the presumption that positive IPV, HIV, and SUD treatment outcomes are determined by multiple tiers within a hierarchy of influence, and that a given client’s outcome successes will be a function of treatment program operations to include IPV screening and referral to trauma-informed care and HIV services (i.e., Grantee level), individual/relationship characteristics (i.e., Client level), and broader community contextual and socio-cultural environmental influences (Community level). Based on this theoretical framework, two interrelated evaluation efforts are supported: a process evaluation and an outcome evaluation. The process evaluation will serve the critical role of establishing the overall evaluation’s context and consequently aids in the interpretation of its findings. The process evaluation will also describe the content of Grantees’ interventions. The process evaluation will provide information about IPV screening and referral and TCE-HIV services were delivered, who delivered them, how they were delivered, to whom they were delivered, and how these services came to modify the Grantee’s overall IPV careening and referral and SUD treatment service delivery system.
The outcome evaluation will provide evidence on the effects of the IPV screening and referral and TCE-HIV treatment interventions on Grantees and clients. In an evaluation such as this VITEL evaluation, the outcome evaluation provides critical information on the Grantee characteristics that are believed to impact VITEL client outcomes. The outcome evaluation will provide evidence on how specific programmatic characteristics relate to effectiveness in reducing IPV risk, substance use, and HIV risk behaviors.
Process and outcome data collection call for different measurement and collection strategies.
To address the different data needs of the process and outcome evaluations, process data collection will be implemented at two distinct points in time, via site visits; outcome data collection will be implemented continuously via counselor administered client surveys and counselor completed interaction forms at 6-month post baseline. The purpose of the site visits is to observe program processes, to conduct semi-structured interviews with project staff and partners/collaborators, to conduct client focus groups, and to provide any technical assistance needed on client survey administration and interaction data completion.
Process measures will be collected through a site visit during the first data collection point (e.g., Month 1). The process measures are designed to capture information on the specific activities associated with implementation and the staff involved in program implementation activities, and the amount of time required for a client to be screened for IPV and if screened-positive, to be referred to trauma-informed services. The second process data collection point (e.g., Month 11) will consist of a follow-up site visit to collect a repeat measure of all data collected during Month 1. To determine the effect of IPV screening and referral to trauma-informed care, steady-state baseline measures will be compared with the same indicators approximately 10 months after the initial measurement. Comparisons between these time points will yield information on community and program changes as measured by these site visit instruments.
Simultaneously, client outcome measures and client interaction data will be collected continuously throughout the evaluation via a counselor administered Client Survey and review of client charts to document service exposure, respectively. The Client Surveys are designed to assess whether and to what extent IPV screening and referral, HIV, and TCE-HIV services are associated with changes in the key constructs of interest (i.e., IPV risk, substance use, HIV risk behaviors) for clients. VITEL counselors will facilitate the completion of the Client Survey at baseline, discharge, and 6-months post baseline. Program staff will complete the Interaction Data Form for each client at 6-month post baseline.
Process Evaluation The process evaluation of the five VITEL Grantees will focus on implementation evaluation questions, primarily, related to Grantees’ service delivery process; evaluating the service delivery process will identify activities required of Grantees to implement and sustain VITEL services. Specifically, process data will assess:
Community /Contextual Conditions
Community Partnerships
Program Characteristics
System Resources
Grantee Activities and Operations
Staffing and Training
Client Conditions (e.g., barriers to receiving services)
Semi-structured interviews with project directors, direct staff, and community collaborators/partners, along with client focus groups will provide the data necessary to conduct the process evaluation. The following presents a description of the process data:
Executive/Administrator Semi-Structured Interviews: (Attachment 1) The goal of this interview will be to collect information regarding the development and changes in program operations, staffing, training and programming; to provide improved understanding of program, agency, and community capacity changes that result from VITEL activities; and to collect documentation of changes in the number or nature of partnerships and collaborations both internal and external to the VITEL program agency. A corresponding consent form (Attachment 2) will be disseminated ahead of any interview.
Individuals who perform administrative tasks related to the VITEL program (e.g., Project Director, Program Manager, and Executive Director) are eligible to be interviewed. It is estimated that one administrative staff member per Grantee site will be interviewed during a given interview session. The administrator interviews will be conducted in two parts. The first part will be conducted with an Executive Staff of the agency; and the second part with the Project Director or Coordinator of the VITEL project. It is likely that one person from the agency may fulfill both roles/positions. If this is indeed the case, the full interview should be conducted with that person.
Direct Staff Semi-Structured Interviews: (Attachment 3). The goal of this interview will be to collect information regarding the development of IPV screening and referral activities and to improve the understanding of program changes that result from VITEL activities. Individuals from the Grantee organization who have direct contact with clients to perform IPV screening and referral related tasks will be eligible to be interviewed. Administrative staff members and direct staff members (e.g., outreach workers, counselors) perform different functions in grantee organizations and can each provide a different vantage point on IPV screening and referral activities within an organization. In order to obtain a balanced set of data, information will be collected from both. Examples of those performing direct service tasks include outreach workers, counselors, and case managers. It is estimated that up to two direct staff will participate in a given interview session. A corresponding consent form (Attachment 4) will be disseminated ahead of any interview.
Partner/Collaborator Semi-Structured Interviews: (Attachment 5) The goal of this interview will be to collect information regarding the development of the relationship between partners and the VITEL Grantee agency; to document the types of activities and services the partners provide in collaboration with the Grantee; and to assess partner’s perceptions of improved client outcomes related to collaboration between partner and Grantee agency. “Partner/collaborator” refers to agencies or organizations that provide services and activities related to the VITEL program. It is estimated that up to two individuals per community partner agency will participate in a given interview session. A corresponding consent form (Attachment 6) will be disseminated ahead of any interview.
Client Focus Groups: (Attachment 7) The goal of the Client Focus Group conducted through the course of VITEL Evaluation site visits will be to collect data from a sub-sample of VITEL clients regarding clients’ satisfaction with the screening and referral services, barriers and facilitators of services, and client level outcomes (i.e., IPV risk, substance use, HIV risk behavior). The Evaluation staff will work collaboratively with Grantee staff to identify client focus group inclusion and exclusion criteria. For example, clients that have been in treatment for less than 14 days may be unable to share information based on the limited time in treatment. Additionally, the recruitment of clients for the focus groups will strive for a balance in participants with regard to gender and age. A specified list of focus group inclusion and exclusion criteria will be developed and shared with the Grantees to aid in client identification. It is estimated that up to nine clients will participate in each focus group session. A corresponding consent form (Attachment 8) will be disseminated ahead of any interview.
Progress Report: (Attachment 9) The purpose of this scheduled (e.g., quarterly, biannually) reporting tool is to assist oversight of Grantee activity towards program implementation of federal award dollars. This report includes both expense related data and quantitative/qualitative information about the program’s impact, as it is applicable to both VITEL and all other HIV grant programs (e.g., TCE-HIV: Minority Women). Additionally, Grantees will be able to relate financial data to performance accomplishments of the federal award as well as share lessons learned and/or promising practices through this reporting mechanism.
Outcome Evaluation While the process evaluation will produce critically useful descriptive information at a community level and grantee level regarding the VITEL Grantees and how their programs are implemented and changes through the course of the one-year funding cycle, the outcome evaluation of the five Grantees will provide evidence on the effects of the VITEL interventions on clients.
Specifically the outcome evaluation will provide data regarding:
Client Characteristics
IPV Risk
HIV Testing and Status
HIV Risk Behavior
Substance Use
Criminal Activity
Education and Housing
Client Satisfaction
Comprehensive Services (Interaction)
The client survey will provide the data necessary to conduct a complete outcome evaluation. The following paragraphs present a description of the client survey.
Client Survey: The goals of the Client Survey are to collect information that will document client substance abuse, HIV risk behavior, and client conditions. This includes measuring variables that impact a client’s situation. For example, clients can be impacted by their quality of life (e.g., educational status, employment status, criminal justice involvement, trauma exposure). In addition, social connectedness measures a client’s use of recovery services, which will also be assessed.
The targeted universe for the VITEL evaluation client survey is clients in a treatment program from the five Grantee treatment programs. The client survey will be administered by Grantee program counselors. All of those receiving the initial baseline survey (Attachment 10) will be asked to complete a follow-up survey at discharge (Attachment 11), and at 6-months post baseline (Attachment 12). Data collection at the two follow-up points is necessary to measure the short- and longer-term client outcomes associated with Grantee treatment approaches.
The client survey is a new instrument developed under the auspices of CSAT specifically for the VITEL Evaluation. The client survey was developed from a combination of new questions (e.g., Sections A-C, Section E13-E25) and from subscales of existing measures. Instruments that were looked at for guidance in the development of items for Sections A-C and Section E13-E25 of the client survey are as follows.
Minority Substance Abuse/HIV Prevention Initiative (Centers for Substance Abuse Prevention (CSAP) (2008)
Addiction Severity Index 5th Edition (ASI) (McLellan, 2002)
Center for Disease Control and Prevention Healthy Days Core Module (Barger, Burke, & Limbert, 2007).
Risk Behavior Assessment Questionnaire (RBA) (ICPSR, 1993)
CSAT GPRA Client Outcome Measures for Discretionary Programs
Sections D, E1-E13, and Section F in the client survey are subscales taken from existing measures. Section D (items D1 – D9) on the client survey is from the Social Support subscale questions from the TCU CEST. Questions for Section E1 – E12 are the Depression and Anxiety symptom dimensions from the BSI. Section F on the Survey are the Taking Steps subscale questions from the SOCRATES A general description of these existing measures and the reliability and validity information for each subscale can be found below.
Texas Christian University Client Evaluation of Self and Treatment (TCU CEST) (Texas Christian University, 2005). The Client Evaluation of Self and Treatment (TCU CEST) was developed as part of NIDA Grant R37 DA13093, Transferring Drug Abuse Treatment and Assessment Resources. The TCU CEST includes a set of assessments that “target” specific needs and status of clients in different stages of change during treatment. The coefficient alpha for the Social Support Scale is .84.
Brief Symptom Inventory (BSI) (Derogatis, 1975). The purpose of this measure is to identify self-reported clinically relevant psychological symptoms in adults. Internal consistency estimates for the two sub-scales are .85 (depression) and .81 (anxiety). Good internal consistency reliability is supported by several other independent studies (Croog et al., 1986; Aroian & Patsdaughter, 1989 in Derogatis, 1993).
Stages of Change Readiness and Treatment Eagerness Scale (SOCRATES) Miller & Tonigan, 1996). Psychometric analyses reveal the following psychometric characteristics of the Taking Steps scale questions:
Cronbach Alpha = .83 - .96
Intraclass Test-Retest Reliability = .91
Pearson Test-Retest Reliability = .93
Upon OMB approval, the client survey will be translated into Spanish for ease of administration for those clients who are Spanish speaking.
Client Interaction Data Form: (Attachment 13) The goal of this form will be to collect data regarding the type and amount of contact/interaction that a client has with the program. Individual clients in the VITEL program will likely have very different types and amounts of contact (due to absences, participation in different components, or dropping out), thus it is critical to have interaction information to accurately assess program effects. This form will be completed by Grantee counselor at 6-month post baseline.
Data collected via semi-structured interviews, interaction forms, client focus groups, and surveys will enable CSAT’s VITEL program to increase its effectiveness in providing IPV screening and referral and HIV services and SUD treatment for minority women. Additionally, the Evaluation will help CSAT achieve the goals of its VITEL program and the Minority AIDS Initiative (MAI). Specifically findings of the evaluation will:
Identify the extent to which IPV screening and referral to trauma-informed services supports the TCE-HIV: Minority Women program, Minority AIDS Initiative (MAI), and National HIV/AIDS Strategy (NHAS) goals and objectives; and
Document implementation of IPV screening and referral services and ascertain which implementation strategy contributes to significant reductions in risk for IPV among substance using racial and ethnic minority women.
Data collection for the outcome evaluation component requires the use of information technology to develop an online data collection system. This online system will provide the three versions of the client survey (i.e., baseline, discharge, 6-months post baseline) to grantee staff for entering these data. Additionally, the Client Interaction Form, which is completed through the review of clients’ records at 6-month post baseline by grantee staff will be available through the online data collection system. Technology will also be used to manage, secure, and store the data to ensure data management control.
Client Surveys: Grantee staff will administer the client survey during face-to-face encounter at baseline, at discharge and at 6 months post baseline. Client survey data will be entered into an online system at the Grantee site that will be password protected. Each day, the VITEL evaluation staff will upload the data over a secure network connection directly to a server where they will also be encrypted and password protected. Using protected electronic data is the most secure form of data management because it eliminates the possibility of paper documents being lost by the survey staff or of data being lost in transit or delivered to an incorrect location. However, not all the Grantees may be equipped to enter this data into the online system, in which case paper copies of the completed surveys from these Grantees will be submitted and data entered onsite by the evaluation contractor. Any paper copies will be stored in a locked file cabinet, with no name/identifying information attached.
Client Interaction Form: Grantee staff will review client treatment records and complete this form for each client at 6-month post baseline. Similar to the client survey, interaction data will be entered into an online system at the Grantee site that will be password protected. On a weekly basis, the VITEL evaluation staff will upload these data over a secure network connection directly to a server where they will also be encrypted and password protected. In the event that grantees are not equipped to enter these data into the online system, paper copies of the completed interaction forms from these Grantees will be submitted with only unique alphanumeric identification and these data hand entered by the evaluation contractor. All paper copies will be stored in a locked file cabinet, with no name/identifying information attached.
Semi-structured Interviews and Client Focus Groups: Interviews and focus groups will be conducted face to face with staff and clients during site visits of the evaluation project. The decision to conduct face-to-face interviews and focus groups is based on the need to develop and maintain relationships between Evaluation staff and Grantees. Additionally, during on-site semi-structured interviews and focus groups, the evaluation staff will have the ability to observe staff composition, appropriateness of settings for clients, etc. Such observations would not be possible were the interviews to be conducted by telephone.
Progress Report: Grantee staff will initially complete the performance progress report, as scheduled, via an electronic fillable PDF and will transition to a web-based system subsequently. Data reflected on this report will be aggregated and will not contain any personally identifiable information that can or could be mapped back to an individual client.
The Evaluation Team has identified items answered through GRPA data collection that are of interest to the VITEL evaluation. To ensure minimal duplication, the Evaluation team has eliminated these items from our Evaluation instruments. This action was taken to remove any additional burden for the grantees. This action, however, was taken with the understanding that CSAT will provide the contractor for the Evaluation with access to the raw VITEL Grantee GPRA data at the three data collection points (i.e., baseline, discharge, 6-months post baseline). It is critical to the VITEL evaluation to have access to raw individual GPRA data as these items include:
Planned services
Client demographics
Military Family and Deployment status
Client substance use/abuse
Family living conditions
Education, employment, and income
Crime and criminal justice status
Mental and physical health
Recovery, Self-Help, and Peer-Support
Violence and Trauma
Social connectedness.
Without access to the raw GPRA data for these measures, the team will be unable to provide answers to VITEL evaluation questions.
The GPRA data will be made available for the purposes of this evaluation, as a supplement to the client survey data. Because the GPRA data represents a secondary client data source, the client survey has been created to supplement the GPRA by including additional measures necessary for evaluating the full spectrum of client outcomes and the moderators and mediators of those outcomes. Grantees will be asked to use the client GPRA identification number on the client survey at the three data collection points, so that the client surveys can be linked to the GPRA data, which contains demographics and other necessary information. Due to the availability of the GPRA data, the client surveys were designed to avoid any duplication. To aid in this process, discussions with grantee staff will be conducted, in order to verify the kind of data being collected.
In lieu of conducting individual performance assessments of their projects, all grantees as an condition of their SAMHSA grant award are required to participate in this evaluation of the VITEL grant program. This participation will allow grantees, SAMHSA, and the evaluation team to assess the progress of individual projects as well as measure the effectiveness of the overall grant program. All grantees will be required to provide data for IPV screening and referral along with original TCE-HIV: Minority Women grant requirements to SAMHSA. In addition, it is possible the evaluation design may necessitate changes in the required data elements and/or timing of data collection or reporting. Grantees will be required to comply with any changes in data collection requirements. Further, training and technical assistance on the evaluation will be provided by CSAT and the contractor at no cost to the grantee.
It is likely that many of the project’s respondents will be from relatively small treatment programs. Information collection for this Evaluation is not anticipated to have a significant impact on the individuals or on the programs or practices with which respondents may be affiliated.
The information to be obtained from respondents is the minimum necessary to achieve the objectives of the evaluation; however, completion of survey instruments, participation in focus groups, and/or participation in semi-structured interviews will likely induce some perception of burden. To reduce this burden, every attempt will be made to move respondents quickly through questions. For example, a screener question will be used in the client survey that asks about the client’s history of injection drug use. If the respondent indicates that he/she has no history of injection drug use, that respondent is skipped to the next category.
During this evaluation, the frequency of data collection from the grantees and the clients they serve is held to the minimum necessary to meet the needs of the evaluation objectives. The data collection points for the process evaluation component are generally acceptable intervals (i.e., in month two and month eleven) for assessing changes in the community and program level constructs that will be assessed in the given time period. Similarly, the outcome data collection points for the client survey are at acceptable intervals (i.e., baseline, discharge, 6-months post baseline) mirroring the GPRA data collection intervals to coincide with GPRA data collection. The one time data collection point for the Client Interaction form is at 6-month post baseline which is ideal for assessing client exposure to IPV screening and referral services.
Client Surveys: Client surveys will be administered to each client receiving IPV screening and/or referral and treatment services from the five treatment Grantees. All clients completing the survey at baseline will also complete the survey at discharge from treatment, and at 6-months following baseline to mirror the GPRA questionnaire administration. Data collection at discharge and 6-months following baseline is necessary to measure the short-term outcomes of the VITEL program.
Data collection at discharge is necessary to measure short-term outcomes related to IPV and referral services provided by VITEL Grantees. These outcomes include: changes in IPV risk, substance use, and HIV risk behaviors. Following up at 6-months post baseline is optimal for producing useful outcome data. Waiting until 6 months after baseline from IPV screening and referral services allows enough time for any effects of VITEL services including changes in IPV risk, substance use, HIV risk behaviors, and/or secondary outcomes, such as education and housing status, criminal involvement, arrests, and other health care utilization. Alternatively, waiting more than 6 months jeopardizes the validity of the data collected. As time passes, self-reported data become less accurate. Moreover, follow-up response rates, especially among much of the population to which VITEL is being delivered, decrease over time.
Semi-Structured Interviews: The semi-structured interviews will be administered to staff and partners/collaborators from each of the five VITEL Grantees during site visits. To determine the effect of VITEL services, the Evaluation will compare baseline measures with the same indicators approximately 10 months after initial measurement. Comparisons between these time points will yield information on community, program, and staff changes as measured at month 11. Collecting follow-up data is optimal for producing useful comparison data, as community and programmatic level changes are likely to be manifested within the 10 months. Alternatively, if the information is collected after more than 10 months, the shortened period for data analysis may result in a lack of important information for CSAT on how best to understand the community and contextual conditions in which the VITEL programs exist and provide services.
Client Focus Groups: Focus groups will be conducted during site visits in month 2 and month 11 with a sub-sample of treatment clients at each of the five Grantees. The goals of the Client Focus Groups are to gather information about client substance abuse treatment history, client behavior (e.g., IPV risk, substance use, HIV risk behavior, quality of life), client perception of social support, and client satisfaction with the VITEL program.
Progress Report: The performance progress report will be completed by Grantees as determined by the request for application and/or notice of award. Collecting progress data less than four times (quarterly) during the project period will remove any opportunity to affect programmatic changes that could be measured. Quarterly reporting provides the optimal timeframe whereby measuring performance and acting on those results can occur ensuring continuous quality improvement.
No technical or legal barriers to reduce burden exist if information is collected less frequently.
This information collection fully complies with the guidelines in 5 CFR 1320.5(d)(2).
The notice required by 5 CFR 1320.8(d) was published in the Federal Register on October 8, 2015 (80 FR 25661). No comments were received in response to this notice.
Client Focus Groups: Gift cards with a value of no more than $30 from major stores (e.g., Wal-Mart or Target) will be used as an incentive for treatment clients to participate in the focus groups. Although participation in the focus groups is voluntary, respondents are likely to perceive a time cost and burden associated with their participation. Survey research literature suggests that monetary incentives increase response rate, with no known adverse effect on reliability (Dillman, 1978, 2000). Pilot tests were conducted with representative participants to determine the appropriateness of the incentive.
SAMHSA will review the evaluation design and procedures to ensure they meet industry standards to protect the participants. This review will also ensure compliance with the spirit and the letter of regulations from the Department of Health and Human Services (DHHS) governing such projects. Systems and procedures for collecting and processing data are designed to help ensure the protection of participants and the data they provide. Documents with data about Grantees or individual clients will be identified by an assigned alpha numeric identification number.
Grantees have been approved to administer the GPRA, thus informed consent is in place for Grantee staff to collect GPRA questionnaire data. Grantees will have to modify their informed consent protocol to include the administration of the client survey. However, it is advised that Grantees also follow local and/or state requirements related to client informed consent. After completing their own data collection (i.e., GPRA), Grantee staff who deliver VITEL services will then briefly explain to the client the reason for an additional survey, describe the survey length and explain the process. The process of administering the client survey is designed to protect privacy, reduce client discomfort and burden, and ensure the collection of quality data.
The client survey will include the OMB approval expiration dates, the statement of survey burden, and the statement that the evaluation is federally sponsored. The Grantee staff will administer the client survey in a private location (e.g., an office) in order to ensure privacy. In cases where literacy is a concern, staff will read each question and their associated responses to the clients.
Client survey data will be entered into an online system that will be password protected. Each day, the VITEL Evaluation staff will upload the data over a secure network connection directly to a server and will also be encrypted and password protected. Using protected electronic data is the most secure form of data management because it eliminates the possibility of paper documents being lost by the survey staff or of data being lost in transit or delivered to an incorrect location. However, not all the Grantees may be equipped to enter this data into the online system, in which case paper copies of the surveys from these Grantees will be provided and manually entered by the contractor. Any paper copies will be stored in a locked file cabinet, with no name/identifying information attached.
The VITEL Evaluation staff will use passwords to safeguard project directories and analysis files containing completed survey data to ensure that there is no inadvertent disclosure of data. The Evaluation staff also will be trained on handling sensitive data and the importance of privacy. As a further precautionary measure, the data being collected will have no identifying information that can be linked back to the client. In keeping with 45 CFR 46, Protection of Human Subjects, the VITEL procedures for data collection, consent, and data maintenance are formulated to protect respondents’ rights and the privacy of information collected.
Data from the VITEL client survey will be kept strictly confidential in compliance with the Privacy Act of 1974 (5 U.S.C. 552a). The privacy of data records will be explained to all respondents during the consent process and in the consent forms.
For clients providing information in the client focus groups, the responses will be kept confidential; that is, no identifying information will be linked with the information provided and the information will be reported in aggregate. This de-identification process will extend to semi-structured interviews completed by Project Directors, Grantee staff, and community/partner collaborators. Any direct quotes used in reporting will not be attributed directly to the speaker, but will be credited only as a Focus Group participant comment.
Client Survey and Client Focus Group: The VITEL multi-site client survey and focus groups, by necessity, will collect sensitive information (e.g., substance use, mental health, and other health and social risk factors) because these are outcomes of interest to SAMHSA. Sensitive information of this nature is always regarded as highly confidential, and privacy for clients in federally assisted treatment programs is assured through strict adherence to Federal Regulation 42 CFR, Part 2. Upon OMB clearance, the survey staff of the Grantee organization and the Evaluation team will apply for certificates of confidentiality. This certificate issued by National Institutes of Health (NIH) and other HHS agencies is designed to protect identifiable research information from forced or compelled exposure. The Evaluation team and Grantee Institutional Review Boards (IRBs) will also ensure that additional appropriate mechanisms and procedures are in place to protect the privacy of the identifiable information to be obtained in the evaluation.
Respondents will be informed about the purpose of the data collection and that responding to all questions is voluntary. They will be assured that they may stop taking the survey at any time. In addition, specific assurances will be provided to respondents concerning the safety and protection of data collected from them.
Semi-structured Interviews: No sensitive information will be collected from the Grantee staff and partners/collaborators. The interview staff of the Evaluation Team will obtain signed consent for participation in the interview data collection. Respondents will be informed about the purpose of the data collection and that responding to all interview questions is voluntary. In addition, specific assurances will be provided to respondents concerning the safety and protection of data collected from them.
Progress Reports: No sensitive information will be collected from the Grantee.
Estimate the annualized hour burden of the collection of information from clients. The total client sample size for the VITEL data collection effort is estimated to be a maximum of 500 adult respondents (e.g., aged 18 and over). The baseline survey is expected to have a response rate of 100%, therefore resulting in 500 respondents completing the baseline survey. The discharge survey is expected to have a response rate of 100%, therefore resulting in 500 respondents completing the discharge survey. While we recognize that it may be difficult to achieve, the set goal response rate for the 6-month post-baseline survey is 100% of the baseline sample, therefore resulting in 500 respondents completing the 6-month post-baseline survey.
The treatment interaction form is expected to have a response rate of 100%, therefore resulting in the treatment interaction forms being completed for 500 treatment clients. The treatment focus groups are expected to have a response rate of 80%, therefore resulting in 36 respondents participating in focus groups. Exhibit 1 presents estimates of annualized burden based on preliminary testing. Although the average burden for the three surveys and interaction form is calculated using the average time from testing, the time required to complete the surveys varies with client characteristics, in particular substance use behaviors. The client survey which is administered at three time points (i.e., baseline, discharge, 6-months post baseline) and interaction form will be timed as part of a field test with a disproportionate number of hypothetical heavy substance users. The time to complete the surveys and interaction form for these clients will be documented. The total estimated time will include time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information.
Exhibit 1. Data Collection Burden for Clients and Grantee Staff
Instrument/Activity |
Number of Respondents |
Responses per Respondent |
Total Response Numbers |
Hours per Response |
Total Burden Hours |
Hourly Wage |
Total Respondent Costa |
Baseline data collection (Clients) |
500 |
1 |
500 |
.42 |
210 |
$19.56 |
$4,107.60 |
Discharge data collection (Clients) |
400 |
1 |
400 |
.42 |
168 |
$19.56 |
$3,286.60 |
6-month post Baseline data collection (Clients) |
400 |
1 |
400 |
.42 |
168 |
$19.56 |
$3,286.60 |
Interaction Form (Client) |
500 |
1 |
500 |
.42 |
210 |
$19.56 |
$4,107.60 |
Treatment Focus Group (Client) |
45 |
2 |
90 |
1.0 |
90 |
$19.56 |
$1,760.40 |
Client Sub-total |
500 |
|
1,890 |
|
846 |
|
$16,548.80 |
Executives and Project Director/Program Manager (Semi-Structured Interviews) |
10 |
1 |
10 |
.75 |
7.5 |
$28.40 |
$213.00 |
Executives and Project Director/Program Manager (Progress Report) |
5 |
1 |
5 |
3.0 |
15 |
$28.40 |
$426.00 |
Direct Staff (Semi-Structured Interviews) |
10 |
1 |
10 |
.75 |
7.5 |
$18.93 |
$141.98 |
Community Collaborators (Semi-Structured Interviews) |
10 |
1 |
10 |
1.0 |
10 |
$18.56 |
$185.60 |
Staff Sub-total |
35 |
|
35 |
|
40 |
|
$966.58 |
TOTAL |
535 |
|
1,925 |
|
886 |
|
$17,515.38 |
aTotal respondent cost is calculated as hourly wage × time spent on survey × number of respondents.
Estimate the annualized hour burden of the collection of information from executives and project directors/program managers. The total executive and project director/program manager sample size for the VITEL multi-site data collection effort is estimated to be a maximum of 15 respondents (5 sites, 3 respondents in each site). Exhibit 1 presents estimates of annualized burden.
Estimate the annualized hour burden of the collection of information from grantee (direct) staff. The total grantee staff sample size for the VITEL multi-site data collection effort is estimated to be a maximum of 10 respondents (5 sites, 2 respondents in each site). Exhibit 1 presents estimates of annualized burden.
Estimate the annualized hour burden of the collection of information from community collaborators. The total community collaborator sample size for the VITEL multi-site data collection effort is estimated to be a maximum of 10 respondents (5 sites, 2 respondents in each site). Exhibit 1 presents estimates of annualized burden.
Estimate the annualized cost burden to the respondent for the collection of information from clients. There are no direct costs to respondents other than their time to participate in the evaluation. The annual cost of the time respondents spend completing these surveys is $18,190.80 (number of total baseline client respondent hours, plus discharge and 6-month post baseline hours, plus treatment interaction form client respondent hours, plus treatment focus group client respondent hours × $19.56, the estimated average hourly wages for adults as published by the Bureau of Labor Statistics, 2007).
Estimate the annualized cost burden to the respondent for the collection of information from executives and project directors/program managers. There are no direct costs to respondents other than their time to participate in the evaluation. The annual cost of the time respondents spend completing these surveys is $639.00 (number of practitioner respondent hours × $28.40, the estimated average hourly wages for individuals working in health-related occupations as published by the Bureau of Labor Statistics, 2007).
Estimate the annualized cost burden to the respondent for the collection of information from grantee (direct) staff. There are no direct costs to respondents other than their time to participate in the Evaluation. The annual cost of the time respondents spend completing these surveys is $141.98 (number of practitioner respondent hours × $18.93, the estimated average hourly wages for individuals working in health-related occupations as published by the Bureau of Labor Statistics, 2007).
Estimate the annualized cost burden to the respondent for the collection of information from community collaborators. There are no direct costs to respondents other than their time to participate in the Evaluation. The annual cost of the time respondents spend completing these surveys is $185.60 (number of practitioner respondent hours × $18.56, the estimated average hourly wages for individuals working in health-related occupations as published by the Bureau of Labor Statistics, 2007).
There are no respondent costs for capital or start-up or for operation or maintenance.
The total estimated cost for the project is $19,157.38 over a one-year period. These costs cover all aspects of meetings and logistics, evaluation design, testing, data collection, site visits, information technology, analysis and reporting. In addition, it is estimated that one full-time SAMHSA staff member will spend 25% of his or her time (520 hours) to manage and administer the project. Assuming an annual salary of $100,000, government personnel costs will be $25,000 over a one-year period.
Total project costs are thus, $44,158.38.
This is a new collection of information.
Time Schedule: Exhibit 2 outlines the key time points for the Evaluation and for the collection of information. The requested period also allows for training and start-up activities associated with the preparation for data collection.
Exhibit 2. Time Schedule for Entire Project
Activity |
Time Schedule |
Obtaining OMB approval for data collection |
December 2015 |
Site Visit Data Collection (Month 2) |
Immediately following OMB approval |
Baseline Client Level Survey Data Collection |
1 month post OMB approval |
Discharge Client Level Survey Data Collection |
2 - 8 months after OMB approval at client discharge (discharge will vary based on scheduled treatment duration) |
Six-month post baseline Client Level Survey Data Collection (Includes Client Interaction Form) |
2 – 8 months after OMB approval |
Site Visit Data Collection (Month 11) |
7 months after OMB approval through 2016 |
Data analysis |
1 month post OMB approval through 2016 |
Dissemination
of findings |
Beginning 6 months post OMB approval through 2016 |
Publications: The VITEL Evaluation is designed to determine the process and outcome of the VITEL services programs, identify the extent to which IPV screening and referral to trauma-informed services supports the TCE-HIV: Minority Women program, MAI, and NHAS goals and objectives; and document implementation of IPV screening and referral services and ascertain which implementation strategy contributes to significant reductions in risk for IPV among substance using racial and ethnic minority women. It is therefore important to prepare and disseminate reports, concept papers, documents, and oral presentations that clearly and concisely present project results so that they can be appreciated by both technical and nontechnical audiences. The VITEL Evaluation Team will:
Produce rapid-turnaround analysis papers, briefs, and reports;
Prepare and submit monthly progress reports and a final VITEL Evaluation report;
Prepare final multi-site findings report, including an executive summary;
Deliver presentations at professional and federally sponsored conventions and meetings; and
Disseminate reports and materials to entities inside and outside CSAT.
Analysis Plan: Process and outcome data collection call for different measurement, collection, and analytical strategies. Both quantitative and qualitative data will be employed in the planned analyses to assess outcomes as well as processes of the VITEL Initiative. The planned approach is to use state-of-the-art statistical methods wherever appropriate to analyze the qualitative and quantitative data available. Qualitative data analysis will take on a thematic approach to uncover underlying themes. The three types of quantitative analyses that will be used are descriptive statistics, multivariate, and multilevel analysis. Finally, triangulation of methods (e.g., qualitative and quantitative data), when feasible will be conducted to examine additional aspects of results that cannot be accomplished with individual methods. Both the qualitative and quantitative data analysis plan is explained in more detail in the remainder of this section.
Qualitative Data Analysis: Qualitative data for this project will come from semi-structured interviews, focus groups, and responses to open-ended survey questions. The information will be utilized to address specific process evaluation questions. To facilitate the systematic analysis of the interview data, which will be collected during Grantee and community interviews, ATLAS.ti (Version 5.5.9, Muhr, 2008) or similar program, a computer-assisted qualitative data analysis software package, will be utilized. Before the analyses begin, Evaluation staff will take part in a training workshop to improve their facility with the capabilities and updated features of the software program.
Semi-Structured Interviews: Interview data will be analyzed to determine themes across Grantee administrators, direct staff and partners/collaborators about their respective programs and services they offer to clients. As a first step in the data analysis process, audio-taped of the interviews will be transcribed; these transcriptions will be the data used for the Evaluation.
Client Focus Groups: Focus group data will be analyzed to determine common themes across clients’ about their substance abuse treatment history, behavior (e.g., IPV risk, substance use/abuse, HIV risk behavior, quality of life), and satisfaction with the VITEL program. As noted for the semi-structured interview data analysis plan, audio-tapes of the focus groups will be transcribed. These transcriptions will be the data used for the Evaluation. The data will then be cleaned to remove any respondent identifying information and all transcription mistakes. Once the data is prepared for analysis, an inductive content analysis will be conducted on qualitative data individually for each group.
The initial step in the analysis process will be reading the raw data (i.e., original text) to discover underlying raw data themes. Raw data themes will then be grouped according to salient responses that correspond to the two main objectives of the Evaluation (see Page 3). These raw data themes will then be grouped into lower order themes based on common topics. Next, following the same coding procedures for grouping raw data themes, lower order themes will be grouped into higher order themes. Finally, higher order themes will be grouped into major categories. Consensus among researchers conducting the analyses will be reached at each step of the analysis process prior to proceeding to the next step (i.e., raw data themes, lower order themes, higher order themes, and major categories) in order achieve inter-coder reliability. Therefore, this process ensures a consistent understanding and interpretation of the data.
Quantitative Data Analysis: VITEL is a multi-level, and multi-component initiative. It therefore calls for multivariate and multilevel analyses to understand the complex relationship among treatment program operations, individual characteristics, and broader community contextual and socio-cultural environmental influences and client outcomes. These analyses address the overall evaluation question of what impact the VITEL Initiative has on client-level outcomes related to IPV risk, substance use/abuse, HIV risky behaviors, HIV testing, quality of life, attitudes toward treatment, and satisfaction with services.
Depending on the availability of the data and the number of waves of data available (data points) for each data element, the approach will be to apply the multivariate or multilevel analyses most appropriate for answering this overall outcome evaluation question and associated sub questions. These multivariate or multilevel analyses are referred to as “cross-site” or “main effect” analyses, as they may be based on pooled data from the five Grantees providing IPV screening and referral and SUD treatment services.
The Evaluation Team will use quantitative data analysis to examine both Grantee and client level outcomes. The analysis of both of these levels are explained in more detail below
Descriptive Statistics: The Evaluation Team will use descriptive statistics to describe the main features of Grantees, their clients, and the communities in which they provide service. Descriptive statistics will allow the reporting of community, Grantee, and client characteristics that will provide a detailed picture of the VITEL program and its participants. A utility of descriptive analysis will be to understand the distribution of variables of interest. Frequencies will be run or the means and standard deviations of each variable will be calculated to examine the central tendency and deviation from the mean, in order to examine the skewness of distribution of the data and to correct the distribution of the data prior to conducting further analyses. Descriptive statistics will also be used to conduct cross tabulations to examine the relationship between the variables. The degree of association between the two variables will be examined for statistical significance. Finally, descriptive statistics will be used to support higher level analyses.
Grantee Outcomes: Grantee characteristics and their effect on program service administration and service delivery can be examined through multilevel analyses. The Grantee outcome evaluation will use information developed in the process evaluation to identify the services most likely to be affected. Quantitative analyses of the Grantee characteristics provide critical information about the services being provided and how these services change over time. Qualitative Grantee data will come from the Administrator/Project Director Semi-structured interview (including the Grantee profile update), and Staff Semi-structured interviews.
Client Outcomes: Client level quantitative analyses will be primarily model-based and will be performed for both individual Grantee populations and the total population pooled across Grantees as appropriate. At the simplest level, regressions will be applied to assess reductions in client substance use and abuse outcomes over time for different number of sessions of substance abuse education (e.g., no sessions to two to three times per month), controlling for different client characteristics (e.g., age, sex, race, substance abuse treatment history, and HIV status).
OMB approval expiration dates will be displayed.
There are no exceptions to the certification statement. The certifications are included in this submission.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Alton J. King |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |