Mai-ss-b_12.21.15

PART B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B1. Respondent Universe and Sampling Methods

For methodological reasons discussed in Section A6, direct service grantees use a census approach and administer the questionnaires to all participants receiving services covered by grant funds. Dosage forms are also completed for all service encounters. SAMHSA’s response universe for this cross-site evaluation includes all active grantees (with relevant participants and objectives) and those initially funded at the end of FY 2015 that provide direct services to participants. In order to ensure accountability for the spending of federal funds, SAMHSA has employed the use of these data as a performance management tool to ensure that grantees are meeting the goals and objectives of the initiative. Data are used to monitor performance throughout the grant period. The Public Health Service Act Sec. 501 [290aa] (d) (13) with respect to grant programs authorized under this title, assure that “all grants that are awarded for the provision of services are subject to performance and outcome data collections.” SAMHSA interprets these requirements to indicate the need for data to be collected on all program participants receiving direct services.

B2. Information Collection Procedures

Common Measures - Youth and Adult Questionnaires (Completed by Program Participants)

The evaluation uses a common protocol for collecting program and participant/client level data and submitting it to SAMHSA via an online, web-based data entry system being developed by the PEP-C Systems Team. PEP-C’s Technical Assistance (TA) Team, in collaboration with the MAI Cross-Site Team, has set up an online Knowledge Base and a help line to assist grantees with data collection and online data entry by providing timely responses to requests for TA. A full description of the data collection protocol is provided in the Overarching Administration Guide (Attachment 5).

Youth and Adult Outcome Questionnaires

Two common questionnaires will be administered to direct-service program participants. The Youth Questionnaire is designed for persons aged between 12 and 17 and the Adult Questionnaire is designed for persons aged 18 and older. For all common measures, administration guides are under development to assist program sites with administering and proctoring the surveys.

The major constructs for the youth outcome questionnaire include demographics, 30-day substance use, age of first use, disapproval of peer substance use, perception of risk of substance use, parental monitoring, availability of emotional support from adults, experience of discrimination, mental health status, perception of peers’ risky behaviors, sexual behavior, sexual self-efficacy, school connectedness, and knowledge of HIV (See Attachment 1 for a copy of the Youth Questionnaire).

The major constructs for the adult questionnaire include demographics, employment, 30-day substance use perception of risk of substance use, sexual behavior, sexual self-efficacy, availability of social/emotional support, perception of risk of unprotected sex, knowledge of HIV, accessibility of health care, experience of discrimination, mental health status, and perception of peers’ risky behaviors. (See Attachment 2 for a copy of the Adult Questionnaire).

The Exhibit below lists the constructs that were deleted from the current questionnaires and the new constructs added in order to address SAMHSA’s newly-emerging priorities.

Exhibit 10: Summary of Revisions to the Youth and Adult Questionnaires

Youth Questionnaire Constructs		Adult Questionnaire Constructs
Deleted	Added	Deleted	Added
Birth month	Disability status	Birth month	Disability status
			College enrollment
Native vs. foreign born	Social support	Nativity	Social norms relating to binge drinking
Drunk driving	Perceived risk of unprotected sexual activity	Drunk driving	Attitude toward peer binge drinking
School enrollment	Perceived risk of injection drug use	Perceived risk of daily cigarette use	Attitude toward peer unprotected sexual activity
Whether or not on summer break	Access to health care	Intent to use alcohol, illegal drugs, or clean needles when injecting drugs	Access to health care
School enjoyment	Discrimination	HIV testing	Discrimination
			Informed about HIV status
Social norms relating to cigarette use	E-cigarette use	Age at first use of cigarettes, other tobacco products, alcohol, marijuana, and other illegal drugs	E-cigarette use
		Past month use of crack/cocaine and methamphetamine (merged under any illegal drug use)
Attitudes toward peer cigarette use	Synthetic marijuana use	Stress relating to substance use	Synthetic marijuana use
Intentions in next 3 months to have sex with more than 1 partner	Prescription drug use	Forced substance use
Ever been tested for HIV; Would be tested for HIV if given opportunity	Access to alcohol	Family relationships	Mental health status
Past month use of inhalants, crack/cocaine and methamphetamine (merged under any illegal drug use)	Emotional health in past month	Age of first child, number of children	Alcohol use disorder screening
Stress relating to substance use	Parental communication about sexual activity	Exposure to substance abuse and/or HIV prevention programs	Forced sexually activity
Age of first use of cigarettes, other tobacco products	Parental monitoring		Coverage under ACA
Prefer employer that does random drug testing		Comfort in completing survey
Lifetime number of sexual partners
Had sex for money, drugs, or other things
Have any children, number of children
Family relationships (retained one item on emotional closeness of family)
Peer behaviors relating to grades, suspensions, inhalant use, volunteer, religious activities, play sports
Sources of prevention messages
Comfort in responding to questionnaires

The revised Youth Questionnaire contains 94 questions, of which 24 relate to HIV/AIDS and the revised Adult Questionnaire contains 79 items, 29 of which relate to HIV/AIDS. The planned analysis will assess the degree to which implemented strategies reduce risks and increase protective factors associated with SA/HIV/AIDS among at-risk populations.

The information obtained from these questionnaires will generate data to determine the effectiveness of the MAI in reducing SA and high-risk sexual behaviors as well as increasing participant knowledge.

Individual and Group Dosage Forms

The Individual Dosage Form includes information on the date of the encounter, the unique identification number of the participant, and up to four service codes with corresponding numbers of minutes of exposure (see Attachment 3 for a copy of the Individual Dosage Form).

The Group Dosage Form contains the same information as the Individual Dosage Form and additionally lists the unique identification numbers of all participants who attended the session (see Attachment 3 for a copy of the Individual Dosage Form).

The Individual and Group Dosage Forms are expected to respectively take three and eight minutes to complete.

Data Collection

As displayed in Exhibit 2, both instruments will be administered at baseline, exit, and follow-up to participants whose services last 30 days or longer; at baseline and exit to participants receiving services lasting between 2 and 29 days; and at exit only for participants whose service duration is one day or less. The data will be collected in a pencil-and-paper self-report format. Exhibit 2 provides the estimated administration times for the questionnaires by the service duration of the respondent. These estimates are based on the burden estimates of the current instruments, adjusted for reductions in the number of items during the revision process.

An Individual Dosage Form will be completed by program staff at each one-on-one service encounter with all participants. A Group Dosage Form will be completed by program staff for every group-format service encounter provided by the grantee.

Data Management

Storage System: Over the life of this initiative, each grantee will be collecting information that must be documented and organized. Each local evaluation team or the person responsible for data management will be required to store:

• Completed instruments (Youth and Adult Questionnaires and Individual and Group Dosage Forms) until they are entered or uploaded into PEP-C’s online data entry system

• Consent forms

• Tracking forms for each program participant.

Before initiating data collection, each site is responsible for setting up a filing and storage system that will accommodate these needs in a manner that safeguards the privacy and anonymity of the participants.

Web-Based Data Entry Upload System: PEP-C’s online data entry system will allow grantees to enter item-by-item or to upload in batch file format all completed instruments. In order to ensure data quality and minimize data cleaning effort, all online instruments include online validation checks that issue error messages and suggestions for correcting data entry errors such as inconsistencies or out-of-range values. PEP-C’s online Knowledge Base will allow authorized grantee staff to download all instruments, administration guides, coding manuals, batch file preparation templates, data submission instructions & user manuals, and PowerPoint documents of all data collection and submission trainings.

The data that have been entered or uploaded to the PEP-C system will be biannually extracted for record linkage cleaning, analysis, and reporting purposes. Once data are entered by the grant site and cleaned by PEP-C, they will be available for download by the grant site for use in local data analysis and reporting.

B3. Methods to Maximize Response Rates

Issues related to response rates, as well as other data collection issues, are discussed at grantee meetings in order for project officers to identify problems and provide technical assistance. In addition, project officers monitor data collection efforts and provide technical assistance to individual grantees as necessary. Because collection of these data is a stipulation of the grants, it is anticipated that all grantees will comply (as appropriate). The participants at each site to whom these measures will be administered are all voluntary respondents; therefore, grantees cannot guarantee full cooperation on the part of participants. Historically, however, participant response rates at exit across grantee sites have averaged around 75 percent. Due to the high-risk nature of the participants, a substantial proportion of whom have historically been transient populations without permanent residence (e.g., homeless, unemployed, or reentry individuals), post-exit follow-up rates have been around 35%. SAMHSA and its contractors continue to provide training and technical assistance to grantees for enhancing these response rates. The newly-funded MSI CBO grantees will be targeting college students and young adults in the surrounding communities. SAMHSA expects that this target population will pose fewer challenges in terms of post-exit follow-up. However, the MAI cross-site team will continue to closely monitor the follow-up rates and will provide training and technical assistance as needed.

B4. Tests of Procedures

CSAP and its contractor have reviewed the Youth and Adult Questionnaires, clarifying terminology and language, and rewriting or eliminating unclear or unnecessary questions.

All new items added to the Youth and Adult Questionnaires as part of the recent revision process have been validated by large-scale data collection efforts conducted by federal and other credible organizations.

The estimates of time to complete the newly revised Youth and Adult Questionnaires are based on the time-to-complete estimates of the currently approved versions of the questionnaires, adjusted for the reduction in the number of items in each questionnaire. The estimated times to complete the Dosage Forms are based on an informal user test of the forms by six members of the PEP-C MAI Cross-Site Team. Each tester completed at least five forms of each type. The mean of the time-to-complete figures reported by each tester was used as the final burden estimate. Individual testers’ estimates were close to each other with a narrow error margin, slightly under one minute on either side.

B5. Statistical Consultants

Sara Azimi-Bolourian, Ph.D.

Public Health Analyst

Performance and Technical Assistance Branch

SAMHSA/CSAP

1 Choke Cherry Road, Room 4-1031

Rockville, MD 20857

Linda D. Youngman, Ph.D.

Branch Chief

Division of Community Programs

SAMHSA/CSAP

1 Choke Cherry Road, Room 4-1115

Rockville, MD 20857

Patricia A. Sabry

Public Health Advisor

Division of Community Programs

SAMHSA/CSAP

1 Choke Cherry Road, Room 4-1111

Rockville, MD 20857

Phillip W. Graham, Dr.PH

Director, Drugs, Violence, and Delinquency Prevention Research Program,

Center for Justice, Safety, and Resilience,

RTI International

PO Box 12194

Research Triangle Park, NC 27709

Elvira Elek, Ph.D

Research Public Health Analyst

RTI International

701 13th St NW #750

Washington, DC 20005

Nilufer Isvan, Ph.D.

Senior Research Fellow

Human Services Research Institute

2336 Massachusetts Avenue

Cambridge, MA 02140

Mindy Herman Stahl, Ph.D.

Research Psychologist,

RTI International

PO Box 12194

Research Triangle Park, NC 27709

Darigg Brown, Ph.D.

Research Public Health Analyst,

RTI International

2951 Flowers Rd South, Suite 119

Atlanta, GA 30341

Jeffrey Guidry, Ph.D.

Associate Professor

Department of Health & Kinesiology

Texas A&M University, Read 159 Bldg.

College Station, TX 77843

William DeJong, Ph.D.

Professor, Community Health Sciences,

Boston University School of Public Health

801 Massachusetts Avenue

Boston, MA 02118

ATTACHMENTS

1. Youth Questionnaire

2. Adult Questionnaire

3. Individual Dosage Form

4. Group Dosage Form

5. Overarching Administration Guide

15