OMB Supporting Statement - METH -5-30-08

OMB Supporting Statement:

Prevention of Methamphetamine Abuse

A. Justification

1. Circumstances of Information Collection

The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Substance Abuse Prevention (CSAP) is requesting approval from the Office of Management and Budget (OMB) for the Youth and Adult Methamphetamine Surveys.

Methamphetamine abuse is on the rise all over the United States, and if this trend continues, it will have a harmful impact on the future of our nation’s health, families, and productivity. The methamphetamine epidemic is on the rise for a variety of reasons. There has been a steady increase in the importation of methamphetamine into different regions by organized trafficking groups. Also, countless users and dealers are manufacturing methamphetamine in homemade “mom and pop” labs found all over the United States in rural, urban, and suburban residences.

Following a hit of methamphetamine, users experience an intense rush of euphoria which is eventually substituted by a feeling of devastating depression. To feel normal again, methamphetamine users must take another hit of the drug, repeating the vicious and deadly cycle, quickly leading to addiction and dependency. The methamphetamine use “high” acts as a diversion from the actual “low” of the effects of using the drug. Methamphetamine abuse is poisonous to the body and brain. Every time a user sparks, inhales, or injects methamphetamine into their body, they are one step closer to heart failure, brain damage, strokes, or psychiatric and psychological symptoms that may lead to suicide and murder. The appeal of this drug is its long lasting “high,” which can last as long as twelve hours. (In comparison, crack cocaine only has a one hour “high.”)

The typical methamphetamine user meets one of two different profiles. The first profile is that of a high school or college student. The second is of a white- or blue-collar worker or an unemployed individual between the ages of 20 and 30. Methamphetamine use is equally divided between males and females; however, although each uses it for a different reason. Many young females use methamphetamine as a way of controlling their appetite and losing weight. Males, on the other hand, use the drug to increase performance, work longer shifts, or to stay energized during parties or “raves.” Unfortunately, due to its low price, users are often under the impression that methamphetamine is not really a drug.

Prevention of methamphetamine abuse is much easier than treatment for a current addiction. Methamphetamine users are classified as “the hardest to treat” of all drug users due to the potency of the drug. Prolonged use of methamphetamine physically changes the brain, causing addicts to constantly feel depressed, confused, and dissatisfied with life when they are not under the influence of the drug. Unlike many other drugs, willpower is not enough to cure methamphetamine addiction. Behavior and support therapies are necessary for success down a long road of recovery.

According to the Monitoring the Future Survey, methamphetamine use among high school seniors has more than doubled during the past six years. As these individuals are the future of our nation, it is imperative that aggressive prevention programs be implemented to reverse these statistics. In fact, focusing our attention on the prevention and treatment of methamphetamine abuse is extremely cost-effective. Treatment for methamphetamine abuse is about one-tenth the cost of having an individual incarcerated. The Prevention of Methamphetamine Abuse (Meth) Initiative grants are necessary to help prevent individuals from starting down the path to addiction, and to help save the lives of those individuals already plagued by their addiction to methamphetamine.

These surveys will be used for the cross site evaluation The Prevention of Methamphetamine Abuse grants program are authorized under section 519E of the Public Health Services Act, as amended. This grant program is awarded to grantees targeting Methamphetamine in there communities from across the United States. The some additional use of this data includes:

1. Conducting community-based prevention programs focused on those populations within the community that are most at risk for methamphetamine abuse and addiction.

2. Assisting local government entities to conduct appropriate methamphetamine prevention activities in rural and urban areas that are experiencing increases in methamphetamine abuse and addiction. This can be documented by local and specific epidemiological, health service use, judicial, and/or environmental data.

3. Training and educating State and local law enforcement officials, prevention and education officials, members of community anti-drug coalitions, and parents on the signs of methamphetamine abuse and addiction and the options for prevention.

4. Planning, administration, and educational activities related to the prevention of methamphetamine abuse and addiction.

5. Monitoring and evaluation of methamphetamine prevention activities, and reporting and disseminating resulting information to the public.

6. Conducting targeted pilot programs with evaluation components to encourage innovative methodologies with drug-endangered children.

The grantees will be collecting data on the approved National Outcomes Measures (NOMs) --OMB No. 0930-0230 -- and program specific questions on youth and adults Methamphetamine use. There are two surveys: one for adults ages 18 and older and another for youths under the age of 18. The adult and youth survey contain 40 and 42 questions respectively with the first 12 questions covering the OMB approved NOMs questions. The focus areas for the adult survey comprise of attitudes toward tobacco, alcohol, and other substances; attitudes and experiences; family relationships, relationships with those around you; future goals; thoughts, beliefs, and experiences related to methamphetamines; and thoughts on possible effects of methamphetamine use. The youth survey focus areas include: general information; attitudes toward tobacco, alcohol, and other substances; attitudes and experiences; family relationships; school experiences; perceived probability to try substances; where they receive substance abuse information; thoughts, beliefs, and experiences relating to methamphetamine; effects of methamphetamine use; and how comfortable they were with answering the survey questions. Additional non-methamphetamine related questions are included to identify risk and protective factors for methamphetamine. These questions identify demographic information which will be useful in categorizing results. Some program specific questions were suggested and agreed upon by the grantees in the review of the surveys.

2. Purpose and Use of Information

The purpose of this data collection is to conduct a cross site evaluation of the impact of the Methamphetamine Program. This Program will address the growing problem of methamphetamine and inhalant abuse and addiction by assisting localities in the conduct of targeted capacity expansions and in the implementation of effective evidence-based prevention interventions. The objectives of this initiative are to:

1. Facilitate effective interventions that prevent, reduce, or delay the use and/or spread of methamphetamine and inhalants.

2. Evaluate the effectiveness of federally funded activities and measure progress toward achieving National goals and objectives.

3. Support targeted capacity expansion for implementing evidence-based and effective prevention interventions tailored for methamphetamine and inhalant use and/or infrastructure development.

4. Prevent further rises in the abuse of methamphetamine and other substances.

To obtain data comparable across grantee sites for program evaluation purposes, two separate surveys for youth and adult program participants were developed and are being submitted for approval. These surveys collect demographic information about the characteristics of individual program participants, along with their attitudes, beliefs, and behaviors related to substance use in general and methamphetamine use in particular.

The surveys are designed for collecting data from participants of direct service interventions lasting 30 days or more at three time points: (a) at program entry, (b) at program exit, and (c) six months after program exit. Comparisons of participant responses to the survey items at these data collection points will provide information about the effects of the program on individuals’ knowledge, attitudes, beliefs, and substance consumption patterns, as well as on the sustainability of those effects over time. The instrumentation will be used by all grantee sites that implement direct service interventions lasting 30 days or longer. Both the adult and the youth surveys ask about previous use, sources of information, attitudes, and expected use in the future, while adults are also asked their perceptions of substance use among youths.

3. Use of Information Technology

It is anticipated that technical infrastructure and data management skills will vary across grantee sites. To maximize data accuracy and reliability, online data entry tools are designed for the surveys being submitted for clearance. These tools will be made available to grantees through CSAP’s Services Accountability and Monitoring System Web portal (CSAMS). The tools are designed to reflect the structure of the surveys and to allow the entry of data from completed surveys directly into the system through the use of radio buttons corresponding to response options. The system automatically quantifies the selected response options and stores the numeric codes in a SQL server for subsequent extraction, cleaning, and analysis.

CSAMS is maintained by CSAP’s Data Information Technology Infrastructure Center (DITIC). The data entered online by grantees are periodically extracted by DITIC and transmitted in encrypted form to CSAP’s Data Analysis Coordination and Consolidation Center (DACCC) for cleaning, record linkage, and analysis. Grantees have two options for accessing the data they enter online. In the first option, grantees can download, in spreadsheet form, the raw data they have entered online, as soon as it is submitted. Grantees can also access their data from the cleaned analysis files prepared by DACCC which are posted on CSAMS under password protection.

Grantees who prefer to create their own data files have the option of uploading complete data files to CSAMS. A grantee choosing this data submission option is required to use a standard codebook while preparing the data, thus ensuring that uploaded data files have the same numeric coding and variable naming conventions as the data entered using the online tools.

The online data entry tools reduce the grantees’ burden by facilitating the data entry process and minimizing coding and variable naming errors. The tools allow grantees without access to data management/analysis software to accurately quantify the information in completed surveys.

4. Efforts to Identify Duplication

In planning this cross-site evaluation, CSAP conducted a comprehensive literature search of completed and ongoing studies of methamphetamine prevention programs targeting youth and adults and found insignificant duplication of the proposed work at a National level. These studies were examined closely to take advantage of applicable methods and to identify any methodological problems that might detract from the validity, generalizability, or application of results. There has neither been an outcome evaluation of methamphetamine prevention programs of comparable scope to this evaluation, nor has there been one that uses the same measures to document methamphetamine prevention outcomes across youth and adults. The present initiative seeks to improve the lessons learned from this research by applying more rigorous data collection and analytical procedures.

In summary, CSAP did not identify any redundancy in that there were no precedents for a cross-site evaluation of projects like the one being proposed. Thus, it is clear that the data to be collected are unique to the CSAP Meth programs, are collected only for the CSAP Meth programs, and are not available elsewhere. The data collected through this multi-site effort will be non-duplicative, minimize burden on respondents, and yield important information for CSAP.

5. Involvement of Small Entities

This data collection will have no significant impact on small entities.

6. Consequences If Information Collected Less Frequently

Baseline, exit, and six-month follow-up surveys will be administered to participants of core prevention intervention programs funded by the Meth Initiative. Data will be collected solely for the purpose of evaluating the effectiveness of these intervention programs. Comparisons between baseline and exit surveys provide information about the effectiveness of the intervention in bringing about changes in targeted attitudes and behaviors. Comparisons between exit and follow-up surveys provide information about the sustainability of program effects over time. Furthermore, follow-up surveys often reveal program effects that were not captured in the exit survey. These typically involve indicators of relatively slow processes of change such as changes in behaviors associated with habit-forming substances. If the baseline and the exit survey data are not collected, CSAP’s ability to judge the effectiveness of its funded programs will be severely limited. Without collecting follow-up data, the sustainability of any short-term program effects cannot be assessed. Absence of follow-up data also may lead to an underestimation of those program effects associated with processes of behavioral change that may take longer to unfold than the typical baseline-to-exit time period.

The data will be collected from participants at three points in time. Failure to collect the information from all participants at these three points in time will result in missed opportunities for lessons learned on how to provide a quality improvement mechanism for CSAP to continually monitor and refine its prevention programs. Data collected at these three time points will provide information as to whether sustainable changes can be maintained over time after the program has ended, and if so, for which types of interventions and populations.

Without this information:

• CSAP will not be able to determine the extent to which it can prevent, reduce, and/or delay methamphetamine use.

• CSAP will not be able to monitor the quality of its prevention programs and determine how they can be improved to ensure continued success at meeting the needs of populations at risk for methamphetamine use.

• CSAP will not be able to fully describe the range of prevention services being used and the efficacy of evidence-based programs.

• CSAP will not be able to ascertain if participants are more knowledgeable about the consequences of methamphetamine use as a result of program participation.

• CSAP will not be able to identify those prevention services that are most effective and identify the potentially unique needs of at-risk groups.

• CSAP will not be able to meet its Federal reporting requirements to DHHS, OMB, and Congress.

7. Consistency With the Guidelines in 5 CFR 1320.5 (d) (2)

This information collection fully complies with 5 CFR 1320.5(d) (2).

8. Consultation Outside the Agency

CSAP has consulted experts from both within and outside of the Agency on refinement of the design, instrumentation, products, and statistical aspects of the cross-site evaluation at critical junctures during the survey design. These consultations enabled CSAP to obtain advice and recommendations on the identification and prioritization of the information to be gathered; to ensure the technical quality, appropriateness, and user relevance of the survey results; to verify the importance, relevance, and accessibility of the information to be sought; and to minimize respondent burden.

a. Consultations Outside of the Agency

The primary outside source of information was The Montana Meth Project. A core effort of the project is an evidence-based, nationally recognized, high-impact advertising campaign that graphically communicates the risks of methamphetamine use. For this evaluation, CSAP adapted the Montana Meth Use and Attitudes Survey. This survey was developed for the Montana Meth Project by the international research company GFK Roper Public Affairs and Media to track attitudes and behaviors related to methamphetamine throughout the State. The original survey consists of three separate schedules designed for administration to teens (ages 12-17), young adults (ages 18-24), and parents of teens. CSAP’s Youth and Adult Methamphetamine survey adopt attitudinal items from the teen and parent schedules of the Montana Meth Use and Attitudes Survey.

b. Additional Consultations Within the Agency:

Design of the multi-site evaluation and surveys were based on initial consultation with SAMHSA experts from CMHS and CSAT, as well as pilot testing of the Meth surveys by the grantees.

Other SAMHSA programs were consulted on the following issues:

• Draft evaluation design plan and data security procedures; plan for coordinating and collecting data; measures to be used to assess outcomes, and mediating factors

• Suitability of proposed assessment surveys

• Materials and nuances of prevention programs that may be relevant to finalizing the methods to be used in conducting the cross-site evaluation and reporting survey findings

• Means of minimizing the burden on project staff and program participants

• Efforts to assess the burden of the surveys (or similar surveys) and the readability of the instructions and the questions

• Identification of efforts to ensure user relevance of results

c. Federal Register Notice

The notice required in 5 CFR 1320.8(d) was published in the Federal Register on Friday, August 31, 2007 on pages 50377-50378). No comments were received.

9. Payment to Respondents

No cash payment will be made to individual program participants from whom data will be collected. Although not a project requirement, some grantee organizations provide in-kind incentives to respondents (such as gift certificates from local vendors), for completing the survey. The decision to provide incentives is left to the discretion of local sites.

10. Assurance of Confidentiality

Given the sensitivity of the topic and the information that is collected, CSAP and its grantees are acutely aware of the need to ensure privacy for the participants’ responses. The survey includes the following introductory text explaining the voluntary nature of the data collection and providing assurance of protection:

“This survey is voluntary. If you choose to take it, you may skip any question you don’t want to answer. This survey asks about your experience and opinion on a number of topics related to alcohol, tobacco, and drug use. Your answers to these questions will be protected. That means no one will connect your answers with your name or other identifying information. To help us keep your answers confidential, please do not write your name on this survey. The information in this survey will be used to learn more about the effectiveness of programs in preventing SA.”

CSAP instructions for administering the survey include the following steps:

1. The Survey administrator prepares survey by filling in respondents’ names and unique identification numbers on the survey’s cover sheet and their unique identification numbers on the first page of the survey.

2. At the beginning of the survey administration, the pre-filled surveys are distributed to the respondents.

3. The respondents are asked to check that their name on the cover sheet is correct and that the identification number on the cover sheet matches the one on the first page.

4. The respondents are then asked to detach the cover sheet from the survey and hand it back to the survey administrator who places all cover sheets in an envelope and seals it.

5. The respondents are instructed not to write their names or any other identifying information except their unique ID numbers on the rest of the survey.

6. Grantee organizations are instructed to keep all documents containing names and ID numbers separate from completed surveys and to keep both in locked cabinets or password-protected electronic files with access limited to senior project staff.

CSAP has designed the multi-site survey data collection strategy so that no identifying information such as names or Social Security Numbers will be requested of participants. The strategy for ensuring confidential data collection is described in the following paragraphs.

Individual Identifier:

An identification number is assigned to each survey participant in both the intervention/treatment and control/comparison groups (if used). A nine- (9- ) digit unique identification number (ID) is used on the forms in order to track the responses of program participants over time and across grantee sites. Each participant’s name and unique 9-digit ID should be written on the face (cover) sheet of the survey and the same 9-digit ID entered on page 2 of the survey. This should be completed by the Administrator prior to handing the survey to the participant. Participant names must not be written on any other page but the face (cover) sheet. The 9-digit ID has the following components:

A B C

--

• Grantee Site Identifier (Field A): Each grantee has been assigned a site identification number by CSAP. The site identification numbers range from 301 to 312. Each grantee’s identifier is a constant.

• Treatment/Comparison Group Type (Field B): This field indicates whether the respondent is receiving the intervention (coded as “1”), or is a control or comparison group member (coded as “2”).

• Individual Participant Identifier (Field C): This 5-digit number serves as the unique Individual Identifier for each program participant. The unique identifier is assigned by the Program Administrator or Project Director at the grantee site. This should be a numeric value (not alphanumeric) and each 5-digit combination should be unique to each individual participant. The 5-digit numbers can range from 00001 to 99999. Programs with multiple service locations may want to consider assigning a range of individual identifiers to each location to allow for easy identification of a participant’s service location. For example, one location could be assigned numbers 10000 to 19999, numbers 20000 to 29999 to a second location, and so forth.

Example: A program participant might have the following ID number:

301-1-13543

The number tells us that this person received intervention services provided by site 301 (Field A=301), that the participant is a member of the intervention group (Field B=1), and that s/he was assigned the unique 5-digit numeric combination 13543 by the grantee site’s Program Director.

The evaluation will involve collecting data from youth and adults, who will complete a self-report survey three points in time: program entry, program exit, and 6 months post- program exit. Several actions will be taken to ensure the privacy and security of the responses by the youth and the adults.

• All data collected will be maintained in a safe and private manner. The DACCC and grantees will conform to all requirements of the Privacy Act of 1974 under the System of Records, Alcohol, Drug Abuse and Mental Health Epidemiologic Data, HHS/SAMHSA/OA, #09-30-0036, and Public Law 107-347 (Sections 512 and 513).

• Grantees will not send identifying information (i.e. name of respondent) to the DACCC. Only a unique participant identification number will be provided. In addition, grantees will not provide identifying information to CSAP.

• Access to the data will be limited to the DACCC staff directly involved in the evaluation. At the end of the grant, a public use dataset will be made available to the public containing the Meth program grantees’ findings, along with detailed documentation. These public use data files will contain no individual identifiers. Further statistical data masking techniques will be applied to the public-use datasets to minimize the risk of intentional or accidental disclosure of individuals’ identities. Reports prepared by the DACCC as contract deliverables will present data in aggregated form only.

• All DITIC and DACCC staff will take a pledge (Attachment 1), administered by the contractors for the DITIC and DACCC contracts, agreeing that all information provided by respondents will be accorded complete privacy and security.

Any compact discs and diskettes that contain project data will be stored in locked files and all electronic data files will be stored in in-house servers under password protection at the DACCC contractor’s offices and will be accessible only to staff directly involved in the data processing aspects of the project. All members of the project will be required to sign a statement of personal commitment (Attachment 1), developed by the contractor to guard the security of data.

11. Questions of a Sensitive Nature

The surveys contain questions of a sensitive nature, such as the use of alcohol, methamphetamines, or other drugs, in order to understand the needs of participants and to measure the impact of services. The surveys also contain a question on sexual orientation. This question is vital to assessing the respondent’s level of risk, given research indicating that gay men are significantly more likely to use methamphetamines and amphetamines than their heterosexual counterparts.¹ In the Methamphetamine surveys, participants are explicitly assured of their privacy and informed that they may skip any question they prefer not to answer.

Grantees are required to have adequate consent procedures in place, and these procedures include obtaining and documenting parental/guardian consent (either active or passive, depending on local regulations) when necessary, and to inform participants of the voluntary nature of the survey, both in its entirety and item by item. SAMHSA review committees will not approve, nor will SAMHSA fund a site, without adequate provisions for meeting Federal policies regarding consent and data security.

12. Estimates of Annualized Hour Burden

Table 1 shows the estimated total burden for data collection. The evaluation data will be collected through survey administered to youth and adult program participants. Each youth and adult will complete surveys three times, taking an average of 50 minutes for baseline, exit, and follow-up surveys. A total of 3,000 youth and adults are expected to respond at baseline. It is expected that 2,400 youth and adults will respond to the exit survey; and a total of 1,680 youth and adults will respond to the follow-up survey. Hence, there will be an estimated total of 27,000 baseline, exit, and follow-up administrations. Therefore, for this five-year program evaluation, the total burden to youth and adults is 5,876 hours and the average annualized burden is 1,175 hours. These numbers reflect the grantee-projected program intake over a three-year period. These revised numbers also reflect a more accurate projection of respondents, as new information has been made available from grantees since the publication of the Federal Register Notice. This burden estimate presented in Table 1 is based on pilot test experience, which showed that each survey would take about 50 minutes to complete.

Table 1. Burden Estimate for Respondents (Combined Youth and Adult).

Description	Number of Respondents	Responses per Respondent	Hours per Response	Total	Hourly Cost	Total Cost
Baseline survey Exit survey 6-month follow-up survey	3,000 2,400 1,680	1 1 1	.83 .83 .83	2,490 1,992 1,394	$6.55* $6.55 $6.55	$16,309.5 $13,047.6 $9,130.7
	7,080			5,876	$6.55	$38,488

*minimum wage

13. Estimates of Annualized Cost Burden to Respondents

For this program there will be no capital, start up, or operation and maintenance costs.

14. Estimates of Annualized Cost to the Government

The total cost to the Government is expected to cost $303,000 for the data collection and the cross-site evaluation. SAMHSA/CSAP has planned and allocated sufficient resources for the efficient and effective management and use of the information to be collected. It is anticipated that the Government Project Officers who oversee the grantees will expend time in assisting their grantees in appropriately responding to the revised measures and CSAP-DACCC in processing and analyzing the submitted data. The TOO overseeing the DACCC and the Project Officer in charge of the Meth Initiative will expend a portion of time overseeing the collection, processing, and analysis of the data, as well as updating the measures as they are developed. Cross-program analyses will be conducted by CSAP-DACCC. Data analysis activities include processing the data received from the various programs, conducting statistical analysis, and developing reports. The annual cost of these activities is projected to be $33,000. These costs are itemized in Table 2 below. Annual hours are based on a 40-hour work week for 48 weeks per year.

Table 2. Estimated Annualized Cost to the Government

Position	Percent FTE	Annual Hours	Rate	Total Annual Cost
Data Analysis Team Senior Analyst (DACCC)	2.5%	50	$80	$4,000
Data Analysis Team Analyst (DACCC)	10%	200	$40	$8,000
Data Management Team Analyst (DACCC)	5%	100	$40	$4,000
Public Health Analyst (CSAP)	10%	200	$30	$6,000
Public Health Analyst (CSAP)	10 %	200	$17.5	$3,500
Public Health Analyst (CSAP)	100 %	2000	$48.75	$ 97,500
IT activities				$180,000
Total		950		$303,000

IT activities related to the Meth Program data collection and data management include: (1) the provision of internet-based outcome data submission through CSAMS; (2) Web site and database maintenance, data backup, user account management, and Web security; (3) system update, bug fixing, system enhancement, new module development, and system integration, and (4) training and technical support.

15. Changes in Burden

This is a new data collection.

16. Time Schedule, Publication and Analysis Plans

Analysis Plan

The defining characteristic of this cross-site evaluation is that all participating grantees share a common protocol, and a common set of performance measures, outcome objectives, and evaluation questions. This evaluation differs from more traditional multi-site clinical trials because each individual grantee will select an Evidence-Based Program (EBP) based on the needs and characteristics of the particular target population, setting, and organization. Multi-site evaluations do not test a single intervention that has different settings, rather they test a category of interventions that have similar outcome objectives but that uses different approaches to accomplish those objectives.

All applicants must describe their evaluation plans in their applications, and funded grantees are required to conduct an evaluation of their projects. The evaluation must be designed to provide regular feedback in order to facilitate project improvements. The evaluation must include both process and outcome components which measure change over time relating to project goals and objectives compared to baseline information. Control or comparison groups are not required.

The grantees will collect data from program participants at three time periods: baseline, exit, and 6- month follow-up. Each methamphetamine grantee will collect program-specific questions in addition to NOMs questions. Similar to the submission process for the Government Performance and Results Act (GPRA), grantees will submit their NOMs/Meth data to their respective program Project Officers as well as to the Center for Substance Abuse Prevention’s (CSAP) Data Analysis Coordination and Consolidation Center (DACCC) two times per year. The OMB approved NOMs incorporate the GPRA measures for reporting and are approved for all PRNS. CSAP, through the DACCC, will be responsible for data analysis across grantee sites, while individual grantees will be responsible for data collection and analysis of their local data.

The analysis of a multiple-site dataset requires a complex set of inter-related tasks. Planning for these tasks must be flexible, allowing adjustments as the opportunities and challenges presented by the empirical realities of the data set are discovered. While multi- site studies provide strong opportunities for knowledge generation (because of the ability to contrast intervention and implementation variation in a single evaluation), they also present significant research challenges. This evaluation recognizes those challenges and anticipated solutions as they will apply to the 13 participating grantees.

Elements of Evaluation Design and Analysis

Sample Size Determination. Individual grantees have proposed their target population sizes. The establishment of sample size at the grantee level depends, to some extent, on financial constraints for program intervention/treatment services, staff allocation, and retention activities for data and evaluation activities.

Control/Comparison Groups. CSAP provides guidelines for grantees to set up comparison groups and to collect comparable data from these groups as an optional survey design element. Wherever feasible, these data will be included in the analysis to provide a more rigorous evaluation of program outcomes than would be possible through the analysis of intervention group data alone. Inclusion of comparison groups in the evaluation design is contingent on the resources available at local evaluation sites.

Data Processing. As a first step in processing the data received from grantees, data fields will be examined for inconsistencies, duplicates, and missing information. Every attempt will be made to obtain the missing information from the grantee to complete the crucial information needed for analysis. Second, the baseline, exit, and follow-up surveys will be linked to create one single data record per program participant containing all of the data collected from that participant. After record linkage is completed, a second data-cleaning step will assess consistency of information across data collection points, inputting missing information wherever possible, and flagging all problematic fields that could not be corrected through imputation. The final step of data processing will involve recoding existing variables and constructing additional variables needed for analysis.

Levels of Analysis. The proposed analysis includes several distinct steps. First, pooled analyses of outcomes will be conducted to assess the presence of significant differences in program outcome indicators measured at baseline, exit, and follow-up time points. Second, matched comparisons will be conducted on individual program participants across data collection points. Third, the heterogeneity of outcomes across sites will be assessed to determine if outcomes for substance use or important risk/protective factors significantly differ across sites. If there are significant differences, hypotheses will be developed to explain those differences and conduct contextual analyses that combine participant-level outcome data with site-level variables hypothesized to contribute to program effectiveness. Additional analyses test the sensitivity of effectiveness models to differences in participant characteristics.

Decisional Balance Scales: According to a widely-accepted model of behavior change, people change their behavior based on a balance between their perceptions of the advantages and disadvantages of engaging in their behavior—change occurs when perceived disadvantages outweigh perceived advantages.² Based on this model, a methamphetamine prevention program is considered to have a positive effect if it decreases participants’ anticipated benefits and increases perceptions of risk of harm from using methamphetamines. To evaluate this dimension of program effects, two scales will be constructed from the survey data, one measuring perceived advantages of methamphetamine use (e.g. “Meth makes people feel attractive”) and the other measuring perceived risks of harm (e.g. “Which of these might happen to people who use methamphetamine? Suffering brain damage”).

Matched-Comparison Measures of Program Effectiveness: Two matched-comparison variables will be constructed for each substance targeted by the programs:

• User Decrease: For each targeted substance, the percentage of baseline users who reported decreased use at exit will be calculated to construct this measure. High values of this measure indicate higher program effectiveness in producing decreases in substance use.

• Non-User Stability: For each targeted substance, the percentage of baseline non-users who also reported non-use at exit will be calculated to construct this measure. High values of this measure indicate program effectiveness in supporting abstinence from substance use. Comparison of non-user stability between intervention and comparison groups provides an assessment of program effectiveness in preventing substance use initiation.

• Increased Perception of Risk: The percentage of program participants whose exit score on the perception of risk scale is higher than their baseline score.

• Decreased Anticipated Benefits: The percentage of program participants whose exit score on the anticipated benefit scale is lower than their baseline score.

Publication Plan

The Methamphetamine Prevention Initiative cross-site evaluation results will be made available to the public through publications and conference presentations. The following journals carry articles on SA prevention and are expected to serve as potential vehicles for distribution of evaluation results: Journal of Substance Abuse Treatment, International Journal of Addictions, Journal of Community Psychology, Journal of Adolescent Research, Journal of Adolescent Health, Preventive Medicine, Evaluation Review, Policy Studies Review, the American Journal of Public Health, Health Psychology, and American Journal of Health Behavior.

The survey results will be distributed through presentations at annual conferences of national and international public health organizations such as the Society for Prevention Research, the American Public Health Association, the National Association of Alcohol and Drug Abuse Counselors, and The National Prevention Network, as well as regional and State SA prevention and treatment associations.

Documents will also be prepared and published on behalf of the Government (CSAP/SAMHSA) through the Government Printing Office (GPO) for Federal agency and public use. Findings will also be available via OMB’s Website (www. expectmore.gov) as well as in annual Reports to Congress and the performance detail sections of annual SAMHSA budgets as they become publicly available.

SAMHSA/CSAP will utilize the data collected using these surveys on an ongoing basis to respond to Federal reporting requirements, as well as to address questions from Congress and the public regarding what types of prevention programs work and for which participants. Further, these data will be used to provide SAMHSA with information to document overall Agency performance requirements. The DACCC will conduct cross-program analyses and analyses of individual programs to help in planning and monitoring SAMHSA/CSAP’s success in meeting its goals. Table 3 below provides examples of the various analyses and reports for which the NOM-SAP data will be used, and when these activities will take place.

Table 3. Timeframe for Evaluation Activities

Activity	Date
GPRA Report to Congress	Yearly, in February
PART Results for OMB	Yearly, in September
Healthy People 2010 Report to DHHS	Yearly, each Fall
Conducting analyses to support CSAP budget proposals to Congress	Yearly, each Spring
Responding to ad hoc requests for analysis from CSAP staff, other Federal agencies, and the public regarding prevention effectiveness	Ongoing

17. Display of Expiration Date

The expiration date will be displayed.

18. Exceptions to Certification Statement

The certification is included in this submission.

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The data collected with these surveys will be used solely for program evaluation purposes, and not for generalizing to a broader population. Therefore, no sampling methods will be used to select respondents. All recipients of core prevention services (that is, intervention programs lasting 30 days or more) provided by Methamphetamine grantees will be administered three surveys using the survey appropriate to their age group at (a) program entry (baseline), (b) program completion (exit), and (c) approximately six months after program completion (follow-up). All respondents will be 12 years old or older. Respondents aged 12-17 will be administered the youth survey; respondents aged 18 and older will be administered the adult survey.

As part of its grant application, each grantee submits an evaluation proposal, which specifies, among other things, the methods to be used in recruiting program participants. Each evaluation proposal includes information on the response universe, recruitment method, and the target number of participants. These proposals are reviewed by a peer review group which assesses the adequacy and appropriateness of the survey design and methods. Only those applicants having technically sound proposals will be funded. In addition, to ensure that grantees carry out their evaluation as planned, SAMHSA/CSAP provides technical assistance as necessary.

2. Information Collection Procedures

While each grantee has its own plan for data collection, processing, cleanup, control, and retention, the work of the grantees is guided by the Youth Survey Administration Guide (Attachment 4) and the Adult Survey Administration Guide (Attachment 5). Each guide describes how uniform data collection will be ensured for the intervention and comparison groups, the timeframe for conducting the assessments over the course of the project, and how participant protection will be assured. As mentioned above, these plans undergo peer review to ensure the adequacy and appropriateness of the survey design and methods.

Common Measures - Youth and Adult Survey (Completed by Program Clients)

The evaluation uses a common protocol for collecting program and participant/client level data via an online, Web-based data entry system (CSAMS). CSAP’s Data Analysis Coordination and Consolidation Center (DACCC) and Data and Information Technology Infrastructure Center (DITIC) will provide technical assistance to grantees with data collection and online data entry. The data collection surveys are described below:

Youth and Adult Outcome e: Two common surveys will be administered to program participants/clients. The Youth Methamphetamine Survey (Attachment 2) is designed for administration to program participants aged 12-17 and the Adult Methamphetamine Survey (Attachment 3) is designed for participants aged 18 or older., Administration guides for both common measures have been prepared to assist program sites with data survey administration and data management, including procedures for maximizing data reliability and secure data storage.

The major constructs for the Youth Survey include demographics, sexual orientation, 30-day substance use (alcohol, tobacco, and other illegal drugs - ATOD), age of first use, disapproval of ATOD use, perception of risk ATOD use, perception of peer behavior, attitudes toward methamphetamine use, knowledge of the consequences of methamphetamine use, and sources for that knowledge. The survey also contains items about school experiences, peer ATOD use, and family cohesion.

The major constructs for the adult survey include demographics, employment, income, household and family structure, family cohesion, sexual orientation, 30-day substance use (alcohol, tobacco, and other illegal drugs - ATOD), age of first use, perception of risk of harm from methamphetamine use, extent and sources of knowledge about the consequences of methamphetamine use, and attitudes toward youth methamphetamine use.

For this evaluation, CSAP adapted the Montana Meth Project’s Montana Meth Use and Attitudes Survey (developed by the international research company GFK Roper Public Affairs and Media) to track attitudes and behaviors related to methamphetamine throughout the State. The survey consists of three separate schedules designed for administration to teens (ages 12-17), young adults (ages 18-24), and parents of teens. The survey was first fielded in Montana during August 2005 to collect benchmark data for the Montana Meth Project. It was repeated in 2006 and 2007 to assess the effects of the program. Trend data indicate that methamphetamine use among Montana’s youth has declined significantly since the launching of the Project.³ In 2007, the survey was also administered to a national sample in an effort to establish national norms for measures derived from the survey.⁴ As noted, CSAP’s Youth and Adult Methamphetamine Survey adopt attitudinal items respectively from the teen and parent schedules of the Montana Meth Use and Attitudes Survey.

Survey Administration Timeline: The common design includes assessments at baseline, program exit, and three to six months post-exit (follow-up). The data collection schedule for intervention programs is as follows:

• Baseline Survey: This survey is administered no earlier than 30 days prior to intake and before core program services begin.

• Exit Survey: This survey is administered within 10 days of program exit or after core program services have ceased.

• Follow-up Survey: This survey is administered within 30 days of the scheduled follow-up data collection. Most evaluation sites have a scheduled follow-up administration of 3 to 6 months after the end of intervention services.

Survey data are collected only from participants of direct services lasting 30 days or longer. The data collection protocol stipulates that there be at least 30 days between the baseline and exit surveys and at least 3 months between exit and follow-up surveys.

The Strategic Prevention Framework

SAMHSA’s Strategic Prevention Framework (SPF) allows for the development of the multiple surveys and data collection procedures used in the evaluation to capture information from grantees as they move through the five SPF steps. In Step 1, the grantee profiles the population needs, resources, and readiness to address gaps and needs. Step 2 involves mobilizing and/or building capacity to address the needs identified in Step 1. Step 3 is the planning phase where the grantee develops a strategic plan and selects the evidence-based intervention(s) to address the population needs identified in Step 1, taking into account the local capacity to provide these services. In Step 4, the grantee implements the selected evidence-based programs and activities. In the final phase of the SPF process, the grantee monitors and determines how effective the programs were in addressing the population’s needs using process and outcome measures. The analysis will address how implementation of the SPF has affected the capacity of the community serving the target populations to deliver evidence-based methamphetamine prevention programs, and how effective the programs were in changing participants’ knowledge, attitudes, and behaviors in the desired direction.

Timing of Data Collection

Baseline data collection must be conducted before the participant has been exposed to any intervention services and can be conducted at any time during the 30 days preceding program entry. The exit survey must be administered after the participant has completed the entire program but no later than 10 days after program completion. The follow-up schedule should be established at the program planning stage and follow-up surveys should be conducted within 30 days of the scheduled follow-up date.

Survey Design Groups

Although not required by CSAP, some grantee sites include a comparison or control group in their survey design. This group consists of individuals similar to the typical program participant in terms of demographics and other program-relevant characteristics, but who do not participate in the program being evaluated. The baseline, exit, and follow-up surveys are administered to this group at roughly the same times as the program participants (the intervention group). Survey designs including comparison/control groups provide a more rigorous assessment of program effectiveness.

Respondent Roster and Data Storage

Grantees are advised to maintain a roster of participant contact information including their unique ID numbers. This or any other documents linking participants’ names to their ID numbers will be kept in a locked cabinet or a password-protected electronic file and access will be limited to a minimum number of senior project staff. Grantees are advised never to store completed survey in the same place with the roster or with any other documents linking ID numbers to participants’ identity.

Survey Administration and Respondent Debriefing

CSAP strongly recommends that the local evaluator and/or evaluation staff administer all surveys. If the evaluation staff cannot administer the survey, it is recommended that program staff with research training conduct the administration. One or more survey administrators (proctors) will be present during the survey in order to explain the process and to answer any questions that may arise. Additional involvement of the proctor in the survey administration will vary depending upon the reading level and language skills of the respondents. Respondents at lower reading levels may be assisted with partial proctoring (certain items read aloud) or full-proctoring (the entire survey is read aloud). In all cases, the respondent will fill out his or her own survey. Grantees are advised to ensure that the room where the survey is conducted has adequate seating to accommodate the group(s) being tested, have adequate lighting and ventilation, and that the seating is spaced to ensure privacy.

The survey administration guidelines that CSAP provides for its grantees also recommend that the survey administrators make brief introductory remarks before and concluding remarks after the survey. The recommended text for introductory remarks includes a brief summary of the purpose of the data collection, its voluntary nature, and the measures taken to ensure respondent anonymity. In addition to thanking the respondents for their participation in the data collection effort, the concluding remarks include the following recommended text:

“Some of the questions on this survey may have raised some troubling questions or issues for some of you. If there is anyone who feels s/he would like to talk to someone about any of your concerns, please see . S/he will be glad to listen to you and provide whatever help s/he can.”

Measures to Ensure Data Reliability

The grantees are provided with a set of common questions from respondents and recommended standard answers that proctors can provide during survey administration. These guidelines ensure maximum standardization in the interpretation of the surveys across grantee organizations and survey sites.

The surveys will be administered by pencil and paper. Grantees will submit data using the online data entry tools on the CSAMS Website. They will also have the option of creating data files using coding and variable naming instructions provided by CSAP, and uploading them via the CSAMS file upload function. CSAP-DACCC will be responsible for conducting validation checks on the data, and for communicating with the grantees to clarify questions about the data.

3. Methods to Maximize Response Rates

Studies are enrolling respondents into specialized programs for allotted amount of times. In some cases, respondents are assigned case managers who maintain close contact with the respondents on a regular basis as a requirement for their program. The ongoing contact is helpful in maximizing response and retention rates. The goal of the data collection is to achieve an 80% response rate. Although not a program requirement, some local grantee sites offer incentives to their program participants (in the form of gift certificates from local vendors) to promote retention within the evaluation through the follow-up survey. Additionally, grantee organizations are trained in strategies for maintaining and updating confidential records of contact information for their program participants. These strategies should increase the likelihood that program participants are successfully contacted and engaged for the follow-up survey.

4. Tests of Procedures

A large majority of the items on the adult and youth surveys are adopted from field-tested and validated surveys. Questions 1-10 in the adult survey and 1-12 in the youth surveys are reproduced from the Adult and Youth NOMs Program data collection (OMB # 0930-0230).

These surveys have been in the field for several years. Data collected through their use have been analyzed for CSAP’s annual reporting purposes. Items adopted from the Montana Meth Use and Attitudes Survey have been piloted and fully fielded, first by the Montana Meth Project, and subsequently by the Arizona and Idaho Meth Projects. In addition, 9 adults and 9 youths were administered the survey to determine the estimated time burden and to ensure that there were no wording problems in the survey administration.

Several of the questions in the surveys are adopted from the Health Promotion in Our Communities Multi-Site Baseline Assessment Survey. These items were developed, field-tested, and validated by the John Snow International Research and Training and Institute, Inc.

Questions adopted by CSAP’s Adult and Youth GPRA data collection (OMB # 0930-0208) have been used to collect program evaluation data from multiple CSAP grantees, and used in several multi-site evaluation studies.

5. Statistical and Information Technology Consultants

Contact Information

Wilma Pinnock

SAMHSA Project Officer

1 Choke cherry Rd

Rockville, MD 20850

Daniel Bailey

SAMHSA Project Officer

1 Choke cherry Rd

Rockville, MD 20850

Human Services Research Institute

2366 Massachusetts Ave.

Cambridge, MA 02140

Datacorp, Inc.

Two Richmond Square
Providence, Rhode Island 02906

Xiaoyan Zhang

KIT Solutions, LLC

5700 Corporate Drive Suite 530

Pittsburgh, PA 15237

Xiaoyanz@kitsolutions.net

Jennifer Caputo

KIT Solutions, LLC

5700 Corporate Drive Suite 530

Pittsburgh, PA 15237

jcaputo@kitsolutions.net

Attachments

1. Methamphetamine DACC-DITIC Statements of Personal Commitment

2. Methamphetamine Youth Survey

3. Methamphetamine Adult Survey

4. Methamphetamine Youth Administration Guide

5. Methamphetamine Adult Administration Guide

1 Dew, B.J., Elifson, K.W., & Sterk, C.E. (2007). Differences in HIV sexual risk behaviors between heterosexual and nonheterosexual male users of methamphetamine. Journal of Drug Issues, 37(2), 281-298; Stall, R., & Wiley, J. (1988). A comparison of alcohol and drug use patterns of homosexual and heterosexual men: The San Francisco men's health study. Drug and Alcohol Dependence, 22, 63-73.

2 Prochaska, J.O., Velicer, W F., Rossi, J.S., Goldstein, M.G., Marcus, B.H., Rakowski, W., Fiore, C., Harlow, L.L., Redding, C.A., Rosenbloom, D., and Rossi, S.R. (1994). Stages of change and decisional balance for 12 problem behaviors. Health Psychology, 13(1), 39-46.

3GFK Roper Public Affairs and Media. (2007). 2007 Montana Meth Use and Attitudes Survey. Missoula, MO: The Meth Project.

4GFK Roper Public Affairs and Media. (2007). The Meth Project: National Use and Attitudes Survey 2007. Palo Alto, CA: The Meth Project.

Page 21