Pep_ss-b-omb_3_30_15_final

PARTNERSHIPS FOR SUCCESS PROGRAM EVALUATION FOR PREVENTION CONTRACT

SUPPORTING STATEMENT

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Respondent Universe and Sampling Methods

The SPF-PFS cross site evaluation will use a census approach to collecting process, programmatic, and implementation data through the instruments at the center of this OMB application, while using existing archival data and data from survey samples for the outcomes measures.

Using a census approach, the targeted universe for the Grantee-Level Instrument–Revised (GLI-R) and the Grantee Project Director (PD) Interview is all Partnerships for Success (PFS) II grantee Project Directors (n=15), all PFS 2013 grantee Project Directors (n=16), and all PFS 2014 grantee Project Directors (n=21), and all future cohorts. All 52 grantee Project Directors are expected to complete both the GLI-R and the PD Interview, as grantees have agreed to participate in cross-site evaluation data collection activities as a condition of funding.

Using a census approach, the targeted universe for the Community-Level Instrument–Revised (CLI-R) is all PFS II subrecipient community Project Directors (n = ~140), all PFS 2013 subrecipient community Project Directors (n = ~250), and all PFS 2014 subrecipient community Project Directors (n=~220), and all future cohorts. All of the approximately 610 subrecipient communities are expected to complete the CLI-R, as grantees have agreed to participate in cross-site evaluation data collection activities as a condition of funding.

A census of all PFS II, PFS 2013, and PFS 2014 grantee Project Directors and subrecipient communities is necessary due to the heterogeneous nature of the SPF-PFS programs. These programs encompass a wide variety of organizational types and structures that are implementing a range of prevention interventions targeted to different populations and with various outcome goals. The variety between the programs makes it critical to the evaluation to capture the details of each program to be able to answer the evaluation questions and assess which program characteristics and mix of interventions are associated with better outcomes for particular demographic groups and types of communities. Additionally, this data will be used by SAMHSA to monitor each program’s performance and grantee and subrecipient communities will also use it to track their ongoing implementation. In order to meet SAMHSA’s annual reporting requirements for GPRA and performance measures, and more frequent reporting requirements related to PFS Health Disparities activities, SAMHSA must obtain data from all grantees and subrecipients, which supports the need for a census approach.

While the process and performance measures will be collected through a census of all grantees and subrecipients on the GLI-R, CLI-R, and PD Interview, all outcomes will come from existing archival data (records of UAD- and PDM-related arrests, vehicle accidents, emergency room visits, and overdose or poisonings) and existing survey data covering such topics as UAD- and PDM-related consumption, perceptions of parental or peer disapproval, perceived risk or harm of use, and family communication. At the grantee level, the related survey estimates generally will come from the National Survey on Drug Use and Health (NSDUH), with some data generated by the Youth Risk Behavior Survey (YRBS) or state, jurisdiction, or tribal surveys. Survey estimates at the subrecipient community level will generally come from state, jurisdiction or tribal surveys. NSDUH and YRBS utilize specified sample design procedures to develop national estimates and also provide estimates at the state and sometimes community (county, region, urban area) levels. Sampling designs vary among the state, jurisdiction, and tribal surveys. Prior to accepting estimates from those sources, PEP-C will review the related survey and sampling designs to ensure adequate generalizability, validity, and reliability of the estimates.

NSDUH provides an example of the type of sampling utilized for the survey-based outcomes measures for the SPF PFS cross-site evaluation. For NSDUH, the surveys are conducted using computer-assisted interviewing methods and a national sample size of 67,500, equally allocated across three age groups: persons aged 12 to 17, persons aged 18 to 25, and persons aged 26 or older (SAMHSA, 2012b). The NSDUH sampling design stratifies the sample by state and geographically partitioned regions within those states, and then randomly selected census blocks within those regions. To select units from the census block segments, NSDUH uses a random start point and interval-based (systematic) selection.

Data from this collection will be used to consider of a sampling approach in the future.

B.2. Information Collection Procedures

Grantee-Level Instrument–Revised and Community-Level Instrument–Revised

The GLI-R and the CLI-R are self-administered, web-based surveys completed through the Program Evaluation for Prevention Contract (PEP-C) online data collection system. GLI-R respondents are the grantee Project Directors and CLI-R respondents are subrecipient community Project Directors. Before data collection for the SPF-PFS evaluation begins, respondents will be provided a unique log-in to enter the data system, where they will be required to create a password. Respondent email addresses for each login will be stored within the system so that automatic alerts and notifications can be sent.

Pending Office of Management and Budget (OMB) approval, the GLI-R will be collected once at the beginning of the grant (or as soon as OMB approval is obtained) and once during the final year of the grant; the CLI-R will be collected every 6 months during the life of the grant (once OMB approval is obtained). Each collection time point will follow the procedures outlined below.

One week before GLI-R or CLI-R submission due dates, predefined, automated emails will be sent to respondents to inform them that the submission deadline is approaching and the data system is open for data entry. A link to enter the system will be included in the email, as will the due date for submission. When grantees or subrecipient communities submit data, in addition to receiving a “thank you” message on the system screen, they will receive email confirmation that the submission was received successfully. Nonresponders will be sent predefined, automated emails 1 day after the deadline and 1 week after the deadline, as needed, to remind them to submit their data. If data still have not been submitted, the grantee or subrecipient community will be contacted by telephone, although the SPF-PFS evaluation team anticipates that this will only occur very rarely, if at all.

The SPF-PFS evaluation will develop user manuals for accessing and navigating the PEP-C online data collection system and question-by-question and frequently asked question (FAQ) guides to help respondents accurately complete the GLI-R and CLI-R. Grantees and subrecipients will also be provided training webinars to: 1) walk through the PEP-C online data collection system, 2) review the GLI-R or CLI-R instrument questions and data collection procedures, and 3) cover specific sections of the instruments, such as cost reporting and reporting on interventions. Within the online data collection system, all manuals, guides, and training webinars will be archived and accessible to respondents for reference at any time.

Availability is important in any data collection system, especially one employed by grantee sites around the country, including multiple time zones and pacific jurisdictions. The online system will be maintained in an available state as much as possible to allow grantees and subrecipient communities to have access for entering data and viewing data cleaning forms by grantees and subrecipient communities, as well as to give the PEP-C team, grantees, and SAMHSA access to reports.

Providing a robust system that is simple and easy to use across all areas is also critically important. To achieve this, the contractor will implement user-friendly features across all functional areas, taking into account the needs of both SAMHSA and grantees. Additionally, every page of the online data system will have a “Help” or “Support” link located in the upper right corner, which will allow the respondent to access the following support resources:

1. Search the Knowledge Base. More comprehensive than a list of FAQs and more organized than a support forum, the Knowledge Base will offer a “layered information” approach so that respondents can search by keyword and then drill down to view material at increasing levels of detail. It will be a curated and easily searchable source of information including items such as

• system documentation,

• user guides,

• policies and procedures,

• protocols,

• training materials, and

• FAQs.

1. Contact Us. Respondents may request assistance either by calling a provided toll-free number or sending an email request, as desired. The toll-free line will be routed to an email system that is checked regularly by members of the training and technical assistance team. Staff responding to technical assistance requests will be trained in use of the system and have ready access to the full Knowledge Base. Training and technical assistance team staff will monitor all submitted tickets to ensure timely response and resolution of technical assistance requests.

Grantee Project Director Interview

As noted above, respondents to the PD Interview telephone interview are grantee Project Directors. The PEP-C evaluation team will contact grantee Project Directors via email (with telephone follow-up) to setup a mutually convenient time for the interview during regularly scheduled business hours. Before conducting the PD Interview, the evaluation team will review grant applications (submitted to SAMHSA by each grantee and given to the evaluation team by SAMHSA) and other documents (e.g., previously completed GLIs) that detail the proposed characteristics of the program and abstract information relevant to the evaluation (e.g., project structure, proposed interventions, subrecipient selection) so that interviewers are familiar with the grantee. This preabstracted information will be used to prepopulate some PD Interview questions to reduce respondent burden. For instance, a list of the grantee’s proposed subrecipient communities will be prepopulated and confirmed or updated with the respondent, as opposed to asking the respondent to generate the list while on the telephone.

Once the interview is scheduled, the contractor will provide the grantee Project Director with an electronic version of the assent form and the partially prepopulated PD Interview and a toll-free, passcode-protected telephone conference number. Before beginning the PD Interview, consent will be requested to record the interview to confirm, if needed, the accuracy of noted responses. A senior evaluator from the contractor’s evaluation team will lead the respondent through the interview while a junior evaluator will record responses and take notes. After the interview, the interviewer and note taker will review the responses for accuracy. Any areas of discrepancy will be validated with the recording (if consented by the respondent); once the responses are considered final, the recording will be deleted. An electronic version of the PD Interview will be maintained on a password protected, secure server accessible only to the contractor’s evaluation team. After the interview, the interviewer will send an email thanking the grantee Project Director for his or her participation.

This procedure will be followed for the follow-up data collection time points.

A procedures manual and the attached PD Interview protocol will be developed for the administration of the PD Interview and a training webinar will be provided to all interviewers and notetakers to walk through interview procedures and questions. The training webinar will be recorded and accessible for later viewing, if needed.

B.3. Methods to Maximize Response Rates

Grantees are required to participate in all SPF-PFS cross-site evaluation activities by the Terms and conditions of the SPF-PFS grant award. The SPF-PFS evaluation team will employ a number of strategies to help ensure grantees and subrecipient communities participate with a 100% response rate.

As described above, the SPF-PFS evaluation will develop user manuals for accessing and navigating the PEP-C online data collection system and question-by-question and FAQ guides to help respondents accurately complete the GLI-R and CLI-R. Grantees will also be provided training webinars to introduce the SPF-PFS evaluation, to walk through the PEP-C online data collection system, to review data collection procedures, and to do a question-by-question review of the GLI-R, CLI-R, and PD Interview. Within the online data collection system, all manuals, guides, and training webinars will be archived and accessible to respondents for reference at any time.

For online web-based surveys, grantees and subrecipient communities will be sent automated, predefined emails to remind them of submission deadlines. Specifically, the following reminder schedule will be followed:

1. One Week before Data Submission Deadline: One week before the data submission deadline, the system will automatically send an email reminder to grantees and subrecipients that have not yet provided their data.

2. One Day After Data Submission Deadline: The data submission system will automatically send a system-generated email to nonsubmitters alerting them that the data submission deadline has passed. When a nonsubmitter is a subrecipient, the grantee will also be notified.

3. One Week After Data Submission Deadline: The data submission system will automatically send a system-generated email to nonsubmitters and their SAMHSA State Project Officers (SPOs) alerting them that the data submission deadline has passed. When a nonsubmitter is a subrecipient, the grantee will also be notified.

4. Two Weeks After Data Submission Deadline: PEP-C will notify the SPO, who will request a telephone call with grantees (or with subrecipients and their respective grantees) who have not submitted their data by 2 weeks after the deadline. Grantees will be expected to monitor their subrecipients’ data submission compliance.

For the PD Interview, the initial email invitation will provide a thorough explanation of the study and its importance, the reasons the respondent is being asked to participate, and means by which they can contact the evaluation team for additional information. The evaluation team will aim to identify the most convenient time for grantee Project Directors to complete the interview. Before the interview, respondents will also be provided the interview topics so they will be knowledgeable about the types of information to be collected. Nonresponders to the initial email invitation will be sent weekly follow-up reminder emails. If needed—although the evaluation team does not anticipate that it will be—the follow-up reminder emails will include the grantee’s SPO.

B.4 Test of Procedures

Three contractor staff completed the GLI-R and the CLI-R, either in paper-pencil form or within word processing software. These staff members have experience with SPF initiatives, including serving as local evaluators for SPF-SIG grantees. The GLI-R is estimated to take 1 hour to complete; this includes 0.5 hours to look up and compile information and 0.5 hours to complete the web survey.

The CLI-R is estimated to take 2.6 hours; this includes time for reading the survey instructions and compiling information needed to respond to survey questions. It is likely that the web-based versions of the GLI-R and CLI-R will take less time than the paper version tested to generate the estimates in this section, as skip patterns will be automated and some items will be prepopulated automatically after initial responses.

The PD Interview was pilot tested with 3 current grantee project directors: two from PFS-II and one from PFS 2013. These interviews were conducted by telephone. Grantee and interviewer feedback from these interviews led to changes in the order of the questions to improve the flow of the interviews. The PD Interview is estimated to take 1.4 hours.

Similar versions of the GLI and CLI were developed and have been implemented in previous SPF State Incentive Grant (SIG) evaluations (OMB No. 0930-0279). Each of the SPF-PFS grantees is a former SPF SIG grantee; thus they will all have experience completing surveys similar in procedure (e.g., entering data into an online data system), length (although the current GLI-R and CLI-R burden times are reduced), and content. Additionally, the SPF-PFS evaluation used lessons learned from the SPF SIG evaluations to improve data collection procedures. In the SPF SIG evaluation, the GLI and the CLI were each split into two separate surveys, which caused respondent confusion over the timing of deadlines for data submission. To resolve this problem, the SPF-PFS evaluation has combined the two parts of the GLI survey (Infrastructure and Implementation) into one GLI-R survey and the two versions of the CLI survey (Parts I and II) into one CLI-R survey, but the PEP-C online data collection system will be programmed to display only items relevant at the time of the response. For example, PFS 2013 subrecipients will respond to items related to their capacity development only at their baseline and in their final years, whereas they will respond to intervention implementation items every 6 months.

The PEP-C evaluation team also has experience implementing data collection procedures similar to those outlined for the PD Interview from a national cross-site evaluation of SAMHSA’s Homeless Programs (OMB No. 0930-0339). During this evaluation, a Project Director interview was conducted with grantee Project Directors with a 100% participation rate. It is important to note that the Homeless Programs Project Director interview was double the length of the SPF-PFS PD Interview, and still each interview was completed with no break-offs or refusals.

B.5 Statistical Consultants

The contractor team comprises several experts who will be directly involved in data collection and statistical analysis. Also, contractor in-house experts will be consulted throughout the program on various statistical aspects of the design, methodological issues, and data analysis, including cost analysis. Finally, the PEP-C project has an External Steering Committee. Members of this External Steering Committee have already provided feedback on the instruments and the evaluation/analysis plan and will continue to provide advice and feedback to the evaluation through scheduled quarterly meetings and ad hoc e-mails as needed. Exhibit 10 provides details of these team members and advisors.

Exhibit 10. Statistical Consultants for the Program Evaluation for Prevention Contract (PEP-C)

Name & Role in Evaluation	Title & Address	Contact Information
PEP-C Evaluation Staff
Laura Dunlap, PhD PEP-C Cost Analysis Team Leader	Director Behavioral Health Economics Program RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 541–7310 Email: ljd@rti.org
Elvira Elek, PhD PEP-C Deputy Director	Research Public Health Analyst Public Health Policy Research RTI International 701 13th Street, NW, Suite 750 Washington, DC 20005	Telephone: (202) 728–2048 Email: eelek@rti.org
Phillip Graham, PhD PEP-C Project Director	Senior Research Public Health Analyst Crime, Violence, and Justice Program RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 485–7752 Email: pgraham@rti.org
Nilufer Isvan Senior Evaluator	Senior Research Fellow Human Service Research Institute (HSRI) 2336 Massachusetts Avenue  Cambridge, MA 02140	Telephone: (617) 844-2505 Email: nisvan@hsri.org
Gillian J. Leichtling Senior Evaluator	Senior Research Associate RMC Research Corporation 111 SW Columbia Street  Suite 1030  Portland, OR 97201-5883	Telephone: (503) 223-8248 x735 Email:GLeichtling@rmccorp.com
Antonio Morgan-Lopez, PhD PEP-C Analysis Team Leader	Senior Research Quantitative Psychologist Risk Behavior and Family Research RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 316–3436 Email: amorganlopez@rti.org
Virginia Mulkern Senior Evaluator	Executive Vice President Human Service Research Institute (HSRI) 2336 Massachusetts Avenue  Cambridge, MA 02140	Telephone: (617) 844-2315 Email: mulkern@hsri.org
Scott Novak, PhD Senior Statistician	Senior Research Public Health Analyst Behavioral Health & Epidemiology RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 541–7129 Email: snovak@rti.org
Steve Sullivan ESC Task Team Leader	Senior Director Cloudburst Consulting Group, Inc. 8400 Corporate Drive, Suite 550 Landover, MD 20785-2238	Telephone: (301) 918-4400 Email: steven.sullivan@cloudburstgroup.com
James Trudeau, PhD Senior Advisor	Director Crime, Violence, and Justice Program RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 485–7751 Email: trudeau@rti.org
Government Project Officers
Pamela Roddy, PhD Contracting Officer’s Representative	Senior Public Health Analyst CSAP, SAMHSA 1 Choke Cherry Road, Room 4-1025 Rockville, MD 20857	Telephone: (240) 276–2422 Email:Pamela.Roddy@samhsa.hhs.gov
Beverlie Fallik, PhD Alternate Contracting Officer’s Representative	Senior Public Health Analyst CSAP, SAMHSA 1 Choke Cherry Road, Room 4-1031 Rockville, MD 20857	Telephone: (240) 276–2438 Email: Beverlie.Fallik@samhsa.hhs.gov
External Steering Committee
Bethany Bray, PhD Methods/Statistics	Research Associate The Methodology Center The Pennsylvania State University 400 Calder Square II State College, PA 16801	Telephone: (814) 865-1225 Email: bcbray@psu.edu
William DeJong, PhD Evaluating Environmental Strategies	Professor Boston University School of Public Health Community Health Sciences 801 Mass Ave Crosstown Center Boston MA 02118	Telephone: (508) 954-0224 Email: wdejong@bu.edu
Brian Flay, DrPH Prevention Science	Professor Oregon State University College Of Public Health and Human Sciences 457 Waldo Hall Corvallis, OR 9733	Telephone: (541) 737-3837 Email: Brian.Flay@oregonstate.edu
Rick Harwood Economics, Cost Analyses	Director of Research and Program Applications National Association of State Alcohol and Drug Abuse Directors, Inc., (NASADAD) 1025 Connecticut Avenue NW, Suite 605 Washington, DC 20036	Telephone: (202) 293-0090, ext. 104 Email: rharwood@nasadad.org
Dottie Natal IT, Data Collection Systems	CEO Imagen Multimedia Corp	Email:dottie@imagenmm.com
Chris Ringwalt, DrPH Intervention Implementation and Dissemination	Public Health Senior Research Scientist Pacific Institute for Research and Evaluation 1516 E. Franklin Street, Suite 200 Chapel Hill, NC 27514-2812	Telephone: (919) 259-0643 Email: ringwalt@PIRE.org;

REFERENCES

Calear, A. L., & Christensen, H. (2010). Systematic review of school-based prevention and early intervention programs for depression. Journal of Adolescence, 33(3), 429–438.

Centers for Disease Control and Prevention. (2012). CDC grand rounds: prescription drug overdoses - a U.S. epidemic. MMWR. Morbidity and Mortality Weekly Report, 61(1), 10–13.

Cheong, J., MacKinnon D. P., & Khoo S.-T. (2003). Investigation of mediational processes using parallel process latent growth curve modeling. Structural Equation Modeling, 10, 238–262.

Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: the simultaneous analysis of multiple data sets. Psychological Methods, 14(2), 81–100.

Curran, P. J., Hussong, A. M., Cai, L., Huang, W., Chassin, L., Sher, K. J., & Zucker, R. A. (2008). Pooling data from multiple longitudinal studies: The role of item response theory in integrative data analysis. Developmental Psychology, 44, 365–380.

Drummond, M. F., Sculpher, M. J., Torrance, G. W., O’Brien, B. J., & Stoddart, G. L. (2005). Methods for the economic evaluation of health care programmes (3rd ed.). Oxford, UK: Oxford University Press.

Feinberg, M. E., Greenberg, M. T., Osgood, D. W., Sartorius, J., & Bontempo, D. (2007). Effects of the Communities That Care model in Pennsylvania on youth risk and problem behaviors. Prevention Science, 8(4), 261–270.

Gold, M. R., Siegel, J. E., Russell, L. B., & Weinstein, M. C. (Eds.). (1996). Cost-effectiveness in health and medicine. New York, NY: Oxford University Press.

Harder, V. S., Stuart, E. A., & Anthony, J. (2010). Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychological Methods, 15, 234–249.

Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). New York, NY: Routledge.

Imai, K., & van Dyk, D. A. (2004). Causal inference with general treatment regimes: Generalizing the propensity score. Journal of the American Statistical Association, 99, 854–866.

Jagers, R. J., Morgan-Lopez, A., Flay, B., & Aban Aya Investigators. (2009). Age and intervention effects on youth violent behavior. Journal of Primary Prevention, 30, 642–658.

Johnston, L. D., O’Malley, P. M., Bachman, J. G., & Schulenberg, J. E. (2013). Monitoring the Future national results on drug use: 2012 overview, key findings on adolescent drug use. Ann Arbor: Institute for Social Research, The University of Michigan.

Lemstra, M., Bennett, N., Nannapaneni, U., Neudorf, C., Warren, L., Kershaw, T., et al. (2010). A systematic review of school-based marijuana and alcohol prevention programs targeting adolescents aged 10–15. Addiction Research and Theory, 18, 84–96.

McCaffrey, D. F., Ridgeway, G. and Morral, A. R. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods, 9, 403–425.

Miller, T. R., Levy, D. T., Spicer, R. S., & Taylor, D. M. (2006). Societal costs of underage drinking. Journal of Studies on Alcohol, 67(4), 519–528.

Muthén, B. (1997). Latent variable modeling with longitudinal and multilevel data. In A. Raftery (Ed.), Sociological Methodology (pp. 453–480). Boston, MA: Blackwell Publishers.

National Institutes of Health. (2011, October). Prescription drugs: Abuse and addiction (NIH Publication No. 11-4881). Retrieved from http://www.drugabuse.gov/sites/default/files/rrprescription.pdf

Office of Juvenile Justice and Delinquency Prevention. (2012, September). Effects and consequences of underage drinking (NCJ 237145). Juvenile Justice Bulletin. Washington, DC: Author. Retrieved from http://www.ojjdp.gov/pubs/237145.pdf

Office of National Drug Control Policy. (2011). Epidemic: responding to America’s prescription drug abuse crisis. Retrieved from: http://www.whitehouse.gov/sites/default/files/ondcp/issues-content/prescription-drugs/rx_abuse_plan_0.pdf

Office of the Surgeon General. (2007). The Surgeon General’s call to action to prevent and reduce underage drinking. Rockville, MD: Author. Retrieved from: http://www.surgeongeneral.gov/library/calls/underagedrinking/calltoaction.pdf

Orwin, R. G., Stein-Seroussi, A., Edwards, J. M., Landy, A. L., & Flewelling, R. L. (2014). Effects of the Strategic Prevention Framework State Incentives Grant (SPF SIG) on state prevention infrastructure in 26 states. Journal of Primary Prevention, 35, 163-180.

Piper, D., Stein-Seroussi, A., Flewelling, R., Orwin, R. G., & Buchanan, R. (2012). Assessing state substance abuse prevention infrastructure through the lens of CSAP’s Strategic Prevention Framework. Evaluation and Program Planning, 35(1), 66–77.

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.

Sacks, J. J., Roeber, J., Bouchery, E. E., Gonzales, K., Chaloupka, F. J., & Brewer, R. D. (2013). State costs of excessive alcohol consumption, 2006. American Journal of Preventive Medicine, 45(4), 474–485.

Shadish, W., Cook, T., & Campbell, D. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin.

Shadish, W. R. (2010). Campbell and Rubin: A primer and comparison of their approaches to causal inference in field settings. Psychological Methods, 15, 3–17.

Spoth, R., Greenberg, M., Bierman, K., & Redmond, C. (2004). PROSPER community-university partnership model for public education systems: capacity-building for evidence-based, competence-building prevention. Prevention Science, 5(1), 31–39.

Spoth, R., Greenberg, M., & Turrisi, R. (2008). Preventive interventions addressing underage drinking: State of the evidence and steps toward public health impact. Pediatrics, 121(Suppl 4), S311–336.

Spoth, R., Greenberg, M., & Turrisi, R. (2009). Overview of preventive interventions addressing underage drinking: State of the evidence and steps toward public health impact. Alcohol Research & Health, 32(1), 53–66.

Spoth, R., Redmond, C., Clair, S., Shin, C., Greenberg, M., & Feinberg, M. (2011). Preventing substance misuse through community-university partnerships: randomized controlled trial outcomes 4(1/2) years past baseline. American Journal of Preventive Medicine, 40(4), 440–447.

Substance Abuse and Mental Health Services Administration. (2012a). National cross-site evaluation of the Strategic Prevention Framework State Incentive Grants for Cohort III, IV, and V: Final evaluation report.

Substance Abuse and Mental Health Services Administration. (2012b). 2011 National Survey on Drug Use and Health: Sample design report. Retrieved from http://www.samhsa.gov/data/2k12/NSDUH2011MRB/NSDUHmrbSampleDesign2011.pdf.

Substance Abuse and Mental Health Services Administration. (2013). Drug Abuse Warning Network, 2011: National estimates of drug-related emergency department visits. (DAWN Series D-39, HHS Publication No. (SMA) 13-4760). Rockville, MD: Author. Retrieved from http://www.samhsa.gov/data/2k13/DAWN2k11ED/DAWN2k11ED.htm#5

Substance Abuse and Mental Health Services Administration. (2013b). Results from the 2012 National Survey on Drug Use and Health: Summary of national findings (NSDUH Series H-46, HHS Publication No. (SMA) 13-4795). Rockville, MD: Author. Retrieved from http://www.samhsa.gov/data/NSDUH/2012SummNatFindDetTables/NationalFindings/NSDUHresults2012.pdf

Substance Abuse and Mental Health Services Administration. (2014). Results from the 2013 National Survey on Drug Use and Health: Summary of national findings (NSDUH Series H-48, HHS Publication No. (SMA) 14-4863). Rockville, MD: Author. Retrieved from http://www.samhsa.gov/data/NSDUH/2013SummNatFindDetTables/NationalFindings/NSDUHresults2013.htm#2.3

Ttofi, M. M., & Farrington, D. P. (2011). Effectiveness of school-based programs to reduce bullying: A systematic and meta-analytic review. Journal of Experimental Criminology, 7, 27–56.

United States Department of Health and Human Services. (2011). National prevention strategy: America’s plan for better health and wellness. Washington, DC: National Prevention Council. Retrieved from: http://www.surgeongeneral.gov/initiatives/prevention/strategy/report.pdf

Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., et al. (2008). Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41(3-4), 171–181.

West, S. G., Biesanz, J. C., & Pitts, S. C. (2000). Causal inference and generalization in field settings: Experimental and quasi-experimental designs. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 40–84). New York, NY: Cambridge University Press.

LIST OF ATTACHMENTS

1: PEP-C Grantee-Level Instrument—Revised

2: PEP-C Community-Level Instrument—Revised

3: PEP-C Grantee Project Director Interview

4: Consultation Outside the Agency

4