SS_B_Rx 6.19.17

Program Evaluation for Prevention Contract: Strategic Prevention Framework for Prescription Drugs

Supporting Statement

B. Collection of Information Employing Statistical Methods

B.1 Respondent Universe and Sampling Methods

Through the instruments at the center of this OMB application, the SPF-Rx data collection will use a census approach to collecting process and programmatic data.

Using a census approach, the targeted universe for the PEP-C SPF-Rx MRT is all SPF-Rx grantee PDs (N = 25) and grantee PDs from all future cohorts. As grantees have agreed to participate in cross-site evaluation data collection activities as a condition of funding, all 25 grantee PDs or other designated grantee staff are expected to complete the SPF-Rx cross-site instruments.

A census of all SPF-Rx grantee PDs is necessary as these data will be used by SAMHSA to monitor each program’s performance, and grantees will also use it to track their ongoing implementation. To meet its annual reporting requirements for GPRA and performance measures, SAMHSA must obtain data from all grantees, which supports the need for a census approach. In addition, the SPF-Rx program grantees have varying data available to them and are subject to varying laws and policies governing access to and use of PDMP data. The variety between the programs makes it critical to the SPF-Rx evaluation to capture the details of each program to be able to answer the EQs and assess which implementation characteristics are associated with better outcomes for particular types of communities.

While grantees will provide aggregated data into the cross-site evaluation tools, all outcomes data will come from existing archival data (e.g., state vital statistics, hospitalization and emergency department records).

B.2 Information Collection Procedures

Web-Based Instruments

With the exception of the SPF-Rx Grantee Interview, all of the instruments are all self-administered, web-based data collection tools completed through the MRT online data collection system. The SPF-Rx AII, Outcomes Module, and Substitute Data Source Request will be accessed through the MRT online data collection system.

Grantee PDs or their designated staff will complete all the instruments. The SPF-Rx AII, Outcomes Module, and Substitute Data Source Request will also be completed by subrecipient community PDs or their designees. Before data collection begins, each respondent will be provided a unique log-in to enter the data system, which will require the creation of a password. Respondent email addresses for each login will be stored within the system so that automatic alerts and notifications can be sent.

For SPF-Rx, the cross-site evaluation team will develop user manuals for accessing and navigating the online data collection systems and question-by-question and FAQ guides to help respondents accurately complete the instruments. Grantees and subrecipients will also be provided training webinars to walk through the online data collection system, review each instrument in detail, and go over data collection procedures. Within the online data collection systems, all manuals, guides, and training webinars will be archived and accessible to respondents for reference at any time. (The PEP‑C team developed these same types of materials for similar SAMHSA projects. See https://PEP‑C.rti.org/HERO/KB/PEP‑C-KB/Default.htm#PFS/PFS Overview.htm%3FTocPath%3DPartnerships%2520for%2520Success%2520(PFS)%7C_____0.)

Availability is important in any data collection system, especially one employed by grantee sites around the country in multiple time zones. The online systems will be maintained in an available state as much as possible to allow grantees to have access for entering data, as well as to give SAMHSA, grantees, and the SPF-Rx cross-site evaluation team access to reports.

The reporting system will be designed to be easy to use. The SPF-Rx evaluation team will implement user-friendly features across all functional areas, taking into account the needs of both SAMHSA and grantees. Additionally, every page of the SPF-Rx MRT online data system will have a Help or Support link, which will allow the respondent to access the following support resources:

1. Search the Knowledge Base. More comprehensive than a list of FAQs and more organized than a support forum, the Knowledge Base offers a layered information approach, so that respondents can search by keyword and then drill down to view material at increasing levels of detail. It will be a curated and easily searchable source of information including items such as

• system documentation,

• user guides,

• policies and procedures,

• protocols,

• training materials, and

• FAQs.

2. Contact Us. Respondents may request assistance by calling a provided toll-free number, sending an email request, or submitting a TA request form as desired. The toll-free line will be routed to an email system that is checked regularly by members of the T/TA team. Staff responding to TA requests will be trained in use of the system and have ready access to the full Knowledge Base. T/TA team staff will monitor all submitted tickets to ensure timely response and resolution of TA requests.

SPF-Rx Grantee Interview

The SPF-Rx cross-site evaluation team will also interview grantee PDs or their designated staff in the second, third, and final years of the grant, following the protocol outlined in Attachment 2. The SPF-Rx evaluation team will contact grantee PDs via email to set up interviews during regularly scheduled business hours. Before conducting the interview, the evaluation team will review grant applications (submitted to SAMHSA by each grantee and given to the evaluation team by SAMHSA) and other reporting data already provided by the grantee (e.g., progress monitoring reports). Relevant information will be abstracted to familiarize the interviewer with the grantee and avoid asking for any information that is already available. This will also reduce respondent burden.

Once the interview is scheduled, the contractor will provide the grantee PD or designated respondent with an electronic version of the assent form and an interview guide.

Before beginning the interview, consent will be requested to record the interview to confirm, if needed, the accuracy of noted responses. A senior evaluator from the contractor’s evaluation team will lead the respondent through the interview while a junior evaluator records responses and takes notes. After the interview, the interviewer will send an email thanking the respondent for his or her participation. The interviewer and notetaker will review the responses for accuracy. Any areas of discrepancy will be validated with the recording (if consented by the respondent). A copy of the notes will also be sent to the respondent for review. Any further comments by the respondent will be added to the notes. Once the responses are considered final, the recording will be deleted. An electronic version of the interview will be maintained on a password-protected, secure server accessible only to the contractor’s evaluation team.

This procedure will be followed for the follow-up data collection time points.

A procedures manual and the attached Grantee Interview protocol will be developed for the administration of the interview. Training on interview procedures and questions will be provided to all interviewers and notetakers. The training will be recorded and accessible for later viewing, if needed.

B.3 Methods to Maximize Response Rates

The terms and conditions of the grant awards require grantees to participate in all SPF-Rx cross-site evaluation data collection activities. The cross-site evaluation team will employ a number of strategies to help ensure that grantees participate with a 100% response rate.

As described above, the evaluation team will develop user manuals for accessing and navigating the MRT online data collection system and question-by-question and FAQ guides to help respondents accurately complete the online instruments. Grantees will also be provided with training webinars to walk through the MRT online data collection system and to review data collection procedures. Within the online data collection system, all manuals, guides, and training webinars will be archived and accessible to respondents for reference at any time.

SAMHSA Project Officers will monitor the MRT online data systems and receive email notifications when their grantees submit individual instruments such as Substitute Data Source Requests. Approximately 1 month after a data submission deadline, the PEP‑C team will provide Project Officers a list of past-due instruments. SAMHSA Project Officers will then follow up with their grantees to ensure submission. This system has already been successfully established and is running smoothly for related SAMHSA grants.

B.4 Test of Procedures

The Grantee Interview was pilot tested with four current SPF-Rx grantee PDs who are or were also PFS PDs. These interviews were conducted by telephone. Grantee and interviewer feedback from these interviews led to changes in the order of the questions to improve the flow of the interviews. The Grantee Interview is estimated to take 1.5 hours to complete; this includes 1.5 hours to complete the interview.

Five current SPF-Rx grantee PDs who are or were also PFS PDs completed the AII in word processing software. PEP‑C staff met with each grantee by telephone to discuss time of completion, questions about information in the instrument, and recommendations for improving the instrument. The AII is estimated to take 2.3 hours to complete; this includes 1.5 hours to look up and compile information and 0.8 hour to complete the web instrument.

Three current SPF-Rx grantee PDs who are or were also PFS PDs completed the Grantee-Level Outcomes Module, the Community-Level Outcomes Module, and the Substitute Data Source Request in word processing software. PEP‑C staff met with each grantee by telephone to discuss time of completion, questions about information in the instrument, and recommendations for improving the instrument. The Grantee-Level Outcomes Module is estimated to take 3 hours to complete; this includes 2 hour to look up and compile information and 1 hour to complete the web instrument. The Community-Level Outcomes Module is estimated to take 3 hours to complete; this includes 2 hours to look up and compile information and 1 hour to complete the web instrument. The Substitute Data Source Request instrument is estimated to take 1 hour to complete; this includes 0.5 hour to look up and compile information and 0.5 hour to complete the web instrument.

Each of the SPF-Rx grantees is a former SPF-PFS grantee; thus they will all have experience completing instruments similar in procedure (e.g., entering data into an online data system), length, and content. Additionally, the SPF-Rx performance measure data collection used lessons learned from the SPF-PFS evaluations to improve data collection procedures.

B.5 Statistical Consultants

The PEP‑C contractor team comprises several experts who will be directly involved in data collection and statistical analysis. Also, contractor in-house experts will be consulted throughout the program on various statistical aspects of the design, methodological issues, and data analysis. Finally, the PEP‑C project has an ESC whose members have already provided feedback on the cross-site evaluation design and instruments and will continue to provide advice and feedback through scheduled quarterly meetings and ad hoc emails as needed. Exhibit 9 provides details of these team members and advisors.

Exhibit 9: Statistical Consultants for the SPF-Rx Cross-Site Evaluation Through the Program Evaluation for Prevention Contract (PEP‑C)

Name & Role in Evaluation	Title & Address	Contact Information
PEP‑C Evaluation Staff
Elvira Elek, PhD PEP‑C Deputy Director	Research Public Health Analyst Public Health Policy Research RTI International 701 13th Street, NW, Suite 750 Washington, DC 20005	Telephone: (202) 728–2048 Email: eelek@rti.org
Phillip Graham, PhD PEP‑C Project Director	Director Drugs, Violence, and Delinquency Prevention Program RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 485–7752 Email: pgraham@rti.org
Gillian J. Leichtling Senior Evaluator	Senior Research Associate RMC Research Corporation 111 SW Columbia Street, Suite 1030  Portland, OR 97201-5883	Telephone: (503) 223-8248 x735 Email: mailto:GLeichtling@rmccorp.com
Monique Clinton-Sherrod, PhD SPF-Rx Project Director	Director Violence and Victimization Research Program RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone (919) 597-5103 Email: mclinton@rti.org
Cheryl Roberts, PhD SPF-Rx Deputy Project Director	Research Sociologist Drugs, Violence, and Delinquency Prevention Program RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone (919) 541-7251 Email: chroberts@rti.org
Jason Williams, PhD PEP‑C SPF-Rx Analysis Team Leader	Senior Research Quantitative Psychologist Risk Behavior and Family Research RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 541-6734 Email: jawilliams@rti.org
Antonio Morgan-Lopez, PhD PEP‑C Analysis Team Leader	Senior Research Quantitative Psychologist Risk Behavior and Family Research RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 316-3436 Email: amorganlopez@rti.org
Nick Peiper	Research Public Health Analyst Behavioral and Urban Health Program RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 316-3314 Email: npeiper@rti.org
Steve Sullivan ESC Task Team Leader	Senior Director Cloudburst Consulting Group, Inc. 8400 Corporate Drive, Suite 550 Landover, MD 20785-2238	Telephone: (301) 918-4400 Email: msteven.sullivan@cloudburstgroup. com
Government Project Officers
Thomas Clarke, PhD Contracting Officer’s Representative	Social Science Analyst Performance and Technical Assistance Branch Division of Systems Development SAMHSA/CSAP 5600 Fishers Lane, Room 16E07 Rockville, MD 20867	Telephone: (240) 276-0493 Email: Thomas.Clarke@samhsa.hhs.gov
Anthony Johnson Alternate Contracting Officer’s Representative	Senior Public Health Analyst Program Management Officer Performance and Technical Assistance Branch Division of Systems Development SAMHSA/CSAP 5600 Fishers Lane, Room 16E01B Rockville, MD 20857	Telephone: (240) 276-0135 Email: anthony.johnson@samhsa.hhs.gov
Sandra Adrovet Contracting Officer’s Representative	Public Health Advisor East Team SAMHSA/CSAP 1 Choke Cherry Road, Room 4-1048 Rockville, MD 20857	Telephone: (240) 276-1489 Email: sandra.adrovet@samhsa.hhs.gov
Kim Nesbitt Contracting Officer’s Representative	Public Health Advisor West Team SAMHSA/CSAP 5600 Fishers Lane, Room 16E65A Rockville, MD 20857	Telephone: 240-276-1742 Email: Kim.Nesbitt@samhsa.hhs.gov
Tonia Gray Contracting Officer’s Representative	Senior Public Health Advisor West Team Lead Division of State Programs SAMHSA/CSAP 5600 Fishers Lane, Room 16E25B Rockville, MD 20857	Telephone: (240) 276-2492 Email: tonia.gray@samhsa.hhs.gov
Stephanie Blake	Social Science Analyst Division of Systems Development SAMHSA/CSAP 5600 Fishers Lane, Room 16E09C Rockville, MD 20857	Telephone: (240) 276-1623 Email: Stephanie.blake@samhsa.hhs.gov
External Steering Committee
Bethany Bray, PhD Methods/Statistics	Research Associate The Methodology Center The Pennsylvania State University 400 Calder Square II State College, PA 16801	Telephone: (814) 865-1225 Email: bcbray@psu.edu
William DeJong, PhD Evaluating Environmental Strategies	Professor Boston University School of Public Health Community Health Sciences 801 Mass Ave Crosstown Center Boston, MA 02118	Telephone: (508) 954-0224 Email: wdejong@bu.edu
Brian Flay, DrPH Prevention Science	Professor Oregon State University College of Public Health and Human Sciences 457 Waldo Hall Corvallis, OR 9733	Telephone: (541) 737-3837 Email: Brian.Flay@oregonstate.edu
Chris Ringwalt, DrPH Instrument Design; Intervention Implementation and Dissemination	Public Health Senior Research Scientist Pacific Institute for Research and Evaluation 1516 East Franklin Street, Suite 200 Chapel Hill, NC 27514-2812	Telephone: (919) 259-0643 Email: ringwalt@PIRE.org
Brenda Rowe	FDA Program Director Alcohol and Tobacco Division GA Department of Revenue 1800 Century Center Boulevard, Suite 400 Atlanta, GA 30345	Telephone: (404) 417-4234 Email: drbcatas@bellsouth.net

LIST OF ATTACHMENTS

1. Annual Implementation Instrument

2. Grantee Interview

3. Grantee-Level Outcomes Module

4. Community-Level Outcomes Module

5. Substitute Data Source Request

6. SPF-Rx Evaluation Plan Checklist

5