Program Evaluation for Prevention Contract (PEPC) Evaluation Activities
Supporting Statement
Using a census approach, the targeted universe for the Program Evaluation for Prevention Contract (PEPC) Strategic Prevention Framework for Prescription Drugs (SPF Rx) fiscal year (FY) 2021 cohort evaluation is all SPF Rx grantee project directors (N = 21), as well as each of the SPF Rx grantees’ subrecipient community partners (N = 110). As grantees agreed to participate in cross-site evaluation data collection activities as a condition of funding, all 21 grantee project directors or other designated grantee staff are expected to complete the SPF Rx cross-site instruments.
To meet its annual reporting requirements for the Government Performance and Results Act (GPRA) and for performance monitoring, the Substance Abuse and Mental Health Services Administration (SAMHSA) strives to obtain data from all grantees. The evaluation team works with SAMHSA to promote timely and accurate responses from grantees and subrecipients. Data sources vary for indicators reported by SPF Rx grantees. For example, some grantees are subject to laws and policies that restrict access to, and use of, Prescription Drug Monitoring Program (PDMP) data. With a larger N, the evaluation team is able to answer the evaluation questions (EQs) and assess the impact of the program, even when grantees and subrecipients encounter data challenges.
Pending OMB approval, the instruments will be collected according to the schedule outlined in Section A.16.
Online Data Collection
The SPF Rx Annual Reporting Tool (ART) and Grantee- and Community-Level PDMP Outcomes Modules are self-administered, web-based data collection tools completed through the SPF Rx Evaluation Cross-Site Data Collection Platform (CS-DCP). The Grantee-Level Interview is a phone interview conducted by an evaluation team interviewer.
Grantee project directors or their designated staff complete the ART to describe their own intervention implementation and both the Grantee- and Community-Level PDMP Outcomes Modules. The ART is completed by subrecipient community project directors or their designees, although grantee staff have the option of completing on behalf of subrecipients. Before data collection begins, each respondent is provided a unique login to enter the CS-DSP, which requires the creation of a password.
For SPF Rx, the cross-site evaluation team will develop two user manuals to assist grantees and subrecipients in completing data collection in the online CS-DCP: one ART manual and one manual that covers the Grantee-and Community-Level Outcomes Modules. The team will also develop a Frequently Asked Questions (FAQ) document addressing common questions and challenges respondents encounter when completing the instruments. Each year, the evaluation team will provide training webinars to grantees and subrecipients to walk through the online data collection system, review each instrument in detail, and go over data collection procedures.
The CS-DCP will be modeled after the platform used for data collection for the SPF-Rx FY 2016 cohort evaluation to enhance familiarity and ease of use. The current PEPC team built the earlier platform and will implement user-friendly features across all functional areas, taking into account the needs of both SAMHSA and grantees.
Respondents can easily request assistance by calling the toll-free PEPC number or sending an email request to the PEPC Help Desk. The toll-free line is routed to an email system that is checked regularly by members of the training and technical assistance team, so voicemail messages are transmitted via email even when PEPC staff are not actively monitoring the phone line. Staff responding to requests are trained in the data collection instruments and use of the CS-DCP. Support staff will have ready access to all available resources (e.g., manuals, FAQ), and have a system for referring difficult data or technology issues to the appropriate PEPC team subject matter experts for further assistance.
Grantee-Level Interviews
The PEPC team will interview grantee project directors or their designated staff in the first, third, and final years of the project following the protocol outlined in Attachment 4. The PEPC team will contact grantee project directors via email to set up interviews during regularly scheduled business hours. Before conducting the initial interview, the PEPC team will review information abstracted from SPF Rx grantee proposals, as well as SAMHSA's Performance Accountability and Reporting System (SPARS) reporting data already provided by the grantee (e.g., progress reports and strategic plans). To reduce respondent burden and redundancy, relevant information will be abstracted to familiarize the interviewer with the grantee. Once the interview is scheduled, the PEPC team will provide the grantee project director or designated respondent with a Grantee-Level Interview protocol for review. This ensures the participant is prepared to answer the questions or can include the correct team member to do so.
Before beginning each interview, the interviewer will request consent to record the interview as a method of confirming, if needed, the accuracy of noted responses. A senior evaluator from the PEPC team will lead the respondent through the interview while a junior evaluator records responses and takes notes. Notes from the interview will be maintained on a password-protected, secure server, accessible only to members of the PEPC team who need access and have completed the required security and privacy training. After the interview, the interviewer will send an email thanking the respondent for their participation. The PEPC team will transcribe the interview and then review the transcript for accuracy. The PEPC team will then send the transcript to the grantee for review. This is to ensure that the transcript is accurate and give the grantee an opportunity to edit or to provide different or additional context for any responses. Once the transcripts are considered final, the recording will be deleted. An electronic version of the transcript will be maintained on a password-protected, secure server accessible only to the PEPC team who require access and have completed the necessary security and privacy trainings.
The terms and conditions of the grant awards require grantees to participate in all data collection activities. The PEPC team will employ several strategies to help ensure that grantees participate in all required data collections for the SPF Rx evaluation.
The SPF Rx program requires participation in data collection as a condition of the grant. For the prior evaluation of the SPF-Rx 2016 grantee cohort, in FY 2020 the PEPC team received outcomes data from 25 of 25 grantees and for 119 of 122 active subrecipients; each grantee and subrecipient submitted their equivalent of the ART. The PEPC evaluation team expects the same response going forward and will employ the below strategies to maximize the response rate.
ART, Grantee- and Community-Level Outcomes
As described above, the PEPC team will develop user manuals for accessing and navigating the CS-DCP online data collection system and an FAQ document to help respondents accurately complete the online instruments. These documents will be disseminated to the SPF Rx grantees and their subrecipients and made available by contacting the PEPC help desk. Grantees will be provided training webinars to walk through the CS-DCP online data collection system and to review data collection procedures. The PEPC team has a help desk email and phone number, with team members available to answer data- and technology-related questions. During the data reporting period, the PEPC team will send regular (at least weekly) reminder emails with resources to encourage on-time reporting.
The PEPC team will work with SAMHSA project officers to identify grantees’ data availability concerns early and identify ways to help grantees obtain and report required data. Approximately a week after a data submission deadline, the PEPC team will provide project officers a list of past-due instruments. SAMHSA project officers will follow up with their grantees to ensure submission. Additionally, during the data cleaning process, the PEPC team will provide data feedback forms to document missing data elements. The PEPC team will follow up regularly with grantees to remind them to finalize data entry, requiring Data Feedback Forms to be completed and returned within 2 weeks. SAMHSA project officers will be copied on these correspondences and alerted about grantees who have not completed data collection. SAMHSA project officers are asked to do additional outreach to their grantees to remind them of the grant’s data reporting requirements and urge them to enter their data.
Grantee-Level Interviews
All grantee project directors are required to participate in the Grantee-Level Interview. The PEPC team will ensure participation by conducting email outreach to grantee project directors to schedule interviews. If there is no reply within 1 week, a second reminder email will be sent. If this does not elicit a response, the PEPC team will follow up with phone calls to the grantee project director. SAMHSA project officers will be copied on all correspondence and will be alerted to grantees who have not replied. If needed, the PEPC team will ask the project officers to reach out to their grantees and urge them to schedule their interviews with the PEPC team.
Each of the SPF Rx data collection instruments were implemented in earlier forms with the SPF Rx FY 2016 grantees. Some revisions were made to the instruments to clarify wording and to incorporate implementation science elements into the ART and Grantee-Level Interviews. However, the data collection procedures will use the processes used in the earlier evaluation.
The PEPC team comprises team members listed in Exhibit 11, who will serve as statistical consultants for the SPF Rx evaluation. Exhibit 11 provides details of these team members and advisors.
Exhibit 11. Statistical Consultants for the SPF Rx Cross-Site Evaluation through the Program Evaluation for Prevention Contract
Name & Role |
Title & Address |
Contact Information |
PEPC Evaluation Staff |
||
Elvira Elek, PhD Project Director |
Senior Research Public Health Analyst RTI International 3040 East Cornwallis Road P.O. Box 12194 Research Triangle Park, NC 27709 |
Telephone: (202) 728-2048 Email: eelek@rti.org |
Jessica Duncan Cance, PhD Associate Project Director |
Research Public Health Analyst RTI International 3040 East Cornwallis Road P.O. Box 12194 Research Triangle Park, NC 27709 |
Telephone: (919) 541-7410 Email: jcance@rti.org |
Jason Williams, PhD Outcomes Evaluation Lead |
Research Public Health Analyst RTI International 3040 East Cornwallis Road P.O. Box 12194 Research Triangle Park, NC 27709 |
Telephone: (919) 541-6734 Email: jawilliams@rti.org |
Sheila Patel, PhD Process Evaluation Lead |
Research Public Health Analyst RTI International 3040 East Cornwallis Road P.O. Box 12194 Research Triangle Park, NC 27709 |
Telephone: (919) 541-6221 Email: svpatel@rti.org |
Michael Bradshaw Data Processing co-Task Lead |
Research Public Health Analyst RTI International 3040 East Cornwallis Road P.O. Box 12194 Research Triangle Park, NC 27709 |
Telephone: (919) 541-7004 Email: mbradshaw@rti.org |
Derek Ramirez, MS Data Processing co-Task Lead |
Data Manager RTI International 3040 East Cornwallis Road P.O. Box 12194 Research Triangle Park, NC 27709 |
Telephone: (919) 485-5517 Email: dramirez@rti.org |
Government Project Officers |
||
Nicole Scott, MPH Contracting Officer Representative |
Social Science Analyst Center for Behavioral Health Statistics Quality, Substance Abuse and Mental Health Services Administration (SAMHSA) 5600 Fishers Lane, #15E29D Rockville, MD 20857 |
Telephone: (240) 276-0565 Email: nicole.scott@samhsa.hhs.gov |
Jon Dunbar-Cooper, MA Alternate Contracting Officer Representative
|
Public Health Analyst Center for Substance Abuse Prevention, Substance Abuse and Mental Health Services Administration (SAMHSA) 5600 Fishers Lane, #16E85D Rockville, MD 20879 |
Telephone: 240-276-2573 Email: jon.dunbar@samhsa.hhs.gov |
Centers for Disease Control and Prevention. (2021). Opioids: Understanding the epidemic. Retrieved January 4, 2022, from https://www.cdc.gov/opioids/basics/epidemic.html.
Damschroder, L. J., Aron, D. C., Keith, R. E. et al. (2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science, 4, 50. https://doi.org/10.1186/1748-5908-4-50
Hedegaard, H., Minino, A. M., &Warner, M. (2018). Drug overdose deaths in the United States, 1999–2017. NCHS Data Brief No. 329. Atlanta, GA: Centers for Disease Control and Prevention.
National Institute on Drug Abuse (2022, January 22). Overdose death rates. https://nida.nih.gov/drug-topics/trends-statistics/overdose-death-rates
Substance Abuse and Mental Health Services Administration (SAMHSA). (2021). Key substance use and mental health indicators in the United States: Results from the 2020 National Survey on Drug Use and Health (HHS Publication No. PEP21-07-01-003, NSDUH Series H-56). Rockville, MD: Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration. https://www.samhsa.gov/data/
U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Behavioral Health Statistics and Quality (2020). National Survey on Drug Use and Health 2020. https://datafiles.samhsa.gov/
Annual Reporting Tool (Attachment 1)
Grantee-Level Outcomes Module (Attachment 2)
Community-Level Outcomes Module (Attachment 3)
Grantee-Level Interview (Attachment 4)
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | King, Summer (SAMHSA/OPPI) |
| File Modified | 0000-00-00 |
| File Created | 2023-08-26 |