Program Evaluation for Prevention Contract: Strategic Prevention Framework for Prescription Drugs
Supporting Statement
The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Prevention (CSAP) is requesting approval from the Office of Management and Budget (OMB) for new data collection activities related to the cross-site evaluation of SAMHSA’s Strategic Prevention Framework for Prescription Drugs (SPF-Rx). The SPF-Rx program is designed to address nonmedical use of prescription drugs as well as opioid overdoses by raising awareness about the dangers of sharing medications and by working with pharmaceutical and medical communities. The SPF-Rx program aims to promote collaboration between states/tribes and pharmaceutical and medical communities to understand the risks of overprescribing to youth age 12–17 and adults 18 years of age and older and enhance capacity for and access to Prescription Drug Monitoring Program (PDMP) data for prevention purposes.
The SPF-Rx program aims to address SAMHSA’s priorities on prevention and reduction of prescription drug and illicit opioid misuse and abuse. Its indicators of success are reductions in opioid overdoses and the incorporation of PDMP data into needs assessments and strategic plans. Data collected through the tools described in this statement will be used for the national cross-site evaluation of SAMHSA’s SPF-Rx program. This clearance package covers continued data collection through 2020, as the evaluation is expected to continue through at least that time; however, the Program Evaluation for Prevention Contract (PEP‑C) is scheduled to conduct a national cross-site evaluation of SPF-Rx through September 2018. The PEP‑C team will systematically collect and maintain an Annual Implementation Instrument (AII) and outcomes data submitted by SPF-Rx grantees through the online PEP‑C Management Reporting Tool (MRT). SAMHSA is requesting approval for data collection for the SPF-Rx cross-site evaluation with the following four instruments:
AII to collect information from primary SPF-Rx grantees and their subrecipient communities about SPF-Rx implementation, including subrecipient communities’ progress through the SPF and the specific prevention interventions being implemented by the subrecipient communities and primary grantees. Information collected will include subrecipients’ organization types, funding, cultural competence, assessments, capacity building, sustainability, strategic planning, prevention intervention implementations, evaluations, and contextual factors (Attachment 1).
Grantee Interview to obtain the perspective of the implementing Project Directors (PDs) or their staff on important topics, including infrastructure and capacity, collaboration, leveraging funding and resources, criteria and use of evidence-informed interventions, monitoring and evaluation, collaboration, challenges, and health disparities. Information from these interviews will help inform SPF-Rx cross-site evaluation reports and will help identify lessons learned and success stories from grantees’ SPF-Rx programs (Attachment 2).
Grantee- and Community-Level Outcomes Modules to collect data on key SPF-Rx program outcomes, including opioid misuse and abuse, opioid overdoses, and opioid prescribing patterns. Grantees will provide outcomes data at the grantee level for their state, tribal area, or jurisdiction, as well as at the community level for each of their subrecipient communities.
Grantee-Level Outcomes Module (Attachment 3)
Community-Level Outcomes Module (Attachment 4)
Substitute Data Source Request to allow grantees to request permission from SAMHSA to use “substitute measures” for their outcomes data—that is, measures that differ from a list of preapproved outcomes measures (Attachment 5).
SAMHSA’s SPF-Rx grant program is authorized under Section 516 of the Public Health Service Act, as amended, and addresses Healthy People 2020 Substance Abuse Topic Area HP 2020-SA. The SPF-Rx grant program also supports SAMHSA’s Strategic Initiative: Prevention of Substance Abuse and Mental Illness. Finally, the SPF-Rx grant program seeks to address behavioral health disparities among racial and ethnic minorities by encouraging the implementation of strategies to decrease the differences in access, service use, and outcomes among the racial and ethnic minority populations served.
Opioid misuse, abuse, and overdose are significant public health issues in the United States. Prescription drug misuse (PDM; including abuse) among young people is the fastest-growing drug problem in the country; after alcohol, prescription drugs are second only to marijuana as the drugs most abused by teenagers (National Institutes of Health, 2011). PDM refers to the use of licit drugs without a prescription to treat issues such as pain, attention deficit disorder, or anxiety; in a way other than prescribed; or because of the feelings the drugs may elicit (National Institutes of Health, 2011). The National Survey on Drug Use and Health (NSDUH) estimates from 2015 indicate that 2.0% of respondents age 12–17 and 2.0% of respondents age 26 or older report current PDM. The problem is especially pronounced among young adults age 18–25, with 5.1% of NSDUH respondents in that age group saying that they are current misusers (SAMHSA, 2016a). The widespread consequences of PDM are similar to those associated with use of alcohol and other drugs (i.e., violence perpetration and victimization along with health, safety, social, emotional, academic, familial, and economic consequences). In 2011, the nonmedical use of pharmaceuticals accounted for nearly a quarter of all drug-related emergency department visits. Approximately 22.5% of these visits involved youth and young adults age 12–24; adults age 25 and older accounted for 76.2% (SAMHSA, 2013). Since 2003, more deaths have been due to opioid analgesic overdoses than to heroin and cocaine combined (Centers for Disease Control and Prevention [CDC, 2012]).
Drug-related overdose is the currently the nation’s leading cause of accidental death, surpassing deaths from motor vehicles beginning in 2008, and deaths from opioid overdose play a significant role in this increase. Opioid overdose fatalities can be attributed to both prescription medications, such as morphine, codeine, oxycodone, and others, and to illegal drugs such as heroin. Opioid overdose deaths attributed to prescription opioids and illicit opioids have quadrupled since 1999 (SAMHSA, 2016a; SAMHSA, 2016b). From 2001 through 2014, the number of overdose deaths involving prescription opioid analgesics more than tripled (SAMHSA, 2016b). In 2014, 40% of drug overdose deaths were associated with prescription opioids, whereas 23.1% were associated with heroin (CDC, 2016a; CDC, 2016b).
A key resource for monitoring prescribing opioids among medical professionals are the PDMPs, state-run databases used to track the prescribing and dispensing of controlled prescription drugs to patients. The PDMPs are designed to monitor this information for suspected abuse or diversion (i.e., channeling drugs into illegal use), and they can give a prescriber or pharmacist critical information regarding a patient’s controlled substance prescription history. All states except Missouri, as well as the District of Columbia and the territory of Guam, have PDMPs to detect high-risk prescribing and patient behaviors.
While prescriber use of PDMPs and program capacity vary from state to state, several case studies suggest that PDMPs are effective in curbing PDM. For example, 70% of Maryland physicians attributed decreases in their opioid prescribing to their use of the state’s PDMP, and the implementation of Florida’s PDMP correlated with significant reductions in both the number of opioid prescriptions written and the number of patients receiving opioids among high-volume prescribers (Lin et al., 2017; Chang et al., 2016). Use of PDMPs has also been associated with reductions in opioid-related overdose deaths, with higher-functioning PDMPs showing greater reductions in death than less robust programs (Patrick et al., 2016).
In FY2017, SAMHSA awarded the 5-year SPF-Rx grant to 25 states, jurisdictions or territories, and tribal organizations. SPF-Rx is grounded in the SPF-based infrastructures that have been developed through the SPF State Incentive Grants (SIGs) that were funded from 2004 to 2010 and the Partnerships for Success (PFS) grants that initiated funding in 2009 and are ongoing. Grantees used the SPF model, which consists of five steps: (1) needs assessment; (2) capacity building; (3) strategic planning; (4) implementation of programs, policies, and practices; and (5) evaluation. Grantees also considered cultural competence and sustainability at each step in the process. The SPF-based programs were established with the goals of preventing the onset and reducing the progression of substance abuse, reducing problems related to substance abuse, and building capacity and infrastructure for prevention. The SPF-Rx program continues the SPF-based model with a specific focus on PDM among youth age 12–17 and adults 18 years of age and older. Through this program, grantees are expected to work with pharmaceutical and medical communities on awareness-raising activities related to dangers of sharing medications and overprescribing and to provide community awareness and prescription drug abuse prevention activities to adults, youth, prescribers, and patients. Training implemented as a part of awareness-raising activities will be informed by SAMHSA’s Opioid Overdose Prevention Toolkit and the CDC’s Policy Guidelines for Prescribing Opioids for Chronic Pain. Additionally, PDMPs will serve as a major data source for identifying the prevalence of PDM and assessing impacts of the program on reductions in misuse.
The SPF-Rx data collection efforts will use an AII, Grantee Interview, and Grantee- and Community-Level Outcomes Modules and Substitute Data Forms. These cross-site measures will provide process data regarding grantee progression through the SPF model, challenges and successes experienced during these steps, PDMP infrastructure building and use, descriptive information about intervention implementation, PDM-related outcomes, training and technical assistance (T/TA) use and needs, and leveraging of funds. This data collection will emphasize the SPF-Rx impact on outcomes related to PDM, including the prevalence of PDM and related consequences such as prescription drug poisonings and overdoses and capacity for and use of PDMP for monitoring prescriber behavior and prevention purposes. The emergence of PDM as a serious public health issue provides a unique opportunity for the SPF-Rx cross-site evaluation to examine the implementation and effectiveness of prevention interventions developed to target this issue.
The SPF-Rx cross-site evaluation and outcomes measurement are expected to have numerous program and policy implications and outcomes at the national, state, and community levels. They will provide valuable information to the prevention field about best practices in real world settings, along with providing guidance to governmental entities and communities as to what types of interventions should be funded and implemented to reduce PDM. Information and guidance about building PDMP capacity and using the data for prevention efforts from the SPF-Rx cross-site evaluation will allow the Federal government, state, tribes, jurisdictions, and local communities to use their resources more effectively and efficiently and to sustain future prevention efforts.
The SPF-Rx evaluation design and measures have been informed by current and previous cross-site evaluation efforts for SAMHSA, drawing heavily from lessons learned through the currently OMB-approved SPF-PFS evaluation (OMB No. 0930-0348). For example, the variability across PFS grantees (which will also apply to SPF-Rx grantees) posed one of the main challenges of evaluation. To minimize this challenge, the SPF-Rx evaluation will focus on assessing the dimensions of variability and on accounting for them in analysis (e.g., control variables in multivariate analysis, moderation analyses), which will allow for accurate description of grantees and characterization of impact. In addition, to take advantage of lessons learned in developing and implementing the SPF-PFS cross-site evaluation instruments, the SPF-Rx cross-site instrument development began with revisions to the SPF-PFS versions of the Community-Level Instrument—Revised (CLI‑R), PD Interview, Outcomes Module, and Substitute Data Request to develop the SPF-Rx AII, SPF-Rx Grantee Interview, SPF-Rx Outcomes Modules, and SPF-Rx Substitute Data Request Form. Because all SPF-Rx grantees are also SPF-PFS grantees, we have reduced reporting burden where possible by eliminating overlapping items that can be abstracted from data collected through SPF-PFS and by streamlining items that were deemed unnecessary for cross-site evaluation purposes for the SPF-PFS project (especially items assessing constructs that appeared less influential on outcomes, such as details on TA received) and items that are less relevant to the SPF-Rx program (e.g., items assessing alcohol-targeted outcomes for the SPF-PFS programs).
During the development of all instruments, input was solicited from grantee-level SPF-Rx PDs and Project Evaluators, SAMHSA, the PEP‑C External Steering Committee (ESC), and other experts and stakeholders (see Exhibit 9, the statistical consultants list, and Section A.8 for consultation outside the agency). After careful review, revisions were made to streamline the instruments, reduce burden, decrease verbosity, increase variation in response options, make items more relevant to the prescription drug focus, and remove cost items. These changes reduced the number of items by more than one-third from the original CLI‑R and simplified the items in many cases.
SPF-Rx AII and outcomes data will be collected in the SPF-Rx MRT. The MRT is a web-based data collection system that will use clickable radio buttons, checkboxes, drop-down choice items, and open-ended text boxes as relevant. The MRT will also allow grantees to upload required documents requested by their Project Officers.
After users access the system, the MRT will direct users to the Home page. The Home page will include the following standard functions on each page throughout the system:
Links to the three landing pages: Contact Information, Annual Implementation Instrument, and Outcome Data. The top navigation menu provides drop-downs that can be used to access pages directly from any other page.
A breadcrumb trail to identify where users are in the system and allow them to move backward to previous sections.
The AII is a web-based survey designed to be completed by grantees and subrecipient community PDs. Data collected from the survey will be used to monitor subrecipient and state, tribal entity, or jurisdiction performance and to evaluate the effectiveness of the SPF-Rx program across states, tribal entities, and jurisdictions. Both grantees and subrecipients will answer questions on prevention interventions that they have implemented. However, subrecipients will also answer questions on their progress through the SPF steps, prevention capacity, and related funding measures. The AII will provide process data related to leveraging of funding, organizational capacity, collaboration with community partners, data infrastructure, planned intervention targets, intervention implementation (categorization, costs, timing, dosage, and reach), evaluation, contextual factors, T/TA needs, and sustainability. The tool will be collected annually; however, not all questions will be answered every time. For instance, SPF-Rx grantees and subrecipients will respond to items related to their targeted populations and outcomes only at baseline unless there are changes, whereas they will respond to intervention implementation items annually (see Exhibit 1 for timing of data collection of various items). Repeated collection of these data is needed to (1) track the grantees’ and subrecipients’ progress and change over time. (2) allow SAMHSA and the grantees to monitor performance and ongoing implementation, and (3) meet new requirements regarding identifying and reducing disparities.
To minimize burden on the subrecipient respondents, all reporting for the AII will be done in a web-based entry form; at each data collection point, only the questions that are required at that time will appear. Responses will also generate skip patterns for later questions in the instrument, where the grantees and subrecipients complete only relevant sets of questions and do not see others. For example, when reporting on their interventions, their reported CSAP strategy type of the intervention later leads respondents to see the questions for that CSAP strategy type sub-form (e.g., prevention education or environmental strategy); they will not see the other sub-forms at all, unless they need to fill them out for another reported intervention. In addition, once completed initially, many items will be automatically prepopulated on later AII administrations. Respondents can keep those prepopulated responses intact or change their answers as relevant. For example, once they provide information about the typology and targets of their interventions, in the future they will see their prior responses and will not need to respond to those items unless their responses have changed.
As described above, the items on the AII were adapted from the previous OMB-approved CLI‑R used for the SPF-PFS evaluation (OMB No. 0930-0348). The AII was developed with substantial input from grantee-level SPF-Rx evaluators, SAMHSA, the PEP‑C ESC, and other stakeholders (see Exhibit 9, the statistical consultants list, and Section A.8, consultation outside the agency). Although items on cultural competency and detailed cost data have been removed, the AII retains or revises items related to use of data; planned intervention targets; community awareness activities; data infrastructure; evaluation; contextual factors; sustainability; and intervention implementation items related to categorization of implemented interventions, timing, dosage, and reach. The AII also includes new items that address SPF-Rx cross-site priorities, such as access to and use of relevant PDM data sources such as PDMP data.
All efforts have been made to minimize respondent burden yet retain the essential information needed to answer the evaluation questions (EQs). This effort is evident when comparing the estimated burden for the CLI‑R to the estimated burden for the current SPF-Rx AII: the CLI‑R burden estimate was 2.6 hours, and the SPF-Rx AII burden estimate is 2.3 hours.
The Grantee Interview is a semi structured telephone interview with grantee staff designed to collect more in-depth information on organizational infrastructure, use of PDMP data, collaboration, leveraging of funds and resources, subrecipient selection, criteria for intervention selection, processes to decrease health disparities, and evaluation activities. The Grantee Interview will be conducted at the beginning of the grant and in the third and final years of the grant; collecting baseline and follow-up data is necessary to assess the grantees’ progress and change over the course of the grant. Depending on the timing of OMB approval and implementation of the interview, the baseline data collection may require grantees to provide retrospective information (e.g., the SPF-Rx grantees will likely be at the beginning of their second year of their grants when they provide their baseline responses). Thus, data will be collected at baseline (typically year 2), year 3, and the final year (typically year 5) for SPF-Rx grantees.
The PEP‑C contractor’s prior experience as state-level SPF SIG evaluators and national cross-site evaluators for SPF-PFS demonstrated the utility of more in-depth interviews with grantee staff. The SPF-PFS evaluation version of the OMB-approved Grantee-Level Interview–Revised (GLI-R; OMB No. 0930-0348) provided essential grantee-level data, but the instrument limited the information the respondents could provide. Supplemental interviews with PDs provided the necessary context to understand changes in infrastructure and outcomes over time. In fact, an earlier version of the GLI (OMB No. 0930-0279) for SPF SIG programs began as an in-depth, in-person interview for SPF SIG Cohorts I and II; it was revised to the survey version for SPF SIG Cohorts III–V to save funds. The SPF-PFS cross-site evaluation retains use of the GLI-R survey and the PD Interview; however, to reduce burden for SPF-Rx grantees, only the Grantee Interview will be used. Having SPF-Rx-specific items for the interview allows for more in-depth discussion and follow-up on important contextual factors. The estimated burden of the Grantee Interview of 1.5 hours is equivalent to estimates for the current PFS PD Interview, but less than the combined burden for the revised PFS GLI-R and PD Interview (at 2.4 hours). The Grantee Interview format will also allow grantees to become better acquainted with PEP‑C SPF-Rx staff, ask PEP‑C staff questions about the cross-site evaluation, and pass along their concerns about any cross-site evaluation activities.
Items on the Grantee Interview were adapted from questions on the OMB-approved PFS PD Interview (OMB No. 0930-0348) that were considered critical for more detailed, in-depth discussion. With an eye toward minimizing duplication and burden, the SPF-Rx evaluation team has made sure that the data collected from the Grantee Interview will be no duplicative and complementary to data that can be gathered from PFS PD Interviews.
Grantee- and Community-Level Outcomes Modules: Grantees will use these instruments to provide required data about consumption, consequence, and intervening variable program outcomes. These outcomes requirements are outlined in Attachment 6. Grantees will provide annual outcome data for each of the following: opioid overdoses (from hospital data, vital statistics, or other sources), opioid prescribing practices and prescribers’ use of PDMPs (from PDMP data), and PDM (if existing survey data are available). The PEP‑C evaluation team has access to NSDUH state-level survey data on opioid misuse and to CDC WONDER [Wide-ranging Online Data for Epidemiologic Research] opioid overdose death data at state and county levels and does not request this information from grantees.
The outcomes provided on the Outcomes Modules come from existing survey and administrative data within the state, tribe, or jurisdiction. For the overdose-related outcomes, the instrument requests some demographic information. For PDM survey data, the instruments request descriptions of the outcome measures (target outcome, data source type and name, reported outcome calculation description, item and response wordings, sample/population age and grade parameters, time frame of data collection, and actual outcome values and variability estimates). The estimated burden for the Outcomes Module is 3 hours for each module. The estimated time primarily stems from the time that it will take to gather the relevant information for completing the modules.
For the Community-Level Outcomes Module, grantees will select one of their subrecipients from a dropdown menu. For a new outcome, grantees will click on the Add a Record button. Once they have added records, they will be able to view previously added records for the selected subrecipient. This will reduce burden on this instrument through two processes:
Grantees will be able to copy information on an outcome from one subrecipient to another, so that grantees will need to provide only the subrecipient-specific information on outcome values and variability.
After the initial data entry for a subrecipient, grantees need to provide information only on the follow-up data point time frame along with the value calculation information at that time point. They will click on an Add Follow-Up Data link provided on the page.
Substitute Data Source Request: This instrument allows grantees to request permission from SAMHSA to use substitute measures for their outcomes—that is, measures that differ from a list of preapproved outcomes measures (see Attachment 6). Grantees will receive the PEP‑C SPF-Rx Outcomes Module Guidance Manual for reference. The experience on the PFS cross-site evaluation suggests that about five grantees will submit a substitute data source request each year.
The instrument requests descriptions of the proposed substitute outcome measures and an explanation for the substitute data source request. For surveys, for example, it asks for the target outcome; data source name; exact wording of proxy item; response options; reported outcome; and the same survey information requested in the Outcomes Module. The estimated burden of the Substitute Data Source Request of 1 hour is equivalent to estimates for the current PFS Substitute Data Source Request.
The AII, Grantee Interview, and Outcome Measures will be used to collect data to measure the main constructs of interest in order to answer the SPF-Rx EQs. Exhibit 1 provides an overview of the evaluation’s main constructs of interest and the data sources and items on the AII, Grantee Interview, and Outcome Measures that will be used to measure them. It also describes the usual timing of data collection for specific sets of items.
Exhibit 1: Evaluation of the Strategic Prevention Framework for Prescription Drugs (SPF-Rx) in the Program Evaluation for Prevention Contract (PEP‑C)—Constructs and Data Sources
EQ1: Was the implementation of SPF-Rx associated with reduced rates of PDM and opioid overdose at state and community levels? NOTE: These outcomes will also constitute the outcomes for each of the remaining EQs |
|||
Construct |
Data Source |
Items |
Timing |
PDM (i.e., past-12-month misuse or abuse of prescription drugs; past 12‑month abuse or misuse of prescription pain killers) |
Secondary data from the NSDUH, provided by the PEP‑C team in the MRT and any other available survey data provided by the grantee |
Outcome measures: NSDUH items—Percentage reporting use of prescription pain relievers in any way that a doctor did not direct during the past 12 months; Percentage reporting use of any prescription drugs in a way that a doctor did not direct during the past 12 months |
Annual or as data become available |
Consequences (i.e., opioid overdose-related emergency department visits; opioid overdose-related hospital admissions; overdose/poisoning; opioid overdose deaths) |
Publicly available secondary dataa; hospital administrative data provided by grantees in the grantee and community outcomes modules |
n/a |
Annual or as data become available |
(continued)
Exhibit 1: Evaluation of the Strategic Prevention Framework for Prescription Drugs (SPF-Rx) in the Program Evaluation for Prevention Contract (PEP‑C)—Constructs and Data Sources (cont.)
EQ1a: What interventions and combinations of interventions did grantees and communities implement for SPF-Rx? How were the types, combinations, and dosages of interventions (supply and demand side) associated with changes in prescription drug outcomes? |
|||
Construct |
Data Source |
Items |
Timing |
Intervention information |
AII (Section 2c) |
Items 26.1–26.7; 27–31; 39–43; 50; 59; 60; 64; 66; 79; 82–83; 87; 89; 90; 91.1–91.2; 94.1–94.2; 96.1; 98; 100; 107–108; 110; 112.1–113.1; 114.1; 118.1; 119; 120.1; 121.1; 122.1; 123; 124.1; 125 |
Annual |
Combination category (i.e., multiple interventions delivered) |
Composites will be created from Intervention Type variables |
Same as above |
Annual |
Timing |
AII (Section 2c) |
Items 26.4, 26.8, 26.9; 49; 58; 71; 106 |
Annual |
Dosage |
AII (Section 2c) |
Items 35; 51–52; 61–63; 65; 74; 84.1–84.2; 85.1–85.2; 96.2–96.4; 112.1–116; 117.2; 118.2; 119.2; 120.2; 121.2; 124.5 |
Annual |
Reach and numbers served |
AII (Section 2c) |
Items 32.7; 36–38; 44–48; 50.2; 53–57; 59; 61; 66–70; 72–74; 76–80; 83; 86; 101–105; 112.5; 113.5; 114.4; 117.2–117.3; 122.2; 124.2–124.3; 126–130 |
Annual |
EQ1b: What other funding sources and types of activities were employed in SPF-Rx states and communities to address PDM and overdose? How was SPF-Rx interventions associated with outcomes above and beyond these other resources? |
|||
Funding sources & leveraging |
AII (Sections 2b and 2c) |
Items 16; 41 |
Annual |
|
GI (Section 2.4) |
Items 18–20 |
Items 18–19 collected baseline and final year; item 20 collected year 3 and final year |
(continued)
Exhibit 1: Evaluation of the Strategic Prevention Framework for Prescription Drugs (SPF-Rx) in the Program Evaluation for Prevention Contract (PEP‑C)—Constructs and Data Sources (cont.)
EQ2: How did SPF-Rx grantees use PDMPs to improve prescription drug outcomes? (Overarching question answered by specific subquestions below.) |
||||||
Construct |
Data Source |
Items |
Timing |
|||
EQ2a: Were changes in PDMP-related capabilities, policies, and practices associated with intermediate outcomes (e.g., increased PDMP registration or queries among physicians) and long-term outcomes (reduced rate of opioid overdose)? |
||||||
PDMP capabilities and practices |
AII (Sections 2a, 2b, 2c) |
Items 6.7–6.14; 6.27; 12; 13; 19–23; 90 |
Annual except item 19 (baseline) |
|||
GI (Sections 2.2, 2.6) |
Items 5–11; 25a–c |
Items 5–6 at baseline only; items 10 and 25a–c at years 3 and 5; items 7–9 and 11 at baseline, year 3, and final year |
||||
PDMP policies |
AII (Section 2c) |
Item 90 |
Annual |
|||
|
GI (Sections 2.2, 2.5, 2.6) |
Items 5–11; 23; 25a–c |
Items 5–6 at baseline only; items 10 and 25a–c at year 3 and final year; items 7–9, 11, and 23 at baseline, year 3, and final year |
|||
EQ2b: How were PDMPs incorporated into the SPF model, including into needs assessment and planning? |
||||||
|
AII (Section 2a, 2b) |
Items 6; 19–23 |
Annual except item 19 (baseline) |
|||
|
GI (Section 2.2) |
Items 5–11; 25a–c |
Items 5–6 at baseline only; items 10 and 25a–c at year 3 and final year; items 7–9 and 11 at baseline, year 3, and final year |
|||
EQ2c: What are characteristics of model PDMPs? |
||||||
|
AII (Section 2b) |
Items 6.7–6.14; 12; 13; 19–23; 90 |
Annual except item 19 (baseline) |
|||
|
GI (Section 2.2) |
Items 7–11; 23; 25a–c |
Items 7–9, 11, and 23 at baseline, year 3, and final year; items 10 and 25a–c at year 3 and final year |
|||
(continued)
Exhibit 1: Evaluation of the Strategic Prevention Framework for Prescription Drugs (SPF-Rx) in the Program Evaluation for Prevention Contract (PEP‑C)—Constructs and Data Sources (cont.)
EQ2d: How did PDMPs’ capacity infrastructure improve over time? How did grantees strengthen workforce development (e.g., credentialing) to ensure use of PDMPs? |
|||
Construct |
Data Source |
Items |
Timing |
|
AII (Section 2b) |
Items 6.7–6.14; 6.27; 12; 13; 19–23; 90 |
Annual except item 19 (baseline) |
|
GI (Section 2.2) |
Items 7–11; 23; 25a–c |
Items 7–9, 11, and 23 at baseline, year 3, and final year; items 10 and 25a–c at year 3 and final year |
EQ3: What barriers and facilitators affected SPF-Rx implementation and outcomes (e.g., characteristics of partnerships, concentration of effort, infrastructure, laws and regulations, state or community contextual factors)? How did grantees address these barriers? |
|||
Infrastructure |
GI (Sections 2.1–2.4; 2.6, 2.8) |
Items 1–20; 25; 33–34 |
GI conducted baseline, year 3, and final year; however, items 5–6 collected only at baseline; items 10, 20, and 25, year 3 and final year; items 18–19, baseline and final year; item 34, final year only |
|
AII (Sections 2a, 2b, 2c) |
Items 6; 12–24; 33 |
Annual, except item 19 asked only at baseline |
Characteristics of partnerships and collaborations |
GI (Sections 2.1, 2.3, 2.4, 2.8) |
Items 1–4; 12–18 |
GI conducted at baseline, year 3, and final year; however, item 18 is asked only at baseline and final interviews |
|
AII (Sections 2b, 2c, 3) |
Items 18; 33; 133 |
Annual |
(continued)
Exhibit 1: Evaluation of the Strategic Prevention Framework for Prescription Drugs (SPF-Rx) in the Program Evaluation for Prevention Contract (PEP‑C)—Constructs and Data Sources (cont.)
EQ3 (cont.): What barriers and facilitators affected SPF-Rx implementation and outcomes (e.g., characteristics of partnerships, concentration of effort, infrastructure, laws and regulations, state/community contextual factors) and how did grantees address these barriers? |
|||
Construct |
Data Source |
Items |
Timing |
Strategy (intervention) selection |
GI (Section 2.5) |
Items 21–24 |
Items 21–22 asked at baseline and year 3; items 23–24 asked baseline, year 3, and final year |
|
AII (Section 2c) |
Items 27–31 |
Annual |
Implementation adaptation |
AII (Section 2c; 2d) |
Items 31; 131–132 |
Annual |
Geography/concentration of effort |
AII (Section 2a, 2c, 3) |
Items 11; 32; 133 |
Annual |
Demographics (gender, age, race, ethnicity) |
AII (Section 2c) |
Items 32.6; 45–48; 50; 54–57; 59; 67–70; 72; 77–80; 83; 102–105; 127–130 |
Annual |
Training and technical assistance |
AII (Section 2b) |
Item 13 |
Annual |
Barriers to implementation and contextual barriers |
GI (Sections 2.2-2.8) |
Items 5–6; 9–11; 14–17; 21; 25; 28–29; 34 |
Baseline, year 3, and final year, except items 5–6 asked baseline only; item 21, baseline and year 3; item 25, year 3 and final year |
|
AII (Sections 2a, 2b, 3) |
Items 133–134 |
Annual for item 133 and baseline and final for item 34 |
(continued)
Exhibit 1: Evaluation of the Strategic Prevention Framework for Prescription Drugs (SPF-Rx) in the Program Evaluation for Prevention Contract (PEP‑C)—Constructs and Data Sources (cont.)
EQ3a: How did grantees implement their quality improvement plans to address access to and use of prevention services for their health disparities populations? |
|||
Strategies employed |
GI (Section 2.7) |
Items 26–29; 31–32 |
Items 26 and 28 collected at baseline; items 27 and 29, baseline and final year; item 30, baseline, year 3, and final year |
Note. AII, Annual Implementation Instrument; EQ, evaluation question; GI, Grantee Interview; MRT, Management Reporting Tool; NSDUH, National Survey on Drug Use and Health; PDM, prescription drug misuse and abuse; PDMP, Prescription Drug Monitoring Program.
a For example, the NSDUH, National Poison Data System, CDC WONDER database with overdose death data for states and counties, and Uniform Crime Reports.
SPF-Rx grantee staff will provide outcomes information via the AII and the Grantee Outcomes Module through SAMHSA’s web-based MRT. Using a web instrument allows for automated data checks as well as for skip procedures and prepopulated fields on the basis of prior responses to certain questions. This will reduce both respondent burden and data entry error, thereby increasing the efficiency of data entry and improving data quality. The automated data checks will ensure that responses follow the expected format (e.g., numbers or dates where those are expected). Web-based systems allow grantees to copy information from one form to another and then change information as needed, such as when they need to provide similar community outcomes data on the same measures for multiple communities, where only the outcomes value differs. Similarly, once completed initially, some items are automatically prepopulated with data entered during a prior reporting period and require editing some fields only if they have changed. Examples in the MRT include the intervention strategies and targeted populations.
Web-based systems also allow SAMHSA SPF-Rx Project Officers and the cross-site evaluation team to review submissions conveniently, request revisions as needed, and then approves grantee submissions as appropriate. A dashboard and other reports for each system will also be available to SAMHSA and the PEP‑C team, as well as to the grantees and communities who submit data, so that they can monitor the overall status of data collection and monitor performance. Grantees will have access to their own data.
Finally, web-based systems allow grantees and SAMHSA Project Officers easy access to the PEP‑C Knowledge Base, which will contain data submission manuals and other relevant documents, a section with responses to frequently asked questions, and a link to a TA submission form. Grantees and Project Officers can also request TA on their SPF-Rx data entry through email and a telephone request system. All TA requests will be routed to an electronic system that tracks requests, follow-ups, and resolutions.
This evaluation is collecting information unique to SPF-Rx program grantees that are otherwise not available to project officers or the PEP‑C cross-site evaluation team.
Participation in this evaluation will not impose a significant impact on small entities. SPF-Rx grantees will usually consist of state agencies, tribal organizations, and other jurisdictions. Some communities may be small entities; however, the MRT is designed to include only the most pertinent information needed to be able to monitor each grantee’s progress and to carry out the evaluation effectively, and the evaluation’s impact will not be significant.
The multiple data collection points for the SPF-Rx cross-site evaluation in the MRT are necessary to track and evaluate grantees’ and communities’ progress and change over time. SAMHSA will use the data for the purposes of the cross-site evaluation for the SPF-Rx programs, and grantees will use these data to track their and their communities’ ongoing implementation. Less frequent reporting will affect SAMHSA’s and the grantees’ ability to do so effectively. For example, SAMHSA is federally required to report on Government Performance and Results Act (GPRA) measures once each year. New Federal health disparities priorities require periodic reports of the activities used to address those priorities through SAMHSA programs such as SPF-Rx.
SAMHSA has made every effort to ensure that data are collected only when necessary and that extraneous collection will not be conducted. For example, the AII tool for SPF-Rx will collect grantee and subrecipient implementation data annually. Exhibit 2 provides information on data collection requirements and timing for the instruments of the PEP‑C SPF-Rx cross-site evaluation in the MRT. It also shows the timing for the Grantee Interview, which will not be collected in the MRT but administered by telephone.
Exhibit 2: Data Collection Requirements and Timing for the MRT Instruments
Instrument |
Requirement |
Timing |
Annual Implementation Instrument |
Yes |
Annually |
Outcomes Data Modules |
Yes |
Annually |
Grantee Interview |
Yes |
Baseline, year 3, and final year |
This information collection fully complies with the guidelines in 5 CFR 1320.5(d)(2).
The notice required by 5 CFR 1320.8(d) was published in the Federal Register on April 18, 2017 (82 FR (18307). No comments were received.
The SPF-Rx tools were developed by SAMHSA and the contractors. The PEP‑C team conducted interviews with grantees before the development phase to obtain feedback that was used to inform the development of the tools. Each instrument was reviewed by three to five SPF-Rx grantee volunteers. Individuals provided feedback on each of the data collection instruments, and the instruments were revised on the basis of their feedback. Revisions ranged from changes in the instructions to simplify them to suggestions for ways to streamline data collected across tools and overlapping grant programs. The External Steering Committee (ESC) was also consulted during tool development. See Exhibit 9 for the list of individuals consulted throughout the development process of the instruments.
No cash incentives or gifts will be given to respondents.
The MRT does not request personal data for the web-based cross-site evaluation instruments. The focus of the data collection is on the programmatic characteristics of the SPF-Rx grantees and subrecipient communities. Grantee staff will provide information about their organizations and their SPF-Rx activities rather than information about themselves personally. The instruments collect programmatic data at the grantee and community levels along with aggregated, nonidentifying individual-level data (e.g., community outcomes data). Sensitive respondent information, such as birthdates and Social Security Numbers, will not be collected.
The MRT team takes responsibility for ensuring that the web and data systems are properly maintained and monitored. Server staff will follow standard procedures for applying security patches and conducting routine maintenance for system updates. Data will be stored on a password-protected server, and access to data in the system will be handled by a hierarchy of user roles, with each role conferring only the minimum access to system data needed to perform the necessary functions of the role.
Although they do not collect individual-level data, evaluation staff are trained on the importance of privacy and in handling sensitive data.
The information reported by respondents for SPF-Rx cross-site evaluation (the AII, Outcomes Module, Substitute Data Source Request, and Grantee Interview) is not sensitive personal information. The instruments focus only on program-related information for the grantees’ projects.
The number of data collection responses will be consistent for grantees but may vary for subrecipients based on the timing of their funding. As such, the burden and respondent cost may vary by year. Exhibit 3 provides an overview of the estimated annual number of responses per grantee, per instrument.
Exhibit 3: Annual Data Collection Responses per Grantee for PEP‑C SPF-Rx Cross-Site Instruments
Instrument |
FY2018 |
FY2019 |
FY2020 |
FY2021–Request OMB Extensiona |
Annual Implementation Instrument |
1 |
1 |
1 |
1 |
Grantee-level outcome data |
1 |
1 |
1 |
1 |
Community-level outcome data |
1 |
1 |
1 |
1 |
Substitute Data Request |
1 |
1 |
1 |
1 |
Grantee Interview |
1 |
|
1 |
1 |
Note. OMB, Office of Management and Budget; PEP-C, Program Evaluation for Prevention Contract; SPF-Rx, Strategic Prevention Framework for prescription drugs.
a FY2021 does not fall within the OMB 3-year approval period; therefore, data collection for those years is not included in the burden estimate.
The SPF-Rx Grantee-Level Outcome Module will have sections that are required of all grantees. We expect that 25 SPF-Rx grantees and grantees in all future cohorts will complete the SPF-Rx Grantee-Level Outcome Data module one time each year, beginning in the second year of their grant. The SPF-Rx Grantee-Level Outcome module is estimated to take 3 hours to complete per response; this includes time to look up and compile information (2 hours) and time to complete the Web-instrument (1 hour). The estimated burden time is based on test instruments completed by evaluation staff members who have experience working with SPF-PFS grantees and input from current SPF-Rx grantees (see Section B.4 for more detail). There are no direct costs to respondents other than their time to complete the instrument. Exhibits 4–6 provide the details of the annual burden for each instrument for FY2018–FY2020, and Exhibit 7 presents estimates of the SPF-Rx Grantee-Level Outcome module annualized burden hours, 75, and the annualized respondent cost, $3,066 (total burden hours × the average hourly wage for State government managers, as reported in the 2015 Occupational Employment Statistics [OES] by the Bureau of Labor Statistics [BLS]; see https://www.bls.gov/oes/current/naics4_999200.htm#11-0000).
All 25 SPF-Rx grantees, and all future cohorts, are expected to complete the Community-Level Outcomes Module one time each year, beginning in the second year of their grants. The Community-Level Outcomes Module is estimated to take 3 hours to complete per response; this includes time to look up and compile information (2 hours) and time to complete the web instrument (1 hour). The estimated burden time is based on test instruments completed by grantees that have experience working with SPF-PFS grants (see Section B.4 for more detail). There are no direct costs to respondents other than their time to complete the instrument. Exhibits 4–6 provide the details of the annual burden for each instrument for FY2018–FY2020, and Exhibit 7 presents estimates of the Community-Level Outcomes Module annualized burden hours (75) and the annualized respondent cost ($3,066 = total burden hours × the average hourly wage for state government managers, as reported in the 2015 Occupational Employment Statistics [OES] by the Bureau of Labor Statistics [BLS]; see https://www.bls.gov/oes/current/naics4_999200.htm#11-00000).
The Substitute Data Source Request instrument is required of grantees only if they want to use an annual required measure in their community outcome reporting that is not preapproved. Usually grantees make these requests in the second or third years of their grants, with most requests occurring shortly before they need to report baseline outcomes data in the second grant year. In FY2018 we expect that five SPF-Rx grantees will complete the Substitute Data Source Request instrument. In FY2019 we expect that half of SPF-Rx grantees will complete the Substitute Data Source Request instrument. In FY2020 we expect that half again SPF-Rx grantees will complete the Substitute Data Source Request instrument. The Substitute Data Source Request instrument is estimated to take 1 hour to complete per response; this includes time to look up and compile information (0.5 hour) and time to complete the web instrument (0.5 hour). The estimated burden time is based on test instruments completed by SPF-Rx grantee staff that have experience working on SPF-PFS grants (see Section B.4 for more detail). There are no direct costs to respondents other than their time to complete the instrument. Exhibits 4–6 provide the details of the annual burden for each instrument for FY2018–FY2020, and Exhibit 7 presents estimates of the Substitute Data Source Request instrument annualized burden hours (3.67) and the annualized respondent cost ($149.89 = total burden hours × the average hourly wage for state government managers, as reported in the 2015 Occupational Employment Statistics [OES] by the Bureau of Labor Statistics [BLS]; see https://www.bls.gov/oes/current/naics4_999200.htm#11-00000).
The AII is required of grantees annually. In FY2018 through FY2020, we expect that 25 SPF-Rx grantees will complete the AII. The AII is estimated to take 2.3 hours to complete per response; this includes time to look up and compile information (1.5 hours) and time to complete the web instrument (0.8 hour). The estimated burden time is based on test instruments completed by SPF-Rx grantee staff that have experience working on SPF-PFS grants (see Section B.4 for more detail). There are no direct costs to respondents other than their time to complete the instrument. Exhibits 4–6 provide the details of the annual burden for each instrument for FY2018–FY2020, and Exhibit 7 presents estimates of the AII annualized burden hours (230) and the annualized respondent cost ($9,384 = total burden hours × the average hourly wage for state government managers, as reported in the 2015 Occupational Employment Statistics [OES] by the Bureau of Labor Statistics [BLS]; see https://www.bls.gov/oes/current/naics4_999200.htm#11-00000).
The Grantee-Level Interview is required of grantees in the baseline, third, and final years of their grant funding. In FY2018 through FY2020, we expect that 25 SPF-Rx grantees will complete the Grantee-Level instrument twice (in their baseline and third years). The Grantee-Level Interview is estimated to take 1.5 hours to complete per response. The estimated burden time is based on test instruments completed by SPF-Rx grantee staff that have experience working on SPF-PFS grants (see Section B.4 for more detail). There are no direct costs to respondents other than their time to complete the instrument. Exhibits 4–6 provide the details of the annual burden for each instrument for FY2018–FY2020, and Exhibit 7 presents estimates of the Grantee-Level Instrument annualized burden hours (25.5) and the annualized respondent cost ($1,042.44 = total burden hours × the average hourly wage for state government managers, as reported in the 2015 Occupational Employment Statistics [OES] by the Bureau of Labor Statistics [BLS]; see https://www.bls.gov/oes/current/naics4_999200.htm#11-00000).
Evaluation Plan Checklist
Evaluation Plan allows grantees to outline their local evaluation plan with core elements required for the cross-site evaluation. Main sections include goals and objectives, performance measures, data analysis plan, and reporting plan. The evaluation plan ensures that grantees align their local infrastructure with the required reporting requirements for the cross-site. In addition, it also provides grantees with technical assistance from the cross-site evaluation team on data collection.
Exhibit 4: FY2018 Annual Burden
Instrument |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Hours per Response |
Total Burden Hours |
Average Hourly Wage |
Total Respondent Costa |
Grantee-Level Outcomes Module |
25 |
1 |
25 |
3 |
75 |
$40.88 |
$3,066 |
Community-Level Outcomes Module |
25 |
1 |
25 |
3 |
75 |
$40.88 |
$3,066 |
Substitute Data Request Form |
12 |
1 |
12 |
1 |
12 |
$40.88 |
$490.56 |
Annual Implementation Instrument |
100 |
1 |
100 |
2.3 |
230 |
$40.88 |
$9,402.40 |
Grantee-Level Interview |
25 |
1 |
25 |
1.5 |
37.5 |
$40.88 |
$1,533 |
Evaluation Plan |
25 |
1 |
25 |
8 |
200 |
$40.88 |
$8,176 |
FY2018 TOTAL |
100 |
|
212 |
|
629.5 |
|
$25,733.96 |
a Total respondent cost is calculated as total burden hours × average hourly wage.
Exhibit 5: FY2019 Annual Burden
Instrument |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Hours per Response |
Total Burden Hours |
Average Hourly Wage |
Total Respondent Costa |
Grantee-Level Outcomes Module |
25 |
1 |
25 |
3 |
75 |
$40.88 |
$3,066 |
Community-Level Outcomes Module |
25 |
1 |
25 |
3 |
75 |
$40.88 |
$3,066 |
Substitute Data Request Form |
12 |
1 |
12 |
1 |
12 |
$40.88 |
$490.56 |
Annual Implementation Instrument |
100 |
1 |
100 |
2.3 |
230 |
$40.88 |
$9,402.40 |
Grantee-Level Interview |
25 |
1 |
25 |
1.5 |
37.5 |
$40.88 |
$1,533 |
FY2019 TOTAL |
100 |
|
187 |
|
429.50 |
|
$17,557.96 |
a Total respondent cost is calculated as total burden hours × average hourly wage.
Exhibit 6: FY2020 Annual Burden
Instrument |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Hours per Response |
Total Burden Hours |
Average Hourly Wage |
Total Respondent Costa |
Grantee-Level Outcomes Module |
25 |
1 |
25 |
3 |
75 |
$40.88 |
$3,066 |
Community-Level Outcomes Module |
25 |
1 |
25 |
3 |
75 |
$40.88 |
$3,066 |
Substitute Data Request Form |
12 |
1 |
12 |
1 |
12 |
$40.88 |
$490.56 |
Annual Implementation Instrument |
100 |
1 |
100 |
2.3 |
230 |
$40.88 |
$9,402.40 |
Grantee-Level Interview |
0 |
0 |
0 |
1.5 |
0 |
0 |
0 |
FY2020 TOTAL |
100 |
|
162 |
|
392 |
|
$16,024.96 |
a Total respondent cost is calculated as total burden hours × average hourly wage.
Exhibit 7: Annualized Data Collection Burden
Instrument |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Hours per Response |
Total Burden Hours |
Average Hourly Wage |
Total Respondent Costa |
Grantee-Level Outcomes Module |
25 |
1 |
25 |
3 |
75 |
$40.88 |
$3,066 |
Community-Level Outcomes Module |
25 |
1 |
25 |
3 |
75 |
$40.88 |
$3,066 |
Substitute Data Request Form |
12 |
1 |
12 |
1 |
12 |
$40.88 |
$490.56 |
Annual Implementation Instrument |
100 |
1 |
100 |
2.3 |
230 |
$40.88 |
$9,402.40 |
Grantee-Level Interview |
17 |
1 |
17 |
1.5 |
25.5 |
$40.88 |
$1,042.44 |
Evaluation Plan |
25 |
1 |
25 |
8 |
200 |
40.88 |
$8,176 |
OVERALL TOTAL |
100 |
|
204 |
|
618 |
|
25,243.40 |
Note. Annualized Data Collection Burden captures the average number of respondents and responses, burden hours, and respondent cost over the 3 years (FY2018–FY2020).
a Respondent cost is calculated as total burden hours × average hourly wage.
There are no respondent costs for capital or start-up or for operation or maintenance.
The total estimated cost to the government for the data collection from FY2018 through FY2020 is $979,981. This includes approximately $496,375 for developing the instruments; programming and maintaining the online data collection system; providing data collection training to grantees and subrecipients; processing, cleaning, and housing data; and analyzing and reporting data. Approximately $55,602 per year represents SAMHSA costs to manage/administer the data collection and analysis for 25% time each by two employees (GS-14-10, $111,203 annual salary). Approximately $105,600 per year represents SAMHSA costs to monitor and approve grantee reporting in these instruments (10% time of 10 Project Officers at $105,600 annual salary). The annualized cost is approximately $326,660.33.
This is a new collection.
Exhibit 8 outlines the key time points for the PEP‑C SPF-Rx data collection.
Exhibit 8: Time Schedule for Data Collection
Activity |
Time Schedule |
Prepare for data collection |
November 2016–January 2017 |
Submit OMB package for approval of data collection |
April 2017 |
Obtain OMB approval of data collection |
September 2017 |
Collect data |
November 2017–September 2021 |
Analyze data |
April 2018–September 2021 |
Disseminate
findings: |
September 2018–September 2020 |
Note. OMB, Office of Management and Budget.
The PEP‑C SPF-Rx evaluation will use the data collected through the PEP‑C SPF-Rx MRT to help SAMHSA reach its diverse stakeholders through targeted products and innovative dissemination venues. The objective for all reports and dissemination products is to provide user-friendly documents and presentations that help SAMHSA successfully disseminate and explain the findings. The dissemination plan includes products in a variety of formats for a variety of target audiences. Audiences for these reports will include Congress, the Office of National Drug Control Policy (ONDCP), SAMHSA Centers, the evaluation’s SAMHSA Contracting Officer’s Representatives (CORs), SPF-Rx grantees, and the broader prescription drug abuse prevention field (i.e., academia, researchers, policymakers, providers). PEP‑C and SAMHSA staff recognizes that different audiences are best reached by different types of report formats. For example, reports to Congress and the ONDCP will require materials that are concise but offer policy-relevant recommendations. Reports created for SAMHSA Centers and the CORs will require more in-depth information, such as substantive background and discussion sections, to supplement the analytic approach. Reports created for SPF-Rx grantees will be concise handouts, with helpful and easy-to-read graphics on performance data rather than lengthy text. The assortment of dissemination products developed using the PEP‑C SPF-Rx MRT data will include short and long analytic reports, congressional briefings, annual evaluation reports, research and policy briefs, ad hoc analytic reports, journal articles, best practice summaries, and conference or other presentations.
The PEP‑C SPF-Rx evaluation uses a series of interdependent analysis frameworks that have been selected to maximize the coverage of key EQs posed for assessing the objectives of SPF-Rx in the prevention of prescription drug and pain reliever misuse and opioid overdose. The evaluation will fully incorporate all data from the cross-site evaluation instruments as well as secondary data, as indicated in Exhibit 3. The analysis plan includes a range of analyses, from basic descriptive analyses of GPRA measures, grantee performance measures, and National Outcomes Measures (NOMs; e.g., means, frequencies, percentages) to sophisticated qualitative analysis and multiple quantitative analytic frameworks and models that reflect complexities that are anticipated to arise with data collected by the PEP‑C.
The SPF-Rx evaluation will use a pre/post design with matched comparison groups when relevant and feasible. The PEP‑C team plans to obtain key county-level characteristics from baseline census, archival, and survey data sources from which to select comparison counties (or communities) for SPF-PFS subrecipients. For some grantees, many of the required estimates will be available through standard public reporting. For others, the PEP‑C team will need to collaborate with grantee-level evaluators to obtain the estimates. In no cases will new data collection be required for the matching process. Follow-up outcomes data for the matched comparison groups will come from the same data sources used for the matching process.
Matched comparison communities will not be completing any of the instruments in the PEP-C SPF-Rx.
Qualitative analyses of the PEP-C SPF-Rx MRT data focus primarily on the Grantee Interview. PEP‑C staff will upload the interview data into a qualitative research software program, NVivo, for coding. Preparation for coding will include developing a dictionary or codebook in which codes will be carefully defined and logged so that coders will be able to follow their meaning and know when to apply the codes to text within an interview. Codes will reflect prominent themes relevant to interpreting evaluation findings. To ensure reliability in the coding process, coders will then be assigned to work independently and concurrently on a subset of the open-ended response data. A kappa coefficient of 0.8 or higher will be maintained on all codes. Any discrepancies will be worked out between coders to ensure consistent application of codes. Upon completion of coding, the findings will be compiled on the basis of the prominence of codes (or themes) and organized around the major research questions and constructs.
Several features of the evaluation design and EQs guided the selection of the analysis frameworks that the SPF-Rx evaluation has proposed to use or adapt. These features include:
Repeated outcomes;
Data from state and tribal grantees;
Data from communities nested within grantees;
Nonrandomized comparison communities within grantee states; and
Nonrandom selection of intervention types that often occur in combination.
Below is an overview of the advanced analytic frameworks that will be used in the SPF-Rx evaluation. The methods below fall into two categories: those well-suited to address the EQs and determine the impact of SPF-Rx and those that maximize the internal and external validity of the evaluation models.
Multilevel latent growth models: One of the primary analysis frameworks will be multilevel latent growth models (MLLGM). The basic linear MLLGM (Muthén, 1997) accounts for variability in changes over time on outcomes. Where possible, a multiple baseline strategy will be employed whereby trends over time on outcomes at the grantee and subrecipient levels before PFS implementation will be compared to post implementation trends (similar to an interrupted time series approach). Adding appropriate predictors of change (as well as interactions of intervention and moderators, such as barrier or site characteristics) will allow us to address all EQs with this method, which focuses on differences in change in outcomes over time. Analyses will focus on predictors of post implementation changes in outcomes over time, such as the type and dosage of interventions supported under SPF-Rx. However, several limitations may arise in these analyses, including small sample sizes at the grantee level, nonrandom assignment of PFS interventions, and variation in how GPRA measures and National Outcomes Measures are reported within and across grantees. As a result, the SPF-Rx evaluation will incorporate alternative or complementary analysis frameworks, or both, in addition to MLLGM.
Meta-regression: A second strategy that can be employed if sample sizes are too small to estimate MLLGM or to estimate scale scores under integrative data analysis is meta-regression (Hox, 2010). Meta-regression uses effect sizes as data, similar to meta-analysis, where effect sizes are extracted from past studies and used as analysis data. Unlike MLLGM, meta-regression does not require that the outcome measure be exactly the same across all analysis units; effect sizes for changes over time from disparate measures of the same construct within grantee (for grantee-level analyses) are sufficient for analysis. In addition to effect sizes, the standard errors for the effect sizes are used to calculate meta-regression weights in a manner similar to that of standard meta-analysis models. Key predictors such as types/combinations of interventions implemented (EQ1) or changes in PDMP use over time (EQ2) can then be used to account for variability in effect sizes as in a standard meta-analysis.
Propensity scoring approaches: Propensity scoring is a statistical approach used to balance measured covariates that influence both the probability of selection into two or more non-experimental groups and treatment outcomes (Rosenbaum & Rubin, 1983; Shadish, Cook, & Campbell, 2002; West, Biesanz, & Pitts, 2000). More recent work has extended propensity scoring to continuous measures of treatment (Imai & van Dyk, 2004). The propensity score (when treatment assignment is categorical) is the predicted probability of assignment to a treatment condition given the key covariates of interest (estimated from a regression model—ordinary least squares for continuous treatment or logistic for categorical treatment), with the resulting probability used as either a sample stratifier or a weight in subsequent outcome analyses. After the propensity score weight is controlled for, covariate distributions should be equal across conditions, which will mimic random assignment to the conditions of interest in the particular EQ. These scores can then be used to weight outcome analyses (e.g., MLLGMs) to produce unbiased estimates of the treatment effect (Harder, Stuart, & Anthony, 2010; McCaffrey, Ridgeway, & Morral, 2004; Rosenbaum & Rubin, 1983; Shadish, 2010).
Integrative data analysis: If concerns arise about the variability in measures across grantees, the SPF-Rx evaluation will employ integrative data analysis (Curran et al., 2008; Curran & Hussong, 2009) to harmonize different measures of PDM (as well as prescriber behavior and consequences measures) across grantees and subrecipients. The harmonization process involves (1) creating a common measure for questions that are worded slightly differently from each other but are comparable and (2) using response scales (e.g., Likert-type scales, ordered categories) that can be condensed to their least common denominator (e.g., ever used/never used). For single-item constructs and measures, the harmonization process is the only step necessary. For constructs that reflect multiple-item scales, confirmatory factor analysis models will be employed to assess which items load on which factors and to derive factor and scale scores via item response theory models, which weight each item according to how common (or rare) a response is and how correlated the item is with other items making up the factor. Note that this step may be more difficult at the grantee level, where sample sizes are small.
OMB approval expiration dates will be displayed.
There are no exceptions to the certification statement. The certifications are included in this submission.
Centers for Disease Control and Prevention (CDC). (2012). CDC grand rounds: Prescription drug overdoses - a U.S. epidemic. MMWR. Morbidity and Mortality Weekly Report, 61(1), 10–13.
Centers for Disease Control and Prevention (2016a). NCHS Data on Drug-poisoning Deaths (National Center for Health Statistics Fact Sheet). Retrieved from https://www.cdc.gov/nchs/data/factsheets/factsheet_drug_poisoning.pdf
Warner, M., Trinidad, J. P., Bastian, B. A., Minino, A. M., & Hedegaard, H., & the Centers for Disease Control and Prevention. (2016b). Drugs most frequently involved in drug overdose deaths: United States, 2010–2014. National Vital Statistics Reports, 65(10), 1–15.
Chang, H.-Y., Lyapustina, T., Rutkow, L., Daubresse, M., Richey, M., Faul, M., . . . Alexander, G. C. (2016). Impact of prescription drug monitoring programs and pill mill laws on high-risk opioid prescribers: A comparative interrupted time series analysis. Drug and Alcohol Dependence, 165, 1–8. http://dx.doi.org/10.1016/j.drugalcdep.2016.04.033
Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14(2), 81–100. http://dx.doi.org/10.1037/a0015914
Curran, P. J., Hussong, A. M., Cai, L., Huang, W., Chassin, L., Sher, K. J., & Zucker, R. A. (2008). Pooling data from multiple longitudinal studies: The role of item response theory in integrative data analysis. Developmental Psychology, 44(2), 365–380. http://dx.doi.org/10.1037/0012-1649.44.2.365
Harder, V. S., Stuart, E. A., & Anthony, J. C. (2010). Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychological Methods, 15(3), 234–249. http://dx.doi.org/10.1037/a0019623
Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). New York, NY: Routledge.
Imai, K., & van Dyk, D. A. (2004). Causal inference with general treatment regimes: Generalizing the propensity score. Journal of the American Statistical Association, 99(467), 854–866. http://dx.doi.org/10.1198/016214504000001187
Lin, D. H., Lucas, E., Murimi, I. B., Jackson, K., Baier, M., Frattaroli, S., . . . Alexander, G. C. (2017). Physician attitudes and experiences with Maryland’s prescription drug monitoring program (PDMP). Addiction (Abingdon, England), 112(2), 311–319. http://dx.doi.org/10.1111/add.13620
McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods, 9(4), 403–425. http://dx.doi.org/10.1037/1082-989X.9.4.403, B. (1997). Latent variable modeling with longitudinal and multilevel data. In A. Raftery (Ed.), Sociological Methodology (pp. 453–480). Boston, MA: Blackwell Publishers. http://dx.doi.org/10.1111/1467-9531.271034
National Institutes of Health. (2011, October). Prescription drugs: Abuse and addiction (NIH Publication No. 11-4881). Retrieved from http://www.drugabuse.gov/sites/default/files/rrprescription.pdf
Office of National Drug Control Policy. (2011). Epidemic: responding to America’s prescription drug abuse crisis. Retrieved from: http://www.whitehouse.gov/sites/default/files/ondcp/issues-content/prescription-drugs/rx_abuse_plan_0.pdf
Patrick, S. W., Fry, C. E., Jones, T. F., & Buntin, M. B. (2016). Implementation of prescription drug monitoring programs associated with reductions in opioid-related death rates. Health Affairs, 35(7), 1324–1332. http://dx.doi.org/10.1377/hlthaff.2015.1496
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. http://dx.doi.org/10.1093/biomet/70.1.41
Shadish, W., Cook, T., & Campbell, D. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin.
Shadish, W. R. (2010). Campbell and Rubin: A primer and comparison of their approaches to causal inference in field settings. Psychological Methods, 15(1), 3–17. http://dx.doi.org/10.1037/a0015916
Substance Abuse and Mental Health Services Administration. (2016a). Key substance use and mental health indicators in the United States: Results from the 2015 National Survey on Drug Use and Health (NSDUH Series H-51, HHS Publication No. (SMA) 16-4984). Rockville, MD: Author. Retrieved from https://www.samhsa.gov/data/sites/default/files/NSDUH-FFR1-2015/NSDUH-FFR1-2015/NSDUH-FFR1-2015.pdf
Substance Abuse and Mental Health Services Administration. (2016b). SAMHSA Opioid Overdose Prevention Toolkit (HHS Publication No. (SMA) 16-4742). Rockville, MD: Author. Retrieved from http://store.samhsa.gov/shin/content//SMA16-4742/SMA16-4742.pdf
West, S. G., Biesanz, J. C., & Pitts, S. C. (2000). Causal inference and generalization in field settings: Experimental and quasi-experimental designs. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 40–84). New York, NY: Cambridge University Press.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | author |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |