Accuracy of Opioid Product Ascertainment (CDER)

15 October 2020

Inflexxion Task Order #5, Aim 2,
Accuracy of Opioid Product
Ascertainment
Study Design and Protocol

Principal Investigator
Jody Green, PhD
Inflexxion, Inc.
Denver, CO
In collaboration with
Susana Peinado, PhD
Ryan Paquin, PhD
Kate Ferriola-Bruckenstein, BA
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
RTI Project Number 0217544.000.001

Contents

Section

Page

1.

Introduction

5

2.

Study Design

6

2.1

Overview ................................................................................................... 6

2.2

Experimental Stimuli ................................................................................... 8

2.3

Randomization .......................................................................................... 10

2.4

Measures ................................................................................................. 11

2.5

Interview Guide ........................................................................................ 12

2.6

ASI-MV Data Access .................................................................................. 12

2.7

Analysis Plan ............................................................................................ 12
2.7.1 Descriptive Analysis ......................................................................... 12
2.7.2 Coding Open-Ended Opioid Product Ascertainment Measures ................ 13
2.7.3 Main Analysis .................................................................................. 13
2.7.4 Analysis of Interview Data ................................................................ 15

3.

Data Collection Procedures

16

3.1

Site Identification ...................................................................................... 16

3.2

Participant Eligibility .................................................................................. 16

3.3

Procedures ............................................................................................... 17
3.3.1 Recruitment and Screening .............................................................. 17
3.3.2 Consent ......................................................................................... 18
3.3.3 Data Collection ............................................................................... 19

4.

Summary

20

5.

References

21

Appendix A. Opioid Products from ASI-MV Included in the Experiment

22

Appendix B.

23

Experimental Design: Replicated Fully Crossed Design

iii

Exhibits

Number

Page

Exhibit 1.

Overview of Study Design ........................................................................... 6

Exhibit 2.

Opioid Product Ascertainment Exercises in Online Assessment ......................... 7

Exhibit 3.

Example of a Free Recall Exercise ................................................................ 9

Exhibit 4.

Example Exercise with a List-of-Product-Names Response Format .................. 10

Exhibit 5.

Randomization Scheme ............................................................................ 11

Exhibit 6.

Example Codebook Questions to Analyze Open-Ended Responses to Free
Recall Questions ...................................................................................... 13

iv

1. Introduction
Data on the use and misuse of prescription opioids are frequently collected via self-report
(Substance Abuse and Mental Health Services Administration, 2019). However, a variety of
respondent characteristics and situational factors can affect the validity of self-reported use
of prescription opioids (Del Boca & Noll, 2000; Smith, Rosenblum, Parrino, Fong, &
Salvatore, 2010). Additionally, data on the use of opioid products have often been grouped
under the broader category of opioids or at the level of active pharmaceutical ingredient
(e.g., hydrocodone), rather than product-specifically (e.g., Vicodin). Having accurate
product-specific information about opioid use would be helpful for determining whether
pharmaceutical interventions, such as abuse-deterrent formulations of prescription opioids,
or other intervention efforts are effective in reducing the rates of abuse of specific products.
Using photographs to identify prescription drugs has been suggested as a promising
strategy to increase the accuracy of prescription drug identification and to capture productspecific information about opioid use. For example, photographs are being used in national
surveys, such as the National Survey of Drug Use and Health (Center for Behavioral Health
Statistics and Quality, 2019). The Addiction Severity Index-Multimedia Version (ASI-MV),
developed by Inflexxion as a tool to assess addiction severity in treatment settings, also
uses photographs to assist patients in reporting their use of prescription drugs.
A small study by Smith and colleagues (2010) provided some evidence, supporting the use
of photographs as a valid strategy for identifying opioid products and reporting use.
However, more research is needed to determine the accuracy of self-report in opioid
product ascertainment; the role of product brand names, active pharmaceutical ingredients,
slang terms, and photos in identifying products; and the overarching strategies used to
identify products.
The purpose of this study is to assess the accuracy of opioid product ascertainment among
patients undergoing treatment for substance abuse. Thus, this study will use a withinsubjects experimental design to examine whether the accuracy of patient identification of
opioid products varies, depending on the information provided about the product (e.g.,
product name, only, vs. product name and active ingredient vs. photo, only). Additionally,
we will investigate individual patient characteristics that affect the accuracy of opioid
product ascertainment. Finally, we will examine decision making and strategies used to
identify opioid products via brief cognitive interviews with patients.

5

Specifically, the primary objectives of this study are to:

▪

Assess the accuracy of opioid product ascertainment among patients in treatment
programs for substance abuse

▪

Determine the influence of photographs on the accuracy of opioid product
ascertainment

▪

Evaluate the impact of nonspecific product endorsements (e.g., by active
pharmaceutical ingredient) and potential analytic approaches to account for
nonspecific data

▪

Identify factors that affect the accuracy of opioid product ascertainment and
strategies employed in the identification of opioid products

This protocol describes the study design and data collection procedures for conducting this
research.

2. Study Design
2.1

Overview

This study will use a mixed methods approach to assess the accuracy of opioid product
ascertainment among patients undergoing treatment for substance abuse as well as learn
about decisional factors and identification strategies that influence the ascertainment
process (Exhibit 1).
Exhibit 1.

Overview of Study Design

Remote
Introduction
(Researcher-led)
► Participant joins call
► Introduce study
► Conduct informedconsent process
► Answer participant
questions

Self-Guided Online
Questionnaire
(Participant-led)

► Complete randomized
product exercises
► Complete questions
assessing patient
characteristics

Remote Cognitive
Interview
(Researcher-led)

► Explore decision-making
process for identifying
opioid products
► Investigate
understanding of
terminology for
referring to opioid
products

First, we will assess the accuracy of opioid product ascertainment among patients
undergoing treatment for substance abuse, using five ascertainment exercises that will be
administered individually to each participant in a within-subjects experiment (Exhibit 2).
We will present participants with photographs of opioid products in random order, and
depending on the type of question, participants will identify the product through free recall
or by multiple choice. For example, by selecting the product name (e.g., Vicodin), the active
pharmaceutical ingredient (e.g., hydrocodone), or both the product name and active

6

pharmaceutical ingredient (e.g., Vicodin [hydrocodone]) from a list of options. For some
exercises, participants will, instead, be shown a specific product name and asked to identify
it from a set of product photographs or to identify the active ingredient contained in the
product. The same five questions will be asked for each opioid product. This self-report data
will be collected, using an online questionnaire. The study design will allow us to meet
project objectives by testing whether ascertainment (i.e., correct identification of opioid
products) differs by the type of ascertainment measure used (free recall, list of product
names, list of product names with active ingredient option, thumbnail photo of product, and
list of active ingredients). The questionnaire will also include questions about patient
characteristics, which we will use to test whether individual differences influence the
accuracy of opioid product ascertainment.
Exhibit 2.

Opioid Product Ascertainment Exercises
in Online Assessment

Following completion of the online questionnaire, we will conduct brief cognitive interviews
to explore patient decision making and strategies for identifying opioid products as well as
patients’ understanding of language used to describe prescription opioids.

7

2.2

Experimental Stimuli

Participants will be shown a sequence of stimuli (opioid products) presented as photographs
or written product names. The product photographs and names will be taken from the ASIMV and programmed to display within the online questionnaire. We will standardize
extraneous details in the product images to eliminate potential confounding factors, like
background color and frame dimensions.
If experimental design considerations allow, a total of 32 prescription opioid pain
medications included in the ASI-MV will be used as stimuli. See Appendix A for a complete
list of opioid products from ASI-MV to be included in the experiment. These 32 opioid
products represent opioid products that include the most common active pharmaceutical
ingredients (hydrocodone, oxycodone, hydromorphone, oxymorphone, morphine, and
tapentadol), single-entity and combination formulations, brand name and generic versions,
short-acting (e.g. immediate-release) and long-acting (e.g. extended-release) formulations,
and nonspecific active pharmaceutical ingredient options (e.g., other hydrocodone IR
product). Stimuli will include products with greater market share and likelihood of being
used in a way not prescribed by a doctor as well as products with a lower market volume. In
this study, to maintain consistency across products, we will include only opioid products
designed to be ingested by mouth in either tablet or capsule form and exclude other
categories of products, such as films or patches.
With 32 opioid products as stimuli and 5 questions per product, a complete within-subjects
design would require a minimum of 160 questions per participant. Since this is a repeated
measures experiment, participant fatigue from repeatedly responding to the same
questions is an important factor to take into account. To reduce burden and corresponding
measurement error that would arise by asking that many questions, we will aim to devise a
fractional design that allows participants to complete a smaller number of questions, while
nonetheless, gathering data on all 32 opioid products across the experiment. Each
participant will respond to a sequence of questions, addressing a smaller subset of the
stimuli, but across all participants, data for the full set of 32 opioid products will be
gathered (see Appendix B). We will also group the free recall questions together on one
screen, similar to the way they appear in the ASI-MV (see Exhibit 3 as an example). This
will reduce the number of screens participants will see throughout the survey. The free
recall exercises will be completed first so that the experience of responding to other
product identification questions does not influence these responses.

8

Exhibit 3.

Example of a Free Recall Exercise

Following the free recall questions, participants will complete the remaining four exercises
for each opioid product. The order of products and response options will be randomized.
However, participants will complete all of one type of exercise before moving to the next
exercise, and the order of exercises that include active ingredients in response options may
need to be completed last to avoid learning effects. To avoid introducing a threat to external
validity, we will present participants with exercises for each product individually, as in
Exhibit 4. Combining multiple questions on one screen is an option for reducing the overall
number of screens participants see, but under this approach, responses to products that are
displayed on the same screen may become correlated through a process of elimination.

9

Exhibit 4.

Example Exercise with a List-of-ProductNames Response Format

Please select the name of the medication shown in the image below.
Note that the image may not be the same as its actual size.

○ Percocet®
○ Roxicodone®
○ Xtampza ER
○ Generic short-acting Vicodin®-type generic
○ Vicodin®
○ Generic extended-release hydromorphone
○ Lorcet®
○ Exalgo
○ Opana®
○ Kadian®
○ None of the above
○ Don’t know
2.3

Randomization

Since each participant will evaluate multiple photographs and product names, it is
important to address the potential issue of incidental effects caused by the serial order in
which stimuli and questions are presented. To mitigate this issue, we will randomize the
order in which stimuli and exercises are presented. The free-recall exercises will always be
completed first to avoid the influence of other questions on these responses, and exercises
that show the product name, along with an inactive ingredient, will always be completed
last. Exercises with the same type of ascertainment measure (e.g., identify name from list of
names) will be grouped together. However, the order in which participants are presented
with the other three measurement types will be randomized. Lastly, although all participants
will view the same set of product photographs and names, the order in which they are
presented within ascertainment exercises will also be randomized (Exhibit 5). We will
implement this randomization scheme by programming it into the online questionnaire.

10

Exhibit 5. Randomization Scheme

Note. Exercises refer to the tasks in Exhibit 2. Exercises 2, 3, and 4 will be presented in random order and will include “Identify
name from list of names,” “Identify photo of product,” and “Identify active ingredient from list of active ingredients.”

2.4

Measures

The primary outcome of this study will be accuracy of opioid product ascertainment, which
we will operationalize by flagging whether participants correctly identify the product
displayed in each ascertainment exercise. The recognition items used in the five types of
exercises differ in response format, but they can all be recategorized and combined into a
single dichotomous variable, indicating whether the product in each exercise was identified
correctly or incorrectly. The free-recall exercises will consist of open-ended measures,
where participants will be asked to type in the name of the opioid product depicted in the
stimulus. These responses will be coded as correct or incorrect, during the data preparation
phase of analysis (see Section 2.5.2). The remaining measures used in the ascertainment
exercises will be modeled on questions from the ASI-MV that ask participants to select
medications they recognize from a list of options. The format of the response options will
differ, depending on the type of exercise (list of product names, list of product names with
active ingredient option, or thumbnail photo of product), but in all cases, participants will be
asked to select one option from a close-ended list.
In addition to the ascertainment questions, we will also include a measure designed to
assess the level of exposure to each product. For the sake of brevity, these measures will
likely have closed-ended response options that indicate increasing levels of familiarity (e.g.,
heard of it, seen it, used it). Level of exposure may be useful as a product-level covariate to
include in main analyses, examining factors that influence opioid product ascertainment.
We will also measure patient characteristics that we expect might impact the accuracy of
opioid product identification, such as lifetime recreational drug use; primary or most
used/abused drug; mental health issues; health literacy; history of medical treatment for
pain; history of prior substance abuse treatment; route of administration (e.g., swallowed,

11

snorted, injected); availability of other opioids; source of opioids used (e.g., prescription,
friend); and demographics.

2.5

Interview Guide

We will develop a guide for conducting brief semi-structured interviews to explore
participants’ decision-making process, while identifying opioid products. The guide will
allow us to probe on the features and characteristics of opioid products that participants
use to identify products. We will also inquire about participants’ level of confidence in the
strategies they use to identify products and the accuracy of their decision making and
identification processes. We will also explore participants’ understanding and interpretation
of terms used to refer to opioid products, such as “prescription opioids” and “medications.”
Interviews will be conducted remotely, using a web-based video conferencing software,
and will take about 10-15 minutes to complete.

2.6

ASI-MV Data Access

In a two-step consent process (see Section 3.3.2), the researcher will ask the participant
for permission to access their responses to the ASI-MV assessment, which participants
would have completed, during intake to their substance abuse treatment program. These
data would be used to gather additional patient characteristics, such as their drug severity
score, to further investigate factors that may influence the accuracy of opioid product
recall. The participant may choose to deny access to their ASI-MV assessment data without
penalty. Analysis will be based upon the subset of participants who agree to allow access to
their ASI-MV assessment information.

2.7

Analysis Plan

The primary outcome of this study will be correct identification of opioid products displayed.

2.7.1

Descriptive Analysis

As a first step, we will produce descriptive statistics for continuous variables (i.e., means,
standard deviations, medians, quartiles, and frequencies) and categorical variables (i.e.,
frequencies and percentages). For composite measures, we will assess internal consistency
among the items for each measure, using Cronbach’s alpha as our metric. For scales that fail
to meet our threshold of 0.75, we will examine whether dropping items will improve
reliability or use a single-item measure.

12

2.7.2

Coding Open-Ended Opioid Product Ascertainment Measures

We will conduct a content analysis of
responses to the open-ended questions to
classify responses as either correctly or
incorrectly identifying the product shown in
each free-recall exercise. We will first
develop a codebook that captures whether
the answers provided constitute correct
ascertainment for each product and the
degree to which answers are correct. For
example, did participants correctly identify
the brand name (if applicable), the active
ingredient, the time release formulation,

Exhibit 6. Example Codebook Questions
to Analyze Open-Ended Responses to
Free Recall Questions

Did participants correctly
identify the brand name?
Did participants correctly
identify the active
ingredient?
Did participants correctly
identify the time release
formulation (ER/IR)?
Did participants correctly
identify the street or slang
name?

Brand Name
Product
Yes/No

Generic
Product
NA

Yes/No

Yes/No

Yes/No

Yes/No

Yes/No

Yes/No

the street or slang name (see Exhibit 6)?
Given the study objective related to evaluating the impact of nonspecific product
endorsements and the related issues when conducting product-specific analyses, we
recommend including an additional code for responses that refer to the product category or
active ingredient, so we can differentiate those from responses that refer to the product by
name. To establish reliability of the codebook, we will draw a random sample of responses,
related to each product (10% of total responses), and two independent coders will classify
responses, using the first draft of the codebook as a guide. We will then compare the
results and calculate Krippendorff’s alpha for each code to assess intercoder reliability
(Hayes & Krippendorff, 2007). The coders will work together to resolve any discrepancies
and revise the codebook as necessary until all codes obtain a Krippendorff’s alpha
coefficient of 0.8 or higher. After the codebook is finalized, we will split the remaining openended responses between the two coders, who will complete the rest of the coding,
independently. The resulting coded variables will be used to construct part of the opioid
product ascertainment outcome variable.

2.7.3

Main Analysis

To understand the effect of stimuli and exercises on product ascertainment, repeated
measures will be conducted on each participant. Whenever within-subjects factors are used
in an experiment, the statistical methods need to adjust for data-correlated errors that are
likely to arise, due to multiple measurements made on the same subject. Hierarchal
models or mixed models will be used to analyze and account for variability, among
participants and within participants, from measure to measure.
The proposed study design can be thought of as a multi-level model with crossed random
effects. Each ascertainment exercise is a Level-1 fixed factor nested both within participants
and products, which are Level-2 random factors. Participants and products are crossed at

13

Level 2 because every participant will rate several products. A general example of this
design, a replicated fully crossed design, is described by Judd, Westfall, & Kenny (2016).
The comparative effects of specific products used as stimuli are not the inferential focus of
this statistical analysis; instead, the products have been chosen for the sake of ecological
validity to represent the various kinds of opioid products that participants might have been
exposed to or used in the past and because they are included on the ASI-MV.
Since our primary outcome measure is dichotomous, we will conduct a multilevel mixedeffects logistic regression. In this model, correct ascertainment is the outcome and is
measured at Level 1 (i.e., the ascertainment exercise level). The main predictors of interest
also occur at Level 1 and relate to the type of measure (free-recall, product list, product list
with active ingredient option, active ingredient list, or photograph). The multilevel
regression equation for the model1 predicting correct ascertainment by type of measure is
logit(CORRECT ijk = 1) = γ 0 + γ 1 (TASK_2 ij ) + γ 2 (TASK_3 ij ) + γ 3 (TASK_4 ij ) + γ 4 (TASK_5 ij ) +
γ 5 (BLOCK_2 ij ) + γ 6 (BLOCK_3 ij ) + γ 7 (BLOCK_4 ij ) +
γ 8 (TASK_2 ij × BLOCK_2 ij ) + γ 9 (TASK_2 ij × BLOCK_3 ij ) + γ 10 (TASK_2 ij × BLOCK_4 ij )
+
γ 11 (TASK_3 ij × BLOCK_2 ij ) + γ 12 (TASK_3 ij × BLOCK_3 ij ) + γ 13 (TASK_3 ij ×
BLOCK_4 ij ) +
γ 14 (TASK_4 ij × BLOCK_2 ij ) + γ 15 (TASK_4 ij × BLOCK_3 ij ) + γ 16 (TASK_4 ij ×
BLOCK_4 ij ) +
γ 17 (TASK_5 ij × BLOCK ij ) + γ 18 (TASK_5 ij × BLOCK_3 ij ) + γ 19 (TASK_5 ij × BLOCK_4 ij ) +
u i0 + u i1 (TASK_2 ij ) + u i2 (TASK_3 ij ) + u i3 (TASK_4 ij ) + u i4 (TASK_5 ij ) +
u 0j + u 1j (TASK_2 ij ) + u 1j (TASK_3 ij ) + u 3j (TASK_4 ij ) + u 4j (TASK_5 ij ) + u ij ,
where CORRECTijk represents the responses of participant i to drug j on task k, γ are fixed
effects estimates, and ui. and u.j are participant and drug random effects, respectively.
TASK_2 – TASK_5 are effects-coded variables, representing the type of measures used in
the ascertainment exercises, and BLOCK_2 – BLOCK_4 are dummy-coded levels of a
factorial variable, representing which group of drug products participants were randomly
assigned as stimuli. The block main effects and task-by-block interaction terms statistically
control for the potential influence that the specific set of 8 drug products used in each
block may have on ascertainment. In all, the model has 20 fixed-effects coefficients and 11
random-effects coefficients.
In the final model, we may also include fixed-effects predictors, related to the Level 2
factors: Participant (demographics, health literacy, product familiarity, drug severity score

We adapted this equation from the canonical analytic model for a replicated fully crossed design
outlined by Judd, Westfall, & Kenny (2016) in the supplementary appendix to their article.

1

14

from the ASI-MV, etc.) or product (category, extended or short release). 2 The main
analyses will exclude the drug severity score from the ASI-MV so that data from all
participants who complete the study exercises will be included. Exploratory analyses,
involving the drug severity scores, will only include the subset of participants who give us
permission to link their ASI-MV drug severity scores with study data. These analyses will
allow us to examine the influence of patient characteristics (e.g., level of exposure, age,
region, etc.) on opioid product ascertainment. We will verify statistical assumptions of the
model by numerically and graphically assessing residuals. If models fail to meet parametric
assumptions, alternative nonparametric models, such as a nonparametric rank-based
mixed models, will be used (Noguchi et al., 2012).
Standard practice in voluntary survey-based research is to allow participants to skip
questions if they choose (American Association for Public Opinion Research, 2014). We will
include a “Don’t know” response option, when applicable. Since participation is voluntary,
participants may withdraw from the study at any point if they choose.

Power Analysis
Assuming repeated measures over 96 participants, with each participant answering
questions for 8 product names, and 5 types of ascertainment measures per participant and
product name in a multilevel design with crossed random effects (i.e., participants and drug
products are nested in 4 blocks with 24 participants and 8 products per block), the study
will have 80% power at α = 0.05 to detect medium-small main effects by type of
ascertainment measure (d = 0.41) (Westfall, Kenny, & Judd, 2014). 3 The design has
considerable flexibility if we need to revise assumptions to include fewer participants. For
example, with only 48 participants and all other assumptions held constant, the study would
still be sensitive to detect conventionally medium-sized effects (d = 0.47).

2.7.4

Analysis of Interview Data

We will use a thematic and iterative approach for analyzing the interview data gathered
from all participants who complete the interviews. These will be the same participants as
those included in the main analyses, described above. This qualitative method involves
identifying, analyzing, and reporting patterns or “themes” within data (Aronson, 1995;

2 Due to the complexity of the study design, limited sample size, and the number of parameters in the
bare-bones analytic model, we will control for potential order effects by design—through
randomization—rather than statistically, by including variables that record the order of the tasks as
model covariates.
3 Cohen’s d is a standardized effect size index defined as the difference between two means divided by
the pooled standard deviation of observations within conditions (Cohen, 1988; Judd, West, & Kenny,
2016). Thresholds for interpreting magnitude are conventional, where effects with d values of 0.20 to
0.50 are small, 0.50 to 0.80 are medium, and 0.80 or greater are large (Cohen, 1988). Cohen’s d can
be converted to other effect size indices (e.g., r, f, η2, OR), using various conversion formulas (Cohen,
1988; Ferguson, 1966; Rosenthal, 1994). For example, d = 0.41 is equivalent to r = 0.20.

15

Braun & Clarke, 2006). A first read of the data serves to gain a high-level understanding of
participant responses. Further analysis is used to identify key themes that emerge across
participants. Two researchers will first examine the data separately and then compare
interpretations to identify areas of agreement and resolve any inconsistencies in the
findings. Additional exploratory analysis may be performed among the participants who
give us permission to link their ASI-MV drug severity scores with study data.

3. Data Collection Procedures
3.1

Site Identification

Inflexxion will identify 4 to 5 substance abuse treatment centers from which to recruit
participants undergoing treatment for opioid abuse (target N = 100; 20-25 participants per
site). These sites will be selected from Inflexxion’s client base of addiction treatment centers
that license the ASI-MV. We will aim for a mix of sites, with respect to the following criteria:

▪

Geographic region—We will attempt to include as diverse a range of sites as possible,
based on geographic location, when recruiting sites from the ASI-MV user base.

▪

Urban/rural location—We will attempt to include at least two sites located in
nonmetropolitan counties and at least two sites in metropolitan counties (Ingram &
Franco, 2014).

▪

Type of treatment program—We will attempt to include at least two sites with
inpatient programs and two with outpatient programs.

Anticipating recruitment challenges, due to the coronavirus pandemic, additional criteria
for identifying sites to approach may help to ensure success in reaching recruitment goals.
Thus, we may want to consider including:

▪

Clinics that have both inpatient and outpatient programs at the same facility to allow
for recruitment from both programs at one site, if needed.

▪

Geographic regions that contain more than one site, in case recruitment challenges
make it necessary to recruit participants from multiple sites in that region.

We will draft a description of the study and information about what would be required from
sites to participate in the study. Inflexxion will contact and inform sites of the study and
inquire about participation. At the completion of data collection, Inflexxion will provide sites
with compensation for time and effort expended by each site to assist with participant
recruitment, scheduling, and data collection.

3.2

Participant Eligibility

Participants will be recruited from sites via convenience sampling. Sites will coordinate
participant recruitment and eligibility screening in partnership with RTI. We will aim to
recruit approximately 100 participants for this study. We will develop a screener that sites
will use to determine whether patients meet the following eligibility criteria:

16

▪

Report past 30-day abuse of at least one prescription opioid intended for the
treatment of pain (e.g., any prescription opioid which contains hydrocodone,
hydromorphone, oxycodone, oxymorphone, morphine, tapentadol)

▪

Not experiencing cognitive or physical symptoms, due to drug use (e.g., currently
high) or the stage of treatment (e.g., withdrawal) that may affect their ability to
understand study procedures, provide informed consent, and complete a 45-minute
questionnaire

▪

18 years of age or older

▪

Ability to read English

▪

No prior experience working in a medical or health-related field

Participants who do not meet all of the above criteria will be excluded from the study.

3.3
3.3.1

Procedures
Recruitment and Screening

Participants will be recruited from inpatient and outpatient treatment programs at study
sites. Inflexxion will work with participating sites to identify a designated lead staff member
(i.e., a site study coordinator) to manage recruitment efforts by providing study information
to potential participants, screening, and scheduling study participants. Site study
coordinators will be staff members at each of the sites who are experienced in working with
this population and the patients at their respective sites.
Potential participants will be contacted by the site study coordinator, who will provide them
with information about the study, gauge their interest, and screen them for eligibility.
Depending on what is most convenient for the study site, recruitment, screening, and
scheduling may be done over the phone or in person. It will likely be more convenient to
contact patients in outpatient programs over the phone, while inpatient recruitment and
screening may be done primarily in person. Eligible participants will then be scheduled for a
60-minute time slot to complete the study. While participants at inpatient clinics will be
provided with a computer and a private room at the clinic facility, outpatient participants will
be allowed to choose whether to participate in the study at the treatment center or at
home. Participants in outpatient clinics who prefer to participate at the clinic will be
encouraged to schedule their study participation to coincide with a planned visit to the
clinic, if possible, to avoid the need to schedule an additional visit.
The screener will guide the site study coordinator through the recruitment and screening
process. It will include a script for the site study coordinator to follow, including a
description of the study, questions for gauging interest and inviting the potential participant
to be screened, screening questions, and questions pertaining to scheduling. Eligible
individuals who are interested in participating will be scheduled immediately following
screening. Participant scheduling will be tracked in an Excel file that will assign a unique ID
17

number to each participant. The Excel file that site study coordinators share with RTI will
include only the participant ID number and screening data and will not include any
personally identifiable information (PII), such as participant names and contact information.
This participant ID number will be used to link screener data to the main study data.
We will monitor screening data to aim for quotas across sites of at least 15% Black
participants and 12% Hispanic participants. These quotas are based on 2019 data from the
ASI-MV of past 30-day use of prescription opioids among adults ages 18 and older.
To ensure we reach our proposed target sample size, study sites will aim to overrecruit and
schedule an additional 3-5 participants beyond the target number for the site. Recruitment
will be adapted at each site, depending on the no-show rate, though this is less likely to be
an issue at inpatient facilities. To increase participation rates, study sites can also contact
participants the day prior to remind them of their upcoming scheduled study participation.

3.3.2

Consent

The informed consent process will be conducted remotely by an RTI researcher, using webbased video conferencing software at the time of participants’ scheduled study participation
on a computer in a private room provided by the site or at the participant’s home. The
consent process will involve two steps. The first step will focus on participation in the study,
ending in a voluntary decision about completing the experiment and interview portions of
the protocol. In the second step, we will request permission from those who give consent to
allow us to link their responses from the ASI-MV assessment completed during intake to
their substance abuse treatment program. Participants who do not grant permission for us
to access their ASI-MV data will, nonetheless, remain eligible to complete the study, but
their data may be excluded from some exploratory analyses, involving drug severity scores
gathered through the ASI-MV. We will explain to participants that granting us access to
their ASI-MV data is optional and that they can still participate in the study and will receive
compensation for their participation, even if they choose not to allow us access to their
ASI-MV data.
An RTI researcher will introduce participants to the study and lead them through the
consent process, allowing participants to ask questions about the study and the
requirements of participation, prior to giving consent. Consent procedures will include
informing potential participants of the purpose of the research, who is conducting the
research, the voluntary nature of participation and the ability to withdraw from the study
at any time, study length and procedures, possible risks to participation, the
confidentiality of participant responses, benefits to participation, a token of appreciation
participants will receive, and contact information in case participants have questions,
following their participation in the study. Consent information provided to participants will
be written clearly, using plain language to avoid confusion and misunderstanding.

18

We will work closely with the IRB and detailed requirements to ensure participants are
adequately informed and consent is given voluntarily. One potential risk to participation
may be some psychological discomfort or distress from viewing photos of opioid products
with which participants have a history of abuse. We will work with study sites to ensure they
have a protocol for monitoring and checking for potential negative psychological effects or
potential triggering effects of exposure to photos of opioids. If there is evidence of these
effects, prior to completion of the study, and participants need to withdraw from the study,
we will work with sites to ensure they follow up with participants and have a protocol in
place for responding to participants’ needs. If incomplete, the participant’s data will likely be
removed from the study.
We will also develop a protocol for sites to implement in the event that participants show up
to participate, while under the influence of drugs or alcohol, in acute withdrawal, or
otherwise impaired. Since these individuals may be incapable of giving informed and
voluntary consent as well as maintaining focus for the duration of the study, they will either
be rescheduled or removed from the study.

3.3.3

Data Collection

Collection Setting
Data collection will occur at participating inpatient and outpatient sites. Eligible participants
at these sites will be directed by clinic staff to a private study room at the scheduled time
to use a computer provided by the clinic. Clinic staff will help participants join the
videoconference call with the RTI researcher.
Participants will first engage in the consent process and, if they consent to participate,
continue with the study by completing the online questionnaire. Participant completion of
the online questionnaire will be entirely self-guided. However, a clinic staff member will be
present and available, either within the room or nearby, to: 1) ensure the participant’s
privacy during questionnaire completion (i.e., no one else enters the room), 2) discern
whether participants may be experiencing any adverse effects from study participation, and
3) troubleshoot any computer or internet connectivity issues that arise.
Data collection with each participant will require approximately 1 hour. Inflexxion will
provide study sites with incentives that the site study coordinator will give to participants in
appreciation of their time.
We will work with study sites to ensure procedures are in place to reduce the risk of
coronavirus transmission and ensure participants are as safe as possible, during study
participation. These procedures are not likely to be above and beyond what clinics are
already doing to reduce transmission. However, we will include a checklist for study sites
to follow, during data collection, to ensure measures, such as the use of face coverings
and cleaning of the computer equipment and other high-touch areas, continue to be
followed.
19

Online Questionnaire
Participant data from the experiment will be collected online via a computer-assisted
questionnaire. Participants will be guided through a series of randomized exercises,
asking them to respond to photographic stimuli as well as answer demographic and
background questions.
We will first develop an annotated draft of the study questionnaire, including programming
notes, in MS Word for review and revision before programming the interactive version for
computer-assisted self-administration. We will program the questionnaire, using Voxco, a
Web survey platform. We will then conduct internal usability testing and quality checks
(QC) to ensure fidelity, for example, that question wording is accurate; that questions and
photos display correctly; and that skip patterns, question randomization, and branching
function as intended. We have detailed procedures for conducting a thorough QC of online
questionnaires, including rigorous testing of the programmed instrument to ensure all
programming logic is correct. Once this initial quality testing is complete, we will provide
Inflexxion with a link to the programmed survey to approve.
We will also plan to review the data early in the data collection process to ensure that
survey data are being captured as intended.

Cognitive Interviews
After completing the questionnaire, participants will rejoin the videoconference call with
the RTI researcher, who will then lead them through a brief interview to explore their
decision-making process and understanding of terminology. Interviews will be conducted
with the video on so that the participant and researcher can see each other to simulate an
in-person conversation. The interviewer will take notes, during the interview. Interviews
will also be recorded as a backup in case we need to fill in any gaps in our notes, prior to or
during the analysis phase.

4. Summary
This protocol describes the study design and data collection procedures for a repeated
measures experiment to examine the accuracy of opioid product identification across five
exercises, using different stimuli. Data will be collected via an online questionnaire from
around 100 participants at 4 to 5 substance abuse centers. Following the experiment, brief
semi-structured interviews will be used to learn about participants’ decision-making
process, while identifying opioid products. Analysis of these data will contribute to the gap
in research, examining the validity of self-report for identifying opioid products and
reporting opioid use.

20

5. References
American Association for Public Opinion Research (AAPOR). (2014). AAPOR guidance for
IRBs and survey researchers. https://www.aapor.org/Standards-Ethics/InstitutionalReview-Boards/Full-AAPOR-IRB-Statement.aspx
Aronson, J. (1995). A pragmatic view of thematic analysis. The qualitative report, 2(1), 1-3.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research
in psychology, 3(2), 77-101.
Center for Behavioral Health Statistics and Quality. (2019). 2019 National Survey on Drug
Use and Health: Prescription drug images for the 2019 questionnaire. Rockville, MD:
Substance Abuse and Mental Health Services Administration.
https://www.samhsa.gov/data/sites/default/files/cbhsqreports/NSDUHPillImages2019.pdf
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale,
NJ: Erlbaum.Del Boca, F. K. & Noll, J. A. (2002). Truth or consequences: The validity of
self-report data in health services research on addictions. Addiction, 95(S3), S347‒S360.
https://onlinelibrary.wiley.com/doi/abs/10.1046/j.1360-0443.95.11s3.5.x
Ferguson, G. A. (1966). Statistical Analysis in Psychology & Education. New York, NY: McGraw-Hill.
Hayes, A. F. & Krippendorff, K. (2007). Answering the call for a standard reliability
measure for coding data. Communication Methods and Measures, 1(1), 77‒89.
https://doi.org/10.1080/19312450709336664
Ingram, D. D., & Franco, S. J. (2014). 2013 NCHS urban-rural classification scheme for
counties. National Center for Health Statistics. Vital and Health Statistics, 2(166).
https://www.cdc.gov/nchs/data/series/sr_02/sr02_166.pdf
Judd, C. M., Westfall, J., & Kenny, D. A. (2016). Experiments with more than one random
factor: Designs, analytic models, and statistical power. Annual Review of Psychology, 68,
17.1–17.25. https://doi.org/10.1146/annurev-psych-122414-033702
Noguchi, K., Gel, Y. R., Brunner, E., & Konietschke, F. (2012). nparLD: An R software
package for the nonparametric analysis of longitudinal data in factorial experiments.
Journal of Statistical Software, 50(12), 1-23.
Rosenthal, R. (1994). Parametric measures of effect size (pp. 231 - 244). In H. Cooper & L.
V. Hedges (Eds.), The Handbook of Research Synthesis. New York, NY: Sage.
Smith, M., Rosenblum, A., Parrino, M., Fong, C., & Colucci, S. (2010). Validity of selfreported misuse of prescription opioid analgesics. Substance Use & Misuse, 45(10),
1509‒1524. https://doi.org/10.3109/10826081003682107
Substance Abuse and Mental Health Services Administration (SAMHSA). (2019). Key
substance use and mental health indicators in the United States: Results from the 2018
National Survey on Drug Use and Health (HHS Publication No. PEP19-5068, NSDUH
Series H-54). Rockville, MD: Center for Behavioral Health Statistics and Quality,
Substance Abuse and Mental Health Services Administration.
https://www.samhsa.gov/data/
Westfall, J. Kenny, D. A., & Judd, C. M. (2014). Statistical power and optimal design in
experiments in which samples of participants respond to samples of stimuli. Journal of
Experimental Psychology: General, 143(5), 2020–2045.
https://doi.org/10.1037/xge0000014

21

Appendix A.
Opioid Products from ASI-MV Included in the Experiment

7.

Product
Lorcet
Lortab
Vicodin
Apadaz
Zohydro ER
Hysingla ER
Other short-acting Vicodintype generic

8.

Percocet

9.

Roxicet
Other short-acting oxycodone
combination
Roxicodone
Other short-acting oxycodone
non-combination IR
Actavis oxycodone IR
New OxyContin (marked with
"OP")
Xartemis XR
Xtampza ER
Other oxycodone ER
Dilaudid

1.
2.
3.
4.
5.
6.

10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.

Exalgo
Other Dilaudid-type generic
Other generic extendedrelease hydromorphone
Reformulated Opana ER
Generic extended-release
oxymorphone (photo #1)
Generic extended-release
oxymorphone (photo #2)
Opana
Nucynta
Nucynta ER
MS Contin
Kadian
Embeda
Arymo ER
MorphaBond ER

Active Ingredient
Category
Hydrocodone
Hydrocodone
Hydrocodone
Hydrocodone
Hydrocodone
Hydrocodone
Hydrocodone

Brand/
Generic
Brand
Brand
Brand
Brand
Brand
Brand
Generic

ShortActing/
LongActing
Short
Short
Short
Short
Long
Long
Short

Short-acting combination
oxycodone IR
Short-acting combination
oxycodone IR
Short-acting combination
oxycodone IR
Single-entity oxycodone IR
Single-entity oxycodone IR

Brand

Short

Combination

Brand

Short

Combination

Brand or
generic
Brand
Generic

Short

Combination

Short
Short

Single
Single

Single-entity oxycodone IR
Oxycodone ER

Generic
Brand

Short
Long

Single
Single

Oxycodone ER
Oxycodone ER
Oxycodone ER
Hydromorphone
Hydromorphone

Long
Long
Long
Short
Long

Combination
Single
Single
Single
Single

Hydromorphone
Hydromorphone

Brand
Brand
Generic
Brand
Brand or
generic
Generic
Generic

Short
Long

Single
Single

Oxymorphone
Oxymorphone

Brand
Generic

Long
Long

Single
Single

Oxymorphone

Generic

Long

Single

Oxymorphone
Tapentadol
Tapentadol
Morphine
Morphine
Morphine
Morphine
Morphine

Brand
Brand
Brand
Brand
Brand
Brand
Brand
Brand

Short
Short
Long
Long
Long
Long
Long
Long

Single
Single
Single
Single
Single
Combination
Single
Single

Combination/
Single
Combination
Combination
Combination
Combination
Single
Single
Combination

22

Appendix B.
Experimental Design: Replicated Fully Crossed Design
There are 32 prescription opioid pain medications included in the ASI-MV that are taken
orally in tablet or capsule form. Presenting 32 products as stimuli to each participant in a
repeated measures design with 5 exercises per product would be too burdensome for
participants. To reduce the number of questions that each participant completes, while
nonetheless, gathering data on all 32 opioid products across the experiment, we
recommend using a replicated fully crossed design (Judd, Westfall, & Kenny, 2016).
Exhibit B1 is a matrix, illustrating the following assumptions:
•

32 opioid products that will be used as stimuli

•

Opioid products will be randomly allocated to 4 blocks, each containing 8 drug
products per block. This represents the number of stimuli that each participant will
see. Participants will also be randomly assigned to the 4 blocks, each containing an
equal number of participants (e.g., using permuted block randomization), to ensure
a balanced design. Participants will complete all 5 exercises for each of the 8
products in their assigned block (a total of 40 exercises per participant).

Exhibit B1. Example arrangement of opioid product stimuli by block and presentation order.
Opioid Product
B
P 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
1
1
× × ×
1
… × × ×
1
24 × × ×
2
25
2
…
2
48
3
49
3
…
3
72
4
73
4
…
4
96
Note. B = Block; P =
1. Opana
2. Kadian
3. Roxicodone
4. Nucynta

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

×
×
×

× × × × × × × ×
× × × × × × × ×
× × × × × × × ×
Participant. For each opioid product, participant cell values represent opioid products included in the ASI-MV:
12. Other short-acting oxycondone combo
23. Xtampza ER
13. Dilaudid-type generic
24. Nucynta ER
14. MS Contin
25. Hysingla ER
15. Xartemis XR
26. Generic extended-release
oxymorphone
5. Generic oxymorphone ER
16. Zohydro ER
27. Dilaudid
6. Actavis oxycodone IR
17. Vicodin
28. Other short-acting Vicodin-type
generic
7. Other oxycondone ER
18. New OxyContin
29. Percocet
8. Other generic extended-release
19. Apadaz
30. Roxicet
hydromorphone
9. Reformulated Opana ER
20. Other short-acting oxycondone non31. Exalgo
combination ER
10. Embeda
21. Lortab
32. MorphaBond ER
11. Lorcet
22.Aryzmo ER

23