Supporting Statement Part B

U.S. Food and Drug Administration
Formative Research Support: Outcomes and Awareness Measurement Research
OMB Control Number 0910-0810
SUPPORTING STATEMENT B
B.

Statistical Methods
1. Respondent Universe and Sampling Methods

The respondent universe for this study is youth aged 13 to 17 years who are able to watch
and hear videos on their electronic device (computer, tablet, or phone). The screener survey will
include questions assessing race, ethnicity, gender identity, and state of residence. To ensure a
reasonably diverse sample, we will set the following soft quotas:
•

Maximum 70% female gender identity

•

Maximum 85% white, non-Hispanic

•

Approximately equivalent distribution across geographic regions

The study will use convenience samples rather than probability samples. We do not
intend to generate nationally representative results or precise estimates of population parameters
from the study; generating a representative sample of the size necessary for this study (e.g., using
random digital dialing or a similar method) would be cost prohibitive. Additionally, it is not
necessary as this is a measures development study.
2. Procedures for the Collection of Information
This section provides an overview of the study procedures, provides information on the
degree of accuracy required for the study, and discusses the estimation procedures.
2a. Study Procedures
Objectives and Research Questions
This study includes the following key objectives and research questions:
1) Objective: Examine the extent to which the distribution and accuracy of self-reported
awareness varies across different awareness assessment approaches
•
•
•

RQ1: To what extent does campaign awareness vary by awareness assessment approach?
RQ2: Which awareness assessment approach is most valid?
RQ3: Which awareness assessment approach is most sensitive to variation in campaign
exposure?
2) Objective: Develop comprehensive, valid, multi-item scales to measure outcomes such
1

as addiction perceptions, expectations of feeling anxious after using tobacco, and other key
predictors of tobacco use that campaign messages aim to change.
•
•

RQ4: What is the factor structure and reliability of the scales?
RQ5: Do the scales relate to other measures of the same construct and/or outcomes that
the scales are hypothesized to be related to?
Study Design and Protocol

To address the objectives and research questions above, we will conduct a
multicomponent study that includes two waves of data collection. The study will be administered
via online surveys conducted among youth (aged 13-17), recruited from adult participants of a
national online panel who are parents of a child in the eligible age range. The Wave 1 survey will
be conducted among a sample of 2,400 youth, and Wave 2 will be conducted approximately 1-2
weeks later among youth from Wave 1 who agree to participate in the follow-up survey.
The study will include the following key components:
1) In Wave 1, we will administer a battery of attitudinal and belief items that campaign
messages aim to change, to facilitate the development and assessment of multi-item
scales for use in media campaign evaluation (Objective 2). These items will be asked
again in Wave 2 to facilitate an analysis of test-retest reliability.
2) Wave 1 will also include a study (Study 1) to address RQs 1 (To what extent does
campaign awareness vary by measurement approach?) and 3 (Which awareness
measurement approach is most sensitive to variation in campaign exposure?) under
Objective 1. Study 1 will use a 3x3 factorial design with factors of recall period (3
levels: 1 week, 1 month, and 3 months) and stimuli format (3 levels: 6-second video,
15-second video, and image collage). Respondents will be randomly assigned to one
of 9 conditions using adaptive randomization to ensure balanced distribution across
conditions. Following assignment to condition, respondents will be shown 3 Real
Cost ads, with variation in stimuli format according to study condition, and asked to
indicate frequency of having seen the ad, with variation in recall period according to
study condition. Ad awareness will be assessed separately for each of 3 ads. Figure 1
illustrates the study design for Study 1.

2

Figure 1. Study 1 design

3) We will conduct a second study (Study 2) to assess RQ2 (Which awareness measurement
approach is most valid?) under Objective 1. This study will use a 2x3 factorial design
with factors of ad exposure (two levels: exposed and unexposed) and stimuli format (3
levels: 6-second video, 15-second video, and image collage). At the conclusion of the
Wave 1 survey, we will randomly assign participants to either an exposure condition, in
which they will be shown 3 new 15-second Real Costs ads that have never aired or have
not aired recently (in randomized order), or a control condition with no ad exposure.
Next, participants who completed Wave 1 will be recontacted approximately 1-2 weeks
later to invite their participation in a second survey. In the Wave 2 survey, participants
will be randomly assigned to a stimuli format condition (6-second video, 15-second
video, and image collage). Following assignment to condition, respondents will be shown
the 3 Real Cost ads that were shown in Wave 1 survey, with variation in stimuli format
according to study condition. Ad awareness will be assessed separately for each of 3 ads.
Figure 2 illustrates the study design for Study 2.

3

Figure 2. Study 2 design

Recruitment and survey administration
Participants will be recruited by an online survey vendor, Lightspeed, that manages a
national online panel of adults. Lightspeed may partner with other survey vendors who maintain
online panels. These partner vendors will only participate in recruitment and administering
incentives to their own panel members; they will not be involved in survey programming, survey
hosting, data collection, or data storage. All aspects of data collection (screener, consent, and
survey administration) will be hosted by Lightspeed, regardless of the panel through which the
participant was recruited.
Lightspeed will identify adults from this database who are likely to have eligible children
based on the number of children in the home and their ages. Lightspeed will send an initial
generic communication to parents through the panelist portal indicating that they have a new
survey available. If a parent clicks on this communication, they are directed to the parental
permission form. If a parent provides permission, we will ask for the child’s assent to participate
and screen them to determine eligibility. Participants will be screened based on age (13-17).
Those who meet the eligibility criteria will continue with the study and otherwise will be
excluded from the study. In appreciation of their child’s participation in the surveys, adult
panelists will receive non-monetary points that can be redeemed for goods, services, or cash.
2b. Degree of Accuracy Required for the Study
For the purposes of estimating statistical power, we assumed a Study 1 sample size of 2,400
participants with equal allocation (n = 267) to each of nine groups. For Study 2, we
conservatively assumed a 50% retention rate, yielding an estimated sample size of 1,200 with
4

equal allocation (n = 200) to each of 6 groups. For Study 1, we estimate being 80% powered to
detect a main effect size of 0.14 (i.e., a difference of 0.14 in mean awareness scores between two
levels of a study factor). For Study 2, we estimate being 80% powered to detect a 9.8%
difference in proportions recalling an ad between stimuli format groups (assuming 50% ad recall
for the comparison group).
2c. Estimation Procedures
Statistical analyses will be conducted to address the study’s primary research questions.
For Objective 1 (Examine the extent to which the distribution and accuracy of selfreported awareness varies across different awareness assessment approaches), we will conduct
descriptive analyses and develop multivariable models to examine statistical associations
between study condition and self-reported ad awareness and recall.
For Objective 2 (Develop comprehensive, valid, multi-item scales to measure outcomes
such as addiction perceptions, expectations of feeling anxious after using tobacco, and other key
predictors of tobacco use that campaign messages aim to change), we will examine the factor
structure of outcome items and assess scale reliability and validity according to best practices in
scale development. 1
3. Methods to Maximize Response Rates and Deal with Nonresponse
To maximize participation, we will incorporate best practices from similar online surveys
into our data collection procedures. These include:
• Implementing a soft launch of the online survey to a small number of selected panel
members to detect and resolve any technical difficulty.
• Keeping the questionnaire at a reasonable length to minimize break-offs.
• Including a brief introduction to the study that identifies FDA as the sponsor, states the
purpose of the study, and provides toll-free telephones numbers for participants to call
RTI with any questions about the study or their rights as a study participant.
• Inviting panel members who appear to be eligible based on their member profile. As part
of the process of registering with the survey panel, panelists provide information about a
range of sociodemographic characteristics, including whether or not they have children,
that can be used to target particular groups. Lightspeed actively manages panelist
profiles, requesting updated information on an ongoing basis to ensure that profile
information is up to date.
• Recruiting verified panelists. Lightspeed uses a double opt-in registration process
whereby panelists are invited to participate and then must sign up through an opt-in
confirmation e-mail. This process protects against fraudulent account registrations and
ensures that panelists are actively motivated to participate in surveys.
• To minimize nonresponse, Lightspeed will conduct ongoing monitoring of response
levels and drop-off rates. Lightspeed will work with RTI project staff to address any
problems that arise throughout the course of the collection of information.
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices
for developing and validating scales for health, social, and behavioral research: a primer. Frontiers in Public Health,
6, 149.

1

5

4. Test of Procedures or Methods to be Undertaken
RTI International will conduct rigorous internal testing of the online survey instruments prior
to fielding. Survey testers will review the online test version of the instrument that we will use to
verify that instrument skip patterns are functioning properly, and that all survey questions are
worded correctly and are in accordance with the instrument approved by IRB and OMB.
Lightspeed will begin data collection with a soft launch during which they will send a generic
communication to a small subset of panel members and review their responses to ensure the
online survey is working properly.
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or
Analyzing Data
The following individuals inside the agency have been consulted on the design and statistical
aspects of this information collection as well as plans for data analysis:
Emily Sanders
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
9200 Corporate Boulevard
Rockville, MD 20850
Phone: 240-402-4269
E-mail: emily.sanders@fda.hhs.gov
Matthew Walker
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
9200 Corporate Boulevard
Rockville, MD 20850
Phone: 240-402-3824
E-mail: matthew.walker@fda.hhs.gov
Erin O’Brien
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-4047
E-mail: erin.obrien@fda.hhs.gov
Lindsay Pitzer
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
6

Silver Spring, MD 20993
Phone: 240-620-9526
E-mail: lindsay.pitzer@fda.hhs.gov
The following individuals outside of the agency have been consulted on questionnaire
development and/or will be collecting and/or analyzing data:
Matt Eggers
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-541-6683
E-mail: meggers@rti.org
Jessica Pepper
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-316-3180
E-mail: jpepper@rti.org
Jim Nonnemaker
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-541-6683
E-mail: jnonnemkaer@rti.org
Ashley Feld
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 781-370-4066
E-mail: afeld@rti.org
Clement Acheampong
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-485-5440
E-mail: cacheampong@rti.org

7