Supporting Statement Part B Flavors Phase 2 Study 2022

U.S. FOOD AND DRUG ADMINISTRATION

How Consumers Use Flavors to Make Inferences About Electronic Nicotine Delivery System (ENDS) Product Qualities and Intentions to Use (Phase 2)

OMB Control No. 0910-NEW

SUPPORTING STATEMENT Part B. Statistical Methods

1. Respondent Universe and Sampling Methods

The study sample will consist of approximately 1,250 current cigarette smokers, 1,250 current e‑cigarette users, 1,250 current dual users, and 1,250 current non-users of tobacco. In each of these tobacco user groups, we seek to enroll 50% young adults (aged 18-24) and 50% youth (aged 13-17). See Exhibit 1 for the sample breakdown.

Exhibit 1. Projected Sample Breakdown by Age and Tobacco Use Status

	Sample Size
Population	Goal
Young Adults (aged 18-24)
Current cigarette users	≈625
Current e‑cigarette users	≈625
Current dual users	≈625
Not current tobacco users	≈625
	≈2,500
Youth (aged 13-17)
Current cigarette users	≈625
Current e‑cigarette users	≈625
Current dual users	≈625
Not current tobacco users	≈625
	≈2,500
Study Total	5,000

Respondents to be included in the study will be members of the Lucid Marketplace online survey panel, or children of current panel members. This panel consists of a sample of over 17 million adults nationwide and over 1 million youth. Adult participants have volunteered to be on the panel and be recruited to participate in surveys. Children are recruited through their adult parents, who are panel participants. All panelists must complete a double opt-in procedure, and parents of youth participants must consent for their child to be on the panel. Adult panelists are recruited through a variety of methods, including referrals, customer databases, and word-of-mouth. The convenience sample provided by the panel is considered suitable for the purposes of this study. This study is based on predetermined quotas. Individuals who are not members of the panel at the time of survey administration will not be invited to participate in this study. Only those who are invited to the study will have the means to access it, so it is not possible for the non-panel members or uninvited panel members to participate.

Invited panel members who decline to participate in the study during the assent/consent process will be excluded from the study. 5,000 respondents who assent/consent will be screened into the study using a series of survey items to determine eligibility based on age and tobacco use. Eligibility assessment will consist of 3 to 8 screener items, depending on participants’ responses. These items will determine whether individuals are within the age range of the study; whether they are current cigarette, e-cigarette, or dual users, or current non-users of tobacco. The screener includes the minimum number of items needed to assess study eligibility and develop the sample needed to achieve study goals. Respondents who choose not to complete a screener item will be ineligible for the study and will be excluded at the point of non-response (rather than after answering all screener items). Furthermore, individuals will be excluded from the study if we have already met the desired sample size for their tobacco use category and age group. The final analytic sample will include only eligible respondents who, after assenting/consenting to participate and completing the screener items, complete all critical elements of the survey.

Study Design and Power Calculations

Each respondent will view five e-cigarette ads. Each ad they see will be of a different type, varying based on the presence or absence of images, name modifiers (e.g., Mango Crush or Glacier Mint), and descriptors: As shown in Exhibit 2, the five Ad Types are: (Ad Type 1) Ad with no image, name modifier, or descriptor; (Ad Type 2) Ad with name modifier and descriptor present (image removed); (Ad Type 3) Ad with name modifier and image present (descriptor removed); (Ad Type 4) Ad with image and descriptor present (name modifier removed); (Ad Type 5) Ad with all 3 features present (no features removed).

For each Ad Type, respondents will be randomized to one of 18 conditions outlined in Exhibit 2. The conditions vary by brand (3 e-cigarette brands: Brand 1, Brand 2, and Brand 3) and flavors (6 flavors: tobacco, mint, menthol, fruit, non-fruit sweet, and unflavored). The stimuli are provided as an attachment.

Respondents will be randomized to five ads, one for each Ad Type. Within each Ad Type, the ad brand and flavor will be randomized. Participants will not view more than 1 ad for each flavor and 2 ads for a single brand.

Thus, the study design includes 90 conditions: 5 Ad Types (within subjects) x 3 Brands (between subjects) x 6 Flavors (between subjects). Each condition will yield data from approximately 138-139 respondents.

Exhibit 2. Conditions and Sample Size for Each Condition (Note: N’s in each cell reflect one age group (i.e., one half of the sample) only. The cell sizes for the other half of the sample are identical.)
		Ad Type
Flavor	Brand	1. No image, name modifier or descriptor (Control)	2. Name modifier + Descriptor (Image removed)	3. Name modifier + Image (Descriptor removed)	4. Image + Descriptor (Name modifier removed)	5. All 3: Image, name and descriptor (No features removed)
Tobacco	Brand1	~138/139	~138/139	~138/139	~138/139	~138/139
	Brand2	~138/139	~138/139	~138/139	~138/139	~138/139
	Brand3	~138/139	~138/139	~138/139	~138/139	~138/139
Mint	Brand1	~138/139	~138/139	~138/139	~138/139	~138/139
	Brand2	~138/139	~138/139	~138/139	~138/139	~138/139
	Brand3	~138/139	~138/139	~138/139	~138/139	~138/139
Menthol	Brand1	~138/139	~138/139	~138/139	~138/139	~138/139
	Brand2	~138/139	~138/139	~138/139	~138/139	~138/139
	Brand3	~138/139	~138/139	~138/139	~138/139	~138/139
Fruit	Brand1	~138/139	~138/139	~138/139	~138/139	~138/139
	Brand2	~138/139	~138/139	~138/139	~138/139	~138/139
	Brand3	~138/139	~138/139	~138/139	~138/139	~138/139
Non-Fruit Sweet	Brand1	~138/139	~138/139	~138/139	~138/139	~138/139
	Brand2	~138/139	~138/139	~138/139	~138/139	~138/139
	Brand3	~138/139	~138/139	~138/139	~138/139	~138/139
Unflavored	Brand1	~138/139	~138/139	~138/139	~138/139	~138/139
	Brand2	~138/139	~138/139	~138/139	~138/139	~138/139
	Brand3	~138/139	~138/139	~138/139	~138/139	~138/139
Total		2,500	2,500	2,500	2,500	2,500

We have investigated the power for the assumed sample sizes for a range of effect sizes. These power analyses suggest that we have sufficient power to detect differences in outcomes for comparisons across the main categories of this study (for example, the difference between Ad Type 2 and Ad Type 1). Alpha was set at 0.05 and desired power was set to 0.80. We estimated power for planned comparisons between two conditions using a corrected alpha of 0.0083 (Bonferroni correction for six hypothesis tests) and Stata’s ‘power two means’ command. These analyses indicate that a mean difference equal to 0.115 and sample size of 2500 per group will achieve a power 80%. For a larger mean difference of 0.2 (e.g., Mean₁=1.5 and Mean₂=1.7), the power is 95.2%. However, for the main comparison between Ad Types, we would still have adequate power to detect a relatively small mean difference. Supplemental analyses for pairwise comparisons to compare ads within each brand are powered at 80% to detect a mean difference of 0.5.

As with any study conducted using opt-in online panels, this study may be subject to several threats to external validity that limit the generalizability of study results. Panelists are recruited into the online panel using convenience sampling methods and thus do not have a known probability of selection into the panel. When inviting panelists to participate in the study, the vendor (SSRS) will seek diversity in age, education, and race/ethnicity to ensure a reasonable degree of inclusiveness on key demographic characteristics. JHU will not generate nationally representative results of the targeted sub-population in this study. Note that generating a representative sample for this study would be costly and is not required for the study’s aims. The study will use convenience samples rather than probability samples, and despite the diversity in the sample, the sample in the study is nevertheless still a convenience sample and not representative of national estimates or necessarily of the underlying study panel used. These limitations in generalizability do not affect the internal validity of the study. Such limitations will be noted in the context of describing the results of the study.

2. Procedures for the Collection of Information

Recruitment, Screening and Consent

Sampling, recruitment, and data collection will be conducted by SSRS. SSRS will recruit study participants through an invitation sent to panelists. The text of this invitation is shown in Attachment 2. Panel members will take the survey in their own homes or other locations that they choose. In the invitation, respondents are presented with a blind link to the survey and presented with the estimated length of survey and the proposed point reward. This helps to ensure screening questions are not given away with an initial description.

Invitations will be targeted towards panelists who, according to the database, have characteristics that match our study criteria (e.g., by age). Adult panelists may be contacted in several ways: Survey invitations are distributed via email, text message, an app and also presented in each panelist’s portal. Youth will be recruited through two methods. First, youth who are current panel members will be invited. Youth panelists became members through their parents—adult panelists can allow their children to become panel members. Second, youth will be recruited through their parent panelists (i.e., parents who are currently members of the existing online panel). The invitation will be sent to parent panelists inviting them to have their child aged 13-17 participate in the study. Parents who are interested in having their child participate can then have their child click on the link in the invitation.

After clicking the online survey link, participants will encounter the assent or consent form. If they assent/consent – by clicking a box indicating that they have read the provided material and agree to participate – they will then complete the study screener. Participants who screen into the survey will view one of the experimental images (see Exhibit 2 above, and Appendix 1) and complete the remaining survey items.

3. Methods to Maximize Response Rates

The survey will use an existing Internet panel to draw a sample. The panel (described in B.1) comprises individuals who share their opinions via the Internet regularly. To help ensure that the participation rate is as high as possible, FDA and JHU:

• have designed a protocol that minimizes burden (short in length, clearly written, and with appealing graphics); and

• will administer the survey over the Internet, allowing respondents to answer questions at a time and location of their choosing.

Furthermore, SSRS will target recruitment to panelists who are likely eligible for the study based on key characteristics including age and, for young adults, tobacco status.

The aim of these methods is to minimize the number of people who spend time on viewing the invitation or completing the screening but are ultimately ineligible to participate in the study based on the inclusion criteria.

4. Test of Procedures or Methods to be Undertaken

SSRS will conduct rigorous internal testing of the online survey instrument prior to data collection. The research team will review the online test version of the instrument that we will use to verify that instrument skip patterns are functioning properly, experimental stimuli exposure is working properly, and that all survey questions are worded correctly and are in accordance with the instrument approved by OMB.

5. Individuals Consulted on Statistical Aspects and Individuals Collection and/or Analyzing Data

The grantee (JHU) will work with SSRS to collect the data and will then analyze the data as a task order under the cooperative agreement No. 3U01FD005942-03S1. Meghan Moran, PhD, phone number 410-614-6872, is the PI for this project. Data analysis will be overseen by the Research Team, including JHU project lead Meghan Moran and FDA project lead Lexie Perreras (301-796-4985).