TRC Copy Testing Memo Part B

Supporting Statement: Part B

0910-0810

The Real Cost Campaign: Online Quantitative Study of Reactions to Rough-Cut Advertising Designed to Prevent Youth Tobacco Use

Statistical Methods

The one-time actual burden figures are listed in Exhibits 1 & 2, Part A.

1. Respondent Universe and Sampling Methods

The primary outcome of this study will be based on a non-random sample of up to 3,000 youth ages 13-17 years old who are either tobacco users or susceptible non-users, and up to 1,000 adults ages 19-54 who are dual users of cigarettes and ENDS and interested in quitting cigarettes. Participants will be randomly assigned to the control group, where they will not view any advertisements (ads), or to the ad-viewing group, where they will be asked to view a randomly assigned ad (see Attachment H: Stimuli) and provide quantitative and qualitative feedback about the ad. All enrolled participants will be asked to answer questions about their knowledge, attitudes, and beliefs about tobacco use as a check for potential unintended consequences of viewing the video ads.

The study is a cross-sectional design, and participants will be enrolled via panel. The screening criteria are based on age, tobacco use status, valid email address, personal or close family or friends’ employment in the tobacco industry, and past participation in tobacco research.

As this study is considered part of formative research for campaign development and planning, these methods are not intended to generate nationally representative samples or precise estimates of population parameters. The sample drawn here is designed primarily to provide information for the FDA’s The Real Cost Campaign. The study results for each advertisement will be assessed individually, as well as compared across related tested advertisements to address the following questions:

• Does the advertisement provide an understandable and engaging message about the harms of tobacco use?

• Does the advertisement have any potential adverse or unintended consequences related to beliefs around the harms of tobacco use?

In instances where there is concern that there may be unintended consequences for adults, stimuli will also be tested with adults. To this end, a subsample of the ads that message on ENDS will be tested on adults to assess if ENDS messages aimed at preventing use among youth have any unintended consequences among adult smokers interested in quitting smoking.

Sampling Methods

This study will recruit both youth and adult participants through online panels. The campaign contractors, FCB and Marketing Workshop, subcontract to the panel provider, Market Cube, which has a proven and demonstrated ability to orchestrate and support the sampling plan specifications of these studies. The panel includes over 600,000 adults age 19-54, over 25,000 teens 13-17 years old, as well as hundreds of thousands of households that include teens. The online panel provider may use social media platforms to recruit additional participants as needed to augment their existing panels. Advertising through social media platforms, can help increase the diversity of the study sample and increase representation of traditionally underrepresented groups (Lane, Armin, & Gordon, 2015; Graham et al., 2008). For many social media platforms, ad targeting can be adjusted in real time, allowing researchers to react to shifting recruitment needs if a demographic is lacking in the overall sample. Social media advertisements will be deployed by the online panel provider based on factors such as age, and geographic location.

Youth:

Upon screener completion, participants will be immediately notified if they qualify. All qualified youth participants will then be prompted to provide their parent/guardian’s email address. Parent/guardian email address will be used to email the parental opt-out form to a potential participant’s parent or guardian. Qualified youth whose parents’ email does not bounce back and whose parents do not opt them out will be emailed a link to complete the questionnaire no less than 24 hours after the parental opt-out form is emailed to their parent/guardian, to allow sufficient time for parental-opt out to occur. Participants will provide informed assent before they begin the questionnaire.

Qualified youth participants who complete the screener, whose parents’ email does not bounce back, and who are not opted-out by their parents/guardians will be contacted via email with a link to complete the questionnaire. Invited youth who do not complete the questionnaire within 24 hours of receiving the link will receive up to two reminders via email to complete the questionnaire.

The screener is designed to determine whether a respondent is qualified to complete the questionnaire. The screener is separated into two phases, with eligibility determined at the conclusion of Phase 1 in order to limit the amount of identifying information that will be collected from respondents who are not qualified to complete the questionnaire. The screener will include the following major components:

• Phase 1

  • Demographic information: age (for verification of 13-17 age range), race/ethnicity and gender (to ensure that sample consists of youth from a variety of demographic backgrounds as set forth by campaign goals),

  • Self-reported tobacco ever-use and lifetime use,

  • Battery of questions to determine openness to tobacco use (if applicable, for “never users”),

  • Background exclusion questions (e.g., recent participation in tobacco research/ tobacco industry affiliation),

  • Zip code (to ensure that participants are within determined geographic targets for the study),

  • Youth email address (to check against all current respondent data to avoid duplicates and reduce fraudulent activity), and

  • IP Address (collected automatically to reduce fraudulent activity, and verify participant country of origin).

• Phase 2 (qualified youth only):

  • Parent email (to email the Parental Opt-Out Form to parents of eligible youth; parental email address will not be used for any other purpose).

Sample Size

To obtain a final sample of up to 3,000 enrolled participants, we estimate that we will need to screen up to 4,500 potential respondents. This estimate is based on the capabilities of the panel provider and previous research conducted within this target audience using similar methodologies.

Adults:

Adults will be recruited via the panel. Reminders are sent to all invitees who have not responded 48 hours after initial invitation. If an invitee does not respond after the reminder is sent, they are considered not available and are not re-approached.

The Screener is expected to take no more than 10 minutes to complete and will collect the following information:

• Demographic information: age (for verification of 19-54 age range), race/ethnicity gender, income level, and education level (to ensure that sample consists of adults from a variety of demographic backgrounds as set forth by campaign goals),

• Self-reported cigarette and other tobacco use status

• Intention to quit smoking questions

• Background exclusion questions (e.g., recent participation in tobacco research/ tobacco industry affiliation),

• Zip code (to ensure that participants are within determined geographic targets for the study),

• Email address (to check against all current respondent data to avoid duplicates and reduce fraudulent activity), and

• IP Address (collected automatically to reduce fraudulent activity and verify participant country of origin).

Sample Size

To obtain a final sample of up to 1000 enrolled participants, we estimate that we will need to screen up to 1,500 potential respondents. This estimate is based on the capabilities of the panel provider and previous research conducted within this target audience using similar methodologies.

2. Procedures for the Collection of Information

Participants will complete both the screener and questionnaire online using their own device. Email addresses are collected on the screener, enabling researchers to contact participants by email with the link to the questionnaire (after the 24-hour parental opt-out waiting period in the case of youth participants), and with up to two reminders reminding them to complete the questionnaire.

Qualified participants will be randomly assigned to either an ad-viewing or non ad-viewing (control) group. Participants in the ad-viewing group will view one of rough cut ad for The Real Cost campaign. Ad-viewing participants will then be prompted to complete a series of questions designed to assess whether the advertisement provides an understandable and engaging message about the harms of tobacco use.

Advertisements shown during copy testing will on average be 30-45 seconds in length. Participants will have the option to replay an ad once it is complete.

All participants will answer a series of questions about household tobacco use, and participant tobacco use, to allow comparison between ad-viewing and control groups to assess the effectiveness of the randomization process and ensure there are no confounding differences between groups. All participants will also answer questions assessing their knowledge, attitudes, and beliefs related to tobacco use, which will be used to assess potential unintended consequences by comparing responses between ad-viewing and control groups. Participants who do not view any advertisements are being included to measure for unintended consequences. Exhibit 5 indicates the variables to be assessed during the questionnaire and the participant groups that will be exposed to these survey items.

Exhibit 5. Structure of the Copy Testing Process and Questionnaire
Action or Variable	Description	Presented to Ad-Viewing Participants	Presented to Control Participants
Introductory questions	Youth questionnaire: Household tobacco use, peer cigarette use, participant lifetime and past 30-day tobacco use, and participant sensation-seeking. Adult questionnaire: Household tobacco use, tobacco use, motivation to quit	X	X
Ad exposure	Each of the ad-viewing participants will view one unique video ad.	X
Ad performance	Immediately following exposure to the video ad, ad-viewing participants will be presented with items to assess ad perceived effectiveness and emotional/attitudinal reactions to the ad. Adult questionnaire will also include questions related to perceptions of who this ad is targeted to.	X
Tobacco-related knowledge, attitudes, and beliefs	Youth questionnaire: Items assessing participants’ knowledge, attitudes, and beliefs related to tobacco use. Adult questionnaire: Questions related to their perceptions of the benefits of quitting smoking, intentions to quit smoking, products they would consider using to quit smoking, and perceptions of tobacco products.	X	X
Attention check	Attention-check question to assist in identifying participants not giving careful consideration to responses.	X	X

2. Methods to Maximize Response Rates and Deal with Nonresponse

Several features of this study have been designed to maximize participant response rate and questionnaire completion.

• Reminders: A series of reminders will be utilized to minimize drop off. Qualified participants who complete the screener but do not complete the questionnaire within 24 hours of receiving the link will receive up to two reminders via email and text message (if the option was selected). These reminder emails will include a unique link to the survey to enable participants to easily complete the questionnaire. These reminders are intended to decrease non-response by ensuring participants have convenient access to complete the questionnaire, and by reminding participants who do not initially complete the questionnaire to complete it before the conclusion of data collection.

• Parental Opt-Out: For youth participants, a parental opt-out approach will be utilized. Due to the target population of this study, traditional active parental permission procedures would discourage participation among the very participants most appropriate for the aims of this study. Many youth who use tobacco or are at risk for using tobacco are unlikely to seek out parental permission or have parents who provide active permission for their children’s participation in prevention programs (Levine, 1995; Pokorny et al., 2001; Unger et al., 2004; Severson and Ary, 1983). Demonstrating this point, there is consistent evidence of quantifiable differences in the characteristics of youth who participate in smoking cessation research when traditional written permission is required compared to waived parental permission, including participant demographics and smoking history (Kearney et al., 1983; Anderman et al., 1995; Severson & Ary, 1983). Utilizing a parental opt-out approach will remove a barrier that might discourage the target audience from completing the questionnaire, thereby reducing non-response.

• Mobile Phone Responsiveness: Both the screener and questionnaire will be optimized for performance on a mobile phone, in addition to other electronic devices such as tablets and laptops. This is especially important as 89% of U.S. teens have or have access to a smartphone and 97% access the internet using mobile devices (Lenhart et al., 2017). Based on this information, we expect that many youth will attempt to complete the screener and questionnaire on a mobile phone. Ensuring that the surveys are optimized for mobile phone performance will reduce non-response and drop-off due to technical issues related to compatibility of the instruments with the mobile phone format.

• Token of Appreciation: In this study, we will use a token of appreciation in the form of points equivalent to $10 total per participant to equalize the burden placed on participants with respect to their time, and to reduce overall burden by increasing questionnaire completion rates among participants who qualify via the screener. As participants often have competing demands for their time, a token of appreciation for participation in research is warranted. The use of a token of appreciation treats participants justly and with respect by recognizing and acknowledging the effort participants expend to participate. Numerous empirical studies have also shown that a token of appreciation can significantly increase response rates in cross-sectional studies and reduce attrition in longitudinal studies (e.g., Abreu & Winters, 1999; Castiglioni, Pforr, & Krieger, 2008; Shettle & Mooney, 1999; Singer, 2002).

4. Test of Procedures or Methods to be Undertaken

The campaign contractors, FCB and Marketing Workshop, have conducted rigorous internal review of the survey instruments. Trained researchers will review the screener and questionnaire to verify that instrument skip patterns are functioning properly, delivery of campaign media materials is working properly, and that all survey questions are worded correctly and are in accordance with the instrument approved by OMB.

4. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following individuals inside the agency have been consulted on the design of the copy testing plan, survey development, or intra-agency coordination of information collection efforts:

Gem Benoza

Office of Health Communication & Education

Center for Tobacco Products

Food and Drug Administration

10903 New Hampshire Avenue

Silver Spring, MD 20993

Phone: 240-402-0088

E-mail: Maria.Benoza@fda.hhs.gov

Mario Navarro

Office of Health Communication & Education

Center for Tobacco Products

Food and Drug Administration

10903 New Hampshire Avenue

Silver Spring, MD 20993

Phone: 240-402-4963

E-mail: Mario.Navarro@fda.hhs.gov

Matthew Walker

Office of Health Communication & Education

Center for Tobacco Products

Food and Drug Administration

10903 New Hampshire Avenue

Silver Spring, MD 20993

Phone: 240-402-3824

E-mail: Matthew.Walker@fda.hhs.gov

The following individuals outside of the agency have been consulted on questionnaire development.

Todd Sussman - FCB New York

SVP, Group Strategy Director

100 West 33rd Street, New York, NY 10001

Phone: 917 685 6580

Email: Todd.sussman@fcb.com

Dimas Adiwiyoto-FCB New York

VP, Account Directot

100 West 33rd Street, New York, NY 10001

Phone: 212 885 3283

Email: Dimas Adiwiyoto@fcb.com

Xiaoquan Zhao

Department of Communication

George Mason University

Robinson Hall A, Room 307B

4400 University Drive, 3D6

Fairfax, VA 22030

Phone: 703-993-4008

E-mail: xzhao3@gmu.edu

References

Anderman, C., Cheadle, A., Curry, S., Diehr, P., Shultz, L., & Wagner, E. (1995). Selection bias related to parental consent in schoolbased survey research. Evaluation Review, 19, 663–674.

CASRO. (2013). CASRO Code of Standards and Ethics, Available at: http://www.casro.org/?page=TheCASROCode. Accessed on: 06/07/2013.

Centers for Disease Control and Prevention (CDC). (2010). Tobacco use among middle and high school students - United States, 2000-2009. Morbidity and Mortality Weekly Report, 59(33), 1063-1068.

Graham, A. L., Milner, P., Saul, J. E., & Pfaff, L. (2008). Online advertising as a public health and recruitment tool: comparison of different media campaigns to increase demand for smoking cessation interventions. Journal of Medical Internet Research, 10(5), e50.

Jamal, A., Gentzke, A., Hu, S.S., Cullen, K.A., Apelberg, B.J., Homa, D.M., & King, B.A. (2017). Tobacco Use Among Middle and High School Students – United States, 2011-2016. Morbidity and Mortality Weekly Report, 66(23), 597-603.

Kearney, K.A., Hopkins, R.H., Mauss, A.L., & Weisheit, R.A. (1983). Sample bias resulting from a requirement for written parental consent. Public Opinion Quarterly, 47, 96102.

Lane, T.S., Armin, J., & Gordon, J. S. (2015). Online recruitment methods for web-based and mobile health studies: A review of the literature. Journal of Medical Internet Research, 17(7), e183.

Lenhart, A., Kantor, L., Malato, D., Benz, J., Tompson, T., Zeng, W., & Swanson, E. (2017). Instagram and Snapchat are Most Popular Social Networks for Teens; Black Teens are Most Active on Social Media, Messaging Apps. Chicago, IL: The Associated Press-NORC Center for Public Affairs Research.

Levine, R.J. (1995) Adolescents as research subjects without permission of their parents or guardians: Ethical considerations. Journal of Adolescent Health, 17(5), 289-297.

Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437-455.

Racial and Ethnic Approaches to Community Health Across the U.S. (REACH U.S.) Evaluation. OMB CONTROL NUMBER: 0920-0805 Report on Incentive Experiments www.reginfo.gov/public/do/DownloadDocument?objectID=30165501. Accessed on February 1, 2016.

Pokorny, S. B., Jason, L. A., Schoeny, M. E., Townsend, S. M., & Curie, C. J. (2001). Do participation rates change when active consent procedures replace passive consent. Evaluation Review, 25(5), 567-580.

Severson, H.H., & Ary, D.V. (1983). Sampling bias due to consent procedures with adolescents. Addictive Behaviors, 8(4), 433–437.

Stevenson, J., Dykema, J., Kniss, C., Black, P., & Moberg, D. P. (2011). Effects of mode and incentives on response rates, costs and response quality in a mixed mode survey of alcohol use among young adults. In annual meeting of the American Association for Public Opinion Research, May, Phoenix, AZ.

Unger, J.B., Gallaher, P., Palmer, P.H., Baezconde-Garbanati, L., Trinidad, T.R., Cen, S., & Johnson, C.A. (2004). No news is bad news: Characteristics of adolescents who provide neither parental consent nor refusal for participation in school-based survey research. Evaluation Review, 28(1), 52-63.

4