B. Statistical Methods
The primary outcome study will consist of a baseline survey and follow-up cross-sectional surveys with an embedded longitudinal cohort in campaign and control cities beginning approximately 6 months after campaign launch. The outcome evaluation data collection will occur in 12 campaign-targeted cities and 12 similar (“comparison”) cities. The 12 campaign and 12 comparison cities were randomly selected from a list of 24 potential campaign cities identified by FDA and their media contractor. We drew a stratified random sample of 12 cities to serve as comparison cities, with the remaining 12 cities selected as campaign markets. All 24 markets were initially grouped into regions. The primary method for this data collection involves conducting intercept interviews at LGBT social venues (e.g., bars, nightclubs, Pride festivals, community centers). Eligible respondents will be young adults who are 18 to 24 years old and who self-identify as LGBT. The sample will include young adults who self-identify as LGBT, as well as young adults who self-identify as being queer, transsexual, gender variant or pansexual (also referred to as omnisexual or trisexual). We will screen potential respondents at these venues. Eligible participants will be invited to complete the outcome survey online (via the participant’s personal computer or mobile device). We will complement this strategy by recruiting LGBT young adults via social media from the same cities. Each wave of the six waves of data collection will consist of a sample size of 3,150 18-24 year olds, with half of the sample (N=1,575) from 12 campaign-targeted cities and half (N=1,575) from comparison cities. Our goal is to recruit 75% of the sample via intercept interviews and 25% via social media at baseline. Baseline respondents will be invited to complete follow-up surveys at 6-month intervals. We are assuming a 50% retention rate at each wave and will recruit additional participants at each follow-up wave to keep the sample size constant at 3,150 18-24 year-olds. Once again, 75% of the newly recruited participants will come from intercept interviews and 25% from social media.
To ensure adequate representation of different subgroups of the young adult LGBT community (e.g., lesbian/gay females, gay males, bisexual females, bisexual males, transgender young adults), social media advertisements will be run in three simultaneous campaigns using ad targeting tools to reach three subgroups of LGBT young adults: gay and bisexual male young adults (i.e., men who are interested in men or men and women), lesbian/gay and bisexual female young adults (i.e., women who are interested in women or women and men) and transgender or genderqueer young adults (identified via keyword targeting using keywords relevant to transgender/genderqueer young adults). Quotas for LGBT subgroups will be programmed in the survey instrument to ensure that adequate numbers of participants in LGBT subgroups are recruited via social media. Once social media quotas are met for each LGBT subgroup, additional members of this subgroup will not be recruited to complete the survey.
For the intercept surveys we will seek input from local LGBT Outreach Coordinators at Health Departments, and Rescue Social Change Group, and conduct online searches to identify LGBT bars, nightclubs, community centers, Pride festivals, and other relevant LGBT-oriented venues and events. We will select a convenience sample of venues and particular dates and times within each sampled bar to maximize likelihood of intercepting the targeted number of participants per city within a brief field period within each city. As needed to establish contact with venue owners, recruitment calls to venue owners and managers will be supplemented by in-person contacts by local field staff. As agreed upon with the venue owners, a small team of interviewers will visit the venue at an established time and will approach patrons who appear to be in the target age range. Participants who agree to participate will be asked to fill out a short screening instrument (Attachment 3) on a tablet.
To supplement the field data collection strategy above, we will recruit a convenience sample of additional LGBT young adults through social media platforms, such as Twitter and Facebook. We will post advertisements to social media and invite young adults 18 to 24 to complete a brief online screener to determine their eligibility (i.e., LGBT, 18 to 24, and living in one of the 24 selected cities). Sample selection for respondents recruited from social media will involve posting advertisements on Facebook and Twitter for viewing by members more likely to be 18 to 24 years old and self-identify as a LGBT young adult, according to their account settings. The advertisements will be geographically targeted in these social media platforms to people living in the 24 selected cities. Eligible respondents will be directed to complete the outcome survey online until the required number of surveys is obtained.
Power Analysis
Statistical power estimates provide guidance on reasonable expectations for observing statistically significant change in outcomes of interest. This process requires an understanding of the study design, planned analyses (i.e., statistical model), expectations about the minimum detectable effect (MDE), as well as characteristics of the population and measures involved.
For the purpose of estimating statistical power for the [LGBT Campaign], we assume data collection will reflect a repeated cross-sectional design among 24 cities, with 12 [LGBT Campaign] cities and 12 cities that will serve as a comparison group. Although the planned data collection includes an embedded longitudinal design, for the purposes of the analyses and power calculations, we are treating each wave of data collection as an independent cross-section.
The proposed impact analysis accounts for the repeated cross-sectional data collection using a generalized linear hierarchical regression model that assesses change in the proportion of young adults that agree with a belief statement related to smoking tobacco (e.g., perceived approval, perceived prevalence, and perceived popularity). The test statistic will involve a two-tailed hypothesis test with a Type I error rate of 0.05 and a Type II error rate of 0.20, yielding 80% statistical power. Our parameter estimates include an intraclass correlation coefficient (ICC) of 0.01 to account for the geographic clustering of respondents and a variance inflation factor of 1.25 to account for potential imbalance across conditions. To some extent, these factors are offset by parameters that will serve to reduce variation. Those parameters include over-time correlation corrections of 0.55 at the cluster levels that account for repeated measures in the same cities as well as a 0.25 variance reduction at the individual level for the inclusion of demographic and socio-economic covariates. These parameter estimates are available in the published literature and support by our experience conducting similar studies (Murray & Short 1997; Murray & Blitstein 2003; Janega, Murray et al. 2004; Farrelly, Davis et al. 2005).
The campaign evaluation’s goal is to be able to identify change of 10 percentage points or greater as statistically significant. There is little available data in the peer-reviewed literature on the level of agreement we can anticipate at baseline. Accordingly, we rely on the conservative assumption that 50% of young adults will agree with campaign messages at baseline.
Given the parameters and assumptions detailed above, the impact evaluation of the [LGBT Campaign] will require data from 1,575 LGBT young adults in total in the 12 campaign cities and 1,575 LGBT young adults total in the 12 comparison cities (N = 3,150) at each wave of data collection. This sample size is predicated on the assumption that agreement with campaign messages is 50% at baseline and increases to 60% at the time of follow up data collection. If actual agreement at baseline is either higher or lower than this value, statistical power is improved and smaller program impacts can be detected with the same sample of respondents. This effect would result in an odds ratio of approximately 1.50, meaning that young adults exposed to the campaign would be 1.50 times more likely than young adults not exposed to the campaign to agree with campaign messages about the effects of smoking tobacco.
This section describes the procedures for in person data collection. Data will be collected in 6 waves beginning in late 2015, prior to the launch of the campaign, and ending 31 months post campaign launch.
RTI staff will attempt to recruit venues for data collection following the Venue Recruitment Guide included as Attachment 7. As needed to establish contact with bar owners, these recruitment calls will be supplemented by in-person contacts by local field staff using the same Recruitment Guide. Calls will be made (and in-person visits, when possible) to the list of bars identified with input from local LGBT Outreach Coordinators at Health Departments, Rescue Social Change Group and online searches. Venue owners will be first be given brief background information about data collection procedures. They will then be asked questions about when and how often LGBT young adults visit their establishments and be provided with information about participant cash incentives (as a benefit of allowing for recruitment in venues). RTI staff will then confirm recruitment times with venue owners who agree to allow recruitment in their establishment.
As agreed upon with bar owners, a small team of interviewers will visit the bar at the established time and will approach patrons who appear to be in the target age range using the talking points and FAQs shown in Attachment 8. RTI staff will introduce themselves and provide the following information to potential participants: incentive amount for completing screener survey ($10 cash), time needed to complete survey (5 minutes), and incentive amount for emailed web survey ($20-25 depending on how quickly they respond). In addition, RTI staff will provide the following information based on questions asked by potential participants: study purpose, reasons for participating, study sponsor information, background information on RTI and FDA CTP, information on privacy and confidentiality, and IRB contact information.
Once a participant agrees to complete the screener, the interviewer will open the case, select the appropriate city and bar, advance the screener to the informed consent screen (Attachment 4) and then pass the tablet to the respondent to complete. Within the screener, respondents will be asked about their zip code, age, sexual identity and sexual orientation (all necessary to establish eligibility including LGBT status, being within the appropriate age range, and living in the recruitment city). Respondents will also be asked two questions about their tobacco use; these tobacco questions are included to allow comparison of eligible respondents who do and do not later participate in the full survey to facilitate nonresponse bias analysis.
Completed screeners will be automatically analyzed within the survey software to determine the young adult’s eligibility. Respondents who screen as eligible will be asked for their email address, cell phone number, and first name so we can confirm we haven’t previously enrolled this person and so we can send an invitation for the full web survey to those who are eligible and new to the study. GPS data will be associated with each screened case to assist in detecting possible falsification by interviewers. Completed screeners that appear to be completed somewhere other than a scheduled venue will be investigated further. Respondents will be paid $10 in cash and will be asked to initial an incentive receipt (Attachment 9) at the end of the screener. Respondents who screen as eligible will be provided with an information card (Attachment 10) advising them that they will receive an email or text message invitation for the full web survey within the next two days.
Intercept respondents will be paid $10 in cash for completing the screener and will be asked to initial an incentive receipt (Attachment 9) at the end of the screener. Those who are eligible will receive an email invitation to complete the full survey. If they choose to complete the full survey they will receive an online gift card of $20.00, with a $5 bonus (total of $25.00) if they complete the full survey within two days of invitation.
Eligible participants who are recruited via social media (procedure described in Section B.2.2 below) will complete the screener integrated with the web survey and will be compensated $20 via an online gift card. Ineligible participants will not be compensated for completing the short screener. We will only be compensating screener participants who participate in LGBT recruitment venues because we need to provide an additional incentive for these participants to complete the survey using the link that they receive via email at a later time. Participants who screen in as eligible via social media ads will proceed immediately to the survey and thus will not need an additional incentive to complete the survey.
All interview data recorded on tablets will be transmitted at least daily via secure encrypted data transmission to RTI’s offices, where the data will be subsequently processed and prepared for analysis, reporting, and data file delivery. Upon transmission to RTI, all survey item data will be automatically wiped from all data collection devices used in the field.
At follow-up, young adults who participated in prior waves of data collection will be re-contacted using a follow-up email (Attachment 11). All other data collection procedures relating to the embedded longitudinal cohort will be the same as those described above.
Respondents may be re-contacted via email to address errors in data collection procedures, such as paying an incorrect incentive or erroneously deeming respondents as ineligible due to errors in data processing (Attachment 11).
To supplement this sample, young adult respondents will also be recruited in 24 U.S. cities (12 campaign and 12 comparison cities) through social media advertisements on Facebook and Twitter targeted at LGBT 18 to 24-year-olds, living in the same 24 U.S. cities.
RTI will place ads on social media platforms Twitter and Facebook. Several examples of these ads are included in Attachment 12. As much as possible, these ads will be targeted toward potentially eligible respondents, who are thought to be 18 to 24 years old, live in the data collection cities, and potentially may self-identify as LGBT. When clicked, the ads will direct the potential participant to a web-based screener instrument (Attachment 3). Respondents who are deemed eligible following completion of the screener will then go on to provide consent (Attachment 6) and complete either the baseline or follow-up instrument (Attachments 1 and 2), which will be administered online. All respondents who complete this survey will receive a virtual gift card valued at $20. Participants recruited via social media will not be paid to complete the screener.
The ability to obtain the cooperation of potential respondents in the baseline survey and maintain their participation across all survey waves will be important to the success of this study. In preparation for launching the baseline data collection, we will review procedures for enlisting respondent cooperation across a wide range of surveys, incorporate best practices from those surveys into the data collection procedures, and adapt the procedures through continuous improvement across the survey waves.
In addition to the $20 incentive (with $5 bonus for intercept participants who complete the survey within 2 days of receipt of the survey link) and $10 incentive for intercept participants to complete the screening survey, the study will use procedures designed to maximize respondent participation. The incentive procedures and amounts for new cross-sectional sample at follow-up waves will be identical to the baseline survey. For the longitudinal sample, participants will receive $20 for completing the follow-up survey and an additional $5 if they complete the survey within 48 hours of the invitation to participate. Prior intercept studies have demonstrated the importance of careful recruitment and training of field staff. Matching the characteristics of the interviewers to the study population has been found to be helpful (Spooner. et. al, 1997). To the extent possible, field interviewers will recruited from the LGBT community within each city. The ideal candidate will be familiar with the venues we will be using for data collection and knowledgeable about times in which 18 to 24 year olds are most likely to be present in high numbers. In addition, the ideal candidate will be reliable, articulate, outgoing, confident, and non-judgmental. Interviewer training will be participatory allowing for ample time to practice approaching respondents to introduce the survey, answer common questions, and overcoming objections. As noted in Section B.2.1, interviewers recruiting bar/nightclub intercept participants will use a document that includes talking points and FAQs (Attachment 8) to encourage participation.
When interviewers transmit their data from completed intercept screenings, the data will be summarized in daily reports posted to a web-based case management system accessed by field supervisors and RTI’s data collection managers. On a daily basis, supervisors will use these reports to review response rates and production levels. This information will allow supervisors to determine progress toward production goals and adjust goals for the remaining venues within each city. Supervisors will discuss information and challenges with their interviewers each week.
RTI will conduct rigorous internal testing of the online survey instrument prior to data collection. Evaluators will review the online test version of the instrument that we will use to verify that instrument skip patterns are functioning properly, delivery of campaign media materials is working properly, and that all survey questions are worded correctly and are in accordance with the instrument approved by OMB.
The following individuals inside the agency have been consulted on the design of the campaign evaluation plan, audience questionnaire development, or intra-agency coordination of information collection efforts:
Gem Benoza
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-0088
E-mail: Maria.Benoza@fda.hhs.gov
David Portnoy
Office of Science
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 301-796-9298
E-mail: David.Portnoy@fda.hhs.gov
Matthew Walker
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-3824
E-mail: Matthew.Walker@fda.hhs.gov
Leah Hoffman
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-743-1777
E-mail: Leah.Hoffman@fda.hhs.gov
Janine Delahanty
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-9705
E-mail: Janine.Delahanty@fda.hhs.gov
Ollie Ganz
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-5389
E-mail: Ollie.Ganz@fda.hhs.gov
The following individuals outside of the agency have been consulted on questionnaire development. Additionally, input has been solicited and received from FDA on the design of this study, including participation by FDA in meetings with OMB:
Matthew Farrelly
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-541-6852
E-mail: mcf@rti.org
Erik Crankshaw
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-316-3809
E-mail: ecrankshaw@rti.org
Jennifer Duke
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-485-2269
E-mail: jduke@rti.org
Jamie Guillory
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-316-3725
E-mail: jguillory@rti.org
Kristine Fahrney Wiant
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-485-5531
E-mail: fahrney@rti.org
Jane Allen
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-597-5115
E-mail: Janeallen@rti.org
Youn Lee
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-485-5536
E-mail: Younlee@rti.org
Amy Henes
RTI International
701 13th Street, NW
Washington, DC, 20005
Phone: 202-974-7821
E-mail: ahenes@rti.org
Patricia LeBaron
RTI International
230 W Monroe Suite 2100
Chicago, IL 60606
Phone: 312-777-5204
E-mail: plebaron@rti.org
Pamela Rao
Akira Technologies, Inc.
1747 Pennsylvania Ave NW Suite 600
Washington, DC 20002
Phone: (202) 517-7187
Email: prao@akira-tech.com
Xiaoquan Zhao
Department of Communication
George Mason University
Robinson Hall A, Room 307B
4400 University Drive, 3D6
Fairfax, VA 22030
Phone: 703-993-4008
E-mail: xzhao3@gmu.edu
Joseph Lee
East Carolina University
Belk Building
Greenville, NC
Phone:
E-mail: leejose14@ecu.edu
Jeff Jordan
Rescue Social Change Group
3436 Ray Street
San Diego, CA 92104
Phone: 619-231-7555 x 150
Email: jeff@rescueagency.com
Mayo Djakaria
Rescue Social Change Group
660 Pennsylvania Avenue SE, Suite 400
Washington, DC 20003
Phone: 619-231-7555 x 120
Email: mayo@rescueagency.com
Dana Wagner
Rescue Social Change Group
660 Pennsylvania Ave SE, Suite 400
Washington, DC 20003
Phone: 619-231-7555 x 331
Email: dana@rescueagency.com
Brandon Tate
Rescue Social Change Group
6463 Sunset Boulevard, Suite 504
Hollywood, CA 90028
Phone: 619-231-7555 x 152
Email: brandon@rescueagency.com
References
Farrelly, M.C., Davis, K.C., Haviland, M.L., Messeri, P., Healton, C.G. (2005). Evidence of a dose-response relationship between "truth" antismoking ads and youth smoking prevalence. American Journal of Public Health, 95(3), 425-431.
Janega, J.B., Murray, D.M., Varnell, S.P., Blitstein, J.L., Birnbaum, A.S., Lytle, L.A. (2004). Assessing the most powerful analysis method for schools intervention studies with alcohol, tobacco, and other drug outcomes. Addictive Behaviors, 29(3), 595-606.
Murray, D.M., Blitstein, J.L. (2003). Methods to reduce the impact of intraclass correlation in group- randomized trials. Evaluation Review, 27(1), 79-103.
Murray, D.M., Short, B.J. (1997). Intraclass correlation among measures related to tobacco- smoking by adolescents: Estimates, correlates, and applications in intervention studies. Addictive Behaviors, 22(1), 1-12.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Matthew Farrelly |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |