“Talk. They Hear You.” Campaign Evaluation: Case Studies
Supporting Statement
The Substance Abuse and Mental Health Services Administration (SAMHSA) Center for Substance Abuse Prevention is requesting approval from the Office of Management and Budget (OMB) for new data collection for the “Talk. They Hear You.” Campaign Evaluation: Case Studies (the “case studies”). This collection includes three instruments:
Parent/Caregiver Pre-Test/Post-Test Survey
Youth Pre-Test and Post-Test Survey
Parent/Caregiver Interview Guide
The case studies collection is part of a larger effort to evaluate the impact of the “Talk. They Hear You.” Campaign. The overall evaluation includes a national survey that will be submitted to OMB for approval under a separate request.
Despite laws prohibiting the consumption of alcohol by individuals under 21 years of age in all 50 states, underage drinking in the United States remains a pervasive threat to public health and safety. Alcohol is the most widely used substance of abuse among youth in the United States,1,2 with significant adverse consequences. Approximately 10 percent of youth aged 9-10 report having consumed more than a few sips of alcohol,3 and by age 15, 50% of youth report having consumed an entire alcoholic beverage.4 Adolescent drinkers are more likely than adults to engage in binge drinking, and are at increased risk of physical and sexual assault, academic problems, drug use and physical illness or death due to alcohol poisoning.5 Additionally, consumption of alcohol during a time of critical brain development may increase alcohol consumption and dependence in adulthood.6
Parental attitudes toward drinking, as well as parental communication, have a significant influence on adolescent alcohol use, particularly among younger adolescents.7,8 Through a direct association with adolescent perceptions and cognition, parental beliefs and communication about the acceptability of underage drinking may positively impact use of alcohol in youth.9 This research suggests that parental interaction with youth regarding underage drinking may provide a unique opportunity for early intervention and prevention.
In 2006, Congress passed the Sober Truth on Preventing Underage Drinking Act (STOP Act), the first comprehensive national legislation addressing the problem of underage drinking. The act mandates a coordinated, multifaceted approach to addressing the prevention and reduction of underage drinking through the modification of enforcement of drinking laws, reduction of alcohol availability to adolescents, increased research and resources for state and local efforts, and an adult-oriented media campaign. Under section 2(d), the STOP Act directs the Secretary of the U.S. Department of Health and Human Services to fund and oversee a national, adult-oriented media public service campaign, and to report annually on the production, broadcasting, and evaluation of this campaign.
In response to the directives set forth in the STOP Act of 2006, SAMHSA’s Center for Substance Abuse Prevention (CSAP) created “Talk. They Hear You.” the Underage Drinking Prevention National Media Campaign. The Campaign is designed to educate and empower parents and caregivers to talk with children about alcohol. To prevent initiation of underage drinking, the campaign targets parents and caregivers of children aged 9-15, with the specific aims of:
Increasing parents’ awareness of the prevalence and risk of underage drinking
Equipping parents with the knowledge, skills, and confidence to prevent underage drinking
Increasing parents’ actions to prevent underage drinking.
The “Talk. They Hear You.” campaign adds to parents’ current knowledge base about underage drinking prevention and empowers parents to address the issue by increasing their level of comfort with the topic and encouraging open communication between them and their children. During campaign development, parents, youth, and stakeholders provided feedback on all aspects of concept and message development. In 2012, SAMHSA conducted a national pilot program to test and refine campaign creative materials and objectives. Then, in 2013, SAMHSA launched the “Talk. They Hear You.” campaign designed to help parents talk to children ages 9–15 about the consequences of underage drinking and introduce skills to help their children avoid situations that result in drinking. This is a national campaign with an ongoing rollout period.
The “Talk. They Hear You.” materials consist of public service announcements, infographics, and a mobile application. These materials show parents using everyday opportunities to talk with their children about alcohol, and reinforce the importance of starting the conversation about alcohol at an early age. They were created and provided to partners to display and distribute to parents and community members. The partners, in turn, generated feedback on campaign successes, such as reach and volume of social media and blog posts, newsletter articles, and direct distribution of campaign materials.
In addition to developing and disseminating the “Talk. They Hear You.” campaign components, SAMHSA is also charged with evaluating the campaign. While message testing during development indicated the potential impact of the materials, and media measures from partner distribution of materials indicate the reach of the campaign, there also needs to be an evaluation of the impact of the campaign as it has been implemented in the general population. SAMHSA’s evaluation design consists of a two-part evaluation mechanism—a large-scale, annual national survey and the supplemental case studies. The large-scale, national survey of parents/caregivers of 9-15 year old children will examine campaign reach and exposure and evaluate whether parents exposed to campaign messaging and materials report changes in attitudes and/or behavior—regarding talking to their children about underage drinking—after seeing the materials. The application package for the national survey will be submitted separately to OMB.
To supplement findings from the national survey, SAMHSA intends to conduct forced-exposure case studies in selected school sites in areas that have currently had limited campaign outreach. These case studies are the current focus of this OMB application package. Because we intend to measure knowledge and attitudes before and after a focused campaign outreach effort in areas that have not had significant exposure to the campaign, this component of the evaluation needs to be started before the national survey.
The unit of analysis for the case studies will be a middle school community, which will involve students; parents; school administrators; and partnering organization(s), such as parent organizations and/or local prevention organizations. There will be two sites selected for the case studies—one site will serve as the experimental group and the other site will serve as the control group. The experimental group will be exposed to the “Talk. They Hear You.” campaign materials using standard campaign materials and dissemination strategies, which will be coordinated through a local partner organization. The control group will not be intentionally exposed to the campaign materials. The case studies will include baseline surveys of both parents/caregivers and children aged 9-15, followed by exposure to campaign materials, and post-exposure surveys of both parents and children. SAMHSA will additionally conduct interviews with parents and caregivers following the post-exposure surveys to obtain more detailed information about the impact of the campaign.
In total there will be three data collection instruments used in the case studies:
Parent/Caregiver Pre-Test/Post-Test Survey – Pre- and post-test survey for use with parents and caregivers of children who are 9-15 years of age. This survey will include questions about respondents’ attitudes and behaviors regarding underage drinking and awareness of the “Talk. They Hear You.” campaign. This survey will be administered electronically and on paper.
Youth Pre-Test and Post-Test Survey – Pre- and post-test survey for use with youth who are in 6th through 8th grade. This survey will include questions about respondent’s attitudes and behaviors regarding underage drinking. This survey will be administered on paper.
Parent/Caregiver Interview Guide – Interview guide for use with parents and caregivers of children who are 9-15 years of age. Parents and caregivers who respond to the post-test survey, described above, will be invited to participate in an individual follow-up interview to discuss, in greater detail, the effect—if any—of the “Talk. They Hear You.” campaign on their attitudes and behaviors regarding underage drinking. The individual interviews will be conducted over the phone.
The purpose of the case studies component of the campaign evaluation is twofold:
To identify evidence regarding the relationship between campaign exposure and changes in parents’ attitudes and behavior regarding underage drinking in the selected sites;
To collect information on whether changes in parent/caregiver attitudes and/or behaviors identified in align with measurable changes in youth attitudes and/or behaviors on underage drinking during the same period.
Whereas national survey data will provide an evaluation of overall campaign information exposure and retention across the United States, these case studies will explore details of how exposure to the “Talk. They Hear You.” campaign impacts adult and student attitudes and behaviors in the selected districts. Together, the results from both components of the evaluation—the national survey and the case studies—will allow an estimation of overall campaign impact in reaching parents/caregivers of 9-15 year old youth, and ultimately of reducing underage drinking. This campaign evaluation will additionally satisfy section 2(d) of the STOP Act, which requires the national, adult-oriented media public service campaign to be evaluated.
The results of the campaign evaluation—as well as progress on implementation of the evaluation—will be reported in an annual report to Congress. Section 2(d) of the STOP Act requires Secretary of the U.S. Department of Health and Human Services to annually report on the production, broadcasting, and evaluation of the “Talk. They Hear You.” Since the STOP Act went into effect, SAMHSA has reported annually on the progress of the campaign and campaign evaluation in fulfillment of this act. SAMHSA will continue to document the progress of the campaign evaluation—and the case study findings, in particular—in future reports to Congress in fulfillment of this law.
SAMHSA will disseminate one of the three case study instruments—the parent and caregiver pre- and post-test survey—electronically. The parent and caregiver pre-test and post-test survey will also be available on paper to accommodate the needs and preferences of the parent/caregiver populations at both the experimental site and the control site. The electronic surveys will be disseminated via email and the paper surveys will sent home with students by the case study middle school sites.
SAMHSA will use the Qualtrics© Survey Suite to develop and disseminate the electronic version of the survey. The use of the Qualtrics© Survey Suite will enable SAMHSA to use branching logic in the electronic survey that provides respondents with different questions based on their responses to previous questions (and effectively enables respondents to skip over the questions that do not apply to them). Such branching logic will reduce the average time of the survey for respondents, thereby reducing overall burden. The use of Qualtrics© also provides SAMHSA with the ability to randomize response options in order to mitigate response option bias. Qualtrics© also includes built-in 508 compliance systems to accommodate respondents with vision impairments and built-in mobile accessibility to accommodate respondents using a range of electronic devices. SAMHSA owns and controls all data collected using Qualtrics©, and the data maintained in the Qualtrics© database will be kept confidential. Qualtrics© uses industry best practices to maintain data security and privacy and have devised proprietary methods to prevent disclosing data to the wrong requester due to programming errors.
The STOP Act required the development and dissemination of a new, adult-oriented media public service campaign, and an evaluation of that campaign. The data collection efforts outlined in this document are specific to the campaign evaluation, and are not duplicated elsewhere.
This data collection will involve the participation of two school districts Involvement will be voluntary as it will align with their respective missions to prevent underage drinking. Further, the summary information from the data collected will be useful to the districts who will be involved in the data collection.
SAMHSA is seeking to conduct the proposed case studies only one time.
This information collection fully complies with 5 CFR 1320.5(d)(2).
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on November 28, 2016 (81 FR 85586). SAMHSA did not receive any comments.
SAMHSA did not consult with outside consultants regarding the case studies component of the “Talk. They Hear You.” campaign evaluation.
No payments will be made to respondents for completing the surveys.
Payments will be made to participants for completing the individual interviews, in the form of gift cards, at a rate of $35.00 per hour.
Both email and paper distributions will be controlled by the schools and no email or student contact information will be shared by the schools with SAMHSA. Only information provided voluntarily by the respondents will be collected by the system as outlined here.
Contact information from a sub-set of the adults participating in the survey is necessary to identify participants for the interview portion of the study. The parent/caregiver post-test survey includes a question prompting respondents to share contact information—first name and a phone number or email address—if they would be willing to be contacted for a follow-up interview. The survey requests this contact information with the following instructions:
Would you be willing to provide more information on your responses to this survey in a follow-up interview on the topic of underage drinking? Interviews will take no more than one hour and will be conducted by phone. For your participation, you will receive a $35 gift card from a local store If you are interested in learning more about the interview or scheduling a time to participate, please share your first name and either a phone number or email address below so we may contact you.
Your name and contact information will NOT BE IDENTIFIED in any reports that result from this survey.
The sole purpose of requesting this contact information is to identify subjects willing to participate in the interview component of the case studies; no names will appear in the Report to Congress or any other report.
Survey data will be stored in password-protected, encrypted files. Access to these files will be limited to the data analyst and supervisor. Upon completion of data collection through the parent/caregiver post-test survey and the follow-up interviews, any identifying information for participants will be purged from the data files.
Our procedures ensure student op-out from the study, should they desire. Before administering the student survey, the teacher from each participating classroom will read the following script: “Shortly, I will be passing out a brief survey. This survey is part of a larger initiative to address underage drinking. The questions are anonymous, so please do not put your name on this survey. Your participation is voluntary. Therefore, if for any reason, before, during, or upon survey completion, you may choose to withdraw from the study and your answers will not be included. If you have any questions or concerns about the survey or find that the survey topic triggers feelings or concerns, please let me know and I will connect you with appropriate supports.”
Questions of a sensitive nature for each of the data collection instruments are detailed in the table below:
Table 1: Questions of a Sensitive Nature
Table 2 indicates the estimated total annual burden on each participant for each data collection activity associated with the “Talk. They Hear You.” campaign evaluation case studies. Table 2 was calculated based on the following assumptions:
There are 575 students, on average, in U.S. middle schools10. There will be two middle school sites included in the case studies (one control site and one intervention site), so SAMHSA estimates approximately 1150 middle school students to be respondents for the pre- and post-test surveys for youth. Since the survey for youth will be administered during class, SAMHSA estimates a 95% response rate (approximately 1093 students). SAMHSA assumes that the pre-test survey and post-test survey will each take approximately 10 minutes to complete. This estimate includes time for distributing the survey, reviewing the instructions, taking the survey, and turning in the completed survey.
SAMHSA estimates one parent/caregiver per student as the base population for potential respondents for the parent/caregiver pre-test and post-test surveys—or 575 parents/caregivers per site. Across both sites—the control and intervention sites—SAMHSA estimates a total 1150 parents/caregivers total as the base population. As the parent/caregiver pre-test and post-test surveys will be administered by the school administration, SAMHSA estimates a response rate of approximately 60%, which is in line with average response rates found in a search for recent surveys administered to parents and caregivers by schools11,12,13 as well as a 2008 meta-analysis of response rates of individuals within organizations participating in research.14 SAMHSA assumes that the pre-test survey and post-test survey will each take approximately 10 minutes to complete. This estimate includes time for distributing the survey, reviewing the instructions, taking the survey, and turning in the completed survey.
SAMHSA aims to conduct hour-long individual interviews with approximately 30 parents and caregivers following the post-test survey. Participants in the interview will be compensated approximately 35.00 dollars per hour for their participation.
Table 2: Estimated Burden for Respondents
Instrument |
Total No. of respondents |
Total Responses/ respondent |
Total responses |
Hrs. per response |
Total hour burden |
Wage rate |
Total hour cost |
Pre-test survey for 6th -8th youth |
1,093 |
1 |
1,093 |
0.17 |
185.8 |
$4.25* |
$789.65 |
Post-test survey for 6th -8th grade youth |
1,093 |
1 |
1093 |
0.17 |
185.8 |
$4.25* |
$789.65 |
Pre-test survey for parents and caregivers |
690 |
1 |
690 |
0.17 |
117.3 |
$7.25** |
$850.42 |
Post-test survey for parents and caregivers |
690 |
1 |
690 |
0.17 |
117.3 |
$7.25** |
$850.42 |
Individual interviews with parents and caregivers |
30 |
1 |
30 |
1 |
30 |
$35 |
$1,050 |
Total |
1,783 |
|
3,596 |
|
636.2 |
|
$4330.15 |
*Federal Youth Minimum Wage Program (http://webapps.dol.gov/elaws/whd/flsa/docs/ymwplink.asp)
**Federal minimum wage per Fair Minimum Wage Act 2007 (http://webapps.dol.gov/elaws/faq/esa/flsa/001.htm)
There will be no capital, startup, operation, or maintenance of services costs to respondents.
The estimated cost to the government for the data collection is $67,212. This includes approximately $65,000 for a 1-year contract for data collection, data analysis, reports, etc. and approximately $2,212 per year represents SAMHSA costs to manage/administrate the survey for 2% of one employee (GS-13). The total annualized cost is approximately $67,212.
This is a new data collection.
Date |
Case Studies Activity |
December 2016-January 2017 |
SAMHSA recruits sites and partner organizations following OMB approval. |
January 2017 |
SAMHSA conducts parent/caregiver and youth pre-tests. |
February 2017 – April 2017 |
SAMHSA works with partner organizations to expose intervention site to campaign materials. |
May 2017 |
SAMHSA conducts parent/caregiver and youth post-tests. |
May 2017 – June 2017 |
SAMHSA conducts parent/caregiver individual interviews. |
July 2017 |
SAMHSA writes internal report on case studies findings. |
February 2018 |
SAMHSA reports findings in the 2018 Report to Congress |
The analysis plan for the case studies is designed to address the following questions:
Is there a relationship between campaign exposure and changes in parent attitudes and behavior about underage drinking at the selected sites?
What are the effects of campaign exposure on parents/caregivers in the selected sites?
Do individual interview data from the school sites suggest a cause-effect relationship between campaign exposure and changes in parent attitudes and behaviors in the experimental school district?
Is there a relationship between parents’ exposure to campaign materials with youth behavior/attitudes about drinking in the experimental school district?
Do any observed changes in overall parent behavior (between the pre-test and the post-test) align with any observed changes in overall youth behavior (between the pre-test and the post-test) in the selected school districts?
Data collected in the case studies will be both numerical and descriptive. SAMHSA will utilize a variety of strategies to analyze these data and will produce a final case study report that integrates the results from each data collection instrument.
The findings obtained from the case studies will be included in Chapter 5 of the annual STOP Report to Congress.
The expiration date will be displayed.
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
1 U.S. Department of Health and Human Services. The Surgeon General’s Call to Action to Prevent and Reduce Underage Drinking. U.S. Department of Health and Human Services, Office of the Surgeon General, 2007.
2 Center for Behavioral Health Statistics and Quality. (2015). Behavioral health trends in the United States: Results from the 2014 National Survey on Drug Use and Health (HHS Publication No. SMA 15-4927, NSDUH Series H-50). Retrieved from http://www.samhsa.gov/data/ on February 1, 2016.
3 Donovan, J., Leech, S., Zucker, R., Loveland-Cherry, C., Jester, J., Fitzgerald, H., et al. (2004). Really underage drinkers: Alcohol use among elementary students. Alcoholism: Clinical and Experimental Research, 28(2), 341-349.
4 SAMHSA (2005) NSDUH
5 Center for Behavioral Health Statistics and Quality. (2015). Behavioral health trends in the United States: Results from the 2014 National Survey on Drug Use and Health (HHS Publication No. SMA 15-4927, NSDUH Series H-50). Retrieved from http://www.samhsa.gov/data/ on February 1, 2016.
6 U.S. Department of Health and Human Services. The Surgeon General’s Call to Action to Prevent and Reduce Underage Drinking. U.S. Department of Health and Human Services, Office of the Surgeon General, 2007.
7 Ennet, S.T., Bauman, K.E., Foshee, V.A., Pemberton, M., & Hicks, K.A. (2001). Parent-child communication about adolescent tobacco and alcohol use: What do parents say and does it affect youth behavior? Journal of Marriage and Family 63, 48-62.
8 Wood, M.D., Read, J.P., Mitchell, R.E., & Brand, N.H. (2004). Do parents still matter? Parent and peer influences on alcohol involvement among recent high school graduates. Psychology of Addictive Behaviors, 18(1), 19-30.
9 Sieving, R.E., Maruyama, G., Williams, C.L., and Perry C.L. (2000). Pathways to adolescent alcohol use: Potential mechanisms of parent influence. Journal of Research on Adolescence, 10(4), 489-514.
10 https://nces.ed.gov/pubs2011/pesschools09/tables/table_05.asp
11 http://onlinelibrary.wiley.com/doi/10.1111/j.1746-1561.1997.tb01287.x/abstract
12 http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0126950
13 http://www.cgu.edu/include/Children's%20Mathematical%20Ach.pdf
14https://www.researchgate.net/profile/Yehuda_Baruch2/publication/228079609_Survey_Response_Rate_Levels_and_Trends_in_Organizational_Research/links/09e415112cd948d7f6000000.pdf
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |