SS-B_THHY Case Study OMB No 0930 0373 11.4.19

“Talk. They Hear You.”® Campaign Evaluation: Case Study

Supporting Statement

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

Respondents will be students and parents/caregivers at two middle schools. One school will be selected to receive forced exposure to the “Talk. They Hear You.”® campaign, and one will act as a control. The two schools will be selected from among middle schools in all 50 states and the District of Columbia. Specifically, the Substance Abuse and Mental Health Services Administration (SAMHSA) will rely on convenience sampling to select the middle school sites according to the following criteria:

• The middle school sites must be in regions that have not been previously targeted for “Talk. They Hear You.” campaign exposure.

• Leadership members at the school districts and schools selected must agree to support the administration of data collection activities.

• Leadership members at the school district and school receiving the intervention must further agree to support the dissemination of “Talk. They Hear You.” campaign materials to their middle school community.

• The two middle school sites that are selected for the case study will be matched based on school characteristics including size, urbanicity, racial/ethnic distribution of students, and percentage of students receiving free and reduced-price lunch.

Requests for participation will be distributed to members of SAMHSA’s network of local educational organizations, focused on partners in regions that have not been exposed to the campaign. Any sites identified by partner organizations will be evaluated based on the above criteria, and two demographically similar sites in two noncontiguous areas will be selected.

2. Information Collection Procedures

After selecting sites and local educational partnering organizations, SAMHSA will take the following procedures to collect data for the case study, according to each of the following data collection instruments:

Youth Survey

SAMHSA will work with the local administrators from the selected middle school sites to coordinate the administration of the pre-test and post-test surveys for youths in both the control site and the intervention site. Information about the survey will be distributed to parents, and they will be given the opportunity to have their child opt out of the study. Ultimately, paper copies of the survey will be distributed to participating middle school students during regular school hours. Administrators from the “Talk. They Hear You.” evaluation team will have paper surveys delivered directly to the school sites. School administrators will determine an appropriate time in the school day to distribute the surveys to students (e.g., during a homeroom period or during assembly) and will coordinate with school staff members to administer the survey. Most likely, school administrators at each site will distribute paper surveys to teachers in their school. Then, teachers will distribute paper surveys to students in class, go over the directions for completion of the survey with students, and collect the completed surveys. Teachers/administrators will use the same “Teacher Script for Youth Survey” guide to standardize the dissemination approach. SAMHSA estimates the amount of time for in-class survey distribution, review of instructions, survey completion, and collection will take approximately 10 minutes. School administrators at each site will collect the completed surveys and be responsible for mailing—using prepaid shipping—the completed surveys back to SAMHSA for data entry and analysis. Data will be entered into Excel spreadsheets. This process will be repeated for the post-exposure survey.

Parent/Caregiver Survey

SAMHSA will work with school administrators from the selected middle school sites to coordinate the administration of the pre-test and post-test surveys online for parents/caregivers in both the control site and the intervention site. To administer the online survey, SAMHSA will create an anonymous URL link to be embedded in an email, and draft language for the email invitation and subsequent email reminders. Then, SAMHSA will provide the email text to the administrators from the selected sites to send to parents/caregivers of students at the school to invite them to take the survey through the anonymous URL. The email will be sent out through the school’s email system so that contact information does not have to be shared with SAMHSA. One week later, SAMHSA will coordinate with administrators from the selected sites to send the approved reminder emails to parents/caregivers. Any survey data submitted online will be downloaded into Excel spreadsheets and made available for data analysis.

In addition to the email invitation and reminder to take the survey electronically, SAMHSA will work with school administrators to send home a paper invitation to take the survey for those parents/caregivers whose primary mode of communication with the school is not email. This letter will include the URL to take the online version of the survey and will also include instructions for how to pick up/return the paper-based version of the survey at the school’s main office. SAMHSA will provide draft language to school administrators for this cover letter. However, school administrators may customize the cover letters as they see fit. Regarding the paper survey dissemination and collection, SAMHSA will provide paper surveys to the selected school sites, which will be stored in the main office of the school. After the deadline for returning completed surveys has passed, school representatives will pick up the collection boxes and mail the completed surveys—using prepaid shipping—to SAMHSA for data entry and analysis.

Parent/Caregiver Interview

In the post-test survey that is administered to parents/caregivers in the school receiving the intervention, adult respondents will be invited at the end of the survey to participate in an hour-long follow-up interview conducted over the phone. This survey question will note that respondents will receive a $35 gift card for their participation. If respondents wish to participate in the follow-up interview, they will respond to the survey question with their first name and a phone number and/or email address where they may be reached to schedule the phone interview.

As the SAMHSA team collects completed post-test surveys from parents/caregivers, SAMHSA will contact any respondents who have agreed to be contacted for the follow-up interview. SAMHSA staff members will call and/or email the respondents to schedule the interview time and share the consent form. To conduct the interview, SAMHSA staff members will call the participant at the agreed-upon time and conduct the interview over the phone. SAMHSA staff members will record the phone interview. The interview recordings will be transcribed and made available for analysis as Word documents.

3. Methods to Maximize Response Rates

In order to maximize response rates, SAMHSA will work closely with its network of local educational organizations to identify middle school sites interested and able to participate as either the intervention or the control site. This will help to obtain “buy-in” from the middle school sites for data collection activities associated with the case study. It is necessary for administrators to be supportive data collection partners with SAMHSA to fully implement this study. By working with willing and eager partners, it is SAMHSA’s intention that the school administration will take ownership over the data collection activities occurring at the school sites, and will send personalized survey invitation emails, cover letters, and reminders to potential respondents. In this way, respondents will receive survey invitations from a known person (e.g., a principal or superintendent), which will encourage their response.

In addition, SAMHSA will work with school administrators to coordinate “Talk. They Hear You.” intervention and data collection activities throughout the school year. SAMHSA will provide access to the parent/caregiver survey using multiple modes—online and paper—to meet the needs of survey respondents with different access preferences. Providing options has been shown to increase response rates.

4. Tests of Procedures

SAMHSA conducted a pilot test in 2012 (OMB No. 0930-0196) of the parent/caregiver survey with approximately 150 parents and caregivers of youths who were 9–15 years of age. Several of the questions included in the youth survey were questions tested and used in the Monitoring the Future Survey, administered by the Survey Research Center in the Institute for Social Research at the University of Michigan (National Institute on Drug Abuse).

The first case study launched in October of 2017 and terminated in June of 2018. The intervention site ended its campaign exposure efforts in April of 2018. During the post-test survey data collection effort, recruitment among the parent population. A total of 784 parents at both sites responded to the pre-intervention survey and 574 to the post-intervention survey. Five key conceptual measures in the parent survey instrument were used to capture parental attitudes about underage drinking: (1) overall concern about underage drinking; (2) general agreement on the importance of discussing underage drinking with their child; (3) confidence in affecting their child’s decisions about alcohol; (4) their relative prioritization of an underage drinking discussion with their child; and (5) their intentions to have a discussion about underage drinking with their child in the near future.

Findings suggest that TTHY had a positive effect on parental attitudes and behaviors around underage drinking at the intervention site. Specifically, the campaign had a significant positive effect on parental confidence in making a difference in their child’s decision about alcohol. It also had a positive directional effect on three other attitudinal categories: concern, importance, and intention to discuss underage drinking. The campaign did not find a significant effect on prioritization. TTHY also had a positive directional effect on parental behavior, including a key outcome: increasing the proportion of parents who had a conversation with their child about underage drinking.

In parallel to parent data collection efforts, SAMHSA collected data regarding student attitudes and behaviors; 1,954 student responses were collected pre-intervention and 1,780 student responses were collected post-intervention.

Student attitudes about underage drinking were marginally affected by the TTHY campaign. Post-intervention, perceptions of binge drinking as being a health risk significantly increased at the intervention site, and disapproval of binge drinking was also affected in a positive direction among students.

The TTHY campaign also mitigated underage drinking behaviors among students. On three behavioral measures (recently tried alcohol, ever tried alcohol, and number of students getting drunk), the TTHY campaign mitigated increases at the intervention site.¹

Key data findings of the TTHY campaign evaluation is provided in Chapter 5, of the “Report to Congress on the Prevention and Reduction of Underage Drinking” and can be found at www.stopalcoholabuse.gov.

5. Statistical Consultants

Contact information

Robert Vincent, M.S.Ed., NCAC II, CDP, NESAP

Public Health Analyst

Center for Substance Abuse Prevention, SAMHSA

Phone: (240) 276–1582

Email: robert.vincent@samhsa.hhs.gov

Ryan Parks

Project Director, Synergy Enterprises, Inc.

8757 Georgia Avenue, Suite 1440

Silver Spring, MD 20910

Phone: (240) 485–3278

Email: RParks@seiservices.com

Melissa Seldin

Senior Research Associate, Synergy Enterprises, Inc.

8757 Georgia Avenue, Suite 1440

Silver Spring, MD 20910

Phone: (240) 485–3286

Email: MSeldin@seiservices.com

List of Attachments & Procedures for Administration/Distribution

Attachment 1 Parent/Caregiver Pre-Test/Post-Test Survey

A request to partake in the pre- and post-test survey will be distributed to parents and caregivers of participating middle schools via email. A Listserv of emails will be provided by the administration of the participating school. A live link for the survey will be embedded within the email request, and can be easily accessed by parents and caregivers wishing to participate in either the pre- and/or the post-test survey data collection efforts.

Attachment 2 Parent/Caregiver Survey Invitation and Reminder

As detailed above, this request/reminder will be distributed via email and will serve as the live link that allows access to the pre- and post-test surveys.

Attachment 3 Youth Pre-Test and Post-Test Survey

This survey will be administered in hard copy potentially by homeroom teachers to students within the participating middle school. Completed surveys will be collected and turned in to the school’s main office for collection and processing.

Attachment 4 Youth Survey Parent/Caregiver Permission and Opt-Out Forms

This form will be sent home from school with children in hard copy. A link to the form will also be embedded via electronic parent/caregiver/principal correspondence. Parents and caregivers will be asked to sign the form and indicate whether they are opting out of the student evaluation effort planned. Forms must be returned to either their child’s middle school teacher or their child’s middle school principal by October 10, 2020, for the request to opt out to be granted.

Attachment 5 Teacher Script for Youth Survey

A hard copy of this form will be distributed to each teacher whose class will be participating in the survey evaluation efforts, along with enough hard copies of the survey itself to be distributed to the students in their classroom setting.

This document explains the purpose of the SAMHSA student evaluation, the procedures for administering the survey to students within the classroom setting, and the procedures for collecting and processing the completed surveys. It also provides a script that teachers are asked to read to their students before disseminating the survey to students for completion.

Attachment 6 Parent/Caregiver Interview Guide

This document will be used by the project evaluation interviewer as guidance for conducting the in-depth interviews held with parents and caregivers following the pre- and post-survey evaluation. These interviews will be conducted by phone.

Attachment 7 Parent/Caregiver Interviews Consent Form

This document serves as the authorization and release form that parents and caregivers will be asked to sign at the end of the post-test survey if they agree to be contacted by telephone to answer follow-up questions regarding their participation in the underage drinking survey. The document outlines the nature of the evaluation being conducted and how the findings will be used. This document also informs them that the call will be recorded, that their participation is entirely voluntary, and that they may withdraw from the study at any time without penalty.

Attachment 8 Campaign Intervention Checklist

In an effort to expand thinking around the methods of exposure that can be utilized for the “Talk. They Hear You.” campaign, the test site location and its participating partners are given this “living document” as they plan for and execute the intervention. They are asked to place a check mark next to the types of methods used during the “Talk. They Hear You.” dissemination period. This checklist will be collected by project evaluators at the end of the study in order to capture all the methods used by the interventionists.

1Note that “ever tried alcohol” increased at both sites, but comparatively less so at the intervention site.

6