TTHY Case Studies OMB SS-B January 20 2017

“Talk. They Hear You.” Campaign Evaluation: Case Studies

Supporting Statement

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

Respondents will be students and parents/caregivers at two middle schools. One school will be selected to receive forced exposure to the campaign and one act as a control. The two schools will be selected from among middle schools in all 50 states and the District of Columbia. Specifically, SAMHSA will rely on convenience sampling to select the middle school sites according to the following criteria:

• The middle school sites must be in regions that have not been previously targeted for “Talk. They Hear You.” campaign exposure.

• The middle school sites must have existing partnerships with their local prevention organizations.

• Leadership at the school districts and schools selected must agree to support the administration of data collection activities.

• Leadership at the school district and school receiving the intervention must further agree to support the dissemination of “Talk. They Hear You.” campaign materials to their middle school community.

• To the extent that it is possible to do so, SAMHSA will select sites for participation that are demographically diverse.

• The two middle school sites that are selected for the case studies will be matched based on school characteristics including: size, urbanicity, racial/ethnic distribution of students, and percentage of students receiving free and reduced price lunch.

Requests for participation will be distributed to members of SAMHSA’s network of local prevention organizations, focused on partners in regions that have not been exposed to the campaign. Any sites identified by partner organizations will be evaluated based on the above criteria and two demographically similar sites in two non-contiguous areas will be selected.

2. Information Collection Procedures

After selecting sites and local partnering organizations, SAMHSA will take the following procedures to collect data for the case studies, according to each of the following data collection instruments:

Youth Survey

SAMHSA will work with the local prevention organization and administrators from the selected middle school sites to coordinate the administration of the pre-test and post-test surveys for youth in both the control site and intervention site. Information will be distributed to parents and they will be given the opportunity to have their child opt-out of the study. Ultimately, paper copies of the 12-question survey will be distributed to participating middle school students during class time. A representative from the local prevention organizations will drop off paper surveys to the school sites. School administrators will determine an appropriate time in the school day to distribute the surveys to students (e.g., during a homeroom period) and will coordinate with school staff to administer the survey. Most likely, school administrators at each site will distribute paper surveys to teachers in their school. Then, teachers will distribute paper surveys to students in class, go over the directions for completion of the survey with students, and collect the completed surveys. SAMHSA estimates the amount of time for in-class survey distribution, review of instructions, survey completion, and collection will take approximately 10 minutes of instructional time. School administrators at each site will collect the completed surveys from teachers and representatives from the local prevention organizations will then pick up all completed surveys. The local prevention organizations will be responsible for mailing—using pre-paid shipping—the completed surveys to SAMHSA for data entry and analysis. Data will be entered into Excel spreadsheets. This process will be repeated for the post-exposure survey.

Parent/Caregiver Survey

SAMHSA will work with the local prevention organization and administrators from the selected middle school sites to coordinate the administration of the pre-test and post-test surveys online for parents/caregivers in both the control site and intervention site. To administer the online survey, SAMHSA will create an anonymous URL link to be embedded in an email, and draft language for the email invitation and an email reminder. Then, SAMHSA will provide the email text to the administrators from the selected sites to send to parents/caregivers of students at the school to invite them to take the survey through the anonymous URL. The email will be sent out through the school’s email system so that contact information does not have to be shared with SAMHSA. One week later, SAMHSA will coordinate with administrators from the selected sites to send the approved reminder emails to parents/caregivers. Any survey data submitted online will be downloaded into Excel spreadsheets and made available for data analysis.

In addition to the email invitation and reminder to take the survey electronically, SAMHSA will work with both the local prevention organizations and school administrators to send home a paper invitation to take the survey for those parents/caregivers whose primary mode of communication with the school is not email. This letter will include the URL to take the online version of the survey and will also include instructions for how to pick up/return the paper-based version of the survey at the school’s main office. SAMHSA will provide draft language to school administrators for this cover letter, however school administrators may customize the cover letters as they see fit. Regarding the paper survey dissemination and collection, representatives from the local prevention organizations will drop off self-sealing paper surveys to the selected school sites, which will be stored in the main office of the school. After the deadline for returning completed surveys has passed, representatives from the local prevention organizations will pick up the collection boxes and mail the completed surveys—using pre-paid shipping—to SAMHSA for data entry and analysis.

Parent/Caregiver Interview

In the post-test survey that is administered to parents/caregivers in the school receiving the intervention, respondents will be invited at the end of the survey to participate in an hour-long follow-up interview conducted over the phone. This survey question will note that they will receive a $35 gift card for their participation. If respondents wish to participate in the follow-up interview, they will respond to the survey question with their first name and a phone number and/or email address where they may be reached to schedule the phone interview.

As the SAMHSA team collects completed post-test surveys from parents/caregivers, SAMHSA will contact any respondents who have agreed to be contacted for the follow-up interview. SAMHSA staff will call and/or email the respondents to schedule the interview time and share the consent form. To conduct the interview, SAMHSA staff will call the participant at the agreed upon time and conduct the interview over the phone. SAMHSA staff will record the phone interview. The interview recordings will be transcribed and made available for analysis as Word documents.

3. Methods to Maximize Response Rates

In order to maximize response rates, SAMHSA will work closely with its network of local prevention organizations to identify middle school sites that have existing partnerships with prevention organizations. This will help to obtain “buy-in” from the middle school sites for data collection activities associated with the case studies. It is necessary for administrators to be supportive data collection partners with SAMHSA to fully implement this study. By working with willing and eager partners, it is SAMHSA’s intention that partnering sites will take ownership over the data collection activities occurring at the school sites and will send personalized survey invitation emails, cover letters, and reminders to potential respondents. In this way, respondents will receive survey invitations from a known person (e.g., a principal or superintendent), which will encourage their response.

In addition, SAMHSA will work with partnering organizations and sites to coordinate data collection activities to occur within the school year. SAMHSA will further provide the survey using multiple modes—online and paper—to meet the needs of survey respondents with different preferences. Providing options has been shown to increase response rates.

4. Tests of Procedures

SAMHSA conducted a pilot test in 2012 (OMB No. 0930-0196) of the parent/caregiver survey with approximately 150 parents and caregivers of youth who are 9-15 years of age. Several of the questions included in the youth survey are questions tested and used in the Monitoring the Future Survey, administered by the Survey Research Center in the Institute for Social Research at the University of Michigan (NIDA).

5. Statistical Consultants

Contact information

Robert Vincent, MS.Ed, NCAC II, CDP, NESAP

Public Health Analyst

Center for Substance Abuse Prevention

SAMHSA

240-276-1582

Elaine Rahbar

Division Director

Synergy Enterprises, Inc.

(240) 485-3606

Deborah Lessne

Senior Research Associate

Synergy Enterprises, Inc.

(240) 485-3286

List of Attachments & Procedures for Administration/Distribution

Attachment 1 Parent/Caregiver Pre-Test/Post-Test Survey

A request to partake in the pre- and post-test survey will be distributed to parents of participating middle schools via email. A listserv of emails will be provided by the administration of the participating school. A live link for the survey will be embedded within the email request and can be easily accessed by parents wishing to participate in either the pre- and/or post-test survey data collection efforts.

Attachment 2 Parent/Caregiver Survey Invitation and Reminder

As detailed above, this request/reminder will be distributed via email and serve as the live link that allows access to the pre- and post-test surveys.

Attachment 3 Youth Pre-Test and Post-Test Survey

This survey will be administered in hardcopy by homeroom teachers to students within the participating middle school. Completed surveys will the collected and turned into the main office for collection and processing.

Attachment 4 Youth Survey Parent Permission and Opt-Out Forms

This form will be sent home from school with children in hardcopy. Parents will be asked to sign the form and indicate whether they are opting out of the student research effort planned. Forms must be returned to either their child’s middle school teacher or principal by January 25, 2017 for the request to opt out to be granted.

Attachment 5 Teacher Script for Youth Survey

A hardcopy of this form will be distributed to each teacher whose class will be participating in the survey research efforts, along with enough hardcopies of the survey itself to be distributed to the students in their classroom setting.

This document explains the purpose of the SAMHSA student research, the procedures for administering the survey to students within the classroom setting, and the procedures for collecting and processing the completed surveys. It also provides a script that they are asked to read to their students before disseminating the survey to students for completion.

Attachment 6 Parent/Caregiver Interview Guide

The following document will be used by the project researcher interviewer as guidance for conducting the in-depth interviews held with parents following the pre- and post-survey research. These interviews will be conducted by phone.

Attachment 7 Parent/Caregiver Interviews Consent Form

This document serves as the authorization and release form that parents will be asked to sign at the end of the post-test survey if they agree to be contacted by telephone to answer follow-up questions regarding their participation in the underage drinking survey. The document outlines the nature of the research being conducted and how the findings will be used. This document also informs them that the call will be recorded, that their participation is entirely voluntary and that they may withdraw from the study at any time without penalty.

Attachment 8 Campaign Intervention Checklist

In an effort to expand thinking around the methods of exposure that can be utilized for the Talk They Hear You (TTHY) Campaign, the test site location and its participating partners are given this “living document” as they plan for and execute the intervention. They are asked to place a check mark next to types of methods used. This checklist will be collected by project evaluations at the end of the study in order to capture all the methods used by the interventionists.

6