OMB Supporting Statement_B

Assessment of the Underage Drinking Prevention Education Initiatives

State/Territory Videos Project

Supporting Statement

B. Statistical Methods

B.1 RESPONDENT UNIVERSE AND SAMPLING METHODS

The process assessment of the State/Territory Videos project relies in part on a convenience sampling strategy. The sample includes all States and Territories that had videos produced under the project between 2007 and 2013, and viewers of the videos between 2011 and 2014. A summary of the sample by phase is provided below.

Phase I - State/Territory Video Contacts Interviews

In 2011-2014, approximately 45 interviews will be conducted with the points of contacts for State/Territory videos that were produced in 2010-2013 (see the table below). No data will be collected on the characteristics of the State/Territory points of contact. The anticipated response rate for this phase is 85 percent.

Phase I – State/Territory Video Contacts Interviews during 2011-2014

Production year of videos	Number of videos produced	Assessment year(s)	Total number of interviews by 2014
2010	8	2011	8
2011	13	2011-2012	13
2012	12	2012-2013	12
2013	12	2013-2014	12
Total:	45	Total:	45

Phase II - Dissemination Updates

State/Territory points of contact for all videos produced since the project began in 2007 will be asked to update the dissemination activities of their State’s/Territory’s video every 6 months, beginning in 2011. A total of 15 videos were produced between 2007 and 2009. These State/Territory points of contact provided initial details of the dissemination activities of their State’s/Territory’s video to the video production team during the post-production phase of the video. They will be included in this data collection phase to determine if the video continues to be utilized to as an effective tool to highlight the State’s/Territory’s UAD prevention activities. If all expected States and Territories produce a video by the end of the project (2014), there will be approximately 60 State/Territory points of contact and 226 dissemination updates (see the table below). No data will be collected on the characteristics of State/Territory points of contact. The anticipated response rate for this phase is 75 percent.

Phase II – Dissemination Updates during 2011-2014

Production year of videos	Number of videos produced	Assessment year(s) (Estimated number of updates)	Total number of updates (x) Number of videos
2007	4	2011 (1), 2012 (2), 2013 (2), 2014 (1)	24
2009	11	2011 (1), 2012 (2), 2013 (2), 2014 (1)	66
2010	8	2011 (1), 2012 (2), 2013 (2), 2014 (1)	48
2011	13	2012 (1), 2013 (2), 2014 (1)	52
2012	12	2013 (1), 2014(1)	24
2013	12	2014 (1)	12
Total:	60	Total:	226

Phase III - Video Viewers Feedback

Persons could view the video produced by the State/Territory in at least three ways—1) in-person (e.g., at a meeting hosted by State Departments of Health or at a professional conference), 2) online (e.g., www.stopacoholabuse.gov, SAMHSA’s YouTube channel, or the State/Territory agency’s website), and 3) electronically (e.g., on DVD or public access television). By 2014, it is estimated that 12,224.40 viewers will complete the Video Viewers Feedback Form. This estimation was calculated by taking the median number of views (122) of current unduplicated videos [17] that were on the SAMHSA YouTube channel over a two month period (September 29, 2010 to December 8, 2010) and multiplying the median by 6 to get the approximate yearly number of views (732). It was further estimated that ten percent (.10) of the yearly viewers would complete the feedback form; therefore, the number of yearly viewers (732) was multiplied by one-tenth (.10) to obtain the estimated number of potential feedback form completes per video in a year (73.20). The estimated number of potential yearly feedback form completes (73.20) was then multiplied by the number of videos produced in each production year and the number of years the videos will be assessed during 2011-2014 (see the table below).

Phase III - Video Viewers Feedback Form Completes during 2011-2014

Production year of videos	Number of videos produced	Number of potential yearly feedback form completes	Number of assessment year(s)	Total number feedback form completes
2007	4	73.20	4 (2011, 2012, 2013, 2014)	1,171.20
2009	11		4 (2011, 2012, 2013, 2014)	3,220.80
2010	8		4 (2011, 2012, 2013, 2014)	2,342.40
2011	13		3 (2012, 2013, 2014)	2,854.80
2012	12		2 (2013, 2014)	1,756.80
2013	12		1 (2014)	878.40
Total:	60		Total:	12,224.40

Demographic information is collected on the viewers; however, these data will be reported in aggregate to SAMHSA. The anticipated response rate for this phase is approximately 75 percent.

B.2 INFORMATION COLLECTION PROCEDURES

Prior to the assessment, a “teaser email” is sent individually to all National Prevention Network (NPN) members whose State/Territory had not previously participated in the State/Territory Videos project. The email is sent under the name of the project’s Government Project Officer with a subject line of—Creating State/Territory Videos on Underage Drinking Prevention. This teaser email announces that an ‘official invitation to produce a video’ will follow, provides frequently asked questions about participating in the State/Territory Videos project, and asks NPN members to consider joining an upcoming conference call so that other questions they may have can be answered. Conference call information is provided at the end of the email as well as contact information for a member of the video production team. A few days prior to the scheduled conference call, a “reminder email” is sent en masse to the NPN members (see Attachment 6, Videos Project Communications Sent Prior to Assessment). It is during these calls that additional details on participating in the project are provided, including information about the assessment.

During the informational conference calls, NPN members and additional State/Territory representatives (e.g., the identified State/Territory points of contact [they are usually on the call]), are informed of the following regarding the assessment—

We intend to conduct a process assessment of the State/Territory Videos project, pending Institutional Review Board and Office of Management and Budget approval. There are three phases to the assessment. The first phase is an interview with the designated State/Territory contact once the video is finalized. The focus of the interview will be around your State’s/Territory’s experience in producing the video, the dissemination efforts of the video, and technical assistance received in producing the video. It is estimated that the interview will take approximately 20 minutes.

The second phase will occur 6 months after the interview. Your designated contact will be sent an email asking him/her to update the dissemination efforts of the video using an online form. It is estimated that it will take 10 minutes to update the dissemination efforts using this online form. We are also asking that the dissemination efforts be updated every 6 months until the end of this project, which is in 2014.

The third phase involves persons that viewed your State’s/Territory’s video. The purpose of this phase is to obtain feedback on the video to determine if the video increased community awareness of the underage drinking prevention efforts of your State/Territory. The feedback form will be located on the ‘State Videos’ page of www.stopalcoholabuse.gov (http://www.stopalcoholabuse.gov/StateVideos.aspx). We are seeking clearance from SAMHSA to place a link to the feedback form on its YouTube channel. If you conduct in-person meetings to showcase your State’s/Territory’s video, you may direct persons to the ‘State Videos’ page of www.stopalcoholabuse.gov to complete the form or we can provide you with a link to the form that can be placed on your agency’s website. In all cases, we will process and share aggregated viewer feedback with you on your video. It is estimated that it will take viewers about 5 minutes to complete the feedback form.

Following the conference call, an “official invitation” to participate in the State/Territory Videos project is sent to the NPN members. NPN members RSVP whether they intend to participate in the project, select which year (2011, 2012, or 2013) would work best to produce the video; and indicate the primary contact person for this project. The name, email address, and phone number of the primary contact person will be forwarded by the video production team to the assessment team.

Phase I - State/Territory Video Contacts Interviews

A member of the assessment team will make contact with the State/Territory point of contact by phone within 45 days after the final copy of the video is provided (this timeframe will be off schedule for the videos finalized in 2010 and those produced in the first half of 2011). If the point of contact is reached, the assessor will start reading the informed consent statement located on the State/Territory Video Contacts Interview Form (see Attachment 1) and inquire if it is a good time to conduct the interview. If it is not a good time, an alternate time will be determined. If it is a good time to conduct the interview, the assessor will continue reading the informed consent statement and proceed with the interview. At the end of the interview, the assessor will thank the contact for their time and note that the contact will be recontacted in about 6 months to update the dissemination efforts of the video. Data entry of information captured on the interview form will be performed by the assessor. Interviews with the State/Territory points of contacts will be conducted according to the schedule in the table below.

Phase I – Schedule of Interviews during 2011-2014

Production year of videos	Number of videos produced	Interview year(s)
2010	8	2011
2011	13	2011-2012
2012	12	2012-2013
2013	12	2013-2014
Total:	45

If the point of contact is not reached on the initial attempt, a voicemail message will be left. The assessor will also email the point of contact. If several attempts to reach the contact are unsuccessful, the assessor will use other methods to reach the contact such as communicating with the NPN.

Phase II - Dissemination Updates

At about 6 months after the interview, the State/Territory points of contact will be sent an email from the assessment team detailing the need to update the dissemination efforts of the video for the past 6 months. The email will include a coded link to the State/Territory Videos Project – Dissemination Update Form (see Attachment 2). This coded link is used to track whether a State/Territory point of contact has responded to the request to complete the form. State/Territory points of contact that do not update the dissemination efforts of their State’s/Territory’s video more than 30 days following the email will be sent a reminder e-mail requesting them to complete the dissemination update form by a defined date. If contacts are unresponsive to emails, the assessor will use other methods to reach the contact such as communicating by phone or contacting the NPN.

As contacts complete each page of the form and click Next, data entered will be automatically saved. Contacts will be allowed to return to the form until they click Done. Once the form is submitted (i.e., Done is clicked), contacts will not be allowed to go back into the form to make changes. The data will be stripped of its association to the coded link and the IP address used to complete the form. An ID code will be assigned to the data and will be used as a running tally of the number of forms received. At the end of the dissemination update form, the contact is thanked and reminded that they will be recontacted in about 6 months to update the dissemination efforts of their State’s/Territory’s video.

Following OMB clearance, an email will be sent to the State/Territory points of contact for videos produced in 2007-2009 noting that OMB clearance has been received and asking them to update the dissemination efforts of the video for the past 6 months for the assessment of the State/Territory Videos project. These State/Territory points of contact provided initial details of the dissemination activities of their State’s/Territory’s video to the video production team during the post-production phase of the video. The aforementioned procedures also apply to these contacts.

Dissemination updates will be collected according to the schedule in the table below.

Phase II – Schedule of Dissemination Updates Collection during 2011-2014

Production year of videos	Number of videos produced	Collection year(s) (Estimated number of updates)
2007	4	2011 (1), 2012 (2), 2013 (2), 2014 (1)
2009	11	2011 (1), 2012 (2), 2013 (2), 2014 (1)
2010	8	2011 (1), 2012 (2), 2013 (2), 2014 (1)
2011	13	2012 (1), 2013 (2), 2014 (1)
2012	12	2013 (1), 2014(1)
2013	12	2014 (1)
Total:	60

Phase III - Video Viewers Feedback

Persons that view the State/Territory videos are among the respondents for this assessment. A short statement about the need for viewer feedback, along with a link to the Video Viewers Feedback Form (see Attachment 3), will be placed on the ‘State Videos’ page of www.stopalcoholabuse.gov (http://www.stopalcoholabuse.gov/StateVideos.aspx). On SAMHSA’s YouTube channel, a short description of the State/Territory Videos initiative accompanies each video that is produced under this project. At the end of the description, visitors are directed to www.stopalcoholabuse.gov for more information or resources. If clearance is granted by SAMHSA, a link to the feedback form will be placed in the description. Additionally, State/Territory points of contact will be notified of the Video Viewers Feedback Form and will be informed that they may direct viewers to www.stopalcoholabuse.gov to complete the form following any in-person showings of the video or they can provide viewers with a link to the form through their agency’s website.

Completion of the Video Viewers Feedback Form is optional and viewers can cancel out of the survey at any time. If viewers choose to submit feedback, they are asked to complete one feedback form for each video viewed. Once viewers select which video they viewed in question 2 of the feedback form, the name of that State/Territory is automatically filled in for the remaining questions that note the name of the State/Territory. As viewers complete each page of the feedback form and click Next, data entered will be automatically saved. Since there is no way to associate a form with a particular viewer, viewers will not be allowed to return to the form that they are completing once the form is submitted (i.e., Done is clicked) or if they exit the URL. The IP address used to complete the form is captured by the host site of the form; however, it is not linked to the feedback form responses. Each form will be assigned an ID code that will be used as a running tally of the number of forms received.

Video viewer feedback will be collected according to the schedule in the table below.

Phase III – Schedule of Video Viewer Feedback Collection during 2011-2014

Production year of videos	Number of videos produced	Number of potential yearly feedback form completes per video	Collection year(s)
2007	4	73.20	2011, 2012, 2013, 2014
2009	11		2011, 2012, 2013, 2014
2010	8		2011, 2012, 2013, 2014
2011	13		2012, 2013, 2014
2012	12		2013, 2014
2013	12		2014
Total:	60

B.3 METHODS TO MAXIMIZE RESPONSE RATES

State/Territory points of contact will be contacted by phone 45 days after the final copy of the video is provided to take part in the interview phase of the assessment. If the points of contact are not reached on the initial attempt, a voicemail message will be left by the assessor and an email will also be sent. Assessors will continue to try and reach contacts by phone and email at various times during the day for up to 2 weeks after initial contact was attempted. After 2 weeks, assessors will use other methods to reach contacts such as communicating with the NPN.

At the end of the interview, assessors note that contacts will be recontacted in about 6 months to update the dissemination efforts of the video. At about 6 months after the interview is completed, State/Territory points of contact will be sent an email from the assessment team detailing the need to update the dissemination efforts of the video for the past 6 months. State/Territory points of contact that do not update the dissemination efforts of their State’s/Territory’s video more than 30 days following the email will be sent a reminder e-mail requesting them to complete the dissemination update form by a defined date. If contacts are unresponsive to emails, assessors will use other methods to reach contacts such as communicating by phone or contacting the NPN. At the end of the dissemination update form, the contact is reminded that they will be recontacted in about 6 months to update the dissemination efforts of their State’s/Territory’s video.

A short statement about the need for viewer feedback, along with a link to the Video Viewers Feedback Form (see Attachment 3), will be placed on the ‘State Videos’ page of www.stopalcoholabuse.gov. The statement will be highlighted so viewers will notice it.

The anticipated response rate for Phase I and II of the assessment is 85 percent and 75 percent, respectively. States/Territories are considered partners in this effort. From inception to conclusion, the State/Territory points of contact are involved in the production process of their State’s/Territory’s video. The response rate for Phase III of the assessment is approximately 75 percent. Given the exposure the videos have on SAMHSA’s YouTube channel and www.stopalcoholabuse.gov, and the number of views on SAMHSA’s YouTube channel in just over a two month period, it is anticipated that many persons will take the time to provide feedback on the videos.

B.4 TESTS OF PROCEDURES

In 2007, a pilot-test of the State/Territory Video Contacts Interview Form (see Attachment 1) was conducted with four States that produced a UAD prevention video (see Attachment 4, Report on Underage Drinking Prevention Videos: A Four-State Pilot Project). A total of five* State/Territory points of contact were interviewed by phone (*one State had two contacts). In 2010, a draft of the State/Territory Videos Project – Dissemination Update Form (see Attachment 2) was sent to three State/Territory points of contact. Additionally, the Video Viewers Feedback Form (see Attachment 3) was sent to three local community members. All individuals were asked to identify any question that they did not understand or thought they would not be able to answer. They were also asked to report the amount of time it took them to complete the instruments. For the interview form, the assessor noted how long it took to complete the interview. All of the questions were easily understood by those individuals, and no questions were identified as being difficult to answer. In light of this input and the simple straightforward nature of the testing procedures, pretesting is not necessary.

B.5 STATISTICAL CONSULTANTS

The following individuals provided statistical consultation in development of the data collection instruments and methodology:

Emily R. Novick, M.P.P. Government Project Officer SAMHSA/CSAP 1 Choke Cherry Road Room 4-1010 Rockville, MD 20857 Phone: (240) 276-1359	Rená A. Agee, M.A. Senior Associate ICF Macro 11785 Beltsville Drive Suite 300 Calverton, MD 20705 Phone: (301) 572-0400	Katia Delrahim Howlett, PhD UADPEI Deputy Project Director URC 11420 Rockville Pike Rockville, MD 20852 (240) 747-4798
Janet D. Griffith, PhD IRB Chair/Senior Fellow ICF 9300 Lee Highway Fairfax, VA 22031 Phone: (703) 225-2243	Ronaldo Iachan, PhD Statistician ICF Macro 11785 Beltsville Drive Suite 300 Calverton, MD 20705 Phone: (301) 572-0538	Lee Harding Technical Specialist ICF Macro 11785 Beltsville Drive Suite 300 Calverton, MD 20705 Phone: (301) 572-0524

References

Centers for Disease Control and Prevention. Youth Risk Behavior Surveillance—United States, 2009. Surveillance Summaries, June 4, 2010. MMWR 2010; 59 (No. SS-5).

Department of Labor, Bureau of Labor Statistics. (2009). 2009 National Occupational Employment and Wage Estimates. Washington, DC: Government Printing Office. Retrieved from http://www.bls.gov/oes/current/oes_nat.htm

Substance Abuse and Mental Health Services Administration. (2010). Results from the 2009 National Survey on Drug Use and Health: Volume I. Summary of the National Findings (Office of Applied Studies, NSDUH Series H-38A, HHS Publication No. SMA 10-4586Findings). Rockville, MD.

U.S. Department of Health and Human Services. (2007). The Surgeon General’s call to action to prevent and reduce underage drinking. Washington, DC: U.S. Department of Health and Human Services, Office of the Surgeon General.

List of Attachments

1. State/Territory Video Contacts Interview Form

2. State/Territory Videos Project – Dissemination Update Form

3. Video Viewers Feedback Form

4. Report on Underage Drinking Prevention Videos: A Four-State Pilot Project

5. ICF Macro Institutional Review Board IRB Review Findings Form

6. State/Territory Videos Project Communications Sent Prior to Assessment

9