Ss_b_dsp_mrt_6_15_17

DIVISION OF STATE PROGRAMS - MANAGEMENT & REPORTING TOOL

SUPPORTING STATEMENT

2. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods

The Division of State Program Management and Reporting Tool DSP-Tool will use a census approach to collecting process and programmatic data along with collecting aggregated existing archival data and data from survey samples for the outcomes measures.

Using a census approach, the targeted universe is all Partnerships for Success (PFS), Strategic Prevention Framework Rx (SPF-Rx), and Prescription Drug Overdose (PDO) grant programs. For PFS, the program includes grantee Project Directors for cohorts 2013 (n=16), 2014 (N=21), 2015 (n=32) and 2016 (N=2). PDO includes Project Directors for cohorts 2016 (N=12) and 2017 (n=11). Finally, SPF-Rx includes Project Directors for cohorts 2016 (N=25). New grant programs may be added in the future to answer standard SPF questions. A census of all Project Directors from all three programs—PFS, SPF-Rx, and PDO--is necessary as this data will be used by SAMHSA to monitor each program’s performance and grantees will also use it to track their ongoing implementation and reporting requirements such as GPRA. In order to meet SAMHSA’s annual reporting requirements for GPRA and performance measures, and more frequent reporting requirements for programs within the Division of State Programs, SAMHSA must obtain data from all grantees, which supports the need for a census approach. In addition, the program grantees encompass a wide variety of organizational types and structures that are implementing a range of prevention interventions targeted to different populations and with various outcome goals.

In addition to the GPRA, data collected by grantees will be used to demonstrate how SAMHSA’s grant programs are reducing disparities in access, service use, and outcomes nationwide.  To accomplish this, SAMHSA expects grantees to utilize their data to (1) identifying subpopulations (i.e., racial, ethnic, sexual/gender minority groups) vulnerable to health disparities and (2) implement strategies to decrease the differences in access, service use, and outcomes among those subpopulations.  There will be subpopulations where sample size is too small to properly sample and where a census would be necessary obtain relevant and reliable outcome data.

B.2. Information Collection Procedures

All instruments are self-administered, web-based data collection tools completed through an online data collection system. All respondents are the grantee Project Directors or their staff. Before data collection performance monitoring begins, respondents will be provided a unique log-in to enter the data system, where they will be required to create a password. Respondent email addresses for each login will be stored within the system so that automatic alerts and notifications can be sent.

Pending Office of Management and Budget (OMB) approval, the estimates in the DSP MRT will be collected according to the schedule outlined in section A-16.

The contractor developed user manuals for accessing and navigating the MRT online data collection system and question-by-question and frequently asked question (FAQ) guides to help respondents accurately complete the instruments.

Within the online data collection system, all manuals, guides, and training webinars will be archived and accessible to respondents for reference at any time on the contractor’s website.

Availability of the system is important in any data collection system, especially one employed by grantee sites around the country, including multiple time zones and pacific jurisdictions. The online system will be maintained in an available state as much as possible to allow grantees to have access for entering data, as well as to give SAMHSA, grantees, and team access to reports.

Providing a robust system that is simple and easy to use across all areas is also critically important. To achieve this, the MRT will implement user-friendly features across all functional areas, taking into account the needs of both SAMHSA and grantees. Additionally, every page of the online data system will have a “Help” or “Support” link located in the upper right corner, which will allow the respondent to access the following support resources:

1. Search the Knowledge Base. More comprehensive than a list of FAQs and more organized than a support forum, the Knowledge Base offers a “layered information” approach so that respondents can search by keyword and then drill down to view material at increasing levels of detail. It will be a curated and easily searchable source of information including items such as

• system documentation,

• user guides,

• policies and procedures,

• protocols,

• training materials, and

• FAQs.

1. Contact Us. Respondents may request assistance by calling a provided toll-free number, sending an email request, or submitting a technical assistance submission form as desired. The toll-free line will be routed to an email system that is checked regularly by members of the training and technical assistance team. Staff responding to technical assistance requests will be trained in use of the system and have ready access to the full Knowledge Base. Training and technical assistance team staff will monitor all submitted tickets to ensure timely response and resolution of technical assistance requests.

B.3. Methods to Maximize Response Rates

Grantees participate in all performance monitoring data collection activities per the Terms and conditions of the grant award. The evaluation team will employ a number of strategies to help ensure grantees participate with a 100% response rate.

As described above, evaluation will develop user manuals for accessing and navigating the online data collection system and question-by-question and FAQ guides to help respondents accurately complete the MRT instruments. Grantees will also be provided training webinars to walk through the MRT online data collection system and to review data collection procedures. Within the online data collection system, all manuals, guides, and training webinars will be archived and accessible to respondents for reference at any time.

SAMHSA Project Officers will monitor the DSP-MRT and receive e-mail notifications when their grantees submit individual monitoring data. Approximately one month after a data submission deadline the contractor will provide Project Officers a list of past due instruments. SAMHSA Project Officers will then follow up with their grantees to ensure submission.

B.4. Test of Procedures

Three contracting staff completed the instrument within a testing version of the system. These staff members have experience with SPF model used in various CSAP programs

The Standard questions in the DSP- MRT are estimated to take 3 hours to complete and the program specific modules—Rx and PDO—are estimated to take 1 hour to complete.

Each of the grantees is a former CSAP grantee; thus they will all have experience completing instruments similar in procedure (e.g., entering data into an online data system), length, and content. Additionally, the SPF performance measure data collection used lessons learned from the earlier data systems to improve data collection procedures.

B.5. Statistical Consultants

The Government and contractor team comprises several experts who will be directly involved in data collection and statistical analysis. Also, contractor in-house experts will be consulted throughout the program on various statistical aspects of the design, methodological issues, and data analysis, including leveraged funding analysis. Finally, SAMHSA has an External Steering Committee. Members of this External Steering Committee have already provided feedback on the performance monitoring instruments and the evaluation/analysis plan and will continue to provide advice and feedback through scheduled quarterly meetings and ad hoc e-mails as needed. The table below provides details of these team members and advisors.

Statistical Consultants for the SPF-PFS Performance Measurement and Contract Staff

Name & Role in Evaluation	Title & Address	Contact Information
Laura Dunlap, PhD Cost Analysis Team Leader	Director Behavioral Health Economics Program RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 541–7310 Email: ljd@rti.org
Elvira Elek, PhD Deputy Director	Research Public Health Analyst Public Health Policy Research RTI International 701 13th Street, NW, Suite 750 Washington, DC 20005	Telephone: (202) 728–2048 Email: eelek@rti.org
Phillip Graham, PhD Project Director	Senior Research Public Health Analyst Crime, Violence, and Justice Program RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 485–7752 Email: pgraham@rti.org
Nilufer Isvan Senior Evaluator	Senior Research Fellow Human Service Research Institute (HSRI) 2336 Massachusetts Avenue  Cambridge, MA 02140	Telephone: (617) 844-2505 Email: nisvan@hsri.org
Gillian J. Leichtling Senior Evaluator	Senior Research Associate RMC Research Corporation 111 SW Columbia Street  Suite 1030  Portland, OR 97201-5883	Telephone: (503) 223-8248 x735 Email:GLeichtling@rmccorp.com
Antonio Morgan-Lopez, PhD Analysis Team Leader	Senior Research Quantitative Psychologist Risk Behavior and Family Research RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 316–3436 Email: amorganlopez@rti.org
Scott Novak, PhD Senior Statistician	Senior Research Public Health Analyst Behavioral Health & Epidemiology RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 541–7129 Email: snovak@rti.org
Steve Sullivan ESC Task Team Leader	Senior Director Cloudburst Consulting Group, Inc. 8400 Corporate Drive, Suite 550 Landover, MD 20785-2238	Telephone: (301) 918-4400 Email: steven.sullivan@cloudburstgroup.com
External Steering Committee
Bethany Bray, PhD Methods/Statistics	Research Associate The Methodology Center The Pennsylvania State University 400 Calder Square II State College, PA 16801	Telephone: (814) 865-1225 Email: bcbray@psu.edu
William DeJong, PhD Evaluating Environmental Strategies	Professor Boston University School of Public Health Community Health Sciences 801 Mass Ave Crosstown Center Boston MA 02118	Telephone: (508) 954-0224 Email: wdejong@bu.edu
Brian Flay, DrPH Prevention Science	Professor Oregon State University College Of Public Health and Human Sciences 457 Waldo Hall Corvallis, OR 9733	Telephone: (541) 737-3837 Email: Brian.Flay@oregonstate.edu
Rick Harwood Economics, Cost Analyses	Director of Research and Program Applications National Association of State Alcohol and Drug Abuse Directors, Inc., (NASADAD) 1025 Connecticut Avenue NW, Suite 605 Washington, DC 20036	Telephone: (202) 293-0090, ext. 104 Email: rharwood@nasadad.org
Dottie Natal IT, Data Collection Systems	CEO Imagen Multimedia Corp	Email:dottie@imagenmm.com
Chris Ringwalt, DrPH Intervention Implementation and Dissemination	Public Health Senior Research Scientist Pacific Institute for Research and Evaluation 1516 E. Franklin Street, Suite 200 Chapel Hill, NC 27514-2812	Telephone: (919) 259-0643 Email: ringwalt@PIRE.org;
Government Officers
Thomas Clarke, PhD. Senior Social Science Analyst Contracting Officer Representative	Senior Public Health Analyst CSAP, SAMHSA 1 Choke Cherry Road, Room 4-1025 Rockville, MD 20857	SAMHSA/CSAP Performance and Technical Assistance Branch1 Choke Cherry Road, Room 16-E03Rockville, MD 20857 sara.azimi-bolourian@samhsa.hhs.gov Phone (240) 276-2708
Stephanie Blake M.A. Advisor	Social Science Analyst CSAP, SAMHSA 1 Choke Cherry Road, Room 4-1031 Rockville, MD 20857	SAMHSA/CSAP Performance and Technical Assistance Branch1 Choke Cherry Road, Room 4-1031Rockville, MD 20857 John.Park@samhsa.hhs.gov

LIST OF ATTACHMENTS

Attachment 1 - Division of State Program--Management Reporting Tool

Attachment 2 - Section A: SPF-Rx

Attachment 3 - Section B: PDO

4