Ss_b_pfs_mrt

2. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods

The SPF-PFS performance monitoring data collection will use a census approach to collecting process and programmatic data along with collecting aggregated existing archival data and data from survey samples for the outcomes measures.

Using a census approach, the targeted universe for the PEP-C PFS MRT is all Partnerships for Success (PFS) 2013 grantee Project Directors (n=16), all PFS 2014 grantee Project Directors (n=21), and all PFS 2015 grantee Project Directors (n=32), and grantee project directors from all future cohorts. As grantees have agreed to participate in performance monitoring data collection activities as a condition of funding, all 69 grantee Project Directors are expected to complete the Contact Information instrument, Quarterly Progress Reports, the Grantee Target Outcome Data instrument, and the Community-Level Outcome Data for Subrecipients instrument. The Substitute Data Source Request instrument is only required of grantees if they want to use an annual required measure in their community outcome reporting that is not pre-approved. The PFS Selected Grantee-Level Outcome Data instrument is voluntarily completed by any grantees who wish to provide grantee-level outcomes data when either none is available for their jurisdiction/tribe or to supplement state-level data provided by PEP-C.

A census of all PFS 2013, PFS 2014, and PFS 2015 grantee Project Directors is necessary as this data will be used by SAMHSA to monitor each program’s performance and grantees will also use it to track their ongoing implementation. In order to meet SAMHSA’s annual reporting requirements for GPRA and performance measures, and more frequent reporting requirements related to PFS Health Disparities activities, SAMHSA must obtain data from all grantees, which supports the need for a census approach. In addition, the SPF-PFS program grantees encompass a wide variety of organizational types and structures that are implementing a range of prevention interventions targeted to different populations and with various outcome goals. The variety between the programs makes it critical to the PEP-C PFS evaluation to capture the details of each program to be able to answer the evaluation questions and assess which program characteristics and mix of interventions are associated with better outcomes for particular demographic groups and types of communities.

While grantees will provide aggregated data into the PFS Selected Grantee-Level Outcome Data and Community-Level Outcome Data for Subrecipients instruments, all outcomes data will come from existing archival data (records of UAD- and PDM-related arrests, vehicle accidents, emergency room visits, and overdose or poisonings) and existing survey data covering such topics as UAD- and PDM-related consumption, perceptions of parental or peer disapproval, perceived risk or harm of use, and family communication. At the grantee level, the related survey estimates generally will come from the National Survey on Drug Use and Health (NSDUH; provided by PEP-C to grantees), with some data generated by the Youth Risk Behavior Survey (YRBS) or state, jurisdiction, or tribal surveys. Survey estimates at the community subrecipient level will generally come from state, jurisdiction or tribal surveys. NSDUH and YRBS utilize specified sample design procedures to develop national estimates and also provide estimates at the state and sometimes community (county, region, urban area) levels. Sampling designs vary among the state, jurisdiction, and tribal surveys. Prior to accepting estimates from those sources, SAMHSA and PEP-C will review the related survey and sampling designs to ensure adequate generalizability, validity, and reliability of the estimates.

NSDUH provides an example of the type of sampling utilized for the survey-based outcomes measures for the SPF PFS performance monitoring and cross-site evaluation. For NSDUH, the surveys are conducted using computer-assisted interviewing methods and a national sample size of 67,500, equally allocated across three age groups: persons aged 12 to 17, persons aged 18 to 25, and persons aged 26 or older (SAMHSA, 2012b). The NSDUH sampling design stratifies the sample by state and geographically partitioned regions within those states, and then randomly selected census blocks within those regions. To select units from the census block segments, NSDUH uses a random start point and interval-based (systematic) selection.

B.2. Information Collection Procedures

All instruments in the PEP-C PFS MRT are self-administered, web-based data collection tools completed through the Program Evaluation for Prevention Contract (PEP-C) online data collection system. All respondents are the grantee Project Directors or their staff. Before data collection for the SPF-PFS performance monitoring begins, respondents will be provided a unique log-in to enter the data system, where they will be required to create a password. Respondent email addresses for each login will be stored within the system so that automatic alerts and notifications can be sent.

Pending Office of Management and Budget (OMB) approval, the estimates in the PEP-C PFS MRT will be collected according to the schedule outlined in section A-12.

PEP-C developed user manuals for accessing and navigating the PEP-C PFS MRT online data collection system and question-by-question and frequently asked question (FAQ) guides to help respondents accurately complete the PEP-C PFS MRT instruments. Grantees and subrecipients will also be provided training webinars to: 1) walk through the PEP-C PFS MRT online data collection system, 2) review the Quarterly Progress Report and Community-Level Outcome Data for Subrecipients instrument and data collection procedures, and 3) cover specific sections of the instruments, such as reporting on leveraged funding. Within the online data collection system, all manuals, guides, and training webinars will be archived and accessible to respondents for reference at any time (see https://pep-c.rti.org/HERO/KB/PEP-C-KB/Default.htm#PFS/PFS Overview.htm%3FTocPath%3DPartnerships%2520for%2520Success%2520(PFS)%7C_____0).

Availability is important in any data collection system, especially one employed by grantee sites around the country, including multiple time zones and pacific jurisdictions. The online system will be maintained in an available state as much as possible to allow grantees to have access for entering data, as well as to give SAMHSA, grantees, and the PEP-C team access to reports.

Providing a robust system that is simple and easy to use across all areas is also critically important. To achieve this, PEP-C will implement user-friendly features across all functional areas, taking into account the needs of both SAMHSA and grantees. Additionally, every page of the online data system will have a “Help” or “Support” link located in the upper right corner, which will allow the respondent to access the following support resources:

1. Search the Knowledge Base. More comprehensive than a list of FAQs and more organized than a support forum, the Knowledge Base offers a “layered information” approach so that respondents can search by keyword and then drill down to view material at increasing levels of detail. It will be a curated and easily searchable source of information including items such as

• system documentation,

• user guides,

• policies and procedures,

• protocols,

• training materials, and

• FAQs.

1. Contact Us. Respondents may request assistance by calling a provided toll-free number, sending an email request, or submitting a technical assistance submission form as desired. The toll-free line will be routed to an email system that is checked regularly by members of the training and technical assistance team. Staff responding to technical assistance requests will be trained in use of the system and have ready access to the full Knowledge Base. Training and technical assistance team staff will monitor all submitted tickets to ensure timely response and resolution of technical assistance requests.

B.3. Methods to Maximize Response Rates

Grantees participate in all SPF-PFS performance monitoring data collection activities per the the Terms and conditions of the SPF-PFS grant award. The SPF-PFS evaluation team will employ a number of strategies to help ensure grantees participate with a 100% response rate.

As described above, the SPF-PFS evaluation will develop user manuals for accessing and navigating the PEP-C online data collection system and question-by-question and FAQ guides to help respondents accurately complete the PEP-C PFS MRT instruments. Grantees will also be provided training webinars to walk through the PEP-C PFS MRT online data collection system and to review data collection procedures. Within the online data collection system, all manuals, guides, and training webinars will be archived and accessible to respondents for reference at any time.

SAMHSA Project Officers will monitor the PEP-C PFS MRT and receive e-mail notifications when their grantees submit individual instruments such as Quarterly Progress Reports or Substitute Data Source Requests. Approximately one month after a data submission deadline PEP-C will provide Project Officers a list of past due instruments. SAMHSA Project Officers will then follow up with their grantees to ensure submission.

B.4. Test of Procedures

Three PEP-C staff completed the PEP-C PFS MRT instruments within a testing version of the system. These staff members have experience with SPF initiatives, including serving as local evaluators for SPF-SIG grantees.

The Contact Information instrument is estimated to take 1 hour to complete; this includes 0.5 hours to look up and compile information and 0.5 hours to complete the web instrument. Quarterly Progress Reports are estimated to take 3 hours to complete; this includes 2 hours to look up and compile information and 1 hours to complete the web instrument. The Grantee Target Outcome Data instrument is estimated to take 0.5 hours to complete; this includes 0.5 hours to complete the web instrument. The PFS Selected Grantee-Level Outcome Data instrument is estimated to take 1 hour to complete; this includes 0.5 hours to look up and compile information and 0.5 hours to complete the web instrument. The Community-Level Outcome Data for Subrecipients instrument is estimated to take 3 hours to complete; this includes 2 hours to look up and compile information and 1 hours to complete the web instrument. The Substitute Data Source Request instrument is estimated to take 1 hour to complete; this includes 0.5 hours to look up and compile information and 0.5 hours to complete the web instrument.

Each of the SPF-PFS grantees is a former SPF SIG grantee; thus they will all have experience completing instruments similar in procedure (e.g., entering data into an online data system), length, and content. Additionally, the SPF-PFS performance measure data collection used lessons learned from the SPF SIG evaluations to improve data collection procedures.

B.5. Statistical Consultants

The PEP-C contractor team comprises several experts who will be directly involved in data collection and statistical analysis. Also, contractor in-house experts will be consulted throughout the program on various statistical aspects of the design, methodological issues, and data analysis, including leveraged funding analysis. Finally, the PEP-C project has an External Steering Committee. Members of this External Steering Committee have already provided feedback on the performance monitoring instruments and the evaluation/analysis plan and will continue to provide advice and feedback through scheduled quarterly meetings and ad hoc e-mails as needed. Exhibit 11 provides details of these team members and advisors.

Exhibit 11. Statistical Consultants for the SPF-PFS Performance Measurement and the Program Evaluation for Prevention Contract (PEP-C)

Name & Role in Evaluation	Title & Address	Contact Information
PEP-C Evaluation Staff
Laura Dunlap, PhD PEP-C Cost Analysis Team Leader	Director Behavioral Health Economics Program RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 541–7310 Email: ljd@rti.org
Elvira Elek, PhD PEP-C Deputy Director	Research Public Health Analyst Public Health Policy Research RTI International 701 13th Street, NW, Suite 750 Washington, DC 20005	Telephone: (202) 728–2048 Email: eelek@rti.org
Phillip Graham, PhD PEP-C Project Director	Senior Research Public Health Analyst Crime, Violence, and Justice Program RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 485–7752 Email: pgraham@rti.org
Nilufer Isvan Senior Evaluator	Senior Research Fellow Human Service Research Institute (HSRI) 2336 Massachusetts Avenue  Cambridge, MA 02140	Telephone: (617) 844-2505 Email: nisvan@hsri.org
Gillian J. Leichtling Senior Evaluator	Senior Research Associate RMC Research Corporation 111 SW Columbia Street  Suite 1030  Portland, OR 97201-5883	Telephone: (503) 223-8248 x735 Email:GLeichtling@rmccorp.com
Antonio Morgan-Lopez, PhD PEP-C Analysis Team Leader	Senior Research Quantitative Psychologist Risk Behavior and Family Research RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 316–3436 Email: amorganlopez@rti.org
Scott Novak, PhD Senior Statistician	Senior Research Public Health Analyst Behavioral Health & Epidemiology RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Telephone: (919) 541–7129 Email: snovak@rti.org
Steve Sullivan ESC Task Team Leader	Senior Director Cloudburst Consulting Group, Inc. 8400 Corporate Drive, Suite 550 Landover, MD 20785-2238	Telephone: (301) 918-4400 Email: steven.sullivan@cloudburstgroup.com
External Steering Committee
Bethany Bray, PhD Methods/Statistics	Research Associate The Methodology Center The Pennsylvania State University 400 Calder Square II State College, PA 16801	Telephone: (814) 865-1225 Email: bcbray@psu.edu
William DeJong, PhD Evaluating Environmental Strategies	Professor Boston University School of Public Health Community Health Sciences 801 Mass Ave Crosstown Center Boston MA 02118	Telephone: (508) 954-0224 Email: wdejong@bu.edu
Brian Flay, DrPH Prevention Science	Professor Oregon State University College Of Public Health and Human Sciences 457 Waldo Hall Corvallis, OR 9733	Telephone: (541) 737-3837 Email: Brian.Flay@oregonstate.edu
Rick Harwood Economics, Cost Analyses	Director of Research and Program Applications National Association of State Alcohol and Drug Abuse Directors, Inc., (NASADAD) 1025 Connecticut Avenue NW, Suite 605 Washington, DC 20036	Telephone: (202) 293-0090, ext. 104 Email: rharwood@nasadad.org
Dottie Natal IT, Data Collection Systems	CEO Imagen Multimedia Corp	Email:dottie@imagenmm.com
Chris Ringwalt, DrPH Intervention Implementation and Dissemination	Public Health Senior Research Scientist Pacific Institute for Research and Evaluation 1516 E. Franklin Street, Suite 200 Chapel Hill, NC 27514-2812	Telephone: (919) 259-0643 Email: ringwalt@PIRE.org;
Government Project Officers
Sara Azimi-Boularian, PhD Contracting Officer’s Representative	Senior Public Health Analyst CSAP, SAMHSA 1 Choke Cherry Road, Room 4-1025 Rockville, MD 20857	LCDR, U.S. Public Health Service Public Health Analyst SAMHSA/CSAP Performance and Technical Assistance Branch1 Choke Cherry Road, Room 4-1031Rockville, MD 20857 sara.azimi-bolourian@samhsa.hhs.gov Phone (240) 276-2708
John Park, PhD Alternate Contracting Officer’s Representative	Senior Public Health Analyst CSAP, SAMHSA 1 Choke Cherry Road, Room 4-1031 Rockville, MD 20857	SAMHSA/CSAP Performance and Technical Assistance Branch1 Choke Cherry Road, Room 4-1031Rockville, MD 20857 John.Park@samhsa.hhs.gov

LIST OF ATTACHMENTS

Attachment 1—Management Reporting Tool