Supporting Statement A
Part A. Justification
The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Prevention (CSAP) requests OMB approval for the cross-site evaluation instruments for the Strategic Prevention Framework State Incentive Grant (SPF SIG) program for Cohorts IV and V. This reinstatement includes:
Submission of the instruments for the cross-site evaluation of the SPF SIG Cohorts IV and V: (a) the two-part Community-Level Instrument (CLI Parts I and II); and (b) the two Grantee-Level Instruments (GLI) — the GLI Infrastructure Instrument and the GLI Implementation Instrument.
Calculation of burden estimates for Cohorts IV and V, 24 and 10 grantees, respectively, for the 2-part CLI and the 2 GLIs. Per guidance from the previous OMB submission for the GLI and CLI Instruments (OMB No. 0930-0279), the number of items have been reduced, resulting in a reduced burden.
CSAP has funded two cross-site evaluations of the Strategic Prevention Framework State Incentive Grant (SPF SIG), one focused on Cohorts I and II and the other on Cohorts III, IV, and V. Collectively, these evaluations of the SPF SIG program provide an important opportunity to inform the prevention field on current practices and their association with community and state level outcomes.
The primary evaluation objective is to determine the impact of SPF SIG on building state prevention capacity and infrastructure, and preventing and reducing the onset and progression of substance abuse, as measured by the SAMHSA National Outcomes Measures (NOMs). Data collected at the grantee, community, and participant levels will provide information about process and system outcomes at the grantee, and community levels as well as context for analyzing population-level and participant-level outcome measures. The continuation of the use of the CLI Part I and Part II and the GLI Infrastructure and Implementation Instruments are included in this OMB review package and are the focus of this request.
The SPF SIG is a major SAMHSA grant program that supports an array of activities to help states and communities build a solid foundation for delivering and sustaining substance abuse prevention services that are effective in reducing the incidence and prevalence of substance use as well as improving risk and protective factors associated with substance use. CSAP provides funding to states/territories, Pacific jurisdictions, and tribal entities to implement the five steps of the strategic prevention framework (SPF), which are:
Step 1: Profile population needs, resources, and readiness to address the problems and gaps in service delivery;
Step 2: Mobilize and/or build capacity to address needs;
Step 3: Develop a comprehensive strategic plan;
Step 4: Implement evidence-based prevention programs, policies, practices and infrastructure development activities; and
Step 5: Monitor process, evaluate effectiveness, sustain effective programs/activities, and improve or replace those that fail.
Cohort IV, funded by CSAP in FY 2009, included 24 grantees — 13 states, 5 jurisdictions, and 6 tribal entities. Cohort V included 10 grantees — 3 states and 7 tribal entities — and was funded in FY 2010. For the purposes of this document, the word grantee will refer to all active funded states/territories, Pacific jurisdictions, and tribal nations. SPF SIG grantees select and fund their own subrecipients; the selection of these subrecipients is not guided by CSAP.
Information on both cross-site evaluations is presented below as context for understanding SAMHSA’s request to continue data collection with the previously approved instruments for Cohorts IV and V. This reinstatement will allow the completion of data collection for the entirety of the SPF SIG program.
A1b1. Cohorts I and II Cross-site Evaluation
The National Institute on Drug Abuse (NIDA) provided support to CSAP to evaluate the impact of the SPF SIG project for Cohorts I and II. Specifically, data have been collected from the 26 states and territories receiving grants initiated in 2004 (Cohort I sites) and 2005 (Cohort II sites) and up to 32 non-Cohort I and II grantee states and territories that served as a comparison group. Data from the CLI (Parts I and II) are used to determine the impact of the SPF SIG on all of the NOMs domains related to prevention (i.e., Abstinence, Education/Employment, Crime and Criminal Justice, Access/Capacity, Retention, Cost Efficiency and Use of Evidence-based Practices). The evaluation has also measured: the effect of establishing and sustaining infrastructure at the state and community-levels to allow for data-based decision-making; the implementation of the SPF; and the environmental factors that affect substance abuse.
A1b2. Cohorts III, IV, and V Cross-site Evaluation
Data have been gathered from the 16 states, Pacific jurisdictions, and tribal territories receiving grants in FY 2006 (Cohort III), the 25 Cohort IV grantees funded in FY 2009, and the 10 Cohort V grantees funded in FY 2010. Data collection for Cohort III has ended; therefore, this data collection request pertains only to Cohorts IV and V. The Cohort III, IV, and V cross-site evaluation team is implementing a multi-level evaluation design encompassing process and outcome data collection at the grantee, community, and participant levels. Data from the two revised GLIs and the revised two-part CLI specifically measure: the effect of establishing and sustaining infrastructure at the grantee and community-levels to allow for data-based decision-making; the implementation of the SPF; and the environmental factors that affect substance abuse. Recognizing that all grantees have prevention activities already underway, the collection of baseline data using these instruments will be used in the analysis to account for pre-SPF SIG activities in estimating the effects of SPF SIG-initiated activities. The process components of the SPF SIG evaluation allow the evaluators to disentangle the effects of various project-related activities and help identify which program and policy elements are effective, under what conditions, and with which target populations.
All SAMHSA grantees are required to collect and report certain data so that the Agency can meet its obligations under the Government Performance and Results Modernization Act of 2010 (GPRAMA). Because SAMHSA programs are often client/participant-level interventions that differ dramatically from one group to another, extracting a sample from the population receiving a specific intervention may be insufficient for evaluating these programs. Populations from each group may not be similar and therefore, it would be inappropriate to infer generalizable findings about the successes or failures of a program due to the uniqueness of each group. Within populations, the number of participants/clients may be too small to extract a representative sample of a population; extracting such a sample could lead to large variance within the sample and could result in inaccurate findings and conclusions about these programs. In order to evaluate programs that differ across groups of clients/participants, it is important to gather data from all participants in order to draw statistically accurate conclusions about how the programs are performing and to identify which components of the programs may contribute to their success and positive client/participant outcomes. For this OMB approval request, SPF-SIG cross-site evaluation participants are the grantees and grantee sub-recipient communities.
The SPF SIG grant program is a major investment by the Federal Government to improve state substance abuse prevention systems, and enhance the quality of prevention programs, primarily through the implementation of the SPF process. The goal of this initiative is to provide states, Pacific jurisdictions, tribal entities, and the communities within them with the tools necessary to develop an effective prevention system with attention to the processes, directions, goals, expectations, and accountabilities necessary for functionality. SAMHSA/CSAP needs to collect information over the course of the remaining grant period to monitor the progress of the SPF SIG initiative, as well as to maintain continuity in the data collection and analytic protocols (for Cohorts IV and V) that have been previously approved by OMB and used by earlier cohorts of SPF SIG grantees in order to complete the data collection for the SPFSIG program as cohort V reaches an end. CSAP will use the findings from the cross-site evaluations to assess the implementation of the SPF, the infrastructure development and implementation at the grantee and community level, and the impact of these activities on community and state level outcomes. Without these data, the extended impact of the SPF process across various types of communities would remain unknown. Additionally, findings from this cross site evaluation will assist CSAP policymakers and program developers as they design and implement future initiatives.
Information on the two GLIs and the CLI Part I and Part II is presented below. SAMHSA is requesting to use the two GLIs and the CLI Parts I and II surveys to complete data collection for Cohorts IV and V.
Two web-based surveys, the GLI Infrastructure Instrument and the GLI Implementation Instrument, were developed for assessing grantee-level efforts and progress (see Attachments A1a and A1b). More specifically, the two GLIs were developed to: a) assess pre-SPF capacity and resources; b) demonstrate processes for leveraging, realigning, and redirecting resources; c) describe how data are managed for performance monitoring; d) provide indicators of movement through the different steps of the SPF; and e) gather information to explain the relationship between SPF implementation and targeted outcomes.
The GLI Infrastructure Instrument is a web based survey that is completed by the grantee when the award is made and again during the fifth year of the SPF. Baseline data after award has already been collected.
The GLI Implementation Instrument is first completed immediately after CSAP’s Division of State Programs approves the grantee’s strategic plan. The instrument is administered again approximately 36 months after approval of the strategic plan. The GLI Implementation Instrument collects information specific to the execution of the five steps that comprise the SPF and is limited to assessing actions that have occurred as a direct result of the SPF.
Both the GLI Infrastructure Instrument and the GLI Implementation Instrument are completed by the grantees’ evaluators twice over the life of the SPF SIG award. Detailed administration Question by Question guides were developed to help improve the reliability and validity of the data collected, thereby ensuring high-quality data with which to evaluate grantee-level progress.
The CLI is a two part, web-based instrument for capturing information about SPF SIG implementation at the community sub-recipient level (communities that receive SPF SIG funds from the Cohort IV and V grantees) (see Attachments A1c and A1d).
Part 1 of the CLI gathers information on the communities’ progress in implementing the five SPF SIG steps and efforts taken to ensure cultural competency throughout the SPF SIG process. Sub-recipient communities receiving SPF SIG awards from the grantees are required to complete the CLI Part I annually. The CLI Part II captures data on the specific prevention intervention(s) implemented at the community level. A CLI Part II is completed for each prevention intervention strategy implemented during the specified reporting period. Information collected on each strategy includes date of implementation, numbers of groups and participants served frequency of activities, and gender, age, race, and ethnicity of the population served/affected. Sub-recipient communities’ partners receiving SPF SIG awards are required to update the CLI Part II every six months.
Data from the CLI (Parts I and II) will allow CSAP to assess the progress of the communities in their implementation of both the SPF and prevention-related interventions funded under the initiative. These data may also be used to assess obstacles to the implementation of the SPF and prevention-related interventions and facilitate mid-course corrections for communities experiencing implementation difficulties. Without these data, it would be impossible to determine whether the SPF SIG initiative had an effect on changes in community- and participant-level NOMs or to identify which components of the SPF process were responsible for the observed changes.
Information technology has been used to minimize respondent burden for Cohorts IV and V in the SPF SIG Cross-site Evaluation. The GLI Implementation and Infrastructure Instruments and the CLI (Parts I and II) are web-based surveys and will be completed online through SAMHSA/CSAP’s Prevention Management and Reporting System (PMRTS). Web-based administration of the instruments increases the efficiency of data submission and improves data quality. Additionally, completion of the instruments online reduces the burden on grantees and communities, as some items will be pre-filled based on information from the initial submission. In the CLI, some items in Part II will be pre-filled with information from Part I of the instrument.
The PMRTS is maintained by CSAP’s Data Collection, Analyses, and Reporting (DCAR). Data entered online by grantees will be periodically extracted and analyzed by the DCAR. Grantees can access the data they entered online by downloading, in spreadsheet form, the raw data they have entered as soon as it is submitted.
Finally, technology is used to facilitate communication and provide updates to SPF SIG personnel. Through a SPF SIG listserv, grantee evaluators, project directors, coordinators, and other key staff will have the opportunity to exchange valuable advice; find guidance and resource materials; and receive announcements and clarifications from CSAP, other SPF SIG grantees, and the cross-site evaluation team. In addition to the listserv, the cross-site evaluation team will also send electronic copies of guidance and resource materials via email and CD to SPF SIG grantees upon request.
The information being collected for the cross-site evaluations for SPF SIG Cohorts IV and V is specific to the program and is not available elsewhere.
The primary entities for the Cohort IV and V studies are states, jurisdictions, and tribal territories and the communities funded within these entities. Because grantees and funded communities involve government or tribal agencies, universities, hospitals, or other large organizations, the evaluation will have no significant economic impact on small entities or small businesses.
The Cross-site Evaluation of the SPF SIG program provides an important opportunity for the field of substance abuse prevention. Not conducting this data collection would significantly impede SAMHSA’s ability to assess the implementation of the SPF SIG process and measure improvements in: strategic planning; capacity and infrastructure development; data-driven decision making; and implementation of evidence-based prevention programs. Less frequent data collection would also impede CSAP’s ability to track changes in both substance use and substance use-related problems. If data collection is not continued, SAMHSA will have an incomplete dataset for the entire program and will lose critical information from tribes who comprise a significant portion of the Cohort IV and V grantees.
The specific schedule of data collection is dependent upon the data being collected. The GLI Infrastructure and the GLI Implementation Instruments are completed twice over the grant period (only once remaining); the CLI (Part I) is collected annually, and the CLI (Part II) is collected every 6 months to assure completeness and accuracy of the data, to assist in monitoring the grantees’ work, to enable report on progress, and to reduce the need to enter years of data at one time. Data from multiple time periods are essential for monitoring the progress of states and communities as they implement the SPF and deliver evidence-based strategies as well as for identifying communities that are experiencing obstacles and may need technical assistance.
The proposed continuation of data collection for Cohorts IV and V fully complies with all guidelines of 5 CFR 1320.5.
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on September 3, 2013 (Vol. 78, Page 54263).
The evaluation design and data analysis plan for this SPF SIG Cross-site evaluation have not changed since the previous OMB approval. The instruments have been revised in consultation with the CSAP SPF SIG Project Officer, DCAR Project Officer, and DCAR team members. The following individuals consulted on these instrument revisions:
Donna Durant Atkinson, Ph.D.
DCAR Project Director
Westat
Rockville, MD
Ilene Klein, Ph.D.
DCAR Associate Project Director
Westat
Rockville, MD
Robert Orwin, Ph.D.
Senior Study Director
Westat
Rockville, MD 20850
Leanne Candura
DACCC Project Manager
Human Services Research Institute
Cambridge, MA
Roy Gabriel
Senior Research Associate
RMC
Portland, OR
Jane Grover
Research Associate
RMC
Portland, OR
Nilufer Isvan, Ph.D.
DACCC Data Analysis Team Lead
Human Services Research Institute
Cambridge, MA
Virginia Mulkern, Ph.D.
DACCC Project Director
Human Services Research Institute
Cambridge, MA
Kelly Vander Ley
Senior Research Associate
RMC
Portland, OR
There is no payment to any respondent.
All information gathered through the administration of the GLI Infrastructure and Implementation Instruments and CLI (Parts I and II) focus on organizational activities undertaken as part of the SPF SIG program, rather than information about individuals. However, all respondents to the GLI and CLI Instruments are required to register with the online survey site where the instrument is completed. As part of this registration, it is necessary to obtain identifying information about these individuals (i.e., name, email address, organizational affiliation, and title/position). This information is used for the creation of a user profile and every attempt is made to keep this information private. After participants have registered with the website, they are provided with a User ID and temporary password and are prompted to create their own secure password. This ensures participants’ survey responses remain private. Additionally, no survey responses will be associated with a specific individual in any reports written using this data.
The GLI and CLI respondents are also provided with the following information before completing the instrument: the purpose of the instrument; an explanation of how the data will be used; notification that responses will be kept private to the extent possible; notification that individual names and positions will not be connected with any responses in any reports prepared from the data; and, notification that all individual responses will be aggregated with the responses of others in all reports based on the data obtained using the GLI and CLI surveys.
No information of a sensitive nature will be directly collected as part of the GLI and CLI.
The estimated annualized hour burden for continuing the Cross-site Evaluations of SPF SIG for Cohorts IV and V is presented in Table 1. Estimates of total and annualized reporting for Cohorts IV and V grantee-level and community-level personnel and participants are based on the completion of the two Grantee-Level Instruments (GLI), and the two-part Community-Level Instrument (CLI). The estimates have been maintained since the previous package and no adjustments have been warranted thus far.
Estimated burden of the GLI Infrastructure and GLI Implementation Instruments is based on the current 24 grantees funded in Cohort IV and 10 grantees funded in Cohort V, all of whom have completed these instruments once at baseline and will complete these instruments one more time during the three year reporting period. Estimated burden of the CLI Part I and CLI Part II assumes an average of 15 communities per grantee, annual completion of the CLI Part I, two instrument updates per year for the CLI Part II, and an average of three distinct prevention intervention strategies implemented by each community during a 6-month period.
Table 1. Estimates of Annualized Hour and Cost Burden to Respondents for Cohorts IV and V
Instrument Type |
Respon-dent |
No. of Respon-dents |
No. of Responses per Respondent |
Total No. of Responses |
Burden per Response (Hrs.) |
Total Burden (Hrs.) |
1Hourly Wage Cost |
Total Cost |
Grantee-Level Burden |
||||||||
GLI Infrastructure Instrument |
Grantee |
34 |
1 |
34 |
2.22 |
75.48 |
$42.59 |
$3,215 |
GLI Implementation Instrument |
Grantee |
34 |
1 |
34 |
1.95 |
66.30 |
$42.59 |
$2,824 |
CLI Part I, 1—20: Community Contact Information —Updates |
Grantee |
34 |
3 |
102 |
0.25 |
25.50 |
$42.59 |
$1,086 |
Total Grantee-Level Burden |
Grantee |
34 |
|
170 |
|
167.28 |
|
$7,125 |
Community-Level |
||||||||
CLI Part I, 21–178: Community SPF Activities –Updates |
Community |
510 |
3 |
1,530 |
0.75 |
1,147.50 |
$32.53 |
$37,328 |
CLI Part II —Updates |
Community |
510 |
18 |
9,180 |
0.50 |
4,590.00 |
$32.53 |
$149,313 |
Total Community-Level Burden |
Community |
510 |
|
10,710 |
|
5,737.50 |
|
$186,641 |
There are no capital/startup costs or operational/maintenance of services costs associated with this project.
The estimated annual cost to the Federal government of collecting cross-site evaluation data for Cohorts IV and V is $529,104. This estimated cost to the Federal government is based on the government contracted cost of the data collection and related evaluation activities along with the personnel cost of government employees involved in oversight and/or analysis. The annual cost to the government for this task within the DCAR contract, (which collects and analyzes the data for which OMB approval is currently being requested), is $500,000. In addition, the annual cost to the government includes 25 percent time for a GS-14 CSAP project officer ($29,104). Thus, the total annual cost to the government for this Cross-site evaluation is $529,104.
This is a reinstatement of the SPFSIG data collection request and therefore is a new collection. There is no change in burden as it is a new collection.
The analyses, tabulation, and publication plans are presented below, along with the evaluation schedule.
Research Questions
Eight research questions guide the SPF SIG cross-site evaluation. They are derived from those in the original design for the Cohorts I and II evaluation and have here been further specified and reinterpreted to reflect both the differences in the later cohorts and the history and results of the Cohorts I and II cross-site evaluation. For example Questions 4a and 4b referring to participant-level improvements are specific to Cohorts III, IV, and V, in which some communities submitted participant-level data. The eight broad questions address whether observed changes in conditions or events in states and communities can be attributed to the SPF SIG grants and programmatic interventions.
1a. Did SPF funding improve grantee-wide performance on NOMs and other outcomes, (e.g., did NOMs improve over the course of the SPF SIG grants)?
1b. What accounted for the variation in NOMs and other outcomes performance across SPF grantees?
2a. Within grantees, did SPF funding lead to community-level improvement on NOMs and other outcomes?
2b. Within grantees, what accounted for variation in NOMs and other outcomes performance across funded communities?
3a. Across grantees, did SPF funding lead to community-level improvement on NOMs and other outcomes?
3b. Across grantees, what accounted for variation in NOMs and other outcomes performance across funded communities?
4a. Did SPF funding lead to participant-level improvement on NOMs and other outcomes?
4b. What accounted for variation in participant-level NOMs and other outcomes performance across funded grantees and communities?
In addition to these eight outcome research questions, which are the central focus of the SPF SIG evaluation, the evaluation design includes a number of research questions related to program process. Data collection related to these questions provides information necessary to interpret the evaluation outcomes. The process questions focus on interpreting the effects of project-related activities; identifying apparently effective program and policy elements including the conditions in which programs appeared to be effective and the populations for whom programs were effective. They also assess some of the contextual factors in states and communities that may effect SPF SIG outcomes. Some examples of the process-related research questions: What changes in allocation of resources for substance abuse prevention programs occurred at the grantee and community-levels? What grantee and community-level mobilization and capacity building activities took place? Has cultural competence been integrated into prevention programs, policies and practices in states, jurisdictions, and tribal entities? To what extent has the prevention infrastructure improved? To what extent were the prevention programs selected by community partners evidence-based? To what extent were the selected programs implemented with fidelity? Is implementation fidelity related to changes in outcomes?
Plans for Tabulation and Analysis
The answers to the evaluation questions will be answered through a number of tabulations and analytical comparisons, including displays of changes over time after grantees are funded and implement SPF steps, and comparisons among grantees and communities, making use of the different data sources in some of the planned analyses as shown in Table 2.
Table 2. Evaluation Questions, Comparisons, Data Sources, and Analyses
Questions |
Comparisons |
Data Sources |
Analyses |
1a. Did SPF funding improve grantee-wide performance on NOMs and other outcomes, that is, did NOMs improve over the course of the SPF SIG grants? |
Change in outcomes before and after SPF funding; |
SEDS Relevant DCAR data Census data Other federal data sources (non-substance abuse related) |
Statistical tests of differences between (pooled) pre and post outcome values; longitudinal analyses as available data series permit. |
1b. What accounted for variation in NOM and other outcomes performance across SPF grantees? |
Comparison with other SPF SIG grantees, within and across cohorts |
Periodic updates of federal data sources |
Cross-tabulations and correlations of outcomes with state characteristics |
Within grantees, did SPF funding lead to community-level improvement on NOMs and other outcomes? |
Changes in outcomes over time and among cohorts |
CLI, community outcome data sources |
Statistical tests of differences between (pooled) pre and post outcome values; longitudinal analyses as available data series permit. |
2b. Within SPF states, what accounted for variation in NOM and other outcomes performance across funded communities? |
Comparisons among communities, their characteristics, and whether they target outcomes |
CLI, NOMs Community outcomes |
Cross-tabulations and correlations of outcomes with state characteristics; HLM when data permit |
3a. Across grantees, did SPF funding lead to community-level improvement on NOMs and other outcomes? |
Comparisons of communities with similar characteristics and available data across states |
CLI, NOMs and other community outcomes |
Pooling community outcomes, comparisons over time pre and post SPF implementation |
3b. Across SPF states, what accounted for variation in NOM and other outcomes performance across funded communities? |
Comparisons of outcomes across states; cohorts |
GLI, NOMs |
Cross-tabulations and correlations of community characteristics and outcomes |
Did SPF funding lead to participant-level improvement on NOMs and other outcomes? |
Changes over time and among communities in submitted PLI* data |
CLI, PLI, NOMs and other community outcomes |
|
4b. What accounted for variation in participant-level NOMs and other outcomes performance across funded grantees and communities? |
Comparisons change over time and across interventions targeting the same outcomes, taking into account both community and individual characteristics |
CLI, PLI |
Crosstabulations and Correlations of program and community characteristics with changes in outcomes; HLM if individual data permit |
* PLI refers to the Participant-level Instrument, this instrument is included in a separate OMB submission (OMB No. 0930-0230 expiration 2/28/13)
Tabulations and analyses will include qualitative and quantitative analyses of data collected and graphic and tabular displays of the key findings. The community-level epidemiological data and participant-level NOMs will be used to answer questions 1a, 2a, 3a, and 4a listed in section A16. The GLI Infrastructure Instrument, the GLI Implementation Instrument, and the CLI (Parts I and II) will be used to gather data to address the effect of the SPF SIG initiative on grantee- and community-level system outcomes and community- and participant-level NOMs outcomes. Specifically, the four questions address the moderators and mediators of outcome variation across SPF SIG funded grantees and their subrecipient communities. Data from the CLI (Parts I and II) in particular will be used to identify similarities and differences in the way SPF SIG is being implemented across grantees and their subrecipient communities including the specific prevention-intervention programs and strategy-types being implemented and the populations served.
Data reduction, scoring and scaling. The first phase of data analysis will consist of reviewing, coding, scoring, and scaling of responses within each domain with the goal of reducing the data to a set of reliable scales that will be used in subsequent analyses. For each domain, a summary score or index will be developed. Attention will be given to developing reliable and valid measures of the constructs in each domain, including assessment of inter-coder reliabilities and relationships among both the items within potential summary scores and between the domains. In addition to our scoring (or quantifying) of the responses, we will use the open-ended responses to provide contextual qualitative data to support and enrich the quantitative scores. The qualitative information will be included as narrative in our reports to help explain the scores regarding the prevention infrastructure and sustainability domains and to provide concrete illustrations of how these domain scores manifest themselves within each state.
Outcome analyses. The data gathered will be used to conduct a variety of quantitative and qualitative analyses related to the eight outcome evaluation questions and the process-related research questions. As part of these analyses, the distributional characteristics of the data, as well as the baseline differences among the groups being compared, will be assessed. Then, within-grantee and across-grantee outcome analyses will be conducted using multilevel statistical modeling methods that account for the “nested” nature of the data (i.e., the data are not independent, they are nested within the communities and within the states).
Basic statistical tests of differences over time and between states and communities being compared will be conducted, including z-tests of differences between survey summary statistics and chi-square analyses of state and community characteristics. Some statistical modeling methods will be performed using Hierarchical Linear Modeling (HLM) Version 6 (Raudenbush et al., 2002)0. The coefficients estimated by the HLM model are applicable to a hierarchical data structure with up to three levels of random variation. In our case, the three levels will be: 1) State, 2) community, and 3) time. HLM also accommodates sampling weights in both linear and nonlinear models. This is relevant to our analysis because 1) most of the NOMs and other outcomes will not meet normality assumptions and therefore require nonlinear models, and 2) states will contribute unequal numbers of communities and population sizes to the cross-site database. Therefore, inverse weighting by these inequalities at the appropriate level will increase the generalizability of the findings. Note that the two GLIs will support analyses of variation at level 3, the CLI will support analyses of variation at level 2, and both will support analyses of variation at level 1 through repeated administrations over time.
One system-level outcome of interest will be changes in prevention infrastructure over time. Data from the two GLIs and CLI (Part I) will be used to measure State systems infrastructure. This includes changes in planning capacity, training capacity, and support for the implementation of evidence-based practices. Thus, data from these instruments will serve as outcome data for grantee systems change and as mediators of changes in population- and participant-level consumption and consequence outcomes. The construct of prevention infrastructure is, however, too complex to be captured by a single summary statistic. In addition to the global index, therefore, indexes will also be developed based on specific infrastructure domains (planning, workforce development, etc.). Analyses of these indexes will help show whether some domains appear more critical to outcomes than others. Other analyses will focus on the relationship between SPF implementation and observed variation in outcomes across grantees.
Community-level analyses conducted with the data gathered from the CLI (Parts I and II) will aim to identify characteristics of community-level interventions that are most effective in producing desired population and participant-level outcomes. These analyses will focus on: 1) comparisons of community-level outcomes from funded communities across multiple states with state and national data; and 2) comparisons of systems-level outcomes across the funded communities, exploring the relationships between different types of community approaches, target populations, levels of implementation and fidelity, mix of strategy types, and aggregated community-and participant-level outcomes. Systems-level outcomes to be included in these analyses include changes in the number and operation of coalitions as assessed by the CLI (Parts I and II). Population outcomes will focus on changes in consumption and consequences NOMs and other outcomes over time.
Evaluation Timeline
Table 3 shows the schedule for the remaining cross-site evaluation activities for Cohorts IV and V.
Table 3. SPF SIG Evaluation Time Schedule
Data Collection, Analysis, and Reporting Timeline Cohorts IV and V |
|
Evaluation Activity |
Date |
GLI Infrastructure Data Collection |
November 2014- IV November 2015 –V |
GLI SPF Implementation Data Collection |
November 2014 - IV November 2015 – V |
CLI (Part 1) Data Collection* |
May 2013 – IV & V or November 2013 – IV & V May 2014 – IV & V or November 2014 – IV & V May 2015 – V or November 2015 – V |
CLI (Part 2) Data Collection |
May 2013 – IV & V November 2013 – IV & V May 2014 – IV & V November 2014 – IV & V May 2015 – V November 2015 – V |
Annual Evaluation Reports |
August 2013 – IV & V August 2014 - IV & V August 2015 – IV & V August 2016 – V |
* Grantees submit CLI (Part I) data in either May or November within a single fiscal year, not in both reporting periods.
The cross-site team will provide CSAP with the reports necessary to determine, in consultation with the relevant federal staff, if the overall quality and quantity of the evaluation data are adequate for public release. Once it is determined that the data will be released, the cross-site team will perform a disclosure analysis of the data to detect both direct and indirect identifiers within the data, as well as the most likely sources for a possible breach of privacy. Based on the standards published by the Standing Review Committee for Disclosure Analysis at the Inter-University Consortium for Political and Social Research (ICPSR), the cross-site team will recommend a plan for each detected identifier. Once the disclosure plan is approved by CSAP, the cross-site team will produce a public use data file in compliance with ICPSR recommendations for public use data. Data will also be made available to the prevention community through the DITIC.
The expiration date for OMB approval will be displayed on all approved instruments.
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
Attachments:
A1a: SPF SIG IV and V GLI Implementation
A1b: SPF SIG IV and V GLI Infrastructure
A1c: SPF SIG IV and V CLI Part I
A1d: SPF SIG IV and V CLI Part II
0 Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods, Second Edition. Newbury Park, CA: Sage.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | RMC Basic Report & Proposal Template |
| Author | RMC Research Corporation |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |