Cross-Site Evaluation of the Minority Substance Abuse/HIV Prevention Program
OMB Supporting Statement
Direct service grantees use a census approach and administer the NOMs form to all participants receiving services covered by grant funds. SAMHSA/CSAP’s response universe for the NOMs renewal includes all active grantees (with relevant participants and objectives) and those initially funded at the end of FY 2011 and beyond that provide direct services to participants. In order to ensure accountability for the spending of federal funds, CSAP has employed the use of these data as a performance management tool to ensure that grantees are meeting the goals and objectives of the program. Data are used to monitor performance throughout the grant period. The Public Health Service Act Sec. 501 [290aa] (d) (13) with respect to grant Programs authorized under this title, assure that-all grants that are awarded for the provision of services are subject to performance and outcome data collections. SAMHSA has operationalized these requirements to indicate the need for data to be collected on all clients served.
Common Measures - Youth and Adult Questionnaires (Completed by Program Clients)
The study uses a common protocol for collecting program and participant/client level data and submitting it to CSAP via an online, web-based data entry system (PMRTS). The DCAR Technical Assistance Center also has been established to assist grantees with online data entry. These data provide the basis for categorizing programs on characteristics of program design and implementation (e.g., one-on-one, group format). These are described below:
Youth and Adult Outcome Questionnaires. Two common questionnaires will be administered to program participants/clients. Each questionnaire has three versions, as follows:
Youth Questionnaire: For persons aged between 12 and 17 - Baseline, Exit, and Follow-up for participants in interventions lasting 30 days or longer, baseline and exit for participants in interventions lasting between 2 and 29 days and at exit only for participants in single session interventions.
Adult Questionnaire: For persons aged 18 and older - Baseline, Exit, and Follow-up for individuals participating in 30 day or longer interventions, baseline and exit for participants in interventions lasting between 2 and 29 days and at exit only for participants in single session interventions.
For all common measures, administration guides have been prepared to assist program sites with implementation.
The major constructs for the youth outcome questionnaire include demographics, 30-day substance use, age of first use, disapproval of substance use, perception of risk, substance use, family cohesion, perception of peer behavior, sexual behavior, school connectedness, knowledge of HIV, and health care (See Attachment 1 for a copy of the Youth Baseline Questionnaire and the Administration Guide.)
The major constructs for the adult questionnaire include demographics, employment, 30-day substance use, age of first use, disapproval of substance use, perception of risk, substance use, age of first use, family cohesion, sexual behavior, knowledge of HIV, health care. See Attachment 2 for a copy of the Adult Baseline Questionnaire and the Administration Guide.)
The GPRA and NOMs measures have already been approved by OMB (OMB No. 0930-0230), and the remaining HIV-related questions have been approved under OMB No.: 0930-0298. The youth questionnaire contains 128 questions, of which 28 relate to HIV/AIDS and the adult questionnaire contains 122 items, 47 of which relate to HIV/AIDS. The planned analysis will assess the degree to which program strategies reduce SA/HIV/AIDS and increase SA, HIV/AIDS protective factors among at-risk populations. Two new questions have been added to address SAMHSA’s need to collect information on binge drinking behavior, not covered under any prior OMB package. Seven new questions have been added to address SAMHSA’ s need to collect information on military families. These seven new questions include five for the adult questionnaire and two for the youth questionnaire.
The binge drinking questions are:
During the past 30 days, on how many days did you have 4 or more drinks on the same occasion? [By 'occasion,' we mean at the same time or within a couple of hours of each other]
and
During the past 30 days, on how many days did you have 5 or more drinks on the same occasion? [By 'occasion,' we mean at the same time or within a couple of hours of each other].
The military questions are:
Adult Questionnaire
Have you ever served in the Armed Forces, in the Reserves, or the National Guard?
No
Yes, in the Armed Forces
Yes, in the Reserves
Yes, in the National Guard
Are you currently on active duty in the Armed Forces, in the Reserves, or the National Guard]?
Yes, in the Armed Forces
Yes, in the Reserves
Yes, in the National Guard
No, separated or retired from Armed Forces, Reserves, or National Guard
Have you ever been deployed to a combat zone [select all that apply?
Never deployed
Iraq or Afghanistan (e.g., Operation Enduring Freedom/Operational Iraqi Freedom/Operation New Dawn)
Persian Gulf (Operation Desert Shield or Desert Storm)
Vietnam/Southeast Asia
Korea
WWII
Deployed to a combat zone not listed above (e.g., Somalia, Bosnia, Kosovo)
Is anyone in your family or someone close to you on active duty in the Armed Forces, in the Reserves, or in the National Guard, or separated or retired from Armed Forces, Reserves, or the National Guard.
Yes
Yes, more than one
No
If yes (answer for up to six people): what is the relationship of that person (Service Member) to you:
Mother/Father
Brother/Sister
Spouse/Partner
Child
Other, Specify___________
Youth Questionnaire
Is anyone in your family or someone close to you on active duty in the Armed Forces, in the Reserves, or the National Guard, or separated or retired from Armed Forces, Reserves, or the National Guard?
Yes
Yes, more than one
No
If yes (answer for up to six people): What is the relationship of that person (Service Member) to you:
Mother/Father
Brother/Sister
Spouse/Partner
Child
Other, Specify___________
The information obtained from these questionnaires will generate data to determine the effectiveness of the program in reducing SA, injection drug use, and high-risk sexual behaviors as well as increasing participant knowledge. Pilot tests have been conducted to assist in controlling respondent burden and ensuring the user-relevance of questions.
Data Collection
Time Points (Efficacy) for Data Administration: The common design for participants in interventions lasting 30 days or longer includes assessments at baseline, program exit, and three to six months post-exit (follow-up). The common questionnaires will be administered to all 30-day intervention (program participants) youth and adults at baseline (first data collection point), program exit (second data collection point), and follow-up (third data collection point).For participants in interventions lasting between 2 and 29 days questionnaires will be administered at baseline and exit. For single session interventions an exit only questionnaire will be administered.
Baseline Questionnaires: This questionnaire (first data collection point), should be administered within 30 days of intake or before core program services begin.
Exit Questionnaires: This questionnaire (second data collection point), should be administered again within 10 days post program exit or after core program services have ceased.
Follow-up Questionnaires: This questionnaire (third data collection point), should be administered within 30 days of the planned follow-up (post-exit). Most study sites have a planned follow-up administration for three to six months after program exit.
The common instrument assessment schedule is depicted in Table 5.
Table 10. Common Instrumentation Assessment Schedule
Intervention Duration |
Instrument |
Baseline |
Exit |
Follow-up |
Single Session Intervention (1day or less) |
Youth and Adult Questionnaire |
|
X |
|
Multiple Session Brief Intervention (Between 2 and 29 days) |
Youth and Adult Questionnaire |
X |
X |
|
Multiple Session Long Intervention (30 days or more) |
Youth and Adult Questionnaire |
X |
X |
X |
This time series analysis is depicted graphically below:
01 X 02 03
Where: 01= Baseline - First Data Collection Point
X = Program Prevention Intervention Activities
02 = Exit - Second Data Collection Point
03 = Follow-Up – Third Data Collection Point
Respondent burden, and intrusiveness have been minimized to be consistent with cross-site analytical objectives. Procedures are employed to safeguard the privacy and security of participant responses. Every effort has been made to coordinate cross-site data collection with local data collection efforts to minimize respondent burden. Pilot study results indicate that the questionnaires require an administration time of approximately 50 minutes.
Data Management
Storage System: Over the life of this initiative, each grantee will be collecting information that must be documented and organized. Each local evaluation team or the person responsible for data management will be required to store:
Individual questionnaires (Youth and Adult and Individual and Group Dosage) until they are entered or uploaded into PMRTS
Consent forms
Tracking forms for each study participant.
Before initiating data collection, each site is responsible for setting up a filing and storage system that will accommodate these needs. They also are responsible for establishing procedures to safeguard the privacy and security of survey responses that they gather. As described in section A.10, as part of the grant application process, SAMHSA/CSAP requires grantees to describe the procedures they will use to ensure the privacy and protection of participant data. These proposals are reviewed for assurances of participant protections by an independent peer review committee, and any identified issues about inadequacies are resolved prior to funding.
Web-Based Data Entry Upload System: Prevention Management Reporting and Training System (PMRTS), an online data entry system which provides prevention information, data collection tools, documents, data entry functions, and access to reporting statistics and tracking. This system is a secure database environment in accordance with the security guidelines defined by the Federal Information Security Management Act (FISMA) and NIST Special Publication 800-53 for Moderate information system security.
All of the HIV instruments can be found in the “Tools” section of this website. Common questionnaires are available in both Microsoft Word and PDF format for individual grant sites to download and make copies for administration to clients or participants. Site evaluators or data collectors are expected to enter client or participant responses to questionnaires through the PMRTS website. However, no personal identifying information is entered into the PMRTS (e.g., names, social security numbers). Sites will also be able to upload response databases through PMRTS that use the appropriate variable/value numbering (questionnaire codebooks are also available on the “Tools” section of PMRTS website). Dosage data will also be entered online via the PMRTS website by the project director or evaluator of the grant site.
Once data have been entered into PMRTS, the DCAR will clean the data and each grant site will have access to their downloadable, clean, electronic data files. Once data are entered by the grant site and cleaned by the DCAR, data will be available for download by the grant site for use in local data analysis and reporting. Sites will be able to enter the data online, or upload if necessary, on a continual basis. The DCAR will abstract data biannually for cleaning, analysis, and reporting purposes; however the data will remain accessible for local evaluations.
CSAP has established a technical assistance center to assist program sites. Any questions regarding the common instruments or submission to the DCAR should be addressed to DCAR Support via telephone, 1 (888) DCAR-4-U or via email, DCARSupport@kitsolutions.net
Each grantee will have established its own follow-up procedures as part of the original protocol. Issues related to response rates, as well as other data collection issues, are discussed at grantee meetings in order for GPOs to identify problems and provide technical assistance. In addition, GPOs monitor data collection efforts and provide technical assistance to individual grantees as necessary. Because collection of the NOMs is a stipulation of the grants, it is anticipated that all grantees will comply (as appropriate). The participants at each site to whom these measures will be administered are all voluntary respondents; therefore, grantees cannot guarantee full cooperation on the part of participants. Historically, however, participant response rates across grantee sites have averaged 40%. Given this attrition rate, data from respondents will be compared with available data on non-respondents to assess non-response bias..
As part of the terms and conditions of the grant award, sites are required to use the NOMs data system, available through the PMRTS website.
CSAP and its contractor have reviewed the adult and youth questionnaires, clarifying terminology and language, and rewriting or eliminating unclear or unnecessary questions. Questionnaires were then pilot-tested on a small sample (less than 10 individuals) to ensure that the multi-site assessment requirements and procedures were consistent with activities conducted at education sites. Pilot-testing was designed to collect information on the overall evaluation design and to draft feedback forms.
All study sites were encouraged to participate in pilot testing because of the diversity in target populations and program settings. (Note: It was only after local sites had completed their pilot testing that project staff learned that not more than nine persons could be included in testing without OMB approval). Pilot testing included administration of draft instruments and solicitation of comments on the instrument from respondents.
The purpose of the pilot testing was to:
Identify instructions and/or questions within the instrument that were unclear or confusing
Obtain suggestions for improving questions or instructions
Identify data collection procedures that were unclear or confusing
Measure respondent burden based on the average time to complete instruments
Identify ways to minimize respondent burden and improve accuracy regarding instrument completion.
Comments on the draft feedback forms included collecting information on the likelihood of obtaining specific responses, overall instrument layout, item flow, and administration.
Joseph Sonnefeld, M.A.
Westat
1600 Research Blvd
Rockville, MD, 20850
240-314-2522
Donna Atkinson, Ph.D.
Westat
1600 Research Blvd
Rockville, MD, 20850
240-453-2761
Ilene Klein, Ph.D.
Westat
1600 Research Blvd
Rockville, MD, 20850
240-314-7539
May Yamate, M.A.
Westat
1600 Research Blvd
Rockville, MD, 20850
240-314-7586
ATTACHMENTS
Youth Questionnaire and Youth Questionnaire Administration Guide
Adult Questionnaire and Adult Questionnaire Administration Guide
Overarching Administration Guide
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | HIV OMB Supporting Statement |
| Author | proddy |
| File Modified | 0000-00-00 |
| File Created | 2021-01-30 |