All 7 MSI HIV grantee sites will take part in the cross-site evaluation. Grantees will be required to participate in all of the data collection activities, including the key informant interviews, and semi-annual outcome evaluation submissions. Data will be collected from MSI staff, partners and students. Except for the pre- and post-tests for the workshops, data will be collected for samples of respondents only.
The number of potential respondents, samples for the data collection, and description of the samples across the MSI sites may be found in the table below.
Type of Data |
Total Potential Sample |
Data Collection Sample |
|
# |
Description of Sample |
||
Annual site visit key informant interviews with MSI staff |
35 * (Estimating 5 program staff at each site) |
14 |
2 staff per site: Principal Investigator and 1 program staff member |
Annual site visit key informant interviews with MSI partners |
35 * (Estimating 5 program staff at each site) |
14 |
2 staff per site: 1 program director and 1 staff member |
Monthly progress reports (activities held, #’s of participants) |
35 * (Estimating 5 program staff at each site) |
14 |
2 staff per site, the Principal Investigator and a program staff member, will be responsible for the inclusion of evaluation data in these reports |
Semi-annual reporting of site outcome evaluation findings |
35 * (Estimating 5 program staff at each site) |
14 |
As above, 2 staff per site, the Principal Investigator and a program staff member, will be responsible for the inclusion of evaluation data in these reports |
Students pre- and post- surveys of educational campaigns (Part of semi-annual reporting of evaluation findings) |
24,000 6,000 students x 4 schools |
400 |
Site evaluations: A sample of 100 students per 4 sites conducting this activity. |
Pre- and post- tests to students enrolled in CDC DEBIs administered on site (Part of semi-annual reporting of evaluation findings) |
420 |
420 |
Site evaluations: All students participating in the workshops will be asked to complete a short questionnaire (10-15 minutes) at the beginning and at the end of the sessions to measure changes in attitudes and knowledge. |
Student focus groups (Part of semi-annual reporting of outcome evaluation findings) |
42,000 (6,000 students x 7 schools) |
350 |
Site evaluations: A maximum sample of 50 students per school (n=7) conducting this activity. |
Annual Site Visit Key Informant Interviews
In-person key informant interviews with MSI staff and partners will be conducted during annual site visits and over the telephone. The interviews will be conducted by the MSI HIV evaluation team. The interviews are expected to take two hours per interviewee.
These interviews will be conducted with the Principal Investigator and an identified program manager at each site. The interviews will focus on the following questions:
To what extent were activities implemented as planned?
What implementation challenges were encountered and how were they addressed?
What resources were needed to carry out the activities?
To what extent did contextual factors – institutional and community conditions – affect implementation?
What lessons learned could be shared with other MSIs implementing similar program components?
During the site visits, the evaluation team will verify information about the target populations and the populations served and the specific activities conducted. These interviews will follow discussion guides which will have been populated with information gathered from existing sources, to the extent possible, in advance of the site visits. Copies of the interview guides may be found in the Appendix A.
The in-person interviews will take place in the Fall 2012, and the telephone interview will be in the Summer 2013.
Monthly Progress Reports
Data that the sites will be submitting for program monitoring purposes will be used for the cross-site evaluation. These will include:
the numbers and types of activities held;
the numbers and descriptions of participants;
program planning and challenges.
The monthly report template may be found in the Appendix.
Semi-Annual Reporting of Site Specific Evaluations
MSI HIV grantees will participate in semi-annual electronic submission of their site-specific evaluation findings. These data will be used to describe the evaluation findings of the most recent six-month reporting period. Although the evaluation will not dictate which MSI HIV staff member will be tasked with providing these data, we anticipate that the local evaluator at each site will assume primary responsibility for this effort. The evaluations will assess changes in students’ knowledge, attitudes and behaviors, including testing, re: HIV/AIDS, as measured by established pre- and post-test methodologies and surveys already developed and validated.
The evaluation instruments that the sites will use are summarized in the table below. The SISTA, SiHLE and Nia curricula are part of the CDC Diffusion of Effective Behavioral Interventions (DEBI) project. This is a national-level strategy to provide high quality training and on-going technical assistance on selected evidence-based HIV/STD/ Viral Hepatitis prevention interventions to state and community HIV/STD program staff. The interventions have been proven effective through research studies that showed positive behavioral (e.g., use of condoms; reduction in number of partners) and/or health outcomes (e.g., reduction in the number of new STD infections). The studies employed rigorous research designs, with both intervention and control groups, so that the positive outcomes could be attributed to the interventions. With input from the researchers, the materials necessary to implement the interventions have been packaged into user-friendly kits. With the appropriate training and intervention package, service providers can increase their opportunities to conduct effective HIV/STD/Viral Hepatitis prevention programs in their communities.
Copies of the forms are also provided in the Appendix.
Type of Data Collection Form |
Specific Data Collection Forms |
Surveys |
|
Pre- and Post-Tests
|
|
Focus Group/Interviews |
|
We will institute a variety of methods to ensure the quality of the data collected for the cross-site evaluation. All Abt evaluation staff that will conduct the annual sites visits will attend a training before the annual key informant interviews, which will cover the site visit protocols, definitions for terms used in the interview guides, and any questions about the data collection instrument or procedures. Staff will review strategies for working with diverse populations and for ensuring that data are collected in a manner that is culturally sensitive and respects the backgrounds and traditions of MSI staff and students.
Evaluation and project team members will work closely with the MSI staff to ensure the quality of the data collected in the regular ongoing electronic data submissions. The data will be reviewed by project staff within 15 days after each submission. Contact will be made with the sites to addresses questions and problems with the submissions. As needed, technical assistance will be provided to the sites.
The evaluation team’s technical assistance pertaining to data collection will ensure that the data provided is of the highest quality. The assistance will focus on helping grantees on the cross-site evaluation data collection protocols, websites, and tracking forms that they will be using to collect and report data. This will include demonstration of these protocols and websites, practice in using them, troubleshooting, and ongoing assistance throughout the evaluation to ensure complete and reliable data collection and submission. Assistance will cover data cleaning and data checks, issues of privacy, and timelines for data submission. A manual will be provided to grantees with detailed instructions for reporting each data element and measure (including definitions).
MSI HIV grantees are required to participate in the cross-site evaluation as a condition of funding. In addition, the project and evaluation team has established relationships with MSI site staff, which are expected to facilitate the submission process. Further, our experience with the sites has demonstrated that they are invested in building strong programs, and learning from their and other grantees’ experiences. For the electronic submissions of data and MSI staff and partner interviews, we expect the response rates to be 100%, above the OMB threshold of 80%.
For the pre- and post-tests given at the beginning and end of student workshops, we also expect the response rate to be 90 - 100%, since the participants will complete these short, 10-15 minute surveys at the sessions. For the focus groups, the staff will identify the students to participate, and will contact the students via e-mail just prior to the sessions. Since there is flexibility in the numbers of students who attend each group (8-12), we will recruit up to 12 per group, expecting that 80-90% will attend. Gift cards for participation in the groups are expected to positively affect the rates.
The Abt team has significant experience developing and implementing data collections instruments and methods. After the instruments and procedures are developed, they will be pilot tested with a small number of respondents (5-8) before being finalized. The pre- and post-tests that are part of the DEBIs have already been tested and validated.
Abt Associates and its sub-contractors, the National Minority AIDS Council (NAMC) and the National Native American AIDS Prevention Council (NNAAPC) are conducting this project under a contract to HHS Office of HIV/AIDS Policy. The plan for this cross-site evaluation was developed by the Abt team. Meg Gwaltney, MBA, designed the evaluation; and Katherine Flaherty, PhD will be carrying out the evaluation. A copy of the evaluation plan is attached in Appendix B. Project Director, Liza Solomon, DrPH and Program Manager, Chanza Baytop, DrPH, provided advice and consultation on the development of the evaluation. Dr. Flaherty is the evaluation contact and may be reached at 617-520-2853 or at Katherine_Flaherty@Abtassoc.com.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Department of Health and Human Services |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |