Evaluation of the Substance Abuse and Mental Health Administration
Primary Care Behavioral Health Integration Grant Program
OMB Supporting Statement
B. Collection of Information Employing Statistical Methods
1. Universe and Respondent Selection Methods to be Used
Intervention and control sites: The ten grantee sites that will serve as intervention sites for RQ2 will be chosen from up to 60 grantees based on five criteria: 1) Potential to identify a well-matched control site, 2) The quality of their data infrastructure, 3) A target population of greater than 750 individuals, 4) A target population that meets the SAHMHSA SMI definition, and 5) The quality of the PBHCI program data submitted to date. Sampling of grantees for RQ will not be done at random. We will specifically select programs that reflect the diversity of the grantees in terms of location, populations served and integration models utilized, as well as those with sufficiently large patient populations and electronic data infrastructure that can support the evaluation. While our research design for RQ2 is quasi-experimental because we will select matched-control sites for each intervention site, we will also treat intervention sites as a series of case studies. Specifically, we will note in our report that sites with the potential to provide a strong exemplar of integrated program implementation. In contrast, the 10 control sites will be chosen based on 1) the absence of primary care services, 2)geographic proximity to the matched intervention sites (within the same state and preferably within the same county), 3) similarity to the intervention site in the size and characteristics of the population served and 4) similarity in the mental health services delivered. One limitation of our design is that there may be differences in the type and degree of match between individual intervention sites and their unique matched control; this is because we can only select intervention sites from the existing pool of community behavioral health clinics already in existence that meet our inclusion/exclusion criteria. In order to ensure that we have the best possible intervention-control site matched pairs, we will consider more than the necessary number of potential sites for the evaluation and then select those with the most potential to support the evaluation with the highest level of scientific rigor. We will also describe these limitations of the study design in any presentation or written summary of our results.
Site visit interviews for leadership, care coordinators, and physical health and mental health providers: The evaluator will work with the Project Directors at each of the sites to be visited to identify individuals to include in each discussion. It is expected that 3-4 sites will be chosen from the first cohort and 6-7 will come from the second cohort.
Client Physical Exam and Survey: PBHCI intervention sites will provide the evaluator with a de-identified list of clients who have enrolled in integrated care services, and the control sites will provide a complete de-identified list of adult clients to the evaluator for participation in the physical exam and survey. The evaluator will select a random sample of n=300 clients for the control sites and n=250 clients for the intervention sites. The evaluator will then provide the information to the site personnel who will prepare a list of names and contact information for the contractor so that patients can be invited to participate in this component of the evaluation. The evaluator will sample n=300 patients from the control sites at baseline so that (assuming ~15% attrition) n=250 participants complete the physical exam and survey per at 1 year follow-up. The evaluator will sample n=250 patients from the intervention sites because they will only complete the physical exam and survey at follow-up (baseline data will be extracted from medical records / registries and the TRAC tool). We propose this sample size because a) we expect that most sites will be able to accrue at least n=250 patients by the middle of their second year of grant activities; b) it will give the evaluator > .80 power (alpha = 0.05) to detect all of the 7 outcomes of interest across all sites (RQ2); and c) it will give the evaluator >.80 power (alpha =0.05) to detect 5 of the 7 outcomes of interest (smoking cessation, high blood pressure, diabetes, and obesity) within clusters of 3 sites. That is, with this level of power, the evaluator can reliably answer questions about which models and model features (RQ3) are related to 5 of the 7 physical health outcomes of interest, provided that the evaluator can identify at least 3 intervention sites that share the model or model feature of interest. Within this size sample, the evaluator might also be able to address hypotheses related to RQ3 and the remaining 2 physical health conditions (hypercholesterolemia, hyperlipidemia) if effect sizes are large.
2. Procedures for the Collection of Information
Site visit interviews for leadership, care coordinators, and physical health and mental health providers: Information will be gathered in small group interviews arranged by the evaluator and the grantees or control site leader, led by a researcher, and recorded by a note-taker. A written statement of informed consent (Attachments 11,12) will be provided to each participant prior to the interview and will be reviewed with the group by the researcher.
Client Physical Exam and Survey: Prospective participants will be asked by a staff person at their behavioral health center if they would like to be contacted in order to learn about a research study. Staff at the behavioral health clinic will compile a list of persons who agree to be contacted and they will send this list with names and contact information to RAND over a HIPAA-compliant secure SharePoint site. Staff at the sites will also keep a tally of the number of clients that have declined to be contacted so that response rates can be calculated for each site. Although we cannot know the demographic and clinical characteristics of the persons who decline to be contacted without creating greater data collection burden, we can and will compare these characteristics of the participating sample to the characteristics of each site’s overall PBHCI clientele (NOMs data can be used for this purpose). This comparison will help us to determine if we have recruited a representative sample of participants for this portion of the study. RAND will securely transmit the list of clients who wish to be contacted to our subcontractor, OnSite Health Diagnostics, who will send a letter (Attachment 13) to prospective participants describing the study. Letters will go out 1-2 weeks prior to the data collection date with information about the study and an invitation for consumers to come to their behavior health clinic on a specific date to participate. OnSite Health Diagnostics will also call prospective participants 1-2 days prior to the data collection date to provide additional information about the study and to further invite consumers to participate in the study (Attachment 16). Participants who do not show-up on the day of data collection but who had indicated that they would like to participate will be called by OnSite Health once more and invited to participate in the study on a second date shortly thereafter.
Individuals who have provided written informed consent (Attachments 14, 15) for primary data collection will participate in a physical health exam. The physical health exam will be conducted by trained and experienced staff from OnSite Health Diagnostics -- a national biometric and diagnostic health screening company (HIPAA compliant). OnSite Health will perform the following patient assessments during the physical health exam: Height, weight, body mass index (calculated from height and weight), blood pressure, waist circumference, and breath carbon monoxide (an indicator of tobacco smoke exposure; participant blows into a tube). OnSite Health Diagnostics’ licensed and trained phlebotomists will also collect a blood sample (approximately 3 tablespoons) for the following tests: blood sugar and HgBA1C (diabetes indicators), cholesterol (total, HDL and LDL), and triglycerides. OnSite staff will also ask participants if they successfully fasted for 8 hours prior to the health exam; an 8 hour fast is necessary prior to the blood tests for diabetes indicators. OnSite staff will issue $20 gift cards (to a local, major retailer e.g., Walmart) to participants who have completed the physical health exam and survey at the end of the 40 minute session.
Client-level Service Utilization Report: Data on client-level service utilization will be extracted from clinical registries and/or medical records (depending on each sites’ data management system) and submitted via an Excel spreadsheet (Attachment 15) or similar database file (e.g., CSV, DBF) to the secure RAND SharePoint site. Sites with electronic medical records and/or registries should be able to automate the process of report generation, which will substantially reduce the burden associated with this task.
Quarterly Reports from grantees: The Project Director from each site will assemble the necessary information and submit it electronically in a Word document to the secure RAND SharePoint site.
Physical Health Indicators to be reported through TRAC: Physical health data will be extracted from grantees’ clinical registries and/or electronic medical records and entered into the TRAC system at the time each consumer completes their TRAC-required bi-annual client interview; entering all TRAC data simultaneously will minimize grantees’ data management burden.
3. Methods to Maximize Response Rates and to Deal with Issues of Non-Response
Client exam and survey: Several elements have been considered to maximize client participation in the exam and survey. In addition to payment for the individual’s time, the exam will take place at the center at which the client usually attends appointments, and it will be scheduled to coordinate with an existing appointment when possible. To address the potential issue of response bias, RAND will compare the characteristics of participants in the client physical exam and survey with the characteristics of sites’ overall PBHCI clientele using NOMs data. This comparison will help us to determine if we have recruited a representative sample of participants for this portion of the study.
For the control sites, it is also an opportunity for clients to be screened for physical health conditions and referred to appropriate care, if necessary. We do, however, project that there will be some attrition in the control group and have increased the initial sample by 15% to account for that.
4. Tests of Procedures or Methods to be Undertaken
The evaluation design team sought to use instruments/questions that have been tested previously. For example, the questions for the site visit interviews came from the Assessment of Chronic Illness Care, the NCQA standards for Patient Centered Medical Homes and the VA Survey of Mental Health Services (VA), among others. The client survey is primarily made up of questions from the TRAC client-level survey with several from the National Health and Nutrition Examination Survey (NHANES) and the National Survey on Prescription Drugs. The quarterly report has been pilot tested with the original 13 grantees during their first year of implementation, and modifications were made based upon their questions and feedback. Further, there will be an opportunity for the evaluator to pilot test the client interview and site visit questions with the first cohort of grantees and control sites before using them with the second and considerably larger cohort.
5. Consultants on Statistical Aspects of the Design and Persons Who Will Collect and Analyze the Information
The evaluation design was conceived by the RAND project team in conjunction with the representatives from SAMHSA and ASPE and with input from the Technical Advisory Group. Members of the RAND project team include (in alpha order): Audrey Burnam, Nicole Eberhart, Bing Han, Susan Lovejoy, Harold Pincus, Marcela Horvitz-Lenon, and Deborah Scharf. Representatives of SAMHSA include (in alpha order): Crystal Blyler, Trina Dutta, Fran Randolph, and Ken Thompson. Representatives of ASPE include (in alpha order): Hakan Aykan, Kirsten Beronio, Richard Frank, and William Marton. The evaluation will be conducted by an independent entity under contract with ASPE.
List of Attachments
Attachment 1: Site visit interview with leadership
Attachment 2: Site visit interview with mental health providers
Attachment 3: Site visit interview with primary care providers
Attachment 4: Site visit interview with care coordinators
Attachment 5: Site visit interview with key staff (control sites)
Attachment 6: Client physical exam and survey
Attachment 7: Client-level service utilization report
Attachment 8: Quarterly reports from grantees
Attachment 9: Physical health indicators to be reported through TRAC
Attachment 11: Consent protocol for site visit interviews-grantee sites
Attachment 12 Consent protocol for site visit interviews-control sites
Attachment 13: Initial contact letter for client exam and survey
Attachment 14: Baseline consent for client exam and survey-intervention sites
Attachment 15: Baseline consent for client exam and survey-control sites
Attachment 16: Telephone script for client exam and survey
Attachment 17: Clinical registry codebook
Attachment 18: RAND IRB package
| File Type | application/msword |
| File Title | Evaluation of the Substance Abuse and Mental Health Administration |
| Author | IST |
| Last Modified By | IST |
| File Modified | 2011-02-11 |
| File Created | 2011-02-11 |