Evaluation of the Primary and Behavioral Health Care Integration Grant Program
Grantee web-based survey. All grantees in cohorts VI–VIII are expected to participate in the web-based survey. The evaluation team will administer the survey to all grantee administrators and selected frontline staff who are conducting integration activities and providing services to PBHCI clients. Sampling methods will not be used.
Telephone interviews. During years 2 and 4, the evaluation team will conduct telephone interviews with staff from select grantees from cohorts VI–VIII. In year 2, the team will select grantees from cohorts VI and VII, who are in the final years of their grant as well as grantees from cohort VIII, who are in the initial stages of implementation. In year 4, the evaluation team will select grantees from cohorts VII and VIII. Some of the grantees may be included in both phases of the telephone interviews to follow up on how implementation issues have developed over time.
In consultation with SAMHSA, the team will select grantees for interviews based on all available information, including their responses to the web survey and initial review of grant applications and other documents, performance management data submitted by grantees to SAMHSA in keeping with grant requirements, and registry/her data. The primary selection criterion will be the presence of program features that have been identified as common sources of implementation challenges. These might be identified explicitly in quarterly reports that grantees submit to SAMHSA, as required by the grant, or uncovered through analysis of performance management data. The team will choose grantees that have reported these challenges as well as grantees that have successfully implemented the same program features. For instance, we may focus on a particular quality indicator that varies widely across grantees and select grantees from across the range of performance for the telephone interviews. In selecting grantees, we will also aim to include a diverse range of integration approaches, communities, populations, and relevant federal and state policy initiatives (for example, communities implementing Medicaid health homes and certified community behavioral health centers).
Site visits. Five grantee sites will be selected to be visited by the evaluation team in each of years 2 and 3 based on data collected from grantee proposals and quarterly report extractions, plus responses to the first wave of the staff web survey.
Site visit interviews with leaders, care coordinators, and physical and mental health providers: The evaluator will work with the project directors at each of the sites to be visited to identify people to include in each discussion. Select staff who have significant roles in PBHCI implementation at each site will take part in these interviews. Information from grantee proposals and ongoing quarterly reports will allow the evaluation team to identify these staff by position. The project directors at each site will provide any updates to the staffing mix, name individuals in positions, and facilitate meeting arrangements. Both individual and group interviews will be conducted as part of the site visit to facilitate inclusion of all staff.
Client focus groups: The evaluation team will work with the grantees to determine the best way to engage 10 to 12 clients for each focus group. Prospective focus group participants will be recruited by grantee staff with whom they are familiar, beginning one or two months before the evaluation site visit. Potential participants will be asked by a staff person at their clinic if they would like to learn about a research study. The evaluation contractor will also ask grantees to have case managers or other direct care providers bring up the study to all eligible clients with serious mental illness (SMI). Case managers or other direct care providers will be told to briefly describe the study (for example, it involves a group discussion with external researchers, and participants will receive a $25 gift card) and to answer any questions clients may have. The evaluation team will provide grantees with recruitment materials, including a focus group description, frequently asked questions, and consent forms. Grantee staff will obtain written informed consent from participants using a standardized form (Attachment K).
Grantee registry/EHR data: All grantees in cohort VIII will be asked to provide registry/EHR data. Sampling methods will not be used.
Comparison groups: The goal of selecting comparison sites is to identify people whose trajectory of outcomes most closely represents what would have happened to PBHCI clients had they not been part of PBHCI. The evaluation team will recruit up to 10 clinics for the comparison group and up to 250 clients per clinic for baseline assessments. The final selection process will depend on the preliminary findings regarding the availability of potential comparison clinics and on input from the technical expert panel (TEP). The factors we will consider in selecting the comparison group include the following:
State policy context: State policies, such as Medicaid reimbursement for physical health services and coverage for community mental health services, influence access to care for the SMI population. Selecting comparison sites within the same states as the PBHCI clinics to which they will be compared will help control for state policy contexts. If it’s not possible to find comparison sites within the same states as the PBHCI clinics, comparison sites may also be selected from other states in which policies are substantially similar, as indicated by overall Medicaid generosity.
Clinic scope of practice: Clinics will be prioritized for selection as comparison clinics if they provide a similar scope of practice to the PBHCI clinics, except for providing PBHCI-funded physical health services. We recognize that the recent spread of integrated care in many forms may make it challenging to identify clinics that provide no care for physical health conditions. Clinics that provide physical health services to some known degree may be included as comparison sites, with appropriate adjustments made to the interpretation of the results.
Caseload size, demographics, and payer mix: Clinics will be selected to match as closely as possible to the PBHCI clinics to which they will be compared. The team will match the clinics on size, the demographic characteristics of the clients they serve, and the mix of payers that cover those clients. The goal of selection on individual client characteristics is not only to match similar clinics but to ensure that comparisons can also be made at the individual level (that is, that comparison clinics have enough clients of similar age, sex, and ethnicity to the PBHCI clinics to allow for a robust comparison).
A total of 2,500 people (about 250 from each clinic) will be included in the comparison group. Power calculations indicate that this sample size would provide adequate power to detect predicted PBHCI effects on most of the health outcomes of interest.
Staff at the behavioral health clinics will compile a list of the names and contact information of people who agree to be contacted and will send it to the evaluation team via a secure SharePoint site. The staff will also keep a tally of the number of clients who decline to be contacted so that response rates can be calculated for each site. The evaluation team will then send a letter to the prospective participants describing the study. The letters, which will go out one or two weeks before the data collection date, will invite clients to come to their behavioral health clinic on a specific date to participate. The team will also call prospective participants one or two days before this date to provide additional information about the study and to encourage clients to participate. Participants who do not show up but who had indicated that they would like to participate will be called once more and invited to participate in the study on a second date shortly thereafter. When a potential participant arrives at the clinic, the evaluation team will solicit written informed consent from him or her to participate. Those who provide consent will take part in a physical health exam.
Some clinics that are otherwise good candidates to participate in the comparison group may have difficulty recruiting 250 clients. Therefore, the evaluation team may accept a smaller number of clients from some clinics if there are no other willing alternate clinics.
PBHCI grantee director and frontline staff surveys: PBHCI grantee directors from all 93 grantee sites will be asked to complete the PBHCI grantee director web survey. The evaluation team will work with grantee project administrators to obtain a list of frontline staff eligible for the PBHCI frontline staff web survey. The team will then reach out to directors and frontline staff via e-mail to invite them to take participate. Participants will access the survey through a web link provided in the invitation e-mail. All survey information will be maintained on a secure system. Respondents will be identified by unique numeric IDs and not names or other identifying information.
Telephone interviews with PBHCI grantee directors: Interviews will be audio-recorded for later reference if participants agree to allow this. Immediately following each interview, the interviewers will complete a structured debriefing form. On the form, the interviewer will summarize the answers to each of the main interview questions, using direct evidence or quotations from the interviewees where possible.
Site visit interviews for leaders, care coordinators, and physical and mental health providers: Information will be gathered in small-group interviews arranged by the evaluator and the grantee project director or comparison site leader. These interviews will be led by a researcher and recorded by a note-taker. Interviews will be audio-recorded for later reference if informants agree to allow this. Immediately following each interview, the interviewers will complete a structured debriefing form. On the form, the interviewer will summarize the answers to each of the main interview questions, using direct evidence or quotations from the interviewees where possible.
Client focus groups: Focus groups will take place at the grantee sites and are expected to take 45 minutes to an hour. Several days before each focus group, grantee staff will place reminder calls to the clients who consented to participate. The focus groups will be conducted using semistructured protocols, with one moderator and one note-taker who have been trained to lead focus groups with people who have SMI. Immediately after each focus group, the moderator will complete a structured debrief template consisting of questions about key topics covered, thereby beginning the analytic process while the material is still fresh.
Grantee registry/EHR data: Grantees will extract data on client-level service use from clinical registries or medical records (depending on each site’s data management system) and submit them via an Excel spreadsheet (Attachment F) or in similar database file format (such as CSV or DBF) to the secure SharePoint site every quarter. The evaluation team will provide technical support to sites regarding data definitions and submission, including individual consultations with sites (as needed) and webinars for all cohort VIII grantees submitting such data.
Comparison group data: client physical exam and health assessment: The physical exam will be conducted by trained and experienced staff who are knowledgeable about HIPAA compliance. Table B.1 shows the patient assessments to be conducted at baseline and at the one-year follow-up.
Evaluation staff will also ask participants if they fasted for eight hours before the health exam, as is required for the blood tests for diabetes. At each comparison clinic, a qualified health provider will be designated to receive the test results and advise clients regarding their results. The evaluation team will also work with clinics to retain clients for the follow-up assessment.
Table B.1. Domains and Measures
Domain |
Measures |
Physical health |
|
Metabolic syndrome |
Waist circumference, “at risk” men >102cm and women >88cm |
Blood pressure |
Diastolic and systolic blood pressure, “at risk” ≥130/85 |
Body mass index |
Calculated from height and weight, “at risk” ≥25 |
Cholesterol levels |
Total cholesterol, HDL-C and LDL-C, “at risk” HDL <40 and LDL ≥130 |
Diabetes screening and management |
HbA1c and fasting plasma glucose, “at risk” ≥5.7 and ≥100 |
Triglycerides |
Triglycerides, “at risk” ≥50 |
Smoking |
Self-report of smoking any or none and carbon monoxide breath analysis “at risk” ≥10ppm |
Behavioral health |
|
Severe psychological distress (Kessler-6) |
In the past month, frequency of psychological distress symptoms (6 items) |
Substance use |
In the past month, frequency of substance use (12 items) |
Social functioning/quality of life |
|
Self-rated health |
Current self-rated overall health (one item) |
Functioning |
In the past month, degree of feeling able to deal with daily life (8 items) |
Global assessment of functioning |
Global assessment of functioning score (one scale, scores range from 0 to 100) |
Housing stability |
In the past month, number of nights somewhere other than a home (2 items) |
Education and employment |
Current enrollment status in a school or job training program and/or current employment status (3 items) |
Criminal justice involvement |
In the past month, number of times arrested (one item) |
Social connectedness |
In the past month, degree of feeling socially connected and supported (4 items) |
Web-based survey: To ensure that we reach an 85% response rate, grantee directors will help recruit eligible staff from their sites. Specifically, grantee directors will receive an e-mail inviting them to participate in the survey, which they will be asked to forward to eligible staff at their site. Grantee directors will also receive a series of e-mails following the original invitation encouraging them to make sure that all appropriate staff complete the web-based survey. At the end of the survey, all respondents will be encouraged to provide the name of their site for tracking purposes (the name will not be linked to their survey responses). The evaluation team will then be able to prompt grantee directors with a personalized e-mail noting if overall response from their site is low. Respondents will have one month to complete the survey following distribution of the original invitation e-mail.
Comparison group data: Several elements have been considered to maximize the participation of comparison clients in the assessments. Besides payment for the individual’s time, the assessment will take place at the center at which the client usually attends appointments, and it will be scheduled to coordinate with an existing appointment when possible. To address the potential issue of response bias, the evaluation team will compare the characteristics of participants in the client physical exam and survey with the characteristics of sites’ overall PBHCI clientele using NOMs data. This comparison will help the team determine whether it recruited a representative sample of participants for this portion of the study.
Director Survey. Cognitive pretesting with eight grantee directors has been conducted using a paper-and-pencil survey. The cognitive pretest sample included grantees from cohorts 7 and 8. It took an average of 45 minutes to complete the survey in its entirety. Based on our debriefings with the directors, we perceive that some directors may have overestimated their completion time because they often stopped and started the survey. Given this information and our experience with decreased completion time when the survey is administered via the web, we expect the burden for completing the director survey to remain at 30 minutes. Survey questions have been revised to improve clarity based on results of these tests.
Frontline Staff Survey. Cognitive pretesting with nine grantee frontline staff has been conducted using a paper-and-pencil survey. The cognitive pretest sample included grantees from cohorts 7 and 8. It took an average of 30 minutes to complete the survey in its entirety. Survey questions have been revised to improve clarity based on results of these tests.
In September 2015, SAMHSA awarded a task order to Mathematica to design and conduct the evaluation (TO HHSS283201200010I/HHSS28342001T). Mathematica designed the evaluation in conjunction with the SAMHSA task order officer, Laura Jacobus Kantor, and with input from the Expert Panel. Members of the evaluation design team include (in alphabetic order): Joshua Breslau (RAND); Jonathan Brown (Mathematica); Audrey Burnam (RAND); Nicole Eberhart (RAND); Harold Pincus (independent consultant); Deb Scharf (independent consultant); Kara Zivin (Mathematica). The evaluation will be conducted by Mathematica Policy Research under contract with SAMHSA.
Attachment A: PBHCI grantee director survey
Attachment B: PBHCI grantee frontline staff survey
Attachment C: Telephone interview protocol
Attachment D: On-site staff interview protocol
Attachment E: Client focus group guide
Attachment F: Electronic data collection tool for grantee registry/electronic health records (EHRs) (PDF version)
Attachment G: Consent to share contact information for research project
Attachment H: Initial client letter for physical exam and health assessment
Attachment I: Consent form for client physical exam and health assessment
Attachment J: HIPAA form for client physical exam and health assessment
Attachment K: Consent form for client focus group
Attachment L: Client health assessment questionnaire
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | PPD_40412-Part A_Initial Package for OMB 040315 |
| Subject | Impact Evaluation of Support for Principals: OMB Data Collection Package |
| Author | Mathematica Staff |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |