2. CCBHC Evaluation OMB_SuppStatementB v2

Section B - Collection of information employing statistical methods

1. Respondent universe and sampling methods

Telephone interviews with state Medicaid and behavioral health officials: The evaluation team will identify the appropriate state respondents based on their knowledge of state demonstration leadership and organizational structure. They will also consult with demonstration leadership within CMS and review state websites as needed.

Telephone interviews with CCBHC site leadership: The evaluation team will conduct interviews with the CEO, medical director, and any other key leadership staff recommended by the CCBHC at up to 15 clinics. The evaluation team will collaborate with ASPE, CMS, and SAMHSA to purposively select a diverse group of CCBHCs in states that encompass the demonstration’s different payment models, organizational structures, and implementation contexts. The team will use responses to the CCBHC survey and information gathered during state official interviews to identify clinics with characteristics of interest. The final list of CCBHCs will likely reflect factors such as geographic location; a mix of states using each of the prospective payment system models available to demonstration states; a mix of original states and new states; the types of partnerships between CCBHCs and other organizations; and successes or challenges with planning and implementation.

Focus groups with clients: The evaluation team plans to recruit clients from the same CCBHCs at which it conducts interviews and conduct up to eight focus groups. The team will leverage the relationships developed with CCBHC leaders during the telephone interviews to help identify clients for the focus groups. Recruiting clients from the same CCBHCs that participate in telephone interviews is an efficient way to help ensure this data collection activity also reflects a balance of CCBHC characteristics, and allows for triangulation of responses across both a CCBHC’s staff and clients. If there are more than eight clinics interested in helping to recruit for focus groups, the evaluation team will consider applying additional selection criteria to choose among them.

Strategies for recruitment of clients for focus groups will be tailored to each clinic, and the evaluation team will consider potential bias introduced by the recruitment procedures. The team will work closely with selected CCBHCs to develop an initial sampling frame and procedures for inviting clients to participate (for example, recruiting among participants receiving services added as a result of CCBHC certification). Prior to the focus group, the evaluation team will work with CCBHCs to obtain informed consent from clients (using consent materials we supply). The evaluation team expects to have CCBHCs obtain written consent from eight to ten clients. Of those eight, the team assumes that approximately three to five will ultimately attend, given their experience with typical no-show rates for focus groups among this population. Ultimately, the evaluation team plans to include up to five clients in each focus group, for a total of approximately 40 clients. If more (or fewer) people attend than expected, the evaluation team will adjust its sampling approach over time.

CCBHC surveys: The evaluation team will ask all CCBHCs participating in the demonstration to submit the survey in the second and third years of the evaluation. The team will not use sampling methods.

2. Procedures for the collection of information

Telephone interviews with state officials: The evaluation team will conduct semi-structured telephone interviews with state Medicaid and behavioral health officials in each demonstration state in evaluation years two through four to gather information on demonstration progress at different stages. Two researchers will conduct each interview, with one leading the interview and the other taking notes. With permission from the interviewee, the researcher will record the interviews to facilitate notetaking.

Telephone interviews with CCBHC site leadership: The evaluation team will conduct an interview by telephone with the CEO and medical director, and any other key leadership staff recommended by the CCBHC at up to 15 clinics during year three of the evaluation. Two researchers will conduct each interview with one leading the interview and the other taking notes. The interviews will be recorded for professional transcription.

Focus groups with clients: The evaluation team will conduct up to eight focus groups with clients of CCBHC services during the third and fourth year of the evaluation. Two researchers will conduct these focus groups with one leading the focus group and the other taking notes. The focus groups will be conducted using WebEx or Zoom and will be recorded and professionally transcribed.

CCBHC survey: The evaluation team will conduct the CCBHC survey in the second and third evaluation years. The survey will be conducted through Confirmit, a user-friendly online platform. The survey team will review daily reports to assess completion rates overall and by state and work closely with state officials to encourage participation among CCBHCs. The survey will include skip patterns to reduce burden on CCBHC respondents and improve consistency of data collection.

3. Methods to maximize response rates and deal with nonresponse

As recipients of CCBHC demonstration funding, states are expected to demonstrate high levels of cooperation and be fully engaged as partners in the evaluation, encouraging CCBHCs to collaborate with the evaluation team regarding the interviews, focus groups, and submission of surveys. The evaluation team will work with state officials and CCBHC leadership to identify a mutually convenient date and time to conduct the interviews. For the focus groups, the evaluation team will work with CCBHCs to identify incentives that will encourage attendance at focus groups and schedule them at dates and times that will allow client participation. The evaluation team will also work closely with state officials to strongly encourage clinic participation in the survey by enlisting state demonstration leaders to send reminders and provide feedback to state officials on response rates.

4. Tests of procedures or methods to be undertaken

There will be no official pre-test for the interview protocols, focus group protocols, or the survey because these instruments are similar to instruments used without difficulty in the previous evaluation of the CCBHC demonstration and other similar evaluations the evaluation team has conducted.

5. Consultants on statistical aspects of the design and people who will collect and analyze the information

In September 2021, ASPE awarded a task order to Mathematica and its subcontractor, the RAND Corporation, to design and conduct the evaluation (Task Order number HHSP233201500035I/ 75P00121F37064). Mathematica designed the evaluation in conjunction with ASPE task order officers Laura Jacobus-Kantor and Judith Dey. The evaluation design team includes Allison Wishon, Kate Stewart, Stefanie Pietras, and Jonathan Brown (Mathematica) and Joshua Breslau and Courtney Kase (the RAND Corporation). The evaluation will be conducted by Mathematica and the RAND Corporation under contract with ASPE.

CCBHC Extension Evaluation, OMB No. 0990-NEW 1