SSA_CCBHC Evaluation_final _OMB

Evaluation of the Certified Community Behavioral Health Clinic Demonstration

Supporting Statement – Section A

Submitted: July 12, 2017

Program Official/Project Officer

Judith Dey, Ph.D. - Social Science Analyst

U.S. Department of Health and Human Services

Office of the Assistant Secretary for Planning and Evaluation

200 Independence Avenue SW, Washington DC 20201

Section A – Justification

1. Circumstances Making the Collection of Information Necessary

The Office of the Assistant Secretary for Planning and Evaluation (ASPE) at the U.S. Department of Health and Human Services (HHS) is requesting Office of Management and Budget (OMB) approval for qualitative data collection activities to support the evaluation of the Certified Community Behavioral Health Clinic (CCBHC) demonstration program. The quantitative data collection activities with their respective OMB approval numbers are described further in the final paragraph of this section.

In April 2014, Section 223 of the Protecting Access to Medicare Act (PAMA) mandated the CCBHC demonstration to address some of the challenges of access, coordination, financing, and quality facing community mental health centers (CMHCs) across the country. The CCBHC demonstration is intended to improve the availability, quality and outcomes of CMHC ambulatory care by establishing a standard definition and criteria for CCBHCs, and developing a new payment system that accounts for the total cost of providing comprehensive services to all individuals who seek care. The demonstration also aims to more fully integrate primary and behavioral health care services; ensure more consistent use of evidence-based practices; and, through enhanced standardized reporting requirements, offer an opportunity to assess the quality of care provided by CCBHCs.

In addition, as required by PAMA, the Centers for Medicare & Medicaid Services (CMS) issued guidance for the establishment of a Prospective Payment System (PPS) for Medicaid-reimbursed mental health services furnished by the CCBHCs participating in the demonstration. A PPS is a method of reimbursement in which Medicare or Medicaid payment is made based on a predetermined, fixed amount. The payment amount for a particular service is derived based on the classification system of that service. States must select among two broad PPS models developed by CMS (although they have some flexibility in exactly how states will operationalize these models). The first model reimburses costs using a fixed daily rate for all services rendered to a Medicaid beneficiary. This model (PPS-1) also includes a state option to provide quality bonus payments to CCBHCs that meet defined quality metrics. The second provides reimbursement using a standard monthly rate per person served, with separate monthly rates that vary based on beneficiaries’ clinical conditions. This PPS-2 model also includes outlier payments for costs above and beyond a specific threshold and quality bonus payments. Both PPS models aim to enhance Medicaid reimbursement to cover the cost of providing services based on the CCBHCs’ anticipated costs.

In the second phase of the demonstration, 8 states were selected from among the 24 that received planning grants to implement their PPSs and provide services that align with the CCBHC certification criteria. These 8 states were selected based on the completeness of the scope of services CCBHCs will offer; their ability to improve the availability of, access to, and engagement with a range of services (including assisted outpatient treatment); and their potential to expand mental health services without increasing federal spending. CCBHCs participating in the demonstration must also provide coordinated care and make a comprehensive range of nine types of services available to all who seek help, including but not limited to those with serious mental illness, serious emotional disturbance, and substance use disorders. Services must be integrated, trauma informed, and encompass whole-person care, and can involve other organizations with which the CCBHC can collaborate.

The demonstration and its evaluation offer an opportunity to examine the implementation and outcomes of CCBHCs. The evaluation will provide critical information to Congress and the larger behavioral health community about innovative ways CCBHCs are attempting to improve care. States are implementing a variety of innovations that will be of general interest to Congress and policy-makers interested in improving behavioral health delivery systems. In addition, the evaluation will assess the effects of a well-defined, comprehensive service array on client outcomes and costs. This well-defined, comprehensive service model may be implemented by other states, even without demonstration authority, and thus is of general interest to Congress and policy-makers. The evaluation of the CCBHC demonstration program is intended to inform annual reports to Congress, which are mandated by Section 223 of PAMA to include assessments of the following:

1. Access to community-based mental health services under the Medicaid program in the area(s) of a state targeted by a demonstration program compared to other areas of that state;

2. The quality and scope of services provided by CCBHCs compared to community-based mental health services provided in states and areas of those states not participating in the demonstration ; and

3. The impact of the demonstration programs on the federal and state costs of a full range of mental health services (including inpatient, emergency, and ambulatory services).

The evaluation is also intended to inform mandated HHS recommendations to Congress (due no later than December 31, 2021) concerning whether the demonstration should be continued, expanded, modified, or terminated.

The ability of ASPE to provide HHS, SAMHSA, CMS, and Congress with the information they need to develop policies that will improve access to high quality community-based mental health care relies, in part, on a rigorously designed, independent evaluation of the CCBHC demonstration. This data collection will provide important feedback on program design, execution, and effectiveness as part of ASPE’s evaluation efforts.

In accordance with the current evaluation contract timeline, the evaluation will take place over 60 months (the evaluation contract began September 29, 2016 and ends September 29, 2021). Evaluation data collection that requires OMB approval will not begin until ASPE receives final OMB approval.

b. Overview of study design and evaluation questions

To learn about the effectiveness of the CCBHC program, the study team will use a mixed-methods approach.

We have conceptualized the evaluation as having two interrelated components: (1) an implementation study that examines how states support the demonstration; CCBHCs’ successes and challenges in maintaining the certification criteria; the costs of CCBHC services; and changes in the accessibility, scope of services, and quality of care that result from the demonstration; (2) an impact study that examines changes in service utilization and costs of serving CCBHC clients relative to comparison groups. The study will address five evaluation questions about the implementation and impacts of the CCBHC demonstration:

1. What activities do CCBHCs implement to improve access to care (including participation in Assisted Outpatient Treatment)? How does access to care in the demonstration area(s) compare to access to care in other parts of the state?

2. How do CCBHCs implement the full scope of services and maintain the certification requirements throughout the demonstration? How does the scope of services provided to CCBHC clients compare with that provided to other populations and in other service settings?

3. What is the quality of care provided to CCBHC clients? How does the quality of care compare with that provided to other populations and in other service settings?

4. Do the PPS models cover the full cost of care for the CCBHCs? What changes do states make in their PPS rates over the course of the demonstration?

5. What is the impact of the demonstration on inpatient, emergency, and ambulatory service utilization rates and state and federal Medicaid costs relative to comparison groups?

The implementation component of the evaluation is designed to assess the extent to which CCBHCs expand their scope of services and access to care, and the quality and costs of CCBHC services. The implementation study will help us understand the services offered by CCBHCs; the costs of these services; the extent to which CCBHCs experience challenges with the certification criteria; their performance on clinical quality measures; and the perspectives of state officials, CCBHCs, and consumer and/or family organizations on the overall successes and challenges of the CCBHC model of care and reimbursement structure. When possible, we will benchmark accessibility and quality of care to other areas of the state and/or national averages.

The evaluation will address questions regarding costs, quality, and impact through analysis of Medicaid claims data obtained directly from CMS, cost reports that states will submit to CMS under a separate OMB approval (OMB 0938-1148 CMS 10398), and quality measures that states are required to submit to SAMHSA under a separate OMB approval (OMB 0938-1148 CMS 10398); thus, this submission focuses solely on qualitative data collection to address implementation questions.

2. Purpose and Use of the Information Collection

Section 223 of PAMA requires the Secretary of HHS to provide annual reports to Congress that include an assessment of access to community-based mental health services under Medicaid, the quality and scope of CCBHC services, and the impact of the demonstration on federal and state costs of a full range of mental health services. In addition, PAMA requires the Secretary to provide recommendations regarding continuation, expansion, modifications, or termination of the demonstration no later than December 31, 2021. The data collected under this submission will help ASPE address the research questions listed above and inform the required reports to Congress. Each proposed data collection instrument is described below, along with how, by whom, and for what purpose the collected information will be used. Table A.1 provides additional detail about how the content areas in each data collection instrument will be used to answer the evaluation’s key questions.

State official telephone interview protocols. We will use the telephone interview protocols to conduct three rounds of semi-structured interviews with state officials to gather information about the demonstration’s progress at different stages of implementation. Each round of interviews will include state Medicaid and behavioral health officials (both mental health and substance abuse officials). Each round of interviews will have a slightly different focus, reflecting the stage of implementation. The first round will be conducted in approximately December 2017 and use the baseline interview protocols for state Medicaid and behavioral health officials to focus on early program implementation, decisions made during the demonstration planning phase, early successes and challenges in fulfilling the certification requirements and data collection and monitoring procedures, and anticipated challenges or barriers to successful implementation. The second round of interviews, conducted in March 2018, will use the demonstration midpoint interview protocols for state Medicaid and behavioral health officials, and will focus on interim successes and challenges since the initial interview; any changes to state organizational structures, policies, or processes that could affect the demonstration; success in implementing demonstration cost reporting procedures and quality measures; and early experiences with the PPS systems.

The final round of interviews, conducted in March 2019, will use the demonstration end protocols for state Medicaid and behavioral health officials to focus on any changes made to the PPS system in the second year of the demonstration, the extent to which states and CCBHCs have been able to use the quality measures to inform quality improvement efforts, and the overall perspectives of state officials regarding the success of CCBHCs in improving care. The final round of interviews will also include interviews with representatives from one or two consumer and/or family organizations within each state to provide their perspectives on the successes and challenges of the demonstration, using the demonstration end consumer representative protocol. These interviews will provide an opportunity to understand the extent to which such organizations have been involved in shaping the delivery of services and how the CCBHC model could be improved.

Site visit interview protocols. During Year 2 of the demonstration, the evaluation team will conduct site visits to clinics in four strategically selected states, with each visit lasting about two business days. We will select states to reflect the different payment models they are using for the demonstration as well as geographic variation. These site visits primarily will provide an opportunity to visit CCBHCs to see their operations firsthand and have in-depth discussions with clinic leadership and frontline clinical staff about how care has changed following the implementation of the demonstration, and how the CCBHC model could be improved. These discussions will focus on access to care at CCBHCs, any challenges in meeting the CCBHC certification criteria, operational and other changes to the clinic resulting from CCBHC certification, and any nuances regarding the progress described in their first clinic progress reports (described below).

CCBHC site leadership interview protocol. We will use questions for CCBHC leadership to systematically derive information on adherence to CCBHC criteria and the demonstration implementation process. For example, the protocol will explore organizational characteristics and financing, scope of services, access to care and CCBHC facilities, electronic health care record use and quality reporting, care coordination, and processes of care to help assess issues of access and quality among CCBHCs. We will assess changes to services and processes of care due to implementation of the demonstration.

CCBHC administration/finance staff interview protocol. We will use questions for CCBHC administration and finance staff to systematically derive information on changes to financing structures as a result of CCBHC certification and use of a PPS, the collection of data associated with that certification, and successes and challenges related to the financing and administration of CCBHC models.

CCBHC frontline provider interview protocol. We will use the questions for the frontline providers to probe about their experience with the new CCBHC structure and processes, and successes and challenges related to access to CCBHC services, the scope of services provided, and the quality of care.

CCBHC care manager interview protocol. We will use the questions for the care managers to systematically derive information on care coordination processes associated with CCBHC certification.

Clinic annual progress report templates. CCBHCs will submit a progress report toward the end of each demonstration year (about March 2018 and 2019). The purpose of the progress reports will be to gather key information over time about clinics’ operations and how their structures, procedures, and services align with the CCBHC certification criteria and promote improved access, quality, and scope of CCBHC services. For example, the reports will collect information about staffing and staff training, scope of services and accessibility, use of health information technology, relationships with other providers for purposes of service delivery and care coordination, and data collection activities. The progress report templates include structured fields to gather comparable information from each CCBHC, using prompts and preset response categories such as check boxes.

Table A.1 Data collection activities, by data source

Data source	Mode, timing, and respondent	Evaluation questions (EQs)	Content	Analysis
Qualitative data sources
Site visits	In Year 2 of the evaluation, the evaluation team will conduct site visits to clinics in four grantee states, each lasting about two business days.	EQ1, EQ2, EQ3, EQ4, EQ5	Site visit interviews: (1) staffing; (2) scope of services; (3) client screening and assessment; (4) collaborations and relationships with other organizations; (5) data sharing; (6) access to care; (7) collection, reporting, and use of data to improve quality; (8) care coordination; (9) successes and challenges	Descriptive analyses
State official telephone interviews	In Years 1, 2, and 3 of the evaluation, the team will conduct telephone interviews with state Medicaid and behavioral health officials.	EQ1, EQ2, EQ3, EQ4, EQ5	(1) Demonstration planning and administration; (2) CCBHC staffing; (3) scope of services and coordination of care; (4) quality of care; (5) cost, payment, and PPS; (6) data availability; (7) demonstration implementation; (8) implementation successes and challenges; (9) sustainment activities	Descriptive analyses
CCBHC annual progress report template	During Years 1 and 2 of the demonstration, all CCBHCs will submit annual progress reports to the evaluation team.	EQ1, EQ2, EQ3, EQ4, EQ5, EQ6	(1) CCBHC staffing; (2) accessibility; (3) care coordination; (4) scope of services; (5) data collection, quality, and other reporting	Descriptive analyses

a. Timeline for the data collection

The evaluation is expected to be completed in five years, with three years of qualitative data collection. Table A.2 shows the schedule of data collection activities.

Table A.2. Timeline for the data collection

Data source	Dates
Site visits	2018-2019
Telephone interviews	2017, 2018, 2019
CCBHC annual progress reports	2018, 2019

3. Use of Improved Information Technology and Burden Reduction

CCHBCs will submit annual progress reports through a secure, password-protected SharePoint site.

4. Efforts to Identify Duplication and Use of Similar Information

In formulating the evaluation plan, ASPE has carefully considered how to minimize burden by supplementing existing administrative data sources with targeted primary data collection. To this end, the evaluation incorporates the following approach:

Using data from existing administrative data sources while conducting supplemental primary data collection: To the extent possible, information regarding demonstration implementation will be gathered through a review of available sources, including, for example, state planning grant applications, demonstration applications, and state Medicaid plans; cost reporting and quality measures states will submit as part of demonstration requirements; and Medicaid claims. However, we expect that the level of detail and consistency of the information provided in these source documents will vary from grantee to grantee. To supplement data gathered from these sources, ASPE is requesting OMB approval to conduct site visits and telephone interviews, and collect progress reports. The evaluation team will use the information gleaned from telephone interviews to clarify and fill in gaps in the data gathered from a document review and progress reports. We will conduct site visits to CCBHCs in four states to see CCBHC operations firsthand and have in-depth discussions with clinic leadership and frontline clinical staff about how care has changed following implementation of the demonstration, and how the CCBHC model could be improved. We have tailored the questions to be asked during these visits to different stakeholders to minimize the time that CCBHC staff must spend in interviews.

5. Impact on Small Businesses or Other Small Entities

The CCBHCs in the participating states vary in size, from small entities to large provider organizations. The qualitative data collection protocols have been designed to minimize burden on these entities. We will make every effort to schedule site visits and interviews at the convenience of these respondents. Evaluation staff will ensure that visits to each facility last no more than two days. We will request the minimum amount of information from CCBHCs that is required to evaluate the CCBHC demonstration effectively.

6. Consequences of Collecting the Information Less Frequently

Each of the data sources provides information needed for the evaluation. If the data are not collected, the evaluator will not have adequate information to answer the five evaluation questions. The inclusion of all planned data sources is needed to glean information about demonstration implementation and obtain a complete picture of the quality of care.

Site visits will take place only once during the evaluation. If they are not conducted, the evaluator will not have adequate information to evaluate whether implementation is consistent with the legislated requirements and/or ensure that the secretary has the information necessary to provide Congress with the information mandated in the legislation. Clinics will submit progress reports twice during the course of the evaluation; repeated reports are needed to explore changes in access, scope of services, and other demonstration requirements over time. Similarly, we will conduct telephone interviews at three points during the demonstration to understand the evolution of demonstration administration; implementation successes and challenges; and changes in access to, costs, and quality of care over time.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

This information collection fully complies with 5 CFR 1320.5(d)(2).

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

This is a new data collection. The 60 day notice was published in the Federal Register on May 4, 2017 (82 FR 20898; Pages 20898-20899; 2017-08973).

9. Explanation of Any Payment or Gift to Respondents

We will not be providing any remuneration or incentive to respondents.

10. Assurance of Confidentiality Provided to Respondents

The Privacy Act does not apply to this data collection. Participants will not be asked about, nor will they provide, individually identifiable information.

Before the start of both telephone and site visit interviews, we will remind all respondents that the information gained will be used for evaluation purposes only and not be attributable to any individual. Responses should not contain private information but will be aggregated to the extent possible so that individual answers will not be identifiable. Because of the limited number of respondents interviewed per state and CCBHC, however, it might be possible to infer individual responses from reports. (For example, there may be only one state Medicaid official participating per state.) For each respondent, we will collect name, professional affiliation, and title, but not Social Security numbers, home contact information, and similar information that could identify the respondent directly.

11. Justification for Sensitive Questions

No information will be collected that is of personal or sensitive nature.

12. Estimates of Annualized Burden Hours and Costs

Table A.3 provides estimates of the average annual burden for collecting the proposed information. Below we provide details on the time and cost burdens for each of the separate data collection activities.

• CCBHC state site visits: During each site visit in the four selected states, the following interviews will occur at each clinic we visit:

• Interview with the clinic’s project director, lasting two hours (4 states x 2 clinics x 1 executive director x 2 hours)

• Interviews with four frontline mental health providers, each lasting one hour (4 states x 2 clinics x 4 mental health providers x 1 hour)

• Interviews with two care managers, each lasting one hour (4 states x 2 clinics x 2 care coordinators x 1.5 hours)

• Interviews with two administration/finance staff, each lasting one hour (4 states x 2 clinics x 2 administrative/finance staff x 1 hour)

• CCBHC state official telephone interviews: We will conduct the following telephone interviews with state officials in all demonstration states at three points during the course of the evaluation, and with consumer and family representatives in the third year:

• Interview with two state Medicaid officials, lasting one hour (8 states x 2 state officials x 1 hour x 3 interview years)

• Interview with two state mental health department officials, lasting one hour (8 states x 2 state officials x 1 hour x 3 interview years)

• Interview with two consumer/family representatives, lasting one hour (8 states x 2 representatives x 1 hour x 1 interview year)

• Completion of CCBHC progress reports: Participating clinics will complete the progress report template twice during the course of the evaluation. ASPE expects the CCBHCs’ executive teams to need four hours to complete all sections of the template (76 clinics x 4 hours x 2 reporting years).

Table A.3: Estimated Annualized Burden Hours and Costs to Respondents

Respondents/activity	Number of sites	Number of respondents per site	Responses per respondent	Total responses	Hours per response	Total hour burden	Average hourly wagea,b	Total hour cost burden ($)
Site visits
CCBHC site leadership staff—site interview	8	1	1	8	2	16	$93.12	$1,489.92
CCBHC frontline providers—site interview	8	4	1	24	1	24	$36.56	$877.44
CCBHC care managers—site interview	8	2	1	16	1	16	$27.87	$445.92
CCBHC administrative/finance staff—site interview	8	2	1	16	1	16	$89.35	$1,429.60
Telephone interviews
State Medicaid official—telephone interview	8	2	3	48	1	48	$73.55	$3,530.40
State mental health official—telephone interview	8	2	3	48	1	48	$73.55	$3,530.40
State consumer/family representative—telephone interview	8	2	1	16	1	16	$59.71	$955.36
Completion of CCBHC annual progress reports
CCBHC site leadership staff—completion of report	76	1	2	152	4	608	$93.12	$56,616.96
Total	132	16	13	178	16	792	$546.83	$68,876.00

^a We drew average hourly wages for site visit interviews and consumer/family representative telephone interviews from the May 2015 National Occupational Employment and Wage Estimates, United States, as reported by the Bureau of Labor Statistics (https://www.bls.gov/oes/current/oes_nat.htm; accessed January 12, 2017).

^b We compiled average hourly wages for state officials from salary information in state CCBHC demonstration planning grant applications and state employee salary databases.

13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers

There will be no direct costs to the respondents other than their time to participate in the data collection.

14. Annualized Cost to the Government

We estimate that two ASPE employees will be involved for 10 percent of their time. Annual costs of ASPE staff time are estimated to be $22,000. Additional costs are 100 percent of the contract awarded for conducting the CCBHC evaluation by ASPE ($2,098,651.00 over five years, or an annualized cost of $419,730.20). The total estimated average cost to the government per year is $441,730.20.

15. Explanation for Program Changes or Adjustments

This is a new data collection.

16. Plans for Tabulation and Publication and Project Time Schedule
    
    

We will incorporate aggregate results from the national evaluation in text and charts in the following documents we will submit to ASPE:

• A delivery system memo that summarizes key features of the behavioral health delivery systems in demonstration states and the state- and clinic-level changes being made to facilitate the demonstration, due in February 2017. These features include, for example, the number of participating CCBHCs, the geographic location of CCBHCs, partnerships between CCBHCs and other providers, Medicaid managed care arrangements in each state, and how the state plans to process Medicaid claims from CCBHCs.

• An initial implementation report memo that will note any baseline findings, summarize the progress of implementation across sites, and synthesize findings from the first year of the demonstration, due in June 2018. These findings will include factors that have facilitated or impeded implementation progress, hiring and staffing of CCBHCs, the types of services that CCBHCS offer, and plans for sustaining the CCBHC model.

• A second implementation report memo that will update findings from the first memo, due in June 2019

• A cost and quality report, due in June 2019

• A final report describing evaluation data collection, analysis, and findings, due in May 2021

Results may also be incorporated into annual reports to Congress required by PAMA.

Table A.4 provides an overview of the evaluation tasks and in which years we will conduct the tasks.

ASPE may also incorporate the aggregate results from the cross-site evaluation into journal articles, scholarly presentations, and congressional testimony related to the outcomes of the CCBHC demonstration program.

Table A.4 Evaluation tasks timeline

Evaluation timeline	2016	2017	2018	2019	2020
Development of evaluation plan and instrumentation	X
OMB Submission		X
Document review: state applications and Medicaid state plans		X
Initial consultations with states		X
State-level telephone interviews		X	X	X
Site visits to select states			X	X
CCBHC progress report collection		X	X
Analysis of administrative cost reports and quality measures				X	X
Analysis of administrative claims data			X	X	X

17. Reason(s) Display of OMB Expiration Date is Inappropriate

We are requesting no exemption.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification. These activities comply with the requirements in 5 CFR 1320.9.

LIST OF ATTACHMENTS – Section A

Note: Attachments are included as separate files as instructed.

1. CCBHC site leadership interview protocol (Attachment A)

2. CCBHC administration/finance staff interview protocol (Attachment B)

3. CCBHC frontline provider interview protocol (Attachment C)

4. CCBHC care manager interview protocol (Attachment D)

5. CCBHC baseline state Medicaid official interview guide (Attachment E)

6. CCBHC baseline state mental health official interview guide (Attachment F)

7. CCBHC midpoint state Medicaid official interview guide (Attachment G)

8. CCBHC midpoint state mental health official interview guide (Attachment H)

9. CCBHC demonstration end state Medicaid official interview guide (Attachment I)

10. CCBHC demonstration end state mental health official interview guide (Attachment J)

11. CCBHC demonstration end consumer and family representative interview guide (Attachment K)

12. CCBHC clinic demonstration year 1 progress report template (Attachment L)

13. CCBHC clinic demonstration year 2 progress report template (Attachment M)

CCBHC Evaluation, OMB No. 0990-NEW