Supporting Statement A_10162024

Supporting Statement A

Evaluation of the Maternal and Child Health Bureau Pediatric Mental Health Care Access and Screening and Treatment for Maternal Mental Health and Substance Use Disorders Programs Project

OMB Control No. 0906-xxxx

Terms of Clearance: None

A. Justification

1. Circumstances Making the Collection of Information Necessary

In compliance with Section 3506(c)(2)(A) of the Paperwork Reduction Act of 1995, this submission requests Office of Management and Budget (OMB) approval of a 3-year clearance for the Health Resources and Services Administration (HRSA) to conduct an evaluation of the Maternal and Child Health Bureau (MCHB) Pediatric Mental Health Care Access (PMHCA) and Screening and Treatment for Maternal Mental Health and Substance Use Disorders (MMHSUD) programs. The PMHCA Program aims to promote behavioral health integration into pediatric primary care by developing and supporting state, regional, and tribal pediatric mental health care teleconsultation access programs; the MMHSUD program aims to support maternity care providers and clinical practices by supporting the development, improvement, and/or maintenance of statewide or regional behavioral health networks. This project will collect data to provide HRSA with information to guide future program decisions regarding increasing health professionals’ (HPs) capacity to address patients’ behavioral health and access to behavioral health services.

Title X, Section 1002 of the 21^st Century Cures Act (Cures Act) (P.L. 114-255) supported increased access to pediatric mental health care and authorized funding to increase access to pediatric mental health care by supporting the development of new and the expansion of existing PMHCA programs. It authorized the appropriation of $9 million each fiscal year (FY) for FYs 2018-2022 for this initiative (42 U.S.C. 254c–19). This legislative authority includes requirements for evaluation, specifically: “A State, political subdivision of a State, Indian tribe, or tribal organization that receives a grant under this section shall prepare and submit an evaluation of activities that are carried out with funds received under such grant to the Secretary at such time, in such manner, and containing such information as the Secretary may reasonably require, including a process and outcome evaluation.”

Section 2712 of the American Rescue Plan Act (P.L. 117-2) allowed for additional funding for PMHCA programs. In addition to amounts otherwise available, it appropriated to the Secretary of Health and Human Services for FY 2021, out of any money in the Treasury not otherwise appropriated, $80 million to remain available, until expended, for carrying out Section 330M of the Public Health Service Act (amended from 42 U.S.C. 254c–19, with requirements for evaluation as noted above).

Section 11005 of the Bipartisan Safer Communities Act (P.L. 117-159) added new program requirements and allowed for additional funding for PMHCA programs. It authorized the appropriation of $31 million each FY for FYs 2023–2027 for this initiative (amended from 42 U.S.C. 254c-19, with requirements for evaluation as noted above).

Title III, Section 317L-1 of the Public Health Service Act (42 U.S.C. § 247b-13a) assisted screening and treatment for maternal mental health and substance use disorders by supporting the establishment, improvement, or maintenance of MMHSUD programs. This section authorized the appropriation of $24 million for each FY for FYs 2023–2027. This legislation indicates that, “The Secretary, based on evaluation of the activities funded pursuant to this section, shall identify and disseminate evidence-based or evidence-informed practices for screening, assessment, treatment, and referral to treatment services for maternal mental health and substance use disorders, including culturally and linguistically appropriate services, for women during pregnancy and 12 months following pregnancy.” See Attachment A1 for a description of the legislation.

PMHCA and MMHSUD programs support:

• HPs in their delivery of high-quality and timely screening, assessment, treatment, and referrals for their targeted populations (i.e., children, adolescents, and young adults for PMHCA programs; pregnant and postpartum people for MMHSUD programs) through the provision of clinical behavioral health consultation, care coordination support/navigation (i.e., resource identification and referrals), and training and education.

• Access to clinical interventions, including by telehealth.

Additionally, the PMHCA and MMHSUD programs focus on achieving health equity related to racial, ethnic, and geographic disparities in access to care, especially in rural and other underserved areas.

This evaluation will allow HRSA to determine the extent to which the PMHCA and MMHSUD programs have met these objectives. JBS International, Inc. (JBS) will implement this evaluation as part of a contract funded by HRSA (Contract No. 75R60219D00046).

2. Purpose and Use of Information Collection

As Section A.1 states, the goal of this project is to provide HRSA with information to guide future program decisions regarding increasing HPs’ capacity to address patients’ behavioral health and access to behavioral health services.

The evaluation uses a mixed-methods design with data collection activities with HRSA MCHB PMHCA and MMHSUD award recipient programs funded in 2021, 2022, and 2023 and their stakeholders. Methodologies for this study include surveys (e.g., online, mailed) and virtual semi-structured interviews (SSIs).

As Exhibit 1 indicates, the project will collect:

• Quantitative data from HPs and practices enrolled and/or participating in the 2021, 2022, and 2023 PMHCA and 2023 MMHSUD programs and from project leadership implementing the programs (e.g., program-level project directors and principal investigators).

• Qualitative data from 2021, 2022, and 2023 PMHCA and 2023 MMHSUD program stakeholders, including program-level behavioral health consultation providers, program-level care coordinators, program champions, community-based and other resources representatives, and project leadership implementing the programs.

We will administer each data collection instrument once in 2025.

Exhibit 1. PMHCA and MMHSUD Data Collection Activities

Tool	2024	2025	2026
HP Survey	N/A	Spring 2025	N/A
Practice-Level Survey	N/A	Spring 2025	N/A
Program Implementation Survey	N/A	Spring 2025	N/A
Behavioral Health Consultation Provider SSI	N/A	Fall 2025	N/A
Care Coordinator SSI	N/A	Summer 2025	N/A
Champion SSI	N/A	Fall 2025	N/A
Community-Based and Other Resources SSI	N/A	Spring 2025	N/A
Program Implementation SSI	N/A	Winter 2025	N/A

Specifically, HRSA is requesting approval for the following:

HP Survey – survey of enrolled/participating 2021, 2022, and 2023 PMHCA and 2023 MMHSUD program HPs, examining their experiences with the programs (e.g., HP training, how the program meets their consultation and care coordination support needs, access to consultations and referrals, capacity to address behavioral health).

Practice-Level Survey – survey of enrolled/participating 2021, 2022, and 2023 PMHCA and 2023 MMHSUD program office managers/office leadership about how their practice is implementing the program (e.g., enrolled/participating practices’ behavioral health screening, consultation, treatment, and referral practices; community linkages; businesses processes; financial sustainability).

Program Implementation Survey – survey of 2021, 2022, and 2023 PMHCA and 2023 MMHSUD program project directors/principal investigators examining program implementation activities, HP enrollment and training, behavioral health service delivery, care coordination support, community linkages, and sustainability.

Behavioral Health Consultation Provider SSIs – interview with 2021, 2022, and 2023 PMHCA and 2023 MMHSUD program-level behavioral health consultation providers about their program’s behavioral health consultation processes and systems to complement and expand on data collected in other data collection activities (e.g., Program Implementation Survey/SSI).

Care Coordinator SSIs – interview with 2021, 2022, and 2023 PMHCA and 2023 MMHSUD program-level care coordinators about their program’s care coordination processes and systems to complement and expand on data collected in other data collection activities (e.g., Program Implementation Survey/SSI).

Champion SSIs – interview with program champions, identified by 2021, 2022, and 2023 PMHCA and 2023 MMHSUD programs, regarding their involvement with the program (e.g., program implementation, health equity, overall interactions) to complement and expand on data collected in other data collection activities (e.g., HP Survey).

Community-Based and Other Resources SSI – interview with community-based and other resources representatives, identified by 2021, 2022, and 2023 PMHCA and 2023 MMHSUD programs, examining their involvement with the awardee program (e.g., collaboration with the program, impacts of collaboration, health equity). The SSIs will be a case study with up to 50 community-based and other resource representatives across all PMHCA and MMHSUD awardees.

Program Implementation SSIs – interview with 2021, 2022, and 2023 PMHCA and 2023 MMHSUD program project directors/principal investigators examining their programs’ implementation to complement and expand on data collected in the Program Implementation Survey.

These data will assist in understanding the implementation and outcomes of the PMHCA and MMHSUD programs. Specifically, the collected data will be used to:

• Summarize the efforts of awardee programs to achieve key outcomes.

• Measure whether and to what extent awardee program activities are associated with changes in HPs’ and practices’ capacity to address patients’ behavioral health and access to behavioral health care.

• Examine changes over time, within and/or across PMHCA and MMHSUD programs, regarding (1) screening, assessment, treatment, and referral for behavioral health conditions among enrolled/participating HPs/practices; (2) provision of behavioral health consultation, care coordination, and training/education; and (3) access to behavioral health services for children, adolescents, and young adults for PMHCA programs and pregnant and postpartum people for MMHSUD programs.

• Provide the data in reports and presentations to HRSA MCHB.

• Develop resources for dissemination by HRSA MCHB.

Supporting Statement B contains additional information on study procedures on the collection of information using these data collection tools, as well as the data collection tools (as attachments).

3. Use of Improved Information Technology and Burden Reduction

The evaluation of the MCHB PMHCA and MMHSUD programs will follow a multimethod approach. Data collection methodologies for this evaluation will use surveys (i.e., web-based, email) and virtual SSIs (e.g., Microsoft Teams, Zoom). All technology used for the survey administration (i.e., web-linked survey administered via email and via survey platform) will meet federal requirements for Section 508 accessibility. Information technology (IT) will be used in the following ways:

• All survey participants will receive the web-linked survey via email. Electronic responses will be downloaded directly into a securely stored server.

• All SSIs will be conducted via a web-based platform (e.g., Microsoft Teams, Zoom). For respondents who agree to be recorded, interviewers will record responses as they are given and upload the recordings to a secured server. For respondents who do not agree to be recorded, a notetaker will record responses and upload call notes to a secured server.

• Reports and materials (e.g., resources) generated from this project may be made available to the public.

We selected the data collection methods for the evaluation because they will reduce participant burden while providing the evaluation with necessary data. Offering a web-based survey reduces burden to participants by eliminating the time it takes to write responses on a paper-based, mail-in survey. In addition, having participants respond to an online survey eliminates the time needed to mail back a paper-based survey. This reduces the burden for respondents participating in interviews via a web-based platform (e.g., Microsoft Teams, Zoom) because they will not have to write down responses to the questionnaires or travel to participate in an in-person interview.

Using protected electronic data is the most secure form of data management because it eliminates the possibility of either paper documents or data being lost in transit or delivered to an incorrect location. However, because not all respondents may prefer to complete a web-based survey (e.g., due to IT security concerns or firewall issues), and to maximize completion rates, we may use alternative forms of administration (e.g., providing a printable PDF to participants). In this case, the respondents can return the printable PDF surveys either as attachments through encrypted emails or via mail or fax, depending on their preference. All hard copies will be submitted to JBS with unique alphanumeric identifiers, and the data will be entered into the online system at JBS. JBS will store hard copies in a locked file cabinet, with no name or identifying information attached.

4. Efforts to Identify Duplication and Use of Similar Information

Since the initial evaluation (September 2018–September 2021) of the PMHCA and Screening and Treatment for Maternal Depression and Related Behavioral Disorders (MDRBD; now MMHSUD) programs funded by HRSA in 2018 and 2019, of which the current evaluation (September 2021–September 2026) is a continuation to incorporate awardees funded in 2021, 2022, and 2023, no evaluations of HRSA MCHB PMHCA and MDRBD/MMHSUD programs among any study population have occurred. The initial evaluation was conducted using information collections approved by OMB under control numbers 0906-0052 and 0906-0074. The lack of evaluative studies is due to the Cures Act legislation authorizing these programs in 2016 and the first cooperative agreement programs funded starting in FY 2018. Because evaluation of these cooperative agreement-funded programs is ongoing, there is no similar or existing evaluation of all HRSA-funded PMHCA and MDRBD/MMHSUD programs (i.e., no other bureaus or agencies are currently evaluating the programs).

An impact study of the HRSA MCHB PMHCA cooperative agreement-funded programs is also being funded by HRSA and conducted by JBS (Contract Number: 75R60219D00046/Task Number: 75R60223F34003) and is ongoing through 2026. However, no duplication of efforts between the evaluation and impact study exists because we developed the data collection instruments for the impact study considering both the HRSA-required data awardees already report and the data collected to support the evaluation of the programs, as detailed in this package.

In addition, there is no duplication of information within this evaluation because we also developed the data collection surveys and SSI guides while considering the data awardees are required to report to HRSA based on their cooperative agreements. We mapped all potential data items to the evaluation questions to ensure no duplication of information and to reduce participant burden.

5. Impact on Small Businesses or Other Small Entities

Data collection efforts (i.e., surveys) for this evaluation include physicians, as part of participating HPs, and participating practices. Although a portion of physicians may be employed by large hospitals or health systems, none of which are considered small businesses, some may be in a private practice or practice in small groups of physicians. Similarly, some participating practices may be part of large systems and, therefore, are not considered small businesses, whereas others may be private practices. Information collected for this evaluation is not anticipated to have a significant impact on physicians or practices.

The information to be obtained from participating physicians and practices is the minimum required for the intended use of the data and to achieve the objectives of the evaluation; however, completion of survey instruments will likely induce minimum burden. To reduce this burden, we have developed the survey to be as short as possible, while still collecting necessary data, and made attempts to move respondents quickly through questions (e.g., adding skip patterns to the surveys so respondents do not need to answer questions not relevant to them).

6. Consequences of Collecting the Information Less Frequently

As Section A.2 notes, the collection of these data is critical to evaluating the PMHCA and MMHSUD programs. We will administer each instrument once, with anticipated data collection in 2025. The frequency of data collection is held to the minimum necessary to meet the needs of the evaluation goals and objectives.

It is important to highlight that OMB approved the current evaluation (September 2021–September 2026) to collect data with the PMHCA programs funded in 2021 and 2022 and their stakeholders (OMB Control No. 0906-0074; expiration December 31, 2025). With this approval, the 2021 and 2022 PMHCA programs’ project directors/principal investigators and enrolled/participating HPs and practices participated in survey data collection twice, in 2023 and 2024. Conducting annual surveys enables the examination of changes over time (e.g., HPs’ capacity to address behavioral health; program provision of behavioral health consultations, care coordination, and training; practice operations). Additionally, 2021 and 2022 PMHCA program-level care coordinators participated in interviews once in 2023 or in 2024, as applicable. Similarly, collecting qualitative data at multiple time points allows for examination of changes in processes and systems among care coordinators overtime.

Following funding of the 2023 PMHCA and MMHSUD programs, we modified the data collection instruments and are submitting this package to (1) include the 2023 PMHCA and MMHSUD awardees in the evaluation and (2) integrate the 2021 and 2022 PMHCA awardees into the revised data collection instruments presented in this package. The previously approved OMB package (OMB Control No. 0906-0074) covering the 2021 and 2022 PMHCA programs should be discontinued following approval of this package.

Since this evaluation concludes in 2026, the PMHCA and MMHSUD awardees and their stakeholders will only be eligible to participate in data collection once in 2025, following approval of this OMB submission.

HP Surveys. The HP Survey will be administered via a web-based platform (i.e., Alchemer) or by paper once to HPs enrolled in 2021, 2022, and 2023 PMHCA and 2023 MMHSUD programs, with anticipated data collection in 2025. The survey will collect information from HPs’ regarding their experience with the program.

Practice-Level Surveys. The Practice-Level Survey will be administered via a web-based platform (i.e., Alchemer) or by paper once to practice managers of practices enrolled/participating in 2021, 2022, and 2023 PMHCA and 2023 MMHSUD programs, with anticipated data collection in 2025. The survey will collect information from office management/leadership, including their practice’s experience with the program.

Program Implementation Surveys. The Program Implementation Survey will be administered via an online platform (i.e., Alchemer) or by paper once to project directors/principal investigators from the 2021, 2022, and 2023 PMHCA and 2023 MMHSUD awardees, with anticipated data collection in 2025. The survey will address program implementation activities, health professional enrollment and training, behavioral health service delivery, care coordination support, and sustainability.

Behavioral Health Consultation Provider SSI. Behavioral Health Consultation Provider SSIs will be administered via a web-based platform (e.g., Microsoft Teams, Zoom) to program-level behavioral health consultation providers from the 2021, 2022, and 2023 PMHCA and 2023 MMHSUD awardees, with anticipated data collection once in 2025. The SSIs will address behavioral health consultation providers’ involvement with the program, behavioral health consultation processes and requests, health equity, and program usefulness.

Care Coordinator SSI. Care Coordinator SSIs will be administered via a web-based platform (e.g., Microsoft Teams, Zoom) to program-level care coordinators from the 2021, 2022, and 2023 PMHCA and 2023 MMHSUD awardees, with anticipated data collection once in 2025. The SSIs will address care coordinators’ involvement with the program, community-based and other resource connections and referrals, health equity, change over time, and lessons learned.

Champion SSIs. Champion SSIs will be administered via a web-based platform (e.g., Microsoft Teams, Zoom) to program champions identified by the 2021, 2022, and 2023 PMHCA and 2023 MMHSUD awardees, with anticipated data collection once in 2025. The SSIs will address champions’ involvement with the program, overall interactions, program implementation, health equity, and program outcomes.

Community-Based and Other Resources SSI. Community-Based and Other Resources SSIs will be administered once via a web-based platform (e.g., Microsoft Teams, Zoom) to community-based and other resources representatives collaborating with PMHCA and MMHSUD programs, with anticipated data collection in 2025. The SSIs will be a case study with up to 50 representatives across all 67 PMHCA and MMHSUD programs funded in 2021, 2022, and 2023. JBS will work with designated staff from the cooperative agreement programs to identify and select representatives who should participate in the SSI. The SSIs will address collaboration with the program, impacts of collaboration, and health equity.

Program Implementation SSIs. Program Implementation SSIs will be administered once via a web-based platform (e.g., Microsoft Teams, Zoom) to project directors/principal investigators from the 2021, 2022, and 2023 PMHCA and 2023 MMHSUD awardees, with anticipated data collection in 2025. Topics will be similar to the Program Implementation Surveys but provide project directors/principal investigators an opportunity to discuss program implementation toward the end of the project period.

There are no legal obstacles to reduce the burden.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

HRSA has elected to implement OMB’s SPD-15 expanded race and ethnicity questions for respondent self-identification questions in the HP Survey. HRSA has elected to include OMB’s SPD-15 race and ethnicity question with minimum categories (i.e., American Indian or Alaska Native, Asian, Black or African American, Hispanic or Latino, Middle Eastern or North African, Native Hawaiian or Pacific Islander, White) in the HP Survey and Practice Level surveys for questions where health professionals are asked to assess the race and/or ethnicity breakdown for patients that they treat. This is because the potential benefit of more detailed data in this collection would not justify the additional burden to the agency nor the public. Asking survey respondents to estimate patient race and ethnicity categories would add additional burden to them and would likely result in unreliable and/or missing data, which would limit the quality of our data. For the SSIs, we will collect participants’ names and emails prior to participation for the purpose of scheduling. Otherwise, the request fully complies with the regulation.

8. Comments in Response to the Federal Register Notice/Outside Consultation

Section 8A

A 60-day Federal Register Notice was published in the Federal Register on May 28, 2024, 89 FR 46143-44 (see Attachment A2). HRSA received two public comments. Attachment A3 provides additional details about the public comments and our responses. No substantive changes were made as a result of the comments to the current information collection described in this package.

A 30-day Federal Register Notice was published in the Federal Register on October 9, 2024, 89 FR 81920-22. One comment was received in support of the overall program, due to the importance of addressing substance use disorders among expecting mothers.Section 8B

Consultations on the evaluation design, data collection instruments (i.e., HP Survey, Practice-Level Survey, Program Implementation Survey, Program Implementation SSI) and protocols, data management, and analysis of the initial evaluation of programs funded by HRSA in 2018 and 2019 (September 2018–September 2021) occurred throughout the planning phase of the initial project. The current evaluation (September 2021–September 2026) is a continuation of the initial evaluation to incorporate additional awardees. We have refined the current evaluation design and data collection instruments based on information learned in the previous evaluation and to account for differences in the new awardee program requirements. These consultations have provided, and will continue to provide, the opportunity to:

• Ensure the technical quality and appropriateness of the overall evaluation design and data analysis plans

• Obtain advice and recommendations concerning the data collection instruments

• Structure the evaluation and instruments to minimize overall and individual response burden

Consultations have occurred with the following individuals in connection with this study:

• Medical Director of Behavioral Health, Chicago Department of Public Health. Years and areas of consultation: 2022, clinical expertise

• Director of Special Projects Massachusetts Behavioral Health Partnership, Co-Founder National Network of Child Psychiatry Access Programs. Years and areas of consultation: 2018‒present; representative of those from whom information is to be obtained

• Professor, Behavioral and Community Health, University of Maryland School of Public Health. Years and areas of consultation: 2019–2021, methodological and analytic expertise

Additionally, for the previous OMB submission for the evaluation (OMB Control No. 0906-0074; expiration 12-31-25), JBS conducted pilot tests of all the data collection tools, except for the Behavioral Health Consultation Provider SSI. Prior to the current OMB submission, we piloted the Behavioral Health Consultation Provider SSI. Supporting Statement B contains additional information on pilot tests of the data collection tools to be used in the evaluation and Attachment B17 provides a summary of pilot test feedback for the Behavioral Health Consultation Provider SSI instrument and outlines the changes made to the data collection tools based on this feedback.

9. Explanation of Any Payment/Gift to Respondents

Respondents will not receive any payments or gifts.

10. Assurance of Confidentiality Provided to Respondents

This data collection effort will collect personally identifiable information (PII) from participating HPs and practice managers in the form of ZIP Codes and email addresses. Collecting email addresses allows us to track survey completion. Zip Codes will be used to link survey responses to secondary data sources for the purpose of identifying variation in implementation outcomes by location. Once data collection is complete, we will deidentify data by removing email addresses from the final analytic file. Survey respondents will be asked to construct a unique ID using the first two letters of the first and last name, and the month of their birthday, which will serve to link individual survey responses over time. The unique ID does not contain PII and will not be stored as PII. Because not all respondents may prefer to complete a web-based survey, and to maximize completion rates, we may use alternative forms of administration (i.e., providing a printable PDF to participants). In this case, the printable PDF surveys can be returned either as attachments through encrypted emails or via mail or fax, depending on the respondent’s preference. All hard copies will be entered into the online system at JBS and stored in a locked file cabinet, with participants’ names and identifying information removed. Additionally, we will collect PII from program implementers, care coordinators, behavioral health consultation providers, champions, and community-based and other resource representatives including their first and last names and email addresses for the purposes of scheduling interviews. Any electronic data collected (i.e., electronic survey data, interview recordings) will be downloaded directly into a securely stored server and will only be accessible to the study team. All data and information from participants will be stored in the secure facilities for 10 years after the study is completed, and we will adhere to Federal requirements regarding collection and storage of PII. No data will not be stored in a way that is retrieved by a personal identifier.

We will assure all respondents that their data will be kept private to the extent allowed by law. In addition, communications to inform participants about the data collection and any other introductory materials about the data collection will indicate HRSA’s Federal status and the purpose of the data collection. Please see Attachments A4-A45 for communication/recruitment materials (e.g., email notifications). Supporting Statement B contains additional information on study procedures related to the communications.

11. Justification for Sensitive Questions

Personally identifiable information (PII), including participants’ names and email addresses, will be collected for administration of the surveys and SSIs. The surveys do not ask for information of a sensitive nature (e.g., sexual practices, alcohol or drug use, religious preference) other than race and ethnicity. Specifically, HP Survey respondents will be asked for their race and ethnicity. Collection of these data are necessary for the evaluation because a diverse workforce is important to patient-clinician communication and to access to care for patients belonging to minority populations.¹

All data and information from participants will be stored in the secure facilities for 10 years after the study is completed, and we will adhere to federal requirements regarding collection and storage of PII.

The study meets the Common Rule definitions for human subjects research (45 CFR 46, Regulations for Protection of Human Subjects); however, the JBS Institutional Review Board (IRB) determined that this research is exempt under 45 CFR 46.101(b)(5) from 45 CFR Part 46 requirements (see Attachment A46).

12. Estimates of Annualized Hour and Cost Burden

This section summarizes the total burden hours for this information collection effort in addition to the cost associated with those hours.

12A. Estimated Annualized Burden Hours

Exhibit 2 and 2a contains estimated response burdens for each subject population participating in the evaluation’s data collection activities. Exhibit 2 shows the estimated burden by type of respondent and Exhibit 2 shows the estimated burden by form.

We calculated estimates for the response-hour burden (1) based on the methodology being used with each respondent population and (2) using the average completion time based on instrument pilot testing. Supporting Statement B contains additional information on pilot tests of the data collection tools to be used in the evaluation, as well as summaries of pilot test feedback and changes made to the data collection tools based on this feedback.

It should be noted that the 60- and 30-day Federal Register Notices were aggregated by types of respondents (e.g., physician; nurse practitioner; physician assistant; counselor, social worker, and other community and social services specialist) and are disaggregated here to more accurately calculate total respondent costs by potential types of respondents who may participate in each data collection instrument.

Exhibit 2. Estimated Annualized Burden Hours (by Type of Respondent)

Type of Respondent	Form Name	No. of Respondents	No. of Responses per Respondent	Average Burden per Response (in hours)	Total Burden Hours
Physician	HP Survey	14,884	1	.33	4,911.72
Nurse Practitioner	HP Survey	4,419	1	.33	1,458.27
Physician Assistant	HP Survey	1,395	1	.33	460.35
Counselor, Social Worker, and Other Community and Social Service Specialist	HP Survey	1,860	1	.33	613.8
Other Health Care Professional/ Support Worker	HP Survey	698	1	.33	230.34
Practice Manager	Practice-Level Survey	6,172	1	.33	2,036.76
Project Director/ Principal Investigator	Program Implementation Survey	67	1	.33	22.11
Psychiatrist	Behavioral Health Consultation Provider SSI	30	1	.75	22.5
Psychologist	Behavioral Health Consultation Provider SSI	10	1	.75	7.5
Health Education Specialist/ Community Health Worker	Behavioral Health Consultation Provider SSI	12	1	.75	9
Counselors, Social Workers, and Other Community and Social Service Specialists	Behavioral Health Consultation Provider SSI	9	1	.75	6.75
Advanced Practice Nurse	Behavioral Health Consultation Provider SSI	6	1	.75	4.5
Health Education Specialist/ Community Health Worker	Care Coordinator SSI	67	1	.75	50.25
Champion	Champion SSI	67	1	.5	33.5
Counselors, Social Workers, and Other Community and Social Service Specialists	Community-Based and Other Resources SSI	50	1	.5	25
Project Director/ Principal Investigator	Program Implementation SSI	134	1	1	134
Total		29,880			10,026.35*

* Rounds down to 10,026 in ROCIS.

Exhibit 2a. Estimated Annualized Burden Hours (by Form)

Form Name	Number of Respondents	Number of Responses per Respondent	Total Responses	Average Burden per Response (in hours)	Total Burden Hours
HP Survey	23,256	1	23,256	0.33	7,674.48
Practice-Level Survey	6,172	1	6,172	0.33	2,036.76
Program Implementation Survey	67	1	67	0.33	22.11
Behavioral Health Consultation Provider SSI	67	1	67	0.75	50.25
Care Coordinator SSI	67	1	67	0.75	50.25
Champion SSI	67	1	67	0.50	33.50
Community-Based and Other Resources SSI	50	1	50	0.50	25.00
Program Implementation SSI	134	1	134	1.00	134.00
Total	29,880		29,880		10,026.35*

* Rounds down to 10,026 in ROCIS.

12B. Estimated Annualized Burden Costs

Exhibit 3 summarizes the estimated, annualized cost burden to respondents of the evaluation. We obtained median hourly wage estimates and occupational profile codes from the Bureau of Labor Statistics using wage estimates from 2023 (the most recently available estimates). We calculated the total respondent cost as (hourly wage rate X 2 [to account for overhead costs]) X (time spent on the instrument X number of responses).

Exhibit 3. Estimated Annualized Burden Costs

Type of Respondent (Occupational Profile Code)	Total Burden Hours	Hourly Wage Rate*	Total Respondent Costs
Physicians (29-1215;29-1216;29-1221)	4,911.72	$207.26	$1,018,003.09
Nurse Practitioners (29-1171)	1,458.27	$121.40	$177,033.98
Physician Assistants (29-1071)	460.35	$125.02	$57,552.96
Counselor, Social Worker, and Other Community and Social Service Specialist (21-000)	613.8	$50.00	$30,690.00
Other Health Care Professionals (29-0000)	230.34	$77.72	$17,902.02
Practice Manager (11-9111)	2,036.76	$106.42	$216,752.00
Project Director/Principal Investigator (19-3099)	22.11	$92.20	$2,038.54
Psychiatrist (29-1223)	22.5	$230.00	$5,175.00
Psychologist (19-3030)	7.5	$89.18	$668.85
Health Education Specialist/Community Health Worker (21-1091, 21-1094)	9	$53.39	$480.51
Counselor, Social Worker, and Other Community and Social Service Specialist (21-000)	6.75	$50.00	$337.50
Advanced Practice Nurse (29-1170)	4.5	$121.40	$546.30
Health Education Specialist/Community Health Worker (21-1091, 21-1094)	50.25	$53.39	$2,682.85
Champions (All Occupations; 00-0000)	33.5	$46.22	$1,548.37
Counselor, Social Worker, and Other Community and Social Service Specialist (21-000)	25	$50.00	$1,250.00
Project Director/ Principal Investigator	134	$92.20	$12,354.80
Total	10,026.35		$1,545,016.77

*SOURCE: U.S. Department of Labor, Bureau of Labor Statistics. (2024, April). Occupational employment and wage statistics. https://www.bls.gov/oes/current/oes_stru.htm

13. Estimates of Other Total Annual Cost Burden to Respondents or Recordkeepers/Capital Costs

Other than time, there is no cost to respondents.

14. Annualized Cost to Federal Government

The cost to the Federal Government for this 5-year project is $4,023,331 or $804,666 per year on average. These costs cover all aspects of survey design, testing, data collection, and analysis. The method used to estimate the cost includes preparation of a detailed line-item budget that specifies all staff/consultant rates and labor hours by task, along with operational and other direct costs (e.g., telephone calls, reproduction).

In addition, it is estimated that 1 full-time equivalent HRSA staff member (Grade 13, Step 5) will spend 20 percent of their time (384 hours) to manage and administer the project. Assuming an annual salary of $200,538 ($133,692 X 1.5 to account for benefits), government personnel costs will be $40,107.60 ($26,738.40 X 1.5 to account for benefits) over a 1-year period.²

15. Explanation for Program Changes or Adjustments

This is a new information collection effort. The previously approved OMB package related to this collection (OMB Control No. 0906-0074; expiration December 31, 2025) covering the 2021 and 2022 PMHCA programs should be discontinued following approval of this package.

16. Plans for Tabulation, Publication, and Project Time Schedule

Project Time Schedule. As Exhibit 4 shows, the project covers a 3-year period commencing upon receipt of OMB approval.

Exhibit 4. Project Time Schedule

Activity	Time Schedule
Obtain OMB approval	Winter 2025 (January-February)
Administer HP Survey, Practice-Level Survey, and Program Implementation Survey	1–2 months following OMB approval
Administer Community-Based and Other Resources SSI	1–4 months following OMB approval
Administer Care Coordinator SSI	4–7 months following OMB approval
Administer Champion SSI and Behavioral Health Consultation Provider SSI	7–10 months following OMB approval
Administer Program Implementation SSI	8–11 months following OMB approval
Data analysis	Beginning 4 months following OMB approval
Dissemination of findings through interim reports, infographics, and final report	Beginning 4 months following OMB approval through 2026

Analysis Plan. The HRSA MCHB evaluation will encompass the use of multiple instruments, collection of information, and analytical strategies. We will collect and analyze both qualitative and quantitative data to assess HPs’ capacity to address patients’ behavioral health and access to behavioral health services among PMHCA and MMHSUD programs. Qualitative data analysis will use a thematic approach to uncover underlying themes among the SSI responses. Quantitative data analyses will include the use of descriptive statistics, univariate analysis, and multivariable analysis. Finally, we will use triangulation of methods (i.e., qualitative and quantitative data), when feasible, to examine additional aspects of program achievements that may not be accomplished with individual methods. The planned qualitative and quantitative data analyses are explained in more detail in the remainder of this section.

Qualitative Data Analysis: Analysis will begin with JBS cleaning transcripts based on recordings of the interviews. Data will be analyzed and coded both deductively (i.e., based on pre-existing concepts) and inductively (i.e., concepts arising from the transcripts) by a team using a process of thematic analysis. To guide analysis, the team will develop a qualitative codebook that contains initial codes (i.e., conceptual tags to apply to chunks of text) derived from the research and guide questions. The codebook will include code descriptions, inclusion and exclusion criteria, and exemplars (i.e., example quotations) from the transcripts. The team lead, a trained qualitative researcher, will develop an analysis protocol, including use of the codebook, upon which the team will be trained.

Analysis will begin with a deep reading of the transcripts to promote overall understanding of the perspectives. A primary analyst will code each transcript, with coding assessed by a reviewer. Primary coders and reviewers will use electronic memos embedded in the transcript files to exchange questions and answers between primary coders and reviewers and to refine coding. Interrater reliability (i.e., consistency between coders) will be accomplished by comparing analyses during weekly team meetings and through the primary coder and reviewer; we will also make use of a software algorithm for interrater reliability using ATLAS.ti. Team use of ATLAS.ti qualitative data analysis software will facilitate the coding and analytic process.

As coding progresses, the analysts will identify potential key themes and subthemes. After finishing coding of the transcripts, a primary analyst and reviewer will be assigned to produce a summary of findings for each applicable research question, based on the coding and identification of themes, including example de-identified quotations from the participants. During the summarization process, the team will discuss fit between the constructed themes and the coded data and refine the summaries as needed.

To enhance reliability and validity, the study team will triangulate findings by data type (i.e., quantitative data, qualitative data) and data source (i.e., qualitative interview source). Study team members will convene a series of analytical working sessions to compare findings derived by data types and sources for each research question to identify potential areas requiring further analysis and refinement.

Quantitative Data Analysis: The quantitative data for this evaluation will come from the HP, Practice-Level, and Program Implementation Surveys and will assess HPs’ capacity to address patients’ behavioral health and access to behavioral health services as a result of the PMHCA and MMHSUD programs. We will collect quantitative data from PMHCA and MMHSUD enrolled/participating HPs, medical practice managers/leadership, and project directors/principal investigators. Selection of statistical analyses are determined by the evaluation questions, measurements of variables, type of sampling, and number of independent variables and outcome variables, as well as by sample size.

Descriptive Statistics: The HRSA MCHB evaluation will use descriptive statistics to describe PMHCA and MMHSUD cooperative agreement-funded programs and their programmatic activities. The purpose of descriptive analysis will be to understand the distribution of variables of interest, as well as to assess the accuracy of measurements, identify sources of error, and provide descriptive information. We will run frequencies or calculate the means and standard deviations of each variable to examine the central tendency and distribution of variables. Knowledge of the distribution of data will inform the use of proper statistical techniques to conduct further analyses moving forward.

We also will conduct analyses to identify random or systematic errors (e.g., instrumental noise) and to assess for missing values, because it is important to determine the potential bias of missing values. To address these potential limitations, the analysis team will consider the imputation of missing values for all variables with a large number of missing values. The team will (1) review, select, and apply the most efficient method, based on careful consideration of the data set and type of missing data, and (2) conduct cross tabulations to examine the relationships between the variables. The degree and statistical significance of association between variables is important not only for reporting relationships of interest, but also for supporting higher level analyses.

Univariate Analysis: Conducting univariate analysis will allow the analysis team to examine associations to identify variables associated with the outcomes of interest. These analyses may include contingency tables and chi-square tests for independence (Pearson’s), t-tests and univariate analysis of variance, linear logistic models when the dependent variable is binary, linear regression when the dependent variable is continuous, and/or Poisson regression when the dependent variable is measured in counts. These analyses will inform the structure and variables included in multivariable analysis.

Multivariable Analysis: The multivariable analysis will provide information about implementation outcomes of PMHCA and MMHSUD programs—direct and indirect and intended and unintended. Based on the evaluation questions and variable identification through the descriptive statistics and univariate analysis, the appropriate multivariable analysis will be determined and applied (e.g., linear or logistic regression, generalized estimating equations, multilevel analysis, cluster analysis, structural equation modeling). When interpreting the results of the analyses, we will evaluate both the statistical significance and the practical importance of the findings. We will compare the magnitude of changes with the literature and with practically meaningful standards.

For example, to determine whether and how HPs’ access to clinical behavioral health consultation has changed over time, a key consideration is the identification of a variable for “health professionals’ access.” The analysis team will review the data from the descriptive statistics and univariate analysis, as well as from a cluster analysis, if performed, and confirm the data time points that can be used to examine change over time. Based on this information, the analysis team will determine the appropriate multivariable analyses.

Publication Plan. As Section A.2 stated, the goal of the evaluation of the PMHCA and MMHSUD programs is to provide HRSA with information to guide future program decisions regarding increasing HPs’ capacity to address patients’ behavioral health and access to behavioral health services. It is therefore important to prepare and disseminate information that clearly and concisely presents evaluation results so that they can be appreciated by both technical and nontechnical audiences. Currently, our publication plan is tentative and includes the following:

• Preparing and submitting to HRSA:

• Resources on best practices and lessons learned incorporating evaluation data. Tentatively, these resources will be released at the end of the contract and published on the public-facing HRSA website. The documents will be provided in Word format and comply with federal requirements for Section 508 accessibility.

• Documents on awardee-level findings for each awardee. Tentatively, these documents will be released at the end of the contract and published on the public-facing HRSA website. Only aggregate PMHCA and MMHSUD findings will be shared (i.e., individual program data will not be shared to ensure program confidentiality). The documents will be provided in Word format and comply with federal requirements for Section 508 accessibility.

• Annual interim evaluation reports. Interim evaluation reports will not be published as they are intended for internal use and are part of our ongoing performance monitoring and quality improvement efforts.

• Final evaluation report. Tentatively, this final report will be released at the end of the contract and published on the public-facing HRSA website. This final report will include aggregate results. This document will be provided in Word format and comply with federal requirements for Section 508 accessibility.

• Hosting annual evaluation design presentations incorporating evaluation data

17. Reason Display of OMB Expiration Date Is Inappropriate

The OMB number and expiration date will be displayed on every page of every form/instrument.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification.

1 U.S. Department of Health and Human Services. (2017, August). Sex, race, and ethnic diversity of U.S. health occupations (2011‒2015). Health Resources and Services Administration, National Center for Health Workforce Analysis. https://bhw.hrsa.gov/sites/default/files/bhw/nchwa/diversityushealthoccupations.pdf

2 SOURCE: Salary Table 2024-DCB. U.S. Office of Personnel Management. (n.d.). https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/2024/general-schedule

21