5. Supporting Statement B N-SUMHSS 2023_final _202308

5. Supporting Statement B N-SUMHSS 2023_final _202308.docx

National Substance Use and Mental Health Services Survey (N-SUMHSS) [CBHSQ]

OMB: 0930-0386

Document [docx]

Download: docx | pdf

National Substance Use and Mental Health Services Survey (N-SUMHSS)

SUPPORTING STATEMENT

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

Respondent Universe and Sampling Methods

The National Substance Use and Mental Health Services Survey (N-SUMHSS) is a multimode (web, paper, and computer-assisted telephone interview [CATI]) data collection of all known and eligible substance use and mental health treatment facilities currently operating in the United States. The N-SUMHSS collects information on the location, characteristics, and utilization of substance use and mental health treatment services at the facility level.

The Inventory of Substance Use and Mental Health Treatment Facilities (I-TF) is the list frame for the N-SUMHSS. The table below outlines the N-SUMHSS respondent universe derived from the I-TF:

Treatment Facilities in the I-TF as of March 22, 2023:

SUBSTANCE

USE FACILITIES

MENTAL HEALTH FACILITIES

TOTAL TREATMENT FACILITIES ON I-TF

24,282

15,394

39,676

N-SUMHSS Inclusion and Exclusion Criteria

Types of substance use and mental health treatment facilities included in the N-SUMHSS include: psychiatric hospitals, general hospitals with a separate inpatient substance use and/or psychiatric unit, state hospitals, Veterans Affairs (VA) medical centers, certified community behavioral health clinics, partial hospitalization/day treatment facilities, outpatient facilities, residential treatment centers for children, residential treatment centers for adults, multi-setting mental health facilities, community mental health centers and other types of residential treatment facilities.

The N-SUMHSS survey universe excluded: (1) Department of Defense (DoD) military mental health treatment facilities; (2) individual private practitioners or small group practices not licensed as a substance use and/or mental health clinic or center; and (3) jails or prisons. Mental health facilities are not eligible for inclusion in the survey universe if they only provide one or more of the following services: crisis intervention services, psychosocial rehabilitation, cognitive rehabilitation, intake, referral, mental health evaluation, health promotion, psychoeducational services, transportation services, respite services, consumer-run/peer support services, housing services, or legal advocacy. Residential facilities whose primary function is not to provide specialty mental health treatment services are also not eligible for inclusion in the survey universe for the N-SUMHSS. Halfway houses that do not provide substance use treatment are included in the I-TF and survey universe so that they could be listed in the National Directory of Drug and Alcohol Use Treatment Facilities and on SAMHSA’s FindTreatment.gov. These facilities are excluded from analyses and public use data files.

Sampling
There is currently no sampling in the N-SUMHSS. All N-SUMHSS eligible treatment facilities are surveyed because the N-SUMHSS is the only source of information for SAMHSA’s FindTreatment.gov, the National Directory of Drug and Alcohol Use Treatment Facilities, and the National Directory of Mental Health Treatment Facilities. Without surveying the respondent universe, SAMHSA would not have adequate information on substance use and mental health facilities for individuals seeking treatment. A sample of treatment facilities would not be feasible for FindTreatment.gov because information for all facilities would not be listed.

Additionally, a full survey for all eligible facilities is needed to gain information on client counts every year. This is important and necessary for the following reasons:

The substance use and mental health fields and states are in need of not only national, but state-level and sub-state statistical data to conduct comparative analyses of access to care (needs assessment) and service utilization across the various components of the substance use and mental health treatment service delivery system.

The behavioral health treatment service delivery system is comprised of various types of treatment through many different operation structures. Within those basic types of treatments are specific types of facilities that serve specific needs or populations. Some facilities are in metropolitan areas and others are in rural areas. Some facilities are in established networks where resources are shared and others are stand-alone facilities. Also, among these many various treatment types and offerings, there are the various operating structures: private for profit, private nonprofit, state government, local/community/county government, tribal government, and federal government, with breakouts of DoD , VA, Indian Health Service, and other federal government.

The ability to generate county- and state-level figures would be lost using a sampling method. There are many different combinations of types of treatment and attributes of the treatment system that would need to be considered.

Accurate numbers of clients are needed by the White House, Congress, states, and counties for trends projections so that adequate resources can be allocated. States and counties use the N-SUMHSS numbers of clients to inform funding decisions and to educate their communities as to if there is a need for increased treatment.

If only a sample of facilities collected client caseload data in the 2024 N-SUMHSS, for example, then the ability to use measure of size calculations for the next N-SUMHSS would be lost. The variables used for measure of size (i.e., reported counts of persons in treatment within a type of treatment as of the survey reference date) would only be available from the 2024 sampled facilities that remain in the universe and not the universe of substance use treatment facilities.

Response rate

The most recent response rate for the N-SUMHSS was about 88% in 2022. SAMHSA expects response rates for the information collection to maintain at a similar level.

Information Collection Procedures

There is no statistical methodology for stratification and sample selection, estimation procedure, or unusual problems requiring specialized sampling procedures. The information collection procedures are as follows:

N-SUMHSS:

The N-SUMHSS will be conducted through an online web survey, with a mail questionnaire option and a computer-assisted telephone interview (CATI) follow-up with non-respondents. The N-SUMHSS Veterans Affairs (VA) Supplement and the Electronic Health Record (EHR) Supplements will be implemented at the same time as the regular N-SUMHSS and follow the same information collection procedures. The N-SUMHSS EHR supplement will be administered once between 2024 and 2026.

Advance Contact with Facilities

Advance letters and emails announcing the upcoming N-SUMHSS administration are sent to treatment facilities in late February, approximately six weeks before the launch of the survey. The letter was printed on SAMHSA letterhead with the N‑SUMHSS wordmark and signed by the Behavioral Health Services Information System (BHSIS) Project Officer. An email version of this letter with the N-SUMHSS wordmark are also sent to the facilities. The advance contact served as an introduction to the N‑SUMHSS to encourage facilities’ participation and reinforce that the effort is nationwide and involves treatment facilities across the country.

Invitation Packet(s)

The first invitation packet is sent via mail and email in late March. The invitation packet includes a personalized cover letter, an N-SUMHSS invitation letter (if received from the states, territories, VA, DOD, IHS), a personalized web survey access flyer with instructions and help desk contact information, FAQs and a Client Counts Worksheet. Facilities that have not responded by around late May and by middle July will be sent a second and third packet including a cover letter, an endorsement letter, a personalized web survey access flyer, a Client Counts Worksheet and a brochure. A hard copy of paper survey along with postage paid business reply envelop will also be included in the third packet.

Thank-you Letters and Reminder Emails

The Thank-you letter distribution starts in May and will continue weekly throughout the survey administration period. Up to 11 reminder emails will be sent to facilities that have not responded to the N‑SUMHSS approximately 1 week, 3 weeks, 9 weeks, 26 weeks, 29 weeks, 31 weeks, and 33 weeks after the first invitation packet is sent.

CATI Follow-ups

SAMHSA uses CATI calls to follow up with facilities that have not completed the N-SUMHSS starting in around July and continues through the end of October. The calls occurred in two phases:

Phase 1: Calls to facilities that had not completed the N-SUMHSS to verify the best contact person for the facility; confirm receipt of the survey materials; and remind them to complete the survey. If the facility requested it, the N-SUMHSS was administered on the phone during this reminder call.
Phase 2: Calls made in an attempt to complete the N-SUMHSS over the phone.

The “Between Cycle” N-SUMHSS (hereinafter referred to as the “Mini N-SUMHSS”) will be conducted periodically as new facilities are identified, using a subset of the N-SUMHSS questions, so that facilities do not have to wait a full year to be listed on FindTreatment.gov. An invitation letter will be sent to the newly identified facilities that were eligible to complete the Mini N-SUMHSS via web or CATI. The letter includes links to additional documentation and user support materials on the N-SUMHSS information website. If a facility did not complete the Mini N-SUMHSS, it will not be listed on FindTreatment.gov. However, as a state-approved facility, it would be included in the next round of N-SUMHSS administration process and be eligible for FindTreatment.gov listing after completing the next round of N‑SUMHSS.

Online I-TF:

The I-TF is continuously updated by states as they identify, license, or certify facilities, decertify, or cancel licenses for facilities, and learn of facilities that have gone out of business or moved. Facility information is updated through an online process. The “I-TF Online State Add Update Form” is used by state representatives to enter or change facility information (see Attachment J).

I-TF Facility Applications:

New facilities can request to be included in the I-TF through an “I-TF Facility Registration Application Form” found on FindTreatment.gov (see Attachment H). I-TF staff reviews all applications and verify the facility’s information to avoid duplication with existing facilities in the I-TF. The information on new facilities will also be passed to their state agency for review, approval and/or identification.

Augmentation Screener:

An annual augmentation process is conducted to identify new substance use and mental health treatment facilities that are not currently included in the I-TF and to determine eligibility for inclusion in the N-SUMHSS data collection process. The facility information provided by states (described above) is augmented by SAMHSA through searches of directories and other databases, such as the American Hospital Association and Data Axle USA. All potential treatment facilities identified are matched to the I-TF to detect duplicates. Facilities are then screened by phone to identify those that provide substance use and/or mental health treatment services using an “Augmentation Screener Questionnaire (see Attachment I).”

Degree of accuracy:

SAMHSA conducts comprehensive data processing and quality control process to ensure satisfactory degree of accuracy for the N-SUMHSS data files and analytic products.

Editing specifications are reviewed, updated, and tested on the response data to identify critical and non-critical data quality issues. SAMHSA reviews the frequencies and output from the test run to determine whether, based on the data, any additional changes to the editing specifications shall be implemented.

Data validation review and follow-up procedures are conducted for the N-SUMHSS data collection cycle. A Data Management Team reviews and updates data validation programs using the editing specifications. The programs automatically generates output for all facility records that contained data quality issues defined as “critical” in the editing specifications; and reads the data validation issues into a spreadsheet for facility review, comment, and correction. In addition, the data validation programs were used to pull revisions entered by facilities back into the survey response data for revalidation and final review.

A Data Validation Team is responsible for contacting facilities where the data validation programs identified one or more critical data quality issues. Initial outreach will be conducted via email and follow-up will be conducted by email and phone. Based on prior experience, most facilities return their survey data (an Excel spreadsheet) by email, which will then be reviewed for completeness and accuracy by the Data Validation Team members. If issues were unresolved or if questions arose from the returned survey data files, the Data Validation Team will follow up again via email or phone until the data issues are resolved.

Use of periodic data collection cycles to reduce burden:

While information from the regular N-SUMHSS is required to be collected annually by legislation (see Section A1 and A6 for details), the Mini N-SUMHSS will be conducted periodically as new facilities are identified, using a subset of the N-SUMHSS questions, to reduce respondent burden and facilities do not have to wait a full year to be listed on FindTreatment.gov. The N-SUMHSS EHR Supplement will be administered at a longer survey interval to reduce burden while still ensuring valuable information related to electronic health record is collected efficiently.

Methods to Maximize Response Rates

N-SUMHSS:

The methods to maximize the N-SUMHSS response rates include:

Establishing early outreach to states. States serve as critical influencers and stakeholders for the N‑SUMHSS and facilities. The N-SUMHSS Support Team establishes early contact with state representatives via outreach and maintains contact with states to strengthen relationships throughout the N-SUMHSS administration. Every two months, throughout data collection, the Support Team sends updates to each state showing its current response rate compared to the national rate and notes the facilities that have not yet responded.

Early outreach and frequent nonresponse follow-up contacts with facilities (see Section B2 for details).
Tracing and locating efforts to determine whether a facility is still in business, closed, or has merged with another facility.
Multiple survey mode options for facilities to participate in the N-SUMHSS and Mini

N-SUMHSS (web, paper and CATI calls) and in both English and Spanish.

An easy-to-navigate survey, the ability to report for a facility network, and pre-filling responses in the online survey for selected questions that have little year-over-year change (e.g., public versus private ownership, hospital type, etc.). The pre-filled responses help reduce burden and improve survey response without impairing the integrity of the data.
Resources to help facilities complete the survey such as a toll-free N-SUMHSS hotline, a designated N-SUMHSS email address, a Help Desk and an information website.
Leveraging Technical Assistance relationships with facilities to resolve records faster. The N-SUMHSS Support Team’s core function is to communicate directly with facilities to answer questions, relay facility updates to the data team, and help troubleshoot technical issues. The Support Team’s ability to pivot from technical support to survey administration, as appropriate, results in a better customer service experience and delivers more immediate resolution for facilities.
Targeting larger networks to identify and address response barriers. Larger networks (e.g., 20+ facilities in a network) are assigned across the N-SUMHSS Support Team to conduct direct outreach via telephone and/or email. This communication supported a variety of efforts including confirming points of contact, verifying the set of network facilities, and helping to address record issues.

I-TF:

The universe of mental health and substance use treatment facilities is not static. Experience with the 2022 N-SUMHSS and I-TF has shown that in a 12-month period, approximately 12 to 15 percent of facilities close and roughly the same number of “new” facilities are identified. Additionally, another 20 percent of facilities change their basic contact information (name, address, telephone number) each year. SAMHSA takes all reasonable measures to ensure that the I-TF is as complete as possible. Since there is no other comprehensive listing of treatment facilities to compare against the completeness of the I-TF, the only avenue available is to collaborate with state agencies to maintain the listing and to implements regular augmentations to identify new facilities that state agencies may not have authority over. Facilities remain on the I-TF until SAMHSA receives evidence that the facility is no longer providing treatment services or that they are otherwise ineligible. Thus, a facility that does not respond to the N-SUMHSS will stay on the I-TF.

Tests of Procedures or Methods to be Undertaken

SAMHSA applies a multistage testing process for the N-SUMHSS web survey. It begins with detailed web programming specifications that included all questions, response options, skip patterns, range checks, and navigation requirements. During this process, random data are generated to verify skip patterns. Manual testing is also conducted using SAMHSA and BHSIS contractor technical and project management staff on multiple browsers and hardware. As a final check, frequencies are run on pretest data to ensure the proper coding of responses. For the N-SUMHSS paper survey, SAMHSA’s BHSIS contractor reviews the formatted materials against the originals. SAMHSA reviews and approves a print proof of the paper questionnaire. A manual review of sampled printed materials is conducted to ensure that the correct files are used before the full printing process begins.

A cognitive testing on the use of electronic health records (EHR) by substance use and mental health treatment facilities in the United States was conducted in the fall of 2019 (OMB #: 0930-0382). Forty pretests were completed where 18 facilities provided mental health treatment services only, 20 provided substance use treatment services only, and two provided both substance use and mental health treatment services. Cognitive testing results were used to reduce respondent burden while simultaneously improving the quality of the data collected using the EHR Supplement Questionnaire (see Attachment E). These objectives are met when respondents are presented with plain, coherent, and unambiguous questionnaires that ask for data compatible with respondents’ memory and/or current reporting and record keeping practices.

If testing of new and/or revised survey items from 10 or more respondents are determined necessary for the N-SUMHSS continuous improvement, a proposed test or set of tests will be submitted for OMB approval separately or in combination with the main collection of information.

Statistical Consultants

SAMHSA’s BHSIS contractor will be responsible for the management of the N-SUMHSS (including programming web instrument, carrying out field work, data cleaning and entry, and data analysis) and management of the I-TF (including accepting and verifying changes to the I-TF, producing a master list for the N-SUMHSS, and conducting the frame augmentation activities). SAMHSA staff will be responsible for overseeing the N-SUMHSS data collection and I-TF operation, reviewing and drafting analytic reports, conducting quality control of all data files and reports, designing and implementing ad hoc analysis and assessment.

Individuals consulted on statistical aspects and individuals collecting and/or analyzing data include:

Heydy Juarez

Survey Statistician, Contracting Officer’s Representative

Office of Treatment Services (OTS)