Appellant Climate Survey OMB Part B - Revised4 Final

Supporting Statement for OMB Clearance for
Office of Medicare Hearings and Appeals (OMHA)
Appellant Climate Survey

Part B: Collections of Information Employing
Statistical Methods

December 4, 2023

Jon Dorman

Department of Health & Human Services
Office of Medicare Hearings and Appeals

Appeals Policy and Operations Division

2550 S Clark St, Suite 2000

Arlington, VA 22202

Telephone: 571-457-7220

Email: Jon.Dorman@hhs.gov

Table of Contents

Part B. Collections of Information Employing Statistical Methods B-1

B.1 Respondent Universe and Sampling Methods B-1

B.2 Procedures for the Collection of Information B-2

B.3 Methods to Maximize the Response Rates and to Deal with Non-Response B-5

B.4 Test of Procedures or Methods to be Undertaken B-7

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting
and/or Analyzing Data B-8

Tables

Table B1. Random Stratified Sample per Year B-3

Table B2. Total Number of Appellants OMHA Contacted Given an 80% Response Rate B-3

Table B3. Research Team Contact Information B-8

Appendices

Appendix A1. Data Collector Confidentiality Agreement Forms

Appendix B1. Appellant Climate Survey-Telephone (English)

Appendix B2. Appellant Climate Survey-Telephone (Spanish)

Appendix B3. Appellant Climate Survey-Mail (English)

Appendix B4. Appellant Climate Survey-Mail (Spanish)

Appendix B5. Appellant Climate Survey-Web (English)

Appendix B6. Appellant Climate Survey-Web (Spanish)

Appendix C1. Introductory Letter (English)`

Appendix C2. Introductory Letter (Spanish)

Appendix C3. Telephone Survey Introduction Script (English)

Appendix C4. Telephone Survey Introduction Script (Spanish)

Appendix C5. Reminder Voicemail Script (English)

Appendix C6. Reminder Voicemail Script (Spanish)

Appendix C7. Reminder Letter with Mail Survey (English)

Appendix C8. Reminder Letter with Mail Survey (Spanish)

Appendix D1. Estimates of Respondent Burden

2. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

B.1.a Sampling

The purpose of this study is to improve the service that the Office of Medicare Hearings and Appeals (OMHA) provides to its appellants. The survey will gauge appellants’ satisfaction with this service along with the overall appeals experience. By identifying areas of success and opportunities for improvement, OMHA will be able to continuously improve its existing processes. The information obtained could lead to reallocation of resources, revisions to certain agency processes, and development of guidance related to the agency’s customer services. OMHA will collect information from two appellant types: (1) individuals disagreeing with and appealing payment decisions regarding Medicare coverage (beneficiaries), and (2) providers or suppliers appealing payment decisions regarding Medicare coverage for items and services furnished to beneficiaries (non-beneficiaries).

The respondent universe includes a total of approximately 7,200 appellants filing one or multiple appeals each fiscal year (FY).¹ The sample for this survey includes all appellants with a “closed” appeal during the first 6 months of the FY (October through March), resulting in a sample of approximately 3,600 appellants. OMHA is focusing on a non-redundant appellant universe so as not to bias this study toward large entities that account for the vast majority of appeals (i.e., providers or suppliers, who file approximately 91 percent of appeals; typically, a single provider or supplier files multiple appeals throughout the year).

Based on analysis of the FY 2022 population, there are 3,579 non-redundant appellants whose cases have been closed in the first 6 months of the FY. Roughly 54 percent of appellants (n = 1,762) in the sample are non-beneficiaries (i.e., providers or suppliers). The remaining (n = 1,817) are individual beneficiaries who may have been represented by a third party. The focus of this study is on appellant satisfaction with the existing process; therefore, it is important to treat each appellant—regardless of type—as an individual entity.

B.1.b Expected Response Rates

Assuming that 80 percent of appellants contacted respond to the Appellant Climate Survey, the total estimated sample size needed is 1,000 appellant records each year. This figure includes 800 respondents and 200 non-respondents. The resulting respondent sample will include approximately 400 beneficiaries and 400 non-beneficiaries.

B.2 Describe the procedures for the collection of information including:

B.2.a Statistical methodology for stratification and sample selection

In order to ensure that the appellant sample is representative of the target universe described in the previous section, OMHA randomly stratifies the appellant sample by appellant type and Field Office (FO). An equal number of appellants will be sampled from each appellant type. The appellant sample will also be stratified by FO in quantities that are reflective of the number of appeals in each FO. Given the percentage distribution of the entire population by these factors, a final stratified random sample includes the number of completed surveys listed below (see Table B1). We will recruit from a total sample of 1,000 appellant records for each FY of the data collection cycle, including beneficiaries and non-beneficiaries (i.e., providers and suppliers), for participation in the survey. Given an 80 percent response rate, a total of 800 contacted appellants is expected in each FY of the data collection cycle (Table B2).

Table B1. Random Stratified Sample per Year²

Stratification Groups	Estimated Sample Counts
	FY 2024	FY 2025	FY 2026
Beneficiary	500	500	500
Non-beneficiary	500	500	500
Total	1000	1000	1000

Table B2. Total Number of Appellants OMHA Contacted Given an 80 Percent Response Rate

Stratification Groups	Estimated Survey Counts
	FY 2024	FY 2025	FY 2026
Beneficiary Status
Beneficiary	400	400	400
Non-beneficiary	400	400	400
Total	800	800	800

B.2.b Estimation procedure

Data collected from the sample of appellants will be used to produce estimates that are representative of all appellants in the target universe. Sampling weights that account for the stratified sampling design and adjust for potential nonresponse and coverage errors will be used to calculate estimates and standard errors and to conduct hypothesis testing.

B.2.c Degree of accuracy needed for the purpose described in the justification

OMHA derived its total sample size to ensure adequate statistical power for calculating estimates representative of the target universe. We will select a sample of 1,000 appellant records each FY (FY 2024, FY 2025, and FY 2026) and expect to obtain data from 800 respondents. This sample is expected to produce representative estimates with 95 percent confidence intervals of +/- 3.4 percentage points or less depending on the outcome. For subgroup analyses occurring at the beneficiary and non-beneficiary levels, the sample will produce estimates with 95 percent confidence intervals of +/- 4.9 percentage points or less depending on the outcome.

B.2.d Unusual problems requiring specialized sampling procedures

OMHA does not anticipate unusual problems that require the usage of specialized sampling procedures.

B.2.e Any use of periodic (less frequent than annual) data collection cycles to reduce burden

OMHA does not plan to use data collection cycles that are less frequent than annual. Data collection will be conducted annually.

B.2.f General data collection procedures

A new sample is drawn in each survey cycle. Appellants may be randomly selected in one or more of the years that the survey is conducted. A stratified random sample based on the two factors identified above—appellant type and FO—is used to obtain the most representative data possible.

The process of collecting data will occur in the following fashion:

1. At the beginning of each survey cycle, OMHA obtains a list of all unique appellants who have had an appeal closed during the first 6 months of the FY. The sample will be stratified by appellant type (beneficiary and non-beneficiary) and 10 FO locations (plus one satellite location) generally proportional to the claims in each location. We will randomly select a total sample size of 1,000 appellant records, including estimated sample sizes of 500 beneficiaries and 500 non-beneficiaries for FY 2024, FY 2025 and FY 2026.

2. Appellants selected for participation in the Appellant Climate Survey will be notified of the study by mail (Appendices C1 and C2) and provided information necessary to complete the web survey. These letters contain official signatures and labels to verify the organization’s intent and professionalism.

3. If the appellant does not complete the web survey within 1 week, appellants may be contacted up to 14 times by telephone (Appendices C3 and C4), and non-beneficiaries may receive up to 3 voicemails (Appendix C5 and C6) as reminders and encouragement to complete the Appellant Climate Survey on the web. The Appellant Climate Survey may be completed by telephone during a reminder telephone call attempt, depending on the appellant’s preference and availability.

4. Appellants who do not complete the survey after 1 month of call attempts will be sent a reminder letter (Appendices C7 and C8) that will include a hard copy of the Appellant Climate Survey and a return envelope with prepaid postage. The letter will encourage the appellant to complete the hardcopy version of the Appellant Climate Survey and return the survey by mail.

5. Telephone and web-based responses will be stored electronically and downloaded into a Microsoft Excel or SPSS database. Mailed responses will be manually entered into the database.

B.3 Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Overall response rate projections are presented above in section B.1.b. Achieving the specified response rates involves using procedures described below to secure participation once the potential respondents have been identified. The Appellant Climate Survey is designed to achieve the highest practical rates of response, commensurate with the importance of survey uses, time constraints, respondent burden, and data collection costs. In the event that an 80 percent response rate is not obtained, a non-response bias analysis will be conducted.

The recruitment procedures designed to maximize the number of sampled appellants who complete the Appellant Climate Survey are described below.

• The introductory letters inviting appellants to participate in the Appellant Climate Survey will be carefully developed to emphasize the importance of this study and how the information will help OMHA improve the appeals process, and to maximize cooperation through advance warning. All letters will include official U.S. Department of Health and Human Services (HHS) OMHA letterhead signed by the Chief Administrative Law Judge to increase study credibility.

• A toll-free number and study email address will be provided so appellants can receive assistance with the survey. English and Spanish speakers will be available to respond to questions and concerns.

• Sampled appellants will have the option of completing the Appellant Climate Survey as a web, telephone, or mail-in survey.

• A training for telephone interviewers will be conducted. The training, specific to this study, will include an overview of the project, a review of the research questions the study will address, a primer on interviewing practices and procedures, and techniques for encouraging respondent candor.

• A core set of interviewers with experience conducting telephone interviews, particularly interviewers who have proven their ability to obtain cooperation from a high proportion of sample members, will be employed.

• Follow-up attempts by telephone will be made with all sampled appellants who do not complete the survey 1 week after the start of data collection. The primary purpose of the call will be to urge appellants to complete the survey. At that point, if the appellants prefer to complete the survey or remaining sections of the survey over the telephone, an interviewer will administer the full survey or any remaining parts of the survey over the telephone.

• Call scheduling procedures that are designed to call numbers at different times of the day (between 8 a.m. and 6 p.m.) and days of the week (Monday through Saturday) will be used to improve response rates.

• A second mailed reminder will be sent to appellants who have not yet completed the survey after the first month of data collection. All letters will include official HHS OMHA letterhead signed by the Chief Administrative Law Judge to increase study credibility. The letter will also include a paper survey and postage-paid return envelope.

B.4 Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The data collection instruments were reviewed for understandability (e.g., confusing wording or layout, questions that were difficult to comprehend or respond to) by a total of four OMHA stakeholders. Instruments were revised, as needed, to incorporate feedback provided by the OMHA stakeholders.

Survey instrument changes from the prior 3-year cycle include the following:

1. Added a new section, “Request for Hearing.” The section focuses on how customers requested a hearing, how satisfied they were with the method they used to request a hearing, and about the clarity of form OMHA-100 (Request for ALJ Hearing or Review of Dismissal).

2. Changed “Hard Copy, Internet and Phone Information” section to “Communications and Web Tools” section. 

3. Added a brief statement about when customers should have received the “Notice of Nondiscrimination” document.

4. Added two satisfaction questions for appellants who used the e-Appeal Portal—one about updates the portal provides on their appeal and another about using the portal for uploading documents electronically.

5. Changed “Telephone Hearing” section to “Hearing.” The appellant is asked what type of hearing they had (telephone or video) and satisfaction with using that method. If they attended a telephone hearing, appellants will be asked whether they were offered the option of a video hearing; if not, they will be asked if they would have participated in a video hearing if offered.

All data collection materials are reviewed by OMHA staff prior to the start of data collection in an effort to minimize respondent burden and improve utility. This study is a continuation of previous data collection and utilizes previously tested instruments with some added measures.

B.5 Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The Contractor, 2M Research, will conduct the Appellant Climate Survey in 2024. The contact information provided below may change depending on the Contractor selected to administer the survey in 2025 and 2026. See Table B3 for current study contact information.
Table B3. Contact Information

Name	Affiliation	Telephone Number	E-mail
Renée A. Johnson	HHS OMHA	(571) 777-2741	renee.johnson@hhs.gov
Jessica Izzo	HHS OMHA	(571) 777-2761	Jessica.Izzo@hhs.gov
Paul Ruggiere	Project Manager, 2M Research	(817) 856-0871	pruggiere@2mresearch.com
Neel Chakraborti	Corporate Officer in Charge, 2M Research	(703) 214-1429	nchakraborti@2mresearch.com

1 The actual number of appeal cases submitted each FY varies; thus, the sample universe will fluctuate each FY.

2 As in previous OMHA Appellant Climate Survey data collection efforts, additional samples may need to be added if response counts are lower than expected.