0990-Section B National Standards for Culturally and Linguistically Appropriate Services (CLAS) in Health and Health Care_June 1

B. Collection of Information Employing Statistical Methods If statistical methods will not be used to select respondents and item 17 on Form 83-I is checked “No” use this section to describe data collection procedures.

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When item 17 on the Form OMB 83-I is checked "Yes," the following documentation should be included in the supporting statement to the extent that it applies to the methods proposed:

1. Respondent Universe and Sampling Methods

CLAS and National CLAS Standards Stakeholders

The study team will compile a list of potential participants who represent the full spectrum of stakeholders to define the potential respondent universe for the stakeholder portion (i.e., Phase 1) of the project, and will use mixed purposive maximum variation (nonprobability) sampling approach to select participants from this universe. The list of potential participants for the Stakeholder portion of the project consists of individuals who are “National CLAS Standards stakeholders” and “CLAS stakeholders.” The “National CLAS Standards stakeholders” are defined as those individuals who were directly involved in the development of the original and/or enhanced National CLAS Standards. More specifically, this sampling frame is identified as members of the National Project Advisory Committee (NPAC) for both the original and enhanced National CLAS Standards, and staff at the HHS Office of Minority Health who were directly involved in the development of the Standards, which represents approximately 46 individuals. “CLAS stakeholders” are defined as those individuals and experts from multiple disciplines who have a demonstrated awareness and knowledge of the delivery of culturally and linguistically appropriate services (CLAS) in health and health care and/or the National CLAS Standards. More specifically, the sampling frame for these individuals includes those individuals who have published (in either peer-review or grey literature) and/or delivered a public presentation about CLAS or the National CLAS Standards since 2001 (when the original Standards were released), and who meet at least one of the following additional criterion: are known/recognized by the National CLAS Standards Evaluation Advisory Group, have been specifically nominated by another CLAS Stakeholder, or have been specifically nominated by the National CLAS Standards Implementation Team Evaluation Subcommittee. “CLAS stakeholders” will be identified via a comprehensive literature and presentation search, consultation with the National CLAS Standard Evaluation Advisory Group and the National CLAS Standards Implementation Team Evaluation Subcommittee, and recommendations from the other National CLAS Standards stakeholders. Using this process, 72 “CLAS stakeholders” and 46 “National CLAS Standards stakeholders” have been identified, and thus comprise the universe of CLAS and National CLAS Standards Stakeholders covered by this collection. Using purposive maximum variation (nonprobability) sampling, the goal is to complete a minimum of 42 Stakeholder interviews (a minimum of 21 NCS Stakeholder interviews and a minimum of 21 CLAS Stakeholder interviews).

Health and Health Care Organization Leadership and Staff

For the health and health care organization interviews, site visits, and staff surveys (i.e., Phase 2), organizations identified will represent a broad array of sizes and characteristics within the following four typologies: public health departments, hospitals, Federally Qualified Health Centers (FQHCs)/clinics, and ambulatory care centers. To define the potential respondent universe of health and health care organization that self-identify as a “CLAS Adopter,” a list of health and health care professional associations within each typology will be developed, and a brief screener survey will be created and disseminated to these professional associations. For the purposes of this project, a “CLAS Adopter” is an organization that, per self-report, implements CLAS, the National CLAS Standards, or both. Twenty-two (22) professional health and health care associations with organizational membership have been identified as potential sampling sources, from which CLAS adopting health and health care organizations would then be identified. The organizational membership of these associations will receive a screener survey, on which CLAS adoption and interest to participate in the project will be assessed. Approximately 50,000 respondents, drawn from the professional associations, will be invited to complete the screener survey.

From this resultant pool of self-reported “CLAS Adopter” health and health care organizations, purposive maximum variation (nonprobability) sampling will be used to select 20 information-rich organizations for participation based on the following criteria: a minimum staff size of at least 125 individuals, age of the organization, inclusion of CLAS and/or cultural diversity in the organization’s mission and/or vision statement, geographic representativeness, variability in organizational structure (e.g., autonomous vs. semi-autonomous departments and units), and patient/client populations served/catchment area. Though the projected sample size for the staff survey is dependent on the size of the organizations included, the goal is to include no more than 20 organizations with at least 125 staff in each organization, with five organizations representing each of the four typologies.

For the Health and Health Care Organization Leadership Interview no more than 5 members of the leadership (defined as the CEO, COO, other senior manager, and/or their designees in health care organizations, and members of the leadership or executive branch [e.g., Director, Deputy Director] in health organizations) from each of the 20 participating organizations, for a possible total of 100 leadership respondents. To improve response rate, only key members of the staff and leadership within the health and health care organizations will be eligible to complete the appropriate version of the staff survey. Two versions of the survey have been developed, one for public health departments, and one for health care organizations (i.e., hospitals, clinics, and ambulatory care centers), to better tailor questions to the target audience. Inclusion criteria for the Health Care Staff Survey include the following: individuals who provide direct clinical care (e.g., physicians, nurses), individuals who provide administrative functions and come into direct contact with patients (e.g., front desk staff, billing), individuals who provide other clinical services (e.g., interpreters, navigators) and department managers. Thus, contract, non-direct care staff, facilities management/engineering staff, maintenance staff, housekeeping staff, nutrition services staff, and IT staff will be excluded from completing the Health Care Staff Survey. Inclusion criteria for the Public Health Department Staff Survey include: administrative staff, leadership at the division/unit level, and managers or administrators of programs and services will be included in the sample. Since health and health care organizations with staff sizes of 125 or greater will be eligible for inclusion, the estimated total number of respondents for the Health and Health Care Organization Staff Survey is 2,500 respondents (125 staff in 20 health and health care organizations).Once health and health care organizations have been selected, a priori and post hoc power analyses will be completed to obtain a minimum required n and to determine the projected and actual power given the final sample.

The maximum respondents for the National CLAS Standards Experience Form are approximately 240,000 (total number of Think Cultural Health newsletter registrants). Data will be collected only once from each participant.

Data Collection for CLAS & National CLAS Standards Stakeholders (Phase 1)
	Total # in Universe	Anticipated Sample Size
CLAS Stakeholders	72	21
National CLAS Standards Stakeholders	46	21
Total	118	42

Universe of Professional Health & Health Care Associations as Sampling Source (Phase 2)
Public Health Departments	6
Federally Qualified Health Centers/Clinics	5
Hospitals	8
Ambulatory Care Centers	3
Total	22

Data Collection for Health & Health Care Organization By Type (Phase 2)
	Anticipated Sample Size
Health & Health Care Leadership Interviews
Public Health Departments	25
Federally Qualified Health Centers/Clinics	25
Hospitals	25
Ambulatory Care Centers	25
Total	100
Health & Health Care Staff Surveys
Public Health Departments	625
Federally Qualified Health Centers/Clinics	625
Hospitals	625
Ambulatory Care Centers	625
Total	2500

2. Procedures for the Collection of Information

CLAS and National CLAS Standards Stakeholders (Phase 1)

As stated in B1, using mixed purposive maximum variation (nonprobability) sampling, a minimum of 42 stakeholder interviews will be completed with National CLAS Standards and CLAS stakeholders (21 National CLAS Standards Stakeholder and 21 CLAS Stakeholder interviews) via phone. Interviews will be conducted by experienced SRA interviewers. Interviews will be audiotaped with a digital tape recorder, and detailed notes will be taken during each interview, to facilitate data analysis and interpretation.

Health and Health Care Organization Leadership and Staff (Phase 2)

Health and health care organizations eligible for participation will be identified via a pre-screening process and engagement processes will begin with selected organizations.

In-person and/or phone interviews will be completed with 3-5 identified health and health care organization’s executive leadership (e.g., CEOs, COOs, Director of Clinical Services, etc.) and other key staff as identified by the organization’s leadership. Interviews will be conducted by experienced SRA interviewers. Interviews will be audiotaped with a digital recorder, and detailed notes will be taken during each interview, to facilitate data analysis. Document and record reviews will be conducted during this phase, during a planned site visits to one organization per typology to supplement information gained from the leadership interviews, and contextualize data collected via the staff survey administration. Documents and records of interest include, but are not limited to, board meeting minutes, staff meeting minutes, documentation of policies and procedures (and changes thereto), organizationally-disseminated communications related to cultural and linguistic competency, culturally and linguistically appropriate services, and the National CLAS Standards.

Based on the size of the staff in each organization, an a priori power analysis will be conducted to determine the minimum required sample to meaningfully interpret the staff survey data. A post hoc power analysis will also be conducted to determine the actual power obtained with the final sample included. Using nonprobability heterogeneity and convenience sampling, the survey will be administered to key leadership and staff in the selected health and health care organizations. Potential survey participants will be invited to participate in the project through a detailed recruitment email sent throughout the organization’s email listserv/database that describes the project, its importance, and its purpose. This recruitment email would be sent from and signed by the senior executive(s) and/or leader(s) of the organization. The recruitment email will include the website where the individual can read more about the project, electronically sign the consent form, and complete the survey. A minimum of 125 staff surveys will be administered at each organization.

3. Methods to Maximize Response Rates and Deal with Nonresponse

To maximize response rates for the Stakeholder Interviews to be completed in Phase 1, an advance letter will be sent via email to the entire universe of stakeholder respondents, which will be signed by a senior HHS OMH agency official and will describe the purpose of the project in detail (Attachment I). The letter will also provide information about participating in the project, including the mode of interview (i.e., via phone), how long the interview is expected to take, that participation is voluntary, how the data will be stored, and how the individual may participate in the project. Approximately one week after the advance letter has been emailed the project team will send a follow-up email to the entire universe of stakeholder respondents, requesting their participation (Attachment J). Interviews will be scheduled as convenient for the respondent. Within one month of the advance letter being emailed to the universe of stakeholder respondents, the project team will convene to review the response rate from this initial wave of recruitment, and determine if a follow-up is necessary. As a follow-up, and to address non-response, respondents will receive a follow-up call from a senior member of the project team, to provide the respondent with additional information about the project and invite the individual to participate in the project. Follow-up will continue, as needed, on a bi-weekly basis after this initial review period, until the minimum number of cases per stakeholder group (n=21) have been interviewed. Response to the stakeholder interviews (Attachment A; Attachment B) conducted in Phase 1 are expected to be high because of widespread acceptance of the National CLAS Standards as a guide to providing culturally competent health services and care that would ultimately decrease health disparities. A response rate of 95% is expected for the stakeholder interviews based OMH’s continuous engagement with stakeholders since the inception of the National CLAS Standards including developing and enhancing the National CLAS Standards.

To maximize response rates for the Health & Health Care Organization Leadership Interviews and the Health & Health Care Organization Staff survey to be completed in Phase 2, an advance letter will be sent via email to an identified organizational point of contact (per the screener survey) and the CEO/Executive of the organization, which will be signed by a senior HHS OMH agency official (Attachment K). This letter will describe the purpose of the project in detail, and provide information about how the organization may choose to participate in the project. Specifically, this letter will detail the data collection procedures for this phase of the project, and request that 1) the organization will identify up to 5 members of the leadership who would consent to be interviewed via phone, and 2) the organization will disseminate and promote the completion of the staff survey among its staff members eligible for inclusion. Approximately one week after the advance email has been sent, a member of the project team will contact the identified organizational point of contact or the CEO/Executive of the organization to provide additional information about the project and to encourage the organization’s participation. During this follow-up phone call and upon agreement to participate, the project team will inquire about the appropriate leaders to interview and the individuals’ contact information, and will provide the organization with a letter about the staff survey to send to their staff via email (Attachment L). The organization’s CEO/Executive will be asked to send the notification email and a reminder email, both with a link to the survey included, immediately, and at two-week intervals thereafter for a six-week period (Attachment M). A member of the project team will then contact the identified leaders via email and phone to schedule leadership interviews at the respondent’s convenience. As a follow-up, and to address non-response, leadership respondents will receive a follow-up call from a senior member of the project team, to provide the respondent with additional information about the project and invite the individual to participate in the project. Follow-up will continue, as needed, on a bi-weekly basis after this initial review period, until the minimum number of cases per organization (n=3) have been interviewed.

The Office of Minority Health (OMH) has previously sponsored studies assessing culturally and linguistically appropriate services in health care organizations. The first study was entitled National Study of Culturally and Linguistically Appropriate Services in Managed Care Organizations (CLAS in MCOs). The second was entitled CLAS in Health Care: Implementation of the National Standards for Culturally and Linguistically Appropriate Services in Health Care at the Alameda Alliance for Health (CLAS in Health Care). The expected response rates for this data collection are primarily informed by these two past studies. Health and Health Care Organization Screener (Attachment F) - For the CLAS in MCOs study, 30% of the managed care organizations approached participated in the study. Overall, it is expected there will be a 30% response rate for ach typology responding to the screener survey. Health and Health Care Organization Staff Survey (Attachment D)-- The CLAS in MCOs study received a 40% response rate for a paper-based questionnaire administered among staff in managed care organizations. Greenlaw and Brown-Welty (2009) found that web-based questionnaires received larger response rates than paper-based (52.46% vs 42.03%, respectively). Given the magnitude of difference between these two questionnaire modes and the response rate from the CLAS in MCOs study, it is anticipated that a 50% response rate will be achieved for the staff questionnaire. Health & Health Care Organization Leadership Interview (Attachment C)— A response rate of 100% is expected for the organizational leadership surveys based upon experience with similar data collections (i.e. the CLAS in MCOs and the CLAS in Health Care studies).

Citation: Greenlaw, C., & Brown-Welty, S. (2009). A comparison of web-based and paper-based survey methods testing assumptions of survey mode and response cost. Evaluation Review, 33(5), 464-480.

4. Tests of Procedures or Methods to be Undertaken

Pilot testing was conducted with nine participants between July 2014 and August 2014. Nine individuals who served on the National Project Advisory Committee of the enhanced National CLAS Standards and/or were known to the project team as knowledgeable about CLAS and/or the National CLAS Standards in health and health care organizations were invited to participate in this pilot test via email in July 2014. All nine invitees opted to participate in the pilot testing.

The pilot test involved the completion of telephone interviews, and for a select subset of participants, completion of an online survey. Upon completion of the telephone interview and survey (if applicable), all participants were also asked questions regarding their opinions and suggestions about the interview protocol (questions) and process, and the burden experienced by completing the interview. Members of the SRA-HDDP evaluation project team then scheduled phone interviews with the invitees via email, and also sent each a consent form to sign and return. To collect feedback about each type of evaluation measure, the nine participants were evenly divided into three groups based on their experience, current work role and environment, and focal areas of expertise: National CLAS Standards Stakeholders, CLAS Stakeholders, and Health and Health Care Organization Leaders. Interview teams consisting of a trained interviewer (who was a senior member of the project team) and an interview note taker conducted all interviews. All interviews were audiotaped, except for one per the participant’s request, and were transcribed by two members of the project team to facilitate data analysis.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or

Analyzing Data

The information collection instruments were developed in 2014 by SRA International in consultation with OMH and a 5-person Advisory Group consisting of content and evaluation experts (Identified in Section A).

HHS consultant for data collection instruments:

Lacreisha Ejike-King, Ph.D., M.S.*

Program Analyst, Office of Regulatory Affairs

U.S. Food and Drug Administration

240-402-2633

Lacreisha.Ejike-King@fda.hhs.gov

*At the time of the development of this submission, Dr. Ejike-King served as a Program Analyst at the Office of Minority Health and the National CLAS Standards Implementation Team Evaluation Subcommittee Co-Lead.

The main SRA personnel on this contract are:

C. Godfrey Jacobs

Program Manager

Health Determinants & Disparities Practice

SRA International, Inc.

(240) 514-2748

Godfrey_Jacobs@sra.com

Crystal L. Barksdale, Ph.D.

Senior Research Analyst

Health Determinants & Disparities Practice

SRA International, Inc.

(240) 514-2991

Crystal_Barksdale@sra.com

The main OMH personnel responsible for receiving and approving contract deliverables

Christine Montgomery

Project Officer; Contracting Officer’s Representative (COR) for contract

Office of Minority Health

240-453-6176

Christine.montgomery@hhs.gov

Rashida Dorsey, Ph.D., MPH

National CLAS Standards Implementation Team Evaluation Subcommittee Co-Lead, deliverables technical reviewer

Director, OMH Division of Policy and Data

Office of Minority Health

240-453-6170

Rashida.Dorsey@hhs.gov

Contractor personnel will implement the sample design, conduct data collection, handle data receipt/editing/keying, produce data files, conduct qualitative analysis, and conduct statistical analysis. OMH will provide direction and review functions to the contractor.