Supporting Statement Part B

Program Evaluation of AHRQ’s MONAHRQ

Expiration Date: 9/15/2015

ICR Supporting Statement

Part B

Agency for Healthcare Research and Quality (AHRQ)

Virginia Mackay-Smith, Associate Director, CDOM, AHRQ

301-427-1404

virginia.mackay-smith@ahrq.hhs.gov

January 14, 2015

B. Collections of Information Employing Statistical Methods.

B.1. Respondent Universe and Sampling Methods

The methodology, content, and administration of Focus Groups with consumer and professional end users of MONAHRQ software will follow recommended best practices. The study population will include a purposive sampling of consumer and professional end users. Participants will be identified by referrals from consumer and professional organizations that focus on health information, and consumers will also be recruited in local libraries (by LTG) in order to capture individual consumers who have no consumer affiliation.

Each identified potential participant will be invited to participate in one of a series of online and in-person focus groups (7-10 in total). This will allow AHRQ to effectively identify what aspects of MONAHRQ meet users’ needs and where the software can most effectively be improved to better serve specific end users’ needs. The focus group topics and questions have been designed for each type of end user: consumer (patients and families), and professionals (including healthcare providers, professional associations, hospital groups, etc.).

Table B.1. Estimated Size of the Respondent Universe

End User Focus Groups
Consumer End Users	3-4 meetings/~8 per = 24-32	Video and in-person focus groups
Professional End Users	4-6 meetings/~8 per = 32-48	Video focus groups

A purposive sampling design, also known as judgmental, selective or subjective sampling, relies on the judgment of the researcher in the selection of units (individuals, cases/organizations) to be studied. The sample is usually small, especially when compared with probability sampling techniques. Thus, when recruiting participants from consumer or professional organizations for focus groups, the evaluation team will request said organizations to select members in order to achieve diversity in ethnicity, age, and socioeconomic status. This sampling approach will be particularly useful to inform the lack of data on awareness, knowledge, experiences, and expectations with regards to the value and utility of MONAHRQ as perceived by consumer and professional end users. In addition, this sampling technique can highlight the potential advantages and challenges of using MONAHRQ that are associated with individuals with particular needs, resources, and abilities. This sampling approach will be used in order to include focus group participants with and without experience with health reporting in general and MONAHRQ generated reports in particular, as well as participants of states with or without statutory mandates to report comparative quality health information.

Focus group data will provide information about needs, expectations, and attitudes towards health information online, and help identify barriers to quality health reporting directed to consumer and professional end users.

B.2. Procedures for the Collection of Information

This section describes data collection procedures. The discussion is divided into three subsections: (1) pre-focus group procedures (2) data collection procedures and (3) quality control procedures.

B.2.A. Pre-Focus Group Procedures

The draft focus group questions are informed by conducting and analyzing key-informant interviews with consumer and professional end users of MONAHRQ.

Participants identified by referrals from consumer and professional organizations will be contacted according to an AHRQ-approved focus group protocol.

For human subjects’ protections purposes an electronic communication will be sent to prospective participants, once they have agreed to participate, explaining the study and its purposes, any foreseeable risks, and asking for informed consent. It will also include the name and telephone number of a person to call with questions regarding Human Subjects protection.

Within one week of the initial mailing, a thank-you/reminder will be sent to each respondent to encourage participation in the focus group. The remainder will include a toll-free number that can be called if respondent has any questions about the focus group.

B.2.B. Data Collection Procedures

Data will be collected utilizing focus groups. Focus groups, are a qualitative research approach, conducted by trained facilitators, in which a group of people from a particular population, in this case consumer and professional end users of comparative health reporting, are asked about their opinions, experiences, behaviors, and beliefs about a particular subject which is of interest to the population from which they were selected. The questions designed for these focus groups are asked in an interactive group setting where participants are free to talk with other group members.

The focus groups will be recorded and the evaluators will perform data extraction and analysis.

B.2.C. Quality Control Procedures

Beginning with study initiation and continuing through all phases of data collection and analysis, steps will be taken to ensure that the data collected are of the highest quality possible. All project staff will be trained to understand the purpose, background, objectives, and importance of the evaluation, as well as their specific role and activities in the study. LTG Associates has standard procedures for management and reporting of confidential data; those procedures will be followed in working with these data.

Completed focus group paper data will be stored in locked file cabinets. All project computer files will be password-protected and access to the files will be limited to authorized project staff.

A management information system will be developed to monitor data collection activities. An emphasis on quality will continue with data editing and data entry.

B.3. Methods to Maximize Response and Participation

Over the past 25 years, LTG Associates has developed culturally and linguistically appropriate procedures that have been successfully used to obtain high-quality data from focus groups in a variety of settings and populations. By focusing on a relatively small number of questions (8-10), the group will be able to explore in depth the topics of importance in the evaluation of MONAHRQ, with sub-issues under each topic.

Focus group data provide the opportunity to analyze the strength with which an individual holds an opinion. If they are presented with opposing opinions or directly challenged, the individual may either modify their position or defend it. Bringing together all the comments that an individual makes enables the researcher to determine whether their view changes in the course of discussion and, if so, further examination of the transcript may reveal which contributions by other focus group members brought about the change.

All participants will be asked to consent to participate. To encourage participation, incentives of $50 in the form of an e-gift card for each participant will be provided.

B.4. Tests of Procedures or Methods to be Undertaken

In developing the focus group questions for the evaluation of MONAHRQ, we sought input regarding the appropriateness and logic of the focus group questions from experts, practitioners and other stakeholders. Most of these experts have experience in quality comparative health reporting.

Prior to requesting OMB clearance, the draft focus group questionnaire was reviewed by AHRQ staff who provided comments and advice about the format, comprehensibility, and ease of response and relevance of the focus group questions. Modifications to the questions and format were made based on comments received during this phase.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

John Ogawa, Ph.D., Senior Research Associate, LTG Associates, Inc.

Cathleen Crain, M.A., Managing Partner, LTG Associates, Inc.

Alberto Bouroncle, Ph.D., Senior Research Associate, LTG Associates, Inc.