SUPPORTING STATEMENT
Part B
Demonstration of Health Literacy Universal Precautions Toolkit
Version: September 14, 2012
Agency for Healthcare Research and Quality (AHRQ)
Table of contents
B. Collections of Information Employing Statistical Methods 3
1. Respondent Universe and Sampling Methods 3
2. Information Collection Procedures and Analysis Plan 8
3. Methods to Maximize Response Rates 13
4. Tests of Procedures or Methods 14
5. Statistical Consultants 14
References 14
B. Collections of Information Employing Statistical Methods
Twelve primary care practices will be recruited to participate in the Demonstration of Health Literacy Universal Precautions Toolkit (Toolkit available in Attachment A). To be eligible, a practice must have an active quality improvement (QI) team, have completed at least one QI program within the past 3 years, have at least one clinician with an interest in health literacy, and have a significant proportion of patients (≥25%) with limited health literacy (as estimated using the Pfizer Health Literacy Prevalence Calculator1).
Practices will be recruited primarily through the American Academy of Family Physicians National Research Network (AAFP NRN), a practice-based research network representing primary care clinicians nationwide, and the State Network of Colorado Ambulatory Practices and Partners (SNOCAP-USA), a practice-based research network based on the University of Colorado Anschutz Medical Campus. Recruitment e-mails (Attachment B) will be sent to all practices on the AAFP NRN and SNOCAP-USA member list-serves, which include 716 primary care practices, and to practices that have expressed interest in participation (which may not be members of these research networks). We also will publish an article describing the study in AAFP News Now, the AAFP’s official newsletter, which is e-mailed to all 94,700 members weekly. Other recruitment activities will include announcements sent to community health centers by the Health Resources and Services Administration and following up with practices who have expressed interest in being a demonstration site after hearing about the project at conference presentations.
Practices expressing interest in participating will be contacted by telephone to obtain descriptive information about the practice and their patients (Practice Screening Calls; see Attachment C). An initial set of 15 practices will be purposively selected, targeting variation along the following dimensions: practice size and type (e.g., Federally Qualified Health Clinic), region of country, urban/rural location, and patient characteristics (e.g., age, race/ethnicity, primary language). Two staff members from each practice will then participate in a follow-up phone call with the project team to discuss responsibilities associated with participating in the project and ensure that the practice is able to participate (Follow-up Recruitment Calls; see Attachment D). After this discussion, a final set of 12 practices will be selected to implement the Toolkit, with two or three practices selected to serve as alternates.
Participating practices will implement four of the tools contained in the Toolkit. Each practice will implement Tool 1, which provides guidance on the formation of a team to oversee Toolkit implementation, and Tool 2, which walks practices through the process of conducting a self-assessment of health literacy-related systems and procedures. Each participating practice also will implement two supplementary tools, which will be selected from a list of ten high-priority tools specified by the project team (i.e., Tools 3, 4, 5, 8, 11, 12, 13, 14, 16, 17, and 20).
As described in Supporting Statement A, a variety of data collection activities will be conducted. For most methods, data will be collected during the month prior to Toolkit implementation (pre-implementation) and again during the last month of the implementation period (post-implementation). The approach we have outlined will allow us to learn a great deal about Toolkit implementation and how the Toolkit may be refined to improve its usefulness for primary care practices. In addition, it will allow us to examine the value of measures developed specifically to examine health literacy-related systems, policies, and performance of health care practices Although we will conduct preliminary analyses examining possible change over time as a result of project implementation, our evaluation approach is exploratory in nature, examining trends in the data as well as factors that may be examined in larger subsequent studies or that may inform improvements to the Toolkit.
In Table B-1, we present information about patient-level data collection, specifically the Patient Survey and the Survey Using Items from the Consumer Assessment of Healthcare Providers and Systems (CAHPS) (Attachments I.1 and J, respectively). We estimate the sample size based on average AAFP NRN practice size (i.e., providers and patients) and describe our sampling approach for each patient-level data collection activity. The response rates for the two patient survey methods warrant specific discussion. The estimated response rate for the Patient Survey is based on prior studies conducted in AAFP NRN practices. For similar surveys requiring that patients remain at the office after a visit is completed, response rates have generally been about 50%. As we will employ methods for enhancing response rates (discussed in Section 3), we project a somewhat higher response rate (55%) for the Patient Survey. The estimated response rate for collection of Consumer Assessment of Healthcare Providers and Systems (CAHPS) data is consistent with expected response rates reported by the Agency for Healthcare Research and Quality, which developed and has extensive experience field testing the CAHPS.2
Table B-1. Patient-Level Data
Method |
Sample Size |
Sample Description |
Sampling Methods |
Projected Response Rate |
Projected Number of Respondents |
Patient Survey |
1091 |
Patients ≥ 18 years who are fluent English speakers and have an office visit during the data collection period – data collected in 6 practices using Tool 17 |
Consecutive patients attending office visits until sample size attained |
55% |
600 |
Survey Using Items from the Consumer Assessment of Healthcare Providers and Systems (CAHPS) |
2182 |
Patients ≥ 18 years with an office visit during the prior 60 days – data collected in two of the six practices using Tool 17 |
Random sample of patients stratified by physician |
55% |
1200 |
The response rates for the patient surveys are not ideal, but are consistent with prior work involving similar surveys. Since we cannot be certain what types of patients will be likely to agree to or decline participation, these lower response rates could lead to some bias in the patient data. For example, patients having had very negative experiences with their providers may be more likely to respond than patients with more neutral encounters. There is no reason to expect, however, that the types of patients likely to participate will change from the pre-implementation to the post-implementation time period.
We also will collect quantitative data from staff members of the participating practices, including the pre- and post-implementation Practice Staff Surveys (Attachments L and M), the Medication Review Form (Attachment K), the pre- and post-implementation Health Literacy Team Leader Surveys (Attachments N and O), the Health Literacy Assessment Questions (Attachment E), and the Implementation Tracking Form (Attachment F). Note that the Health Literacy Assessment Questions will be completed as part of Toolkit implementation at the pre-implementation time period. However, we will ask practices to complete the items again at the conclusion of the implementation period so that we may explore whether this Toolkit resource suggests the possibility of changes over time in staff perceptions of the practice environment. In addition to the data collection methods included in Table B-2, the project team will examine participating practices’ health literacy environments, assessing written patient materials and signage in the offices as well as observing naturally occurring communications between patients and front desk staff. As the review of practices’ health literacy environments will be completed based on project team observation and assessment, it will involve no burden to participating practices.
Table B-2. Practice-Level Quantitative Data
Method |
Sample Size |
Sample Description |
Sampling Methods |
Projected Response Rate |
Projected Number of Respondents |
Practice Staff Survey (pre/post implementation) |
576 |
All practice staff |
Universe survey (census) |
75% |
432 |
Medication Review Form |
120 |
12 clinical staff across 3 practices using Tool 8 will collect 10 times each |
Providers conducting medication reviews as part of routine care |
100% |
120
|
Health Literacy Team Leader Survey (pre/post implementation) |
24 |
Leader of each practice’s Health Literacy Team |
Universe survey (census) |
100% |
24 |
Health Literacy Assessment Questions |
160 |
Staff from different areas of each practice (e.g., doctors, front office staff) |
Purposive sample selected by practices |
75% |
120 |
Implementation Tracking Form (completed 6 times) |
72 |
Leader of each practice’s Health Literacy Team |
Universe survey (census) |
100% |
72 |
Because a primary objective of the demonstration is to identify approaches for improving the Toolkit, we will conduct qualitative interviews with practice staff members regarding the Toolkit implementation process. Note that these activities emphasize improving the quality of the Toolkit and Toolkit implementation, as opposed to research evaluation. The interviews with the Health Literacy Team Leader (pre- and post-implementation; Attachments Q and R) and Team Member (Attachment T), as well as the Practice Staff Member interviews (Attachment U) will be conducted during site visits with each practice and will be audio recorded. For the Practice Staff Member Interview, practices will be asked to nominate one or two clinical staff members who were not involved in the Health Literacy Team, but who have knowledge of the changes made as part of Toolkit implementation. All other communications will be conducted by telephone. Project staff will obtain written informed consent from interview participants (Attachment P). For the Health Literacy Team Leader, consent will be obtained at the pre-implementation site visit and will include consent to complete the pre- and post-implementation interviews, as well as the Health Literacy Team Leader Survey, and Check-in Phone Calls, and the Implementation Tracking Form.
Also to collect data regarding the implementation process, the project team will conduct Check-in Phone Calls with all practices at regular time points throughout the project (Attachment S). These calls will occur two weeks, one month, two months, and four months into the implementation period. Calls will be audio recorded and transcribed.
Table B-3. Qualitative Interviews
Method |
Sample Size |
Sample Description |
Sampling Methods |
Projected Response Rate |
Projected Number of Respondents |
Practice Screening Calls |
20 |
Practice staff member contacting project team regarding interest in project participation |
Staff member who initiates contact |
100% |
20 |
Health Literacy Team Leader Interview (pre/post implementation) |
24 |
Leader of each practice’s Health Literacy Team will be interviewed at pre- and post-implementation |
Universe survey (census) |
100% |
24 |
Health Literacy Team Member Interview |
12 |
One member from each practice’s Health Literacy Team will be interviewed at post-implementation |
Purposive sample selected by practices |
100% |
12 |
Practice Staff Member Interview |
24 |
One or two staff members who are not involved in the Health Literacy Team will be interviewed during post-implementation period |
Purposive sample selected by practices |
100% |
24 |
Check-in Phone Calls |
48 |
Leader of each practice’s Health Literacy Team will be interviewed 2 weeks, and 1, 2, & 4 months into implementation |
Universe survey (census) |
100% |
48 |
In this section, we briefly describe the approach to collecting each type of data and summarize key analysis questions. Because this project represents a small pilot demonstration and emphasizes assessment of Toolkit implementation, with the goal of identifying potential improvements to the Toolkit, our quantitative data analyses will be exploratory and limited in nature. Analyses will not be intended to provide definitive results that can be generalized to the full population of primary care practice settings, but instead will focus on assessing the value of the measures for examining change over time as a result of Toolkit implementation. As such, we do not provide formal power calculations.
Patient-Level Data
Patient Survey. The Patient Survey will be collected in English using a combination of touch screen tablets and paper surveys. The approach used will depend on factors related to the practice (e.g., the practice’s comfort with the technology) and the patients (e.g., some patients might prefer to complete a paper survey at home). The Patient Survey will be collected only by practices implementing Tool 17. We estimate that six of the participating practices will choose to implement this Tool. During the pre- and post-implementation data collection periods, each of these six practices will invite consecutive patients to complete the survey until 50 surveys have been collected per time point. Patients will complete the survey following their doctor’s visit. For paper surveys, patients will seal the completed form in a self-addressed, postage-paid envelope, either putting it in the mail or returning it to a designated practice staff member. Identifying data will not be collected about participants, nor will patients be compensated. If tablets are used, practices will sample the next consecutive patient once the tablet is no longer in use by the prior participant. Tablets will be programmed to ensure that practice staff cannot open the database in which Patient Survey data are stored or otherwise view patient responses. Regardless of the administration technique, the survey will contain a cover page indicating that participation is voluntary and that individual participant’s answers will not be shared with members of the practice.
Consumer Assessment of Healthcare Providers and Systems (CAHPS): Selected items from the CAHPS Clinician and Group Survey will be administered in two practices, with the goal of obtaining 1200 completed surveys across practices and data collection time points. Data will be collected by mail and phone, in English and Spanish. The two practices for which these data will be collected will be selected from those practices choosing to implement Tool 17. Participating practices will provide the vendor administering the CAHPS items with each physician’s patient panel for the past 60 days. At each time point, the survey will be sent by mail to a random sample of 546 patients from each of the two practices. A reminder postcard will be sent 4-10 days after the questionnaire was mailed, followed by a second questionnaire 28-30 days after the original mailing, and a second reminder post-card 4-10 days after the second questionnaire was mailed. If the survey is not returned, the potential participant will be contacted by phone to conduct the survey. A maximum of six phone calls will be made. Surveys will be conducted until a sample of 1200 is reached.
Medication Review Form: Practices implementing Tool 8 from the HL Toolkit will complete the Medication Review Form. We expect that four clinic staff per participating practice will complete this form and each will complete it approximately five times at each time point (pre-implementation and post-implementation). Therefore, a total of 12 clinical staff will complete a total of 120 Medication Review Forms.
Medication Review Forms will be completed for patients identified by practice staff, as a result of their normal medication refill and scheduling procedures, to require a visit to review medications. During these visits, providers will complete a paper copy of the Medication Review Form, which will not include patient identifying information.
Data Analysis: Given our small sample sizes, analyses of the patient data will be exploratory in nature and will focus on the value of given measures in assessing the health literacy environments of participating practices. We will use the Patient Survey data to assess patients’ perceptions of possible changes over time in practice performance related to eight specific tools in the Toolkit (i.e., Tools 4, 5, 8, 11, 12, 14, 16, 17, and 20). For example, using data collected related to Tool 4 (Tips for Communicating Clearly), we will examine whether there appears to be a tendency for patient ratings of the quality of their verbal communications with practice staff to be better at post-implementation than pre-implementation. We will assess suggestive changes over time in patient responses to items related to specific tools and also will explore changes over time in patients’ overall impressions of practices’ health literacy environments (i.e., using composite scores computed from the items included in the Patient Survey).
Using CAHPS data collected for two practices, we will conduct exploratory analyses to assess whether CAHPS items have the potential to show change over time in response to practices’ health literacy-related quality improvement efforts. Like with the Patient Survey data, we will explore changes in overall impressions of practice performance as well as patient perceptions of performance related to specific tools. We will compare these data to the Patient Survey data to explore whether there may be similar trends over time in the two sets of patient survey data.
When possible, we will compare Patient Survey and CAHPS data with other measures addressing the same tools. For example, for Tool 11 (Design Easy-to-Read Material), we will have the ability to explore whether patient perceptions of the understandability of written patient materials is consistent with the project team’s review of the readability of patient materials disseminated by participating practices.
The Medication Review Form will allow us to explore possible change over time in patients’ comprehension of their medication regimens. Because not all practices will implement Tool 8, the sample size for data collected using the Medication Review Form will be limited. Therefore, our analyses will be descriptive only. At pre-implementation and post-implementation, we will summarize the implementation of Brown Bag Medication Reviews, documenting the extent and types of problems found, and indicating whether there was any sign that problems identified became less frequent over the course of the implementation period.
Practice-Level Quantitative Data
Practice Staff Survey: We will request that all staff members from participating practices complete the Practice Staff Survey. The survey will be administered in writing and will be collected anonymously. Surveys will be completed at pre-implementation and post-implementation and sealed in an envelope before being returned to the Health Literacy Team leader for submission to the project team.
Health Literacy Team Leader Survey: The leader of each practice’s Health Literacy Team will complete this survey in hardcopy at pre-implementation and post-implementation. This survey is intended to provide detailed information about practice policies relevant to health literacy and to collect detailed process information about implementation of the Toolkit (e.g., development of the Health Literacy Team, meeting schedule, membership). Some items are only collected by practices implementing specific tools (i.e., items related to Tools 4 and 5).
Health Literacy Assessment Questions: As part of implementation of Tool 2, which guides practices in conducting a self-assessment of their health literacy-related systems and procedures, practices will complete the Health Literacy Assessment Questions at the beginning of the project period. Although not required as part of Toolkit implementation, we will request that they complete the items again following implementation so that we may examine whether these items may be able to capture change over time. Practices will be asked to collect responses from staff representing different components of the practice (e.g., clinicians, front desk staff) and to obtain input from members of the Health Literacy Team as well as staff members who are not directly involved in Toolkit implementation. Practices will provide the project team with data from both time points.
Implementation Tracking Form: The Health Literacy Team leader from each practice will complete and routinely update this form to identify planned and actual timing of Toolkit implementation. These data will help us to monitor the timing of Toolkit implementation.
Review of the Health Literacy Environment: The project team will evaluate each practice’s health literacy environment, examining the reading level of written materials provided to patients, observing interactions between patients and office staff during planned site visits, and assessing the ease of navigating the office environment (Attachment H). These data will be collected with no burden to practices.
Data Analysis: We will use practice-level data to describe each practice’s health literacy environment, either from the perspective of practice staff members or from the observation of the project team. We will assess change over time in responses to the Practice Staff Survey, Health Literacy Team Leader Survey, and Health Literacy Assessment Questions and performance on the Review of the Health Literacy Environment to explore whether these measures are able to show change over time as a result of implementing the Toolkit. As with the patient-level data, we will examine change over time in items related to specific tools as well as composite scores assessing overall practice performance related to health literacy. We also will explore whether tool-specific responses show consistent patterns across different source of patient- and practice-level data.
Analyses related to the Health Literacy Assessment Questions serve an additional objective. This set of items is included in the Toolkit itself and is meant to be a resource through which practices can evaluate their health literacy environments, plan their quality improvement activities, and potentially assess their progress over time. To guide possible revisions to the Toolkit, we will explore whether responses to the Health Literacy Assessment Questions suggest changes over time and whether they show a similar pattern of change as related data, such as Patient Survey data.
The Implementation Tracking Form will be used to track the planned and actual timing of practices’ implementation activities. The form will be updated regularly and will be discussed during the regular check-in calls that the project team will conduct with participating practices. Data from these forms will be used for descriptive purposes, in helping us to understand how long specific components of the implementation process take.
Practice-Level Qualitative Data
As a major objective of the project is to identify barriers to implementation and possible improvements to the Toolkit, the project team will collect extensive qualitative data regarding the implementation process.
Practice Screening Calls: As part of the practice recruitment process, we will conduct phone calls with all practices interested in participating in the demonstration. Through these calls, we will collect important information to allow us to select a diverse set of practices for participation (e.g., diverse practice types and patient characteristics). In addition, data from these calls will provide important descriptive information about the eventual sample of participating practices (e.g., size, percent patients speaking a language other than English).
Qualitative Interviews with the Health Literacy Team Leader and Members: To gain detailed information about the implementation process and suggestions for Toolkit revision, we will interview each practice’s Health Literacy Team Leader at pre-implementation and post-implementation and another member of the team at post-implementation. These interviews will take place in person during project team visits to each site. If a scheduled interview has to be cancelled (e.g., due to practice staff member illness), we will conduct the interview by phone.
Qualitative Interviews with Practice Staff Members: To gain insight into project implementation from outside of the implementation team, project staff also will interview one or two practice employees who are not members of the Health Literacy Team. These interviews will occur during the post-implementation site visit, but can be conducted by phone if a scheduled interview must be cancelled due to illness or other circumstances.
Check-in Phone Calls: So that we may closely monitor implementation progress, the project team will conduct routine check-in calls with the Health Literacy Team leader. Calls will be initiated by the project team two weeks, one month, two months, and four months into the six-month implementation period (all Check-in Calls will be digitally recorded and transcribed). Practices will be welcome to contact us at any other time with questions. For all contacts with the participating practices, the project team will track who initiated the contact, by what means (e.g., email, phone), what tool was the topic of the conversation, what the main purpose of the contact was, what information was provided by the project team, what the resolution of the call was, and the length of the contact in minutes (for phone contacts). Data regarding Check-in Calls and other technical assistance communications will allow us to estimate the time spent providing technical assistance and to identify tools requiring more or less facilitation. In addition, these data will serve as the basis for the Technical Assistance Guide and will contribute to identification of additional resources that may be useful to include in the Toolkit.
Data Analysis: The goal of the qualitative analysis will be to examine practice personnel’s perspectives on and experiences with Toolkit implementation, with the ultimate goal of identifying refinements that might make the Toolkit more readily useable in the primary care setting. Using transcriptions from the pre-implementation and post-implementation qualitative interviews as well as documentation from Check-in Calls, site-visit observation notes, and practice-initiated technical assistance calls, the experiences of practices in implementing the Toolkit will be described in Case Studies.
The qualitative data software package ATLAS.ti will be used to analyze all qualitative data. This software will allow the project team’s two qualitative researchers to review and identify meaningful segments of text, assign code words, and identify emerging concepts. As a theory-building qualitative package, ATLAS.ti will be used to code the data, to help the investigators record memos and insights about the data, and to build and test theories. Once all data are coded, the qualitative researchers will synthesize the data, triangulating the findings from interviews, Check-in and technical assistance calls, and site visit observations. This process will allow for refinements in explanatory models and themes, comparisons across different practices, and answering questions about facilitators and barriers to implementation. The triangulation process will involve meetings of the full research team to engage in reflexive team analysis, including jointly reviewing findings, examining contradictory data, and considering the possibility of social desirability underlying apparent discrepancies.
We will use a variety of methods to enhance response rates.
Patient-Level Data: Although we cannot provide incentives to participants for completing the Patient Survey, we will take measures to enhance response rates. First, we will ensure that the survey is not excessive in length. In addition, we will allow patients to complete the survey at the doctor’s office or mail it later using a self-addressed, stamped envelope (if the data are collected using a paper survey). We also will train practice staff thoroughly to ensure that they emphasize the importance of the survey, highlighting that the answers provided by participating patients will help the practice to improve the way they communicate with patients and assuring patients that their personal responses will not be seen by anyone at the practice.
Response rates for the CAHPS survey will be maximized through repeated mailings and phone calls. Data from the developers of CAHPS suggest that the method we will be employing, which involves repeated mailings and phone calls, is likely to result in a response rate of approximately 55%.2
In implementing the Medication Review Form, we will instruct practices to recruit patients they have identified through normal clinical activities (e.g., medication refill processes) as requiring a medication review. Incorporating project-related medication reviews into routine practice will ensure that the reviews conducted are of clinical value, thus enhancing buy-in of providers and patients. We will encourage the practice staff coordinating Toolkit implementation to emphasize with providers the importance of completing the Medication Review Form as part of the review process. In addition to supporting evaluation activities, these data will provide the practices themselves with detailed information about the types and pervasiveness of medication regimen errors among their patients. The form itself also can be incorporated into the chart to supplement documentation related to the review process. To ensure that scheduled medication reviews proceed as planned, we also will encourage practices to utilize strategies for ensuring that selected patients bring all medications to the scheduled medication review visit (e.g., postcard and phone reminders).
Practice-Level Data: Practices recruited to participate in the demonstration will have an inherent interest in implementing the Toolkit and will have been informed in detail about the data collection requirements. Further, the project team will be working primarily with the leader of the practice’s Health Literacy Team, who will oversee Toolkit implementation the project at each practice. This individual or his/her designated project coordinator will be responsible for ensuring that all project-related data are collected and all qualitative interviews are conducted.
Quantitative statistical support for the project will be provided by L. Miriam Dickinson, Ph.D., a biostatistician in the University of Colorado Department of Family Medicine. All qualitative analysis will be directed and conducted by Karen Albright, PhD, an Assistant Professor in the Colorado School of Public Health.
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT |
| Author | wcarroll |
| Last Modified By | ctac |
| File Modified | 2012-08-21 |
| File Created | 2012-08-21 |