Supporting Statement Part A - Health IT Tool Evaluation 6-2-2011

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see Attachment A), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:

1. Research that develops and presents scientific evidence regarding all aspects of health care; and

2. Synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

3. Initiatives to advance private and public efforts to improve health care quality.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

The Agency for Healthcare Research and Quality (AHRQ) is a lead Federal agency in developing and disseminating evidence and evidence-based tools on how health IT can improve health care quality, safety, efficiency, and effectiveness.

In support of the health IT initiative, AHRQ developed the National Resource Center (NRC) for Health IT Web site. This site contains a range of information and evidence-based tools that support the health IT initiative’s work and aims.

With this project AHRQ is conducting an evaluation to assess whether these tools are reaching their intended audiences, are easy to use, and provide the information that users expect and need. The current project is an evaluation of one of the tools available on the NRC site: the Health IT Survey Compendium. The Health IT Survey Compendium is a searchable resource that contains a set of publicly available surveys to assist organizations in evaluating health IT. The surveys in the Health IT Survey Compendium cover a broad spectrum, including user satisfaction, usability, technology use, product functionality, and the impact of health IT on safety, quality, and efficiency.

The audiences included in this evaluation are health IT researchers (ranging in experience and expertise from research assistants to more senior investigators such as university professors) and health IT implementers (e.g., clinical champions and IT staff at provider organizations, IT implementation consultants and experts). In the course of conducting this evaluation, AHRQ will evaluate both users and non-users (defined as not current but possible users) of the Health IT Survey Compendium.

The goals of this project are to determine whether the Health IT Survey Compendium is reaching its intended audiences, whether it is meeting the information needs and expectations of these audiences, and whether it is easy to use.

To achieve these goals AHRQ will conduct the following activities:

1) Screening & demographics questionnaire – used to recruit research participants for the needs assessment interviews, usability testing and discussion groups, which are described below. The questionnaire also has a demographics section to collect some basic demographic information for those persons that “screen-in.” See Attachment B.

2) Needs assessment interviews – consisting of semi-structured interviews with non-users of the Health IT Survey Compendium. The purpose of these interviews is to discover and then assess the relative importance of information needs of the intended audiences of the Compendium. These interviews will provide the perspective of non-users of the Compendium in order to elicit unbiased feedback about information needs. After thoroughly exploring information needs, each interviewee will be shown the Health IT Survey Compendium and asked to provide feedback about how it addresses their needs for surveys and data collection instruments. See Attachment C.

3) Usability testing – focusing on the navigation, ease of use, and usefulness of the Health IT Survey Compendium. These interviews will include both current users and non-users of the Health IT Survey Compendium. See Attachment D.

4) Discussion groups -- consisting of eight groups of 6-8 participants each (a maximum of 64 participants across all eight groups). The majority of the session time will be spent showing the Health IT Survey Compendium to the participants, and the moderator will elicit reactions to and opinions about the Health IT Survey Compendium, its features, and the surveys offered. Current users and non-users of the Health IT Survey Compendium will be interview separately using different interview guides. See Attachments E1 and E2.

Contact materials for recruiting participants and scheduling interviews are included as Attachments F, G and H.

This study is being conducted by AHRQ through its contractors, Westat and Mosaica Partners, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to health care technologies. 42 U.S.C. 299a(a)(5).

2. Purpose and Use of Information

The outcome of the evaluation will be a report including recommendations for enhancing and improving the Health IT Survey Compendium. The report will provide results about both the perceived usefulness and the usability of the Health IT Survey Compendium. Results will be presented for individual audience segments as well as for the user population as a whole. The report will also include specific suggestions on how to revise and extend the Health IT Survey Compendium to make it more useful to health IT researchers and implementers, and will discuss the general implications of the Health IT Survey Compendium evaluation for the development and evaluation of other tools available on the NRC Web site.

3. Use of Improved Information Technology

AHRQ will collect data through an established qualitative evaluation methodology, which includes telephone interviews with study respondents. Because most interview questions are open-ended to allow for in-depth exploration of issues, electronic submission of responses is not a viable option.

While AHRQ will not be using technology to capture responses, meeting software technology, specifically the WebEx online meeting software, will be used to facilitate the various interviews and discussions.

4. Efforts to Identify Duplication

To date, AHRQ has not conducted a systematic evaluation of its Health IT Survey Compendium. As far as AHRQ is aware, no external entity has assessed the Health IT Survey Compendium.

5. Involvement of Small Entities

As noted above, the audiences to be included in the assessment include health IT researchers and implementers who have used the surveys and other tools in the Health IT Survey Compendium or representatives of these intended audiences. It is expected that some of these individuals may be members of small businesses, but at this time it is unclear what portion of respondents would be part of a small entity. Study participation is voluntary and AHRQ will be respectful of study participants’ time. Interviews will be scheduled at times convenient for respondents.

6. Consequences if Information Collected Less Frequently

This is a one-time collection.

7. Special Circumstances

This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

As required by 5 CFR 1320.8(d), notice was published in the Federal Register on March 11^th, 2011 for 60 days (see Attachment I). One Comment was received (see Attachment J).

8.b. Outside Consultations

AHRQ consulted with its evaluation contractors, Westat and Mosaica, in developing the recruiting, interview, and moderator guides. No other outside consultants contributed to the formation of the study design and instruments.

9. Payments/Gifts to Respondents

AHRQ will offer eligible persons $75 as an incentive to participate in the Information Needs Assessment Interviews, Usability Testing, and Audience Discussion Groups. This is currently the standard payment for participation in interviews and focus groups lasting up to two hours. In order to take part in these activities, participants will have to join a phone call and WebEx session at a previously determined time. Although it may be possible to conduct this study while providing a smaller incentive amount to participants, experience suggests that doing so would increase the resources needed for recruiting participants with the desired background and experience, thus increasing the overall project cost to the government.

10. Assurance of Confidentiality

Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose without their prior consent.

The study will collect information from respondents about the usefulness and usability of the Health IT Survey Compendium. AHRQ will collect the respondent’s name, phone number, age range, organizational affiliation, title. This information will be used for case tracking purposes or for clarification call backs.

All respondent involvement will be voluntary. Only oral consent for participation will be obtained from respondents. Respondents will be informed that: (1) the project team will not share their name, their organization’s name or copies of the interview notes with anyone outside of the team; (2) respondent comments may be included in reports and publications but will not be attributed to specific individuals or organizations; and (3) the interviewers have a system to mark specific comments in interview notes as off-limits for reports and publications when notified to do so by the respondent.

All electronic files will be password protected and accessible only from a secured network. When not in use by project staff, all printed information or materials that could potentially identify participants in the study will be stored in locked cabinets that are accessible only to project team members.

11. Questions of a Sensitive Nature

No questions of a sensitive nature will be asked. Further, during the introduction to the interview, respondents will be informed that their participation is voluntary and that they can refuse to answer any question.

12. Estimates of Annualized Burden Hours and Costs

Exhibit 1 shows the estimated annual burden hours for each respondent’s time to participate in this evaluation. The screening questionnaire will be completed by as many as 120 persons and will take 3 minutes to complete on average (only those persons that “screen-in” will complete the demographics section). The needs assessment will be completed by 18 persons and requires one hour. Usability testing will involve 18 persons and is estimated to take one and a half hours. Eight discussion groups with no more than 8 persons each will be held and will last for about 90 minutes. The total annual burden is estimated to be 147 hours.

Exhibit 2 shows the estimated annual cost burden associated with the respondent time to participate in this evaluation. The total annual burden is estimated to be $7,454.

Exhibit 1. Estimated Annualized Burden Hours

Interview Type	Maximum Number of respondents	Number of responses per respondent	Max. Hours per response	Total burden hours
Screening Questionnaire	120	1	3/60	6
Needs Assessment	18	1	1.0	18
Usability Testing	18	1	1.5	27
Discussion Groups	64	1	1.5	96
Total	120	na	na	147

Exhibit 2. Estimated Annualized Cost Burden

Interview Type	Maximum Number of respondents	Total burden hours	Average hourly wage rate*	Total cost burden
Screening Questionnaire	120	6	$50.71	$304
Needs Assessment	18	18	$50.71	$913
Usability Testing	18	27	$50.71	$1,369
Discussion Groups	64	96	$50.71	$4,868
Total	120	147	NA	$7,454

*The hourly wage for the participants across the four data collections (screening questionnaire, needs assessment interview, usability testing interviews, and discussion group interviews) is based upon the mean of the average hourly wages for Social science research assistants (19-4061; $19.39 per hour);Postsecondary Health Specialties Teachers (25-1071; $53.88 per hour);Management analysts (13-1111; $40.70 per hour); Computer and Information Systems Managers (11-3021; $58.00 per hour); Family and General Practitioners Teachers (29-1060; $81.03 per hour);Pharmacists (29-1051; $51.27 per hour). May 2009 National Occupational Employment and Wage Estimates, United States, U.S. Bureau of Labor Statistics Division of Occupational Employment Statisticshttp://www.bls.gov/oes/current/oes_nat.htm#29-0000

13. Estimates of Annualized Respondent Capital and Maintenance Costs

Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.

14. Estimates of Total and Annualized Cost to the Government

The estimated total cost to the Federal Government for this project is $411,641.00 over a two-year period from September 8, 2010 to September 7, 2012. The estimated average annual cost is $205,821. Exhibit 3 provides a breakdown of the estimated total and average annual costs by category.

Exhibit 3. Estimated Total and Annual Cost* to the Federal Government

Cost component

Total cost

Annualized cost

Project Management and Coordination Activities

$ 58,140

$ 29,070

Evaluation Plan and Protocol Development

$ 44,908

$ 22,454

OMB Submission Package

$ 12,362

$ 6,181

Conduct Evaluation **

$ 159,991

$ 79,996

Data Analysis, Report and Briefing

$118,081

$59,041

Documentation and 508 Compliance

$ 18,159

$ 9,080

Total

$ 411,641

$ 205,821

*Costs are fully loaded including overhead, G&A and fees.

** These activities include the data collections described in this submission

15. Changes in Hour Burden

This is a new collection of information.

16. Time Schedule, Publication and Analysis Plans

Time schedule and publication plans. The anticipated schedule for this project is shown in Exhibit 4. Once clearance from the Office of Management and Budget is obtained, AHRQ will begin identifying appropriate respondents and scheduling and conducting interviews.

Exhibit 4. Anticipated Schedule

Activity	Estimated timeline following OMB clearance
Recruit for Interviews	Months 1 – 3
Conduct Interviews	Months 4 – 6
Analyze Results	Month 7
Report Results	Month 8

There are no plans for publication of the evaluation results. Rather, AHRQ will use results from the evaluation to inform its improvement efforts for the Health IT Survey Compendium. The public will see improvements to the Health IT Survey Compendium over time, with updates being made as resource constraints allow.

Analysis plans.

As stated in Section 1, the goals of this project are to determine whether the Health IT Survey Compendium is reaching its intended audiences, whether it is meeting the information needs and expectations of these audiences, and whether it is easy to use. The general research question categories include:

Who are the intended user audiences for the Health IT Survey Compendium?
Are the user audiences aware of and interested in the Health IT Survey Compendium and its surveys?
Do the user audiences find the Health IT Survey Compendium to provide relevant surveys that are useful and fit their needs?
Do the user audiences find the Health IT Survey Compendium to be easy to use?

On a rolling basis over the course of the project, the project team will review interview notes and meet regularly to discuss the study’s key findings. Using an iterative process, the team will identify new themes as they emerge, and explore and shape already identified themes in greater depth.

The data that result from this study will be qualitative in nature, the respondents may not be representative of the broader population, and the findings will be based on a limited number of respondents (no more than 100). However, the data will still be useful in providing input into the continuous improvement process of the Health IT Survey Compendium. Below is the analysis plan for each of the three interview methods.

Needs Assessment. The Needs Assessment interviews will provide a good understanding of the types of challenges faced by potential users of the site, and the types of surveys and survey data that would help them address their questions. These needs can be ranked by importance for the researcher and health IT implementer audience segments. The results will also provide AHRQ with relative importance data or ranked listings of the types of information tools that are considered important by the audience segments. The data will be analyzed to determine underlying dimensions, such as stage of health IT implementation, that relate to information needs. Analysis will be based on project team review of interview notes and/or interview recordings. The project team will discuss key findings and identify themes across interview participants. These discussions may also lead the team to develop a coding scheme that could be used to reduce the qualitative data into a more measurable form. A coding scheme could be used to identify issues and recommendations and ultimately be used to help quantify the magnitude of specific themes emerging from the interviews. This coding scheme would likely evolve over the course of the analysis as new themes and issues emerge in later interviews.

The informational needs reported in the needs assessment will be compared with the domain of surveys actually provided by the Health IT Survey Compendium. This information will be used to assess where the Health IT Survey Compendium successfully addresses the identified needs and where it does not. Based on the interviewees’ initial reactions to the Health IT Survey Compendium, a final report that will include recommendations for possible improvements to the Health IT Survey Compendium will be developed.

Usability Testing. Once the usability sessions are complete, results will be compiled in two ways—across tasks and participants. Data will be reviewed to identify themes and usability issues within and across the audience segments and user/non-user subgroups. Qualitative data and debriefing responses will be reviewed for insights that will inform recommendations for improving the Health IT Survey Compendium’s usability. A final report will identify specific aspects of the current site that helped or hindered the participants from completing each task accurately and quickly. The report will document ways that the tool could be redesigned to overcome any identified problems. The findings from this method will be included in a briefing and in the final report and recommendations. The results will serve as the basis for actionable recommendations for improving the Health IT Survey Compendium’s usability. The final report will discuss the barriers that participants encountered while attempting to perform the test tasks and will offer solutions to mitigate those challenges. Wherever relevant, the report will point out which challenges and solutions may be generalized to the other health IT tools on the NRC site.

Discussion Groups. Session notes will be reviewed and integrated, extracting main themes across groups, audience segments (i.e. health IT researcher or implementer), and user/non-user groups. Any segment-specific, group-specific, or user/non-user specific findings that emerge from the group discussions will be identified. For instance, the final report will document whether, based on first experience with the Health IT Survey Compendium, non-users feel that it would be a useful resource for them and at what stages of IT implementation. The final report will document the groups’ preferences and suggestions for making improvements to the Health IT Survey Compendium. These suggestions may range from adding content areas to providing additional ways to filter and refine searches for surveys of interest.

17. Exemption for Display of Expiration Date

AHRQ does not seek this exemption.

List of Attachments

Attachment A – Healthcare Research and Quality Act of 1999

Attachment B – Screening and Demographics Questionnaire

Attachment C – Needs Assessment Interview Guide

Attachment D – Usability Testing Interview Guide

Attachment E1 – Discussion Groups Interview Guide for Users

Attachment E2 – Discussion Groups Interview Guide for Non-Users

Attachment F – Recruiting Communication – Request to Organizations

Attachment G – Recruiting Communication – Flyer

Attachment H – Recruiting Communication – Scheduling Protocol

Attachment I – Federal Register Notice

Attachment J – Public Comment and AHRQs Response

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-02-01

Supporting Statement Part A - Health IT Tool Evaluation 6-2-2011