Responses to Public Comments
on the Health IT Tool Evaluation Project
In accordance with the Paperwork Reduction Act of 1995, 44 U.S.C. 3506(c)(2)(A), the Agency for Healthcare Research and Quality (AHRQ) published a 60-day notice in the Federal Register on March 11, 2011 regarding the project ‘‘Health IT Tool Evaluation.’’
AHRQ received comments from the following individuals and organizations during this period (in order of receipt):
Laura J. Fochtmann, M.D., Professor of Psychiatry and Behavioral Science Stony Brook University
AHRQ appreciates these parties’ interest in the project and their thoughtful comments. AHRQ also appreciates that the parties express support for the study. Dr. Fochtmann’s comments primarily focus on the issues of recruitment, screening, and burden.
AHRQ provides a specific response to the individual comment, which is included below.
RESPONSE TO PUBLIC COMMENT #1 (From Dr. Laura J. Fochtmann)
ISSUE: “it is not clear how those individuals will be identified for possible screening”
RESPONSE: As described in Supporting Statement Part B section 2, participants will be recruited by a consulting firm specializing in health information technology (IT). This firm works with a variety of public and private organizations to develop their health information exchange (HIE) business plans and implementation strategies. The consulting firm will use its various contacts for networking purposes to recruit participants. The consulting firm will use flyers, e-mail form letters, and telephone scripts, included as Attachments F to H, to guide its evaluation staff in identifying and recruiting participants for the various study methods. The consulting firm will also conduct outreach to target organizations to solicit candidates for participation.
ISSUE: “Since the sampling for project participants appears to be opportunistic rather than a random sample of a broad range of individuals in potential user categories, it is hard to see how the data would allow a detailed assessment of the extent to which the compendium is reaching its intended audience.”
RESPONSE: It is correct that these three types of interviews will be unable to generate statistics about the size of the “user” population. AHRQ is using other secondary data sources to answer this question (e.g., using web log data). A random sample of this population would be unrealistic because there is no sampling frame of health IT professionals (as described in section 1 of Supporting Statement Part B). However, we anticipate that the interviews will help provide some qualitative sense of who the potential audience might be, which could help AHRQ in developing outreach strategies.
ISSUE: “From the standpoint of cost, the opportunistic sampling approach may be more realistic but the goals of the project should clearly delineate the specific sampling method(s) that are being used and the strengths and weaknesses of those methods.”
RESPONSE: As described in section 1 of Supporting Statement Part B, these are intended as qualitative interviews and are not intended to generate statistics. The goal is to provide AHRQ with feedback about this tool (the Health IT Survey Compendium) to help facilitate continuous improvement activities.
ISSUE: “It is possible that the non-users are simply unaware of the Health IT Survey resources. This may drive non-use rather than issues with usability, satisfaction or functionality. If it is determined that the tools are not reaching the intended audience, the reasons for that gap should also be determined. This may be implicit in the project goals but it may be worth making this issue more explicit.”
RESPONSE: Thank you. Based on the questions included in Attachments E1 and E2, we anticipate that the group discussions will help elicit information on how individuals locate health IT tools and whether there are obstacles.
ISSUE: “In terms of the estimates of annual time and cost burden, it would be useful to express the burden on a per individual basis as well as on a total basis for both hours and cost. It is not clear whether the needs assessment, discussion groups and usability testing will involve overlapping groups of individuals or whether individuals who pass the screening process will be divided into mutually exclusive groups for these 3 aspects of the study. The screening criteria are also somewhat unclear.”
RESPONSE: As indicated in Attachment I, each respondent will be asked to participate in only one interview. Those respondents that participate in the needs assessment will face an average burden of 63 minutes or $53.25, which includes 3 minutes for the screening interview and 60 minutes for the needs assessment interview. Those respondents who participate in the usability testing or discussion groups will face an average burden of 93 minutes or $78.60, which includes 3 minutes for the screening interview and 90 minutes for the usability testing or discussion group.
As shown in Attachment B (Recruiting– Demographic Screener Questionnaire), screening information includes (for example):
Experience in evaluating health IT systems (criteria for inclusion in study)
Visits to the AHRQ Health IT Web site “Health IT Tools” or “Knowledge Library”
Interaction with the “Health IT Survey Compendium”
ISSUE: “It is not clear whether individuals who participate in the study are being compensated for their time or whether the estimates of burden are simply a requirement of the AHRQ process.”
RESPONSE: As discussed in the section 9 of Supporting Statement Part A, as a thank you for their participation, participants will receive an incentive payment of $75.00.
ISSUE: “The rationale for choosing average figures across highly disparate salary levels versus weighted averages (based on likely sample composition) or median values is not clear. Also, it is not clear why figures for post-secondary teachers in general were chosen rather than the figures for college/university faculty. Similarly it is not clear why figures for physicians' salaries choose family medicine physicians who tend to have low salaries relative to many (but not all) specialist physicians.”
RESPONSE: As described in section 1 of Supporting Statement Part A, we anticipate that interviews will be conducted with participants ranging in experience and expertise. For this reason, we utilized a variety of wage categories when developing the burden estimates. Salary data were gathered from the May 2009 National Occupational Employment and Wage Estimates, United States, U.S. Bureau of Labor Statistics (BLS) Division of Occupational Employment Statistics. The BLS wage data are based on very specific categories and the nomenclature BLS uses does not always line up in the expected ways. For example, they use the term “post secondary” when describing university faculty. To see the available categories please refer to http://www.bls.gov/oes/current/oes_nat.htm#29-0000.
Attachment 1: Public Comment #1 in Response to 60-day Federal Register Notice:
The proposed information collection project will likely yield useful information about the Health IT Survey Compendium, however, I believe there are several aspects of the proposal that deserve comment.
The plan includes interviewing of users and non-users as part of the project, but it is not clear how those individuals will be identified for possible screening. Since the sampling for project participants appears to be opportunistic rather than a random sample of a broad range of individuals in potential user categories, it is hard to see how the data would allow a detailed assessment of the extent to which the compendium is reaching its intended audience. From the standpoint of cost, the opportunistic sampling approach may be more realistic but the goals of the project should clearly delineate the specific sampling method(s) that are being used and the strengths and weaknesses of those methods.
It is possible that the non-users are simply unaware of the Health IT Survey resources. This may drive non-use rather than issues with usability, satisfaction or functionality. If it is determined that the tools are not reaching the intended audience, the reasons for that gap should also be determined. This may be implicit in the project goals but it may be worth making this issue more explicit.
In terms of the estimates of annual time and cost burden, it would be useful to express the burden on a per individual basis as well as on a total basis for both hours and cost. It is not clear whether the needs assessment, discussion groups and usability testing will involve overlapping groups of individuals or whether individuals who pass the screening process will be divided into mutually exclusive groups for these 3 aspects of the study. The screening criteria are also somewhat unclear.
It is not clear whether individuals who participate in the study are being compensated for their time or whether the estimates of burden are simply a requirement of the AHRQ process. The rationale for choosing average figures across highly disparate salary levels versus weighted averages (based on likely sample composition) or median values is not clear. Also, it is not clear why figures for post-secondary teachers in general were chosen rather than the figures for college/university faculty. Similarly it is not clear why figures for physicians' salaries choose family medicine physicians who tend to have low salaries relative to many (but not all) specialist physicians.
In sum, the project looks like a positive one that may help individuals and organizations make better use of existing survey approaches for assessing health IT implementations. Consideration of the above issues may help to clarify some aspects of the design for readers and participants.
Thank you for your efforts in helping to advance better use of health information technologies.
Sincerely,
Laura J. Fochtmann, M.D.
Professor of Psychiatry and Behavioral Science Stony Brook University
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | giambo_p |
| File Modified | 0000-00-00 |
| File Created | 2021-02-01 |