OMB_StatementA_PCORTFEval 01272020

The mission of the Agency for Healthcare Research and Quality (AHRQ) is to produce evidence to make health care safer, higher quality, more accessible, equitable, and affordable, and to work within the U.S. Department of Health and Human Services and with other partners to make sure that the evidence is understood and used.

According to its authorizing legislation (Healthcare Research and Quality Act of 1999; see http://www.ahrq.gov/hrqa99.pdf), AHRQ shall promote health care quality improvement by conducting and supporting the following:

1. Research that develops and presents scientific evidence regarding all aspects of health care;

2. The synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policymakers, and educators; and

3. Initiatives to advance private and public efforts to improve health care quality

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

AHRQ Authorization to Provide Researcher Training in Comparative Effectiveness Research/Patient-Centered Outcomes Research (CER/PCOR) Methods

Section 6301(b) of the Patient Protection and Affordable Care Act, Public Law 111-148 (the “Affordable Care Act”), enacted Section 937(e) of the Public Health Service Act, authorizes AHRQ to build capacity for comparative effectiveness research (CER) by establishing grant programs that provide training for researchers in methods used to conduct research. It also notes that, “[at] a minimum, such training shall be in methods that meet the methodological standards adopted [by PCORI] under 1181(d)(9) of the Social Security Act.” In addition, AHRQ is charged with dissemination of patient-centered outcomes research (PCOR) and CER into practice, Section 937(a). To this end, AHRQ’s PCOR Trust Fund training program (PCORTF-TP) responds to Congress’ direction to AHRQ to invest in training grants that build researchers’ skills and enhance research capacity in these areas.

PCOR is research that assesses the benefits and harms of preventive, diagnostic, therapeutic, palliative, or health delivery system interventions. This research helps clinicians, patients, and caregivers make decisions about health care choices by highlighting comparisons and outcomes that matter to people, such as survival, function, symptoms, and health related quality of life. The AHRQ PCORTF-TP supports individuals and academic institutions to train researchers and clinicians in CER methods applied within the context of PCOR (CER/PCOR) via mentored career development award mechanisms for emerging independent investigators as well as targeted skill development and applied experiences via research grant mechanisms for independent researchers. PCORTF-TP grants support training for recent graduates, mid-career professionals, and established professionals in research and clinical settings. The program prioritizes expanding capacity in underserved and predominantly minority communities.

AHRQ recognizes the importance of ensuring that its training activities are useful, well implemented, and effective in achieving their intended goals. Therefore, the PCORTF-TP evaluation reflects AHRQ’s commitment to ensuring responsible stewardship. The PCORTF-TP evaluation comprises analysis of grantee progress reports, a bibliometric analysis of grantee publications, key informant interviews with AHRQ program staff responsible for managing PCORTF-TP grants, focused discussions with a PCORTF-TP evaluation Stakeholder Working Group, and surveys of grantees and mentors.

The purpose of this evaluation is to assess the outputs, outcomes, and impact of AHRQ’s PCOR Career Development (K) Award training and infrastructure capacity-building programs. The evaluation will address the following questions:

What is the nature of PCORTF–TP activities for scholar/investigator development?
Which activities for PCORTF–TP scholars/investigators have the greatest influence on intended outcomes (e.g., PCOR careers)?
How have PCORTF–TP and partner institutions developed the capacity for PCOR training and mentoring, and in what ways is this sustainable?
What do mentors and mentees perceive to be the most important ways that the program has contributed to the field of CER//PCOR?

To achieve the goals of this project, the evaluator will survey PCORTF–TP awardees, scholars, and mentors. Online surveys (Attachment A: K Awardee Survey/K12 Scholar Survey and Attachment B: K Awardee /K12 Scholar Primary Mentor Survey) will be used to: 1) collect non-identifying demographic information; and 2) ask respondents about their training activities and outcomes. Key informant interviews (Attachment C: Key Informant Interview Guide) will be used to collect qualitative data about program processes, outcomes, and lessons learned from K12 scholar program directors.

The PCORTF–TP evaluation is being conducted by AHRQ through its contractor, AFYA, Inc., pursuant to (1) 42 U.S.C. 299b-7; (2) AHRQ’s authority to conduct and support research on health care and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness, and value of health care services, 42 U.S.C. 299a(a)(1); and (3) AHRQ’s authority to support the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policymakers, and educators, 42 U.S.C. 299(b)(2).

2. Purpose and Use of Information Collection

AHRQ will use the information collected through this Information Collection Request to assess progress toward achieving the PCORTF-TP aims. The information collected will facilitate program planning. Results will indicate whether grantees are conducting activities relevant to CER/PCOR training and whether those activities are increasing CER/PCOR capacity. Two surveys, each tailored for four respective PCORTF-TP respondent groups as well as key informant interviews will yield data on training activities, trainees’ career plans, trainees’ research and clinical activities relevant to CER/PCOR, and primary mentor experiences. The surveys (refer to Attachments A and B) are designed to capture primarily quantitative data with some qualitative data. The interview guide (Attachment C) is designed to collect qualitative data.

No claim is made that the results from this study will be generalizable in the statistical sense. Rather, this evaluation is aimed at determining the effectiveness of this particular program.

3. Use of Improved Information Technology

This evaluation study will rely on self-administered Web-based survey instruments (Attachments A and B), which will be deployed using low-burden, and respondent-friendly survey administration processes and instruments. Additionally, the technology to be employed can be configured to allow participants to complete as much of the questionnaire as desired in one sitting or to continue the questionnaire at another time. The technology also minimizes the possibility of participant error by electronically skipping questions that are not applicable to a particular participant, thus minimizing participant burden. Key informant interviews (Attachment C) will be conducted via telephone and digitally recorded.

AHRQ IT Survey Application

OVERVIEW

The Survey application is an online web-based tool that provides customizable surveys for the Agency for Healthcare Research and Quality (AHRQ). The Survey application allows the AHRQ staff and programs to create and manage private and public surveys. Key features include customizable survey questions and webpage, communication modules for reaching out to survey responders, and data analysis and visual representation tools.

The Survey application collects and stores survey data and survey respondent information in a secure Survey database behind the AHRQ network firewall. AHRQ Survey administrators access the Survey application data through a secure two-factor single sign-on authentication. External administrators access the application through a secure online interface which requires a user identification and password entered through the Survey Login screen. All Survey data are manually deleted from the system 30 days after the survey end date.

FEATURES

Create and Manage Survey
Create and Publish Survey Webpage
Pre-register Survey Respondents
Email Notifications to Survey Respondents
Search and Filter Survey Data
Export to Spreadsheet
Survey Analytics & Reporting
Store Survey Data in the Secure Survey Database
Manual Delete/Purge of Survey Data

4. Efforts to Identify Duplication

The current request supports the evaluation of a specialized CER/PCOR training program for the Agency and, therefore, the instruments are specific and unique to this project. While, in 2016, AHRQ completed a similar, though smaller-scale grantee survey effort of its general health services research Career Development (K) Award training program, no other training program like the one associated with this effort exists, so we are assured that the evaluation of this project is not a duplicative effort.

5. Involvement of Small Businesses or other Small Entities

This collection request does not involve burden to small businesses or other small entities.

6. Consequences if Information Collected Less Frequently

Data will be obtained only once for this study. Survey data collection is critical to the evaluation design as it is the one method by which evaluators will ask individual trainees and mentors these questions about program activities and results.

7. Special Circumstances

This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

As required by 5 CFR 1320.8(d), a notice was published in the Federal Register on December 13, 2019, Vol. 84, No. 240, page 68170 for 60 days, and again on January 27, 2020, Vol. 85, No. 17, page 4666 for 30 days (see Attachment D). No public comments were received during the 60-Day review period.

8.b. Outside Consultations

A comprehensive formative research effort was conducted by AFYA Inc., to inform the design of the evaluation and data collection instruments. Formative research included review of program progress reports, key informant interviews with AHRQ PCORTF-TP program staff, and a small focus group with grantees. These data were analyzed to determine which information should be collected through surveys as well as how questions should be phrased to be appropriate for PCORTF-TP grant participants.

A grantee stakeholder working group also reviewed and provided feedback on draft data collection instruments. There are no unresolved issues.

9. Payments/Gifts to Respondents

No honoraria or other gifts will be provided to respondents in exchange for their participation in the survey.

10. Assurance of Confidentiality

Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.

No confidential or protected health information will be collected for this project.

11. Questions of a Sensitive Nature

This project includes no questions of a sensitive nature. The evaluation instruments do not contain any questions concerning sexual behavior and attitudes, religious beliefs, income, or proprietary business information.

12. Estimates of Annualized Burden Hours and Costs

Exhibit 1 shows the estimated annualized burden hours for the respondents’ time to participate in this evaluation. The survey will be completed by approximately 288 awardees, scholars, principal investigators (PI), and mentors. The surveys (Attachments A and B) will each require approximately 30 minutes to complete. The key informant interview (Attachment C) will be conducted with approximately 13 PIs. These interviews are expected to take one hour each. The total hour burden is expected to be 150.0 hours for this participant data collection effort.

Exhibit 1: Estimated annualized burden hours

Form Name	Number of Respondents	Number of responses per respondent	Hours per response	Total Burden hours
*K Awardee/K12 Scholar Survey**	147	1	0.5	73.5
K Awardee/K12 Primary Mentor Survey	128	1	0.5	64
Key Informant Interview Guide for K12 Program Directors	13	1	1	13
Total	288			150.5

*K Awardee/K12 Scholar survey = K01/K08/K99/K18 Awardees and K12 Scholars

Exhibit 2 shows the estimated annualized cost burden based on the respondents’ time to participate in this project. The total cost burden is estimated to be $11,134.34.

Exhibit 2: Estimated annualized cost burden

Form Name	Number of Respondents	Total Burden hours	Average Hourly Wage Rate*	Total Cost Burden
K Awardee/K12 Scholar Survey	147	73.5	$74.43*	$5,434.59
K Awardee/K12 Primary Mentor Survey	128	64	$74.43*	$4,732.16
Key Informant Interview Guide for K12 Program Directors	13	13	$74.43*	$967.59
	288	150.5		$11,134.34

* Average hourly wage ($73.94) based on the average annual salary for three categories of Health Specialties Teachers, Postsecondary (25-1071; Scientific Research and Development Services-$178,090; General Medical and Surgical Hospitals-$153,790; and Colleges, Universities, and Professional Schools-$126,890). Data Source: National Occupational Employment and Wage Estimates in the United States, May 2018, “U.S. Department of Labor, Bureau of Labor Statistics” (available at http://www.bls.gov/oes/current/naics4_621400.htm)

13. Estimates of Annualized Respondent Capital and Maintenance Costs

There are no capital and maintenance costs to respondents for this effort.

14. Estimates of Annualized Cost to the Government

The total cost to the Government for this evaluation activity is estimated to be $596,162.57. This amount is the total contract value for this project being conducted by AFYA under contract with AHRQ to evaluate the PCORTF Training Program. This amount is itemized in Exhibit 3 on an annualized basis, and includes costs for developing and revising the evaluation plan (task 1, 3, and 4); the conduct of formative research and secondary data collection and analysis (task 2), primary data collection activities and data processing and analysis (task 5); results dissemination (task 6); and administrative support activities and reporting (task 7). The average annualized cost for this evaluation activity is estimated to be $198,720.86.

Exhibit 3: Estimated Total and Annualized Cost

Cost Component	Total Cost	Annualized Cost FY2019	Annualized Cost FY2020	Annualized Cost FY2021
Developing and revising the evaluation plan	$48,134.14	$48,134.14	$0.00	$0.00
Formative research and secondary data collection/analysis	$61,122.73	$61,122.73	$0.00	$0.00
Prima Analysis	$295,723.40	$74,914.30	$144,992.06	$75,817.05
Results dissemination	$54,272.14	$4,228.11	$0.00	$50,044.04
Administrative Support and Reporting	$136,910.15	$43,284.92	$45,382.57	$48,242.66
Total	$596,162.57	$231,684.20	$190,374.63	$174,103.74

15. Changes in Hour Burden

This is a new collection of information.

16. Time Schedule, Publication, and Analysis Plans

The project timeline is shown in Exhibit 4 below.

Exhibit 4: Project Timeline

Data Collection and Analysis	Timeframes
Prepare data collection instruments (e.g., phone interview guide, survey instrument) and share with AHRQ TOO for review/approval	8 months
Prepare and submit OMB clearance package for data collection instruments	9 months (for process and complete approval)
Administer AHRQ TOO approved web-based survey	5 weeks
Conduct secondary data review and bibliometric analyses	24 months

Publication Plan

Study results will be disseminated through peer-reviewed publications; presentations at professional conferences, including relevant AHRQ conferences; and AHRQ’s Web site. Manuscripts and presentations will clearly state the limitations of the study findings including the lack of generalizability of the specific results associated with the research methods.

Analysis Plan

Descriptive statistics will be calculated for all survey items. Survey responses will be aggregated and frequency distributions will be compiled. Bonferroni correction will be calculated for multiple comparisons. Chi-square or Fisher’s exact tests will be performed to compare differences for categorical data. Responses to open-ended questions will be coded using content analytic techniques.

17. Exemption for Display of Expiration Date

AHRQ does not seek this exemption.

List of Attachments

Surveys and Survey Follow-up Interview Guide

Attachment A: K Awardee Survey/K12 Scholar Survey

Attachment B: K Awardee /K12 Scholar Primary Mentor Survey

Focus Group and Key Informant Interview Guides

Attachment C: Key Informant Interview Guide for K12 Program Directors

Federal Register Notice

Attachment D: Federal Register Notice

Respondent Invitation Messaging

Attachment E: AHRQ Survey Introduction Message and Invitation for Awardees/Scholars

Attachment F: AHRQ Survey Introduction Message and Invitation for Mentors

Attachment G: Interview Participation Introduction and Invitation for K12 Program Directors

ATTACHMENT E: AHRQ Survey Introduction Message and Invitation for Awardees/Scholars

Invitation message to be sent by AHRQ:

Dear AHRQ PCOR Training Program Participant,

In an effort to understand your experience with AHRQ’s Patient-Centered Outcomes Research (PCOR) Career Development (K) Award grant program, we have partnered with AFYA, Inc., an independent research and evaluation firm, to conduct an online survey. The purpose of this survey is to improve AHRQ’s understanding of program outcomes, which will inform AHRQ’s program planning and resource allocation as well as outcomes reporting.

Participation in the survey is voluntary. Choosing to participate or not will have no affect your current or future receipt of AHRQ funding. We will protect your privacy to the extent permissible by law by separating the responses you provide from your name, contact information and other identifiers (such as your computer’s IP address, your institution’s name). Your answers will be aggregated with the responses of all other participants before being reported to AHRQ.

Your perspective is important to us and I hope you take the time to complete this survey, which should take approximately 30 minutes.

Over the next 2 weeks, you will be receiving an email from AFYA, Inc. with further instructions and a link to the survey.

If you have any questions, please do not hesitate to contact me at (insert email).

Thank you,

[insert contact information]

Invitation message to be sent by AFYA, Inc.:

Dear Dr. _______________________________,

We are inviting you to complete an online survey related to your participation in the AHRQ Patient-Centered Outcomes Research (PCOR) Career Development (K) training program. You will be asked questions about your experience with and impressions of the program.

Participation in the survey is voluntary.

It should take you approximately 30 minutes to complete the survey. The information that you provide is very important to AHRQ and we hope that you will find the time to participate. We request that you complete the questionnaire by DATE.

If you have any questions, please do not hesitate to contact me by telephone (XXXXXXXXX) or email (insert email).

Regards,

[insert contact information]

ATTACHMENT F: AHRQ Survey Introduction Message and Invitation for Mentors

Invitation message to be sent by AHRQ:

Dear AHRQ PCOR Training Program Mentor,

In an effort to understand your experience as a primary mentor on an AHRQ Patient-Centered Outcomes Research Career Development (K) Award grant, we have partnered with AFYA, Inc., an independent research and evaluation firm, to conduct an online survey. The purpose of this survey is to improve AHRQ’s understanding of training mentor experience, which will inform AHRQ’s program planning and resource allocation as well as outcomes reporting.

Your perspective is important to us and I hope you take the time to complete this survey, which should take approximately 30 minutes.

Over the next 2 weeks, you will be receiving an email from AFYA, Inc. with further instructions and a link to the survey.

If you have any questions, please do not hesitate to contact me at (insert email).

Thank you,

[insert contact information]

Invitation message to be sent by AFYA, Inc.:

Dear Dr. _______________________,

We are inviting you to complete an online survey related to your participation as a mentor in the AHRQ Patient-Centered Outcomes Research career development award grant program. You will be asked questions about your experience with and impressions of the program.

Participation in the survey is voluntary.

If you have any questions, please do not hesitate to contact me by telephone (XXXXXXXXX) or email (insert email).

Regards,

[insert contact information]

ATTACHMENT G: Interview Participation Introduction and Invitation for K12 Program Directors

Invitation message to be sent by AHRQ:

Dear PCORTF K12 grant Program Director,

In an effort to understand your experience with AHRQ’s Patient-Centered Outcomes Research (PCOR) K12 grant program, we have partnered with AFYA, Inc., an independent research and evaluation firm, to conduct interviews with PCOR K12 Program Directors. The purpose of these interviews is to improve AHRQ’s understanding of training program processes and outcomes, which will inform AHRQ’s program planning and resource allocation as well as outcomes reporting.

Participation in the interview is voluntary. Choosing to participate or not will have no affect your current or future receipt of AHRQ funding. We will protect your privacy to the extent permissible by law by separating the responses you provide from your name, contact information and other identifiers (such as your institution’s name). Your answers will be aggregated with the responses of all other participants before being reported to AHRQ.

Your perspective is important to us and I hope you take the time to complete this survey, which should take approximately 60 minutes.

Over the next 2 weeks, you will be receiving an email from AFYA, Inc. with further instructions on scheduling an interview.

If you have any questions, please do not hesitate to contact me.

Thank you,

[insert contact information]

Invitation message to be sent by AFYA, Inc.:

Dear Dr. ___________________________,

We are inviting you to participate in a key informant interview about your experience as an AHRQ Patient-Centered Outcomes Research (PCOR) K12 Award Program Director. You will be asked questions about your experience with and impressions of the program. Participation is voluntary.

The interview is expected to take approximately 60 minutes to complete. The information that you provide is very important to AHRQ and we hope that you will find the time to participate. We request that you schedule an interview by DATE.

If you have any questions, please do not hesitate to contact me by telephone (XXXXXXXXX) or email (insert email).

Regards,

[insert contact information]

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	SUPPORTING STATEMENT
Author	Jenice James
File Modified	0000-00-00
File Created	2021-01-14