0990 Kidney X SSA_comments removed

Evaluation of the Kidney Innovation Accelerator (KidneyX)

Supporting Statement – Section A

Submitted – May 31, 2019

Program Officer/Project Officer

Sandeep Patel, Ph.D. | Open Innovation Manager

U.S. Department of Health & Human Services (HHS)

Office of the Chief Technology Officer (CTO)

Address: 200 Independence Avenue S.W. | Washington, D.C. 20201 | Rm. 624D.2

Telephone: 202-205-0355

Fax: 202-690-6772

Email: Sandeep.Patel@hhs.gov

Part A: Justification

1. Circumstances making the collection of information necessary

   1. . Overview

The Department of Health and Human Services (HHS) Office of the Chief Technology Officer (CTO) is seeking approval by the Office of Management and Budget (OMB) on a new Information Collection Request (ICR 201905-0990-002) for data collection to support the independent evaluation of the Kidney Innovation Accelerator (KidneyX). This data collection is authorized under Section 301 of the Public Health Service Act (42 U.S.C 241) and is necessary to ensure the success of the KidneyX initiative.

KidneyX was launched in October 2018, after nearly 2 years of discussion, as a joint effort between HHS and the American Society for Nephrology (ASN) to spur new innovations in the prevention, diagnosis, treatment, and possibly cure of kidney disease. One motivation for the initiative is the poor quality of life (QoL) that patients with end-stage renal disease (ESRD) now experience. The two most common forms of current treatment, kidney transplantation and kidney dialysis, have substantial limitations and debilitating side effects. KidneyX seeks to improve dramatically the existing technologies addressing renal conditions and diseases and to foster new solutions providing improved outcomes for patients via less burdensome treatment options. KidneyX seeks to elicit new innovations and accelerate the development of treatments and interventions addressing kidney diseases and disorders through a series of innovation prize competitions. An innovation prize competition defines a desired innovation or innovation domain, and the parameters defining qualified participants, the nature and format of submission, and the process for judging submissions. At the end of the submission period, expert judges will review qualifying submissions and award a prize to one or more submissions that best address the desired objective. KidneyX will provide non-dilutive funds to prizewinners, and also offer coaching and other support in working with components of HHS that facilitate commercialization, such as the Food and Drug Administration and the Center for Medicare and Medicaid Services. KidneyX is also leading efforts within HHS to streamline the approval and availability of new kidney technologies through intra-agency coordination.

This data collection will support an external evaluation of the KidneyX competitions and other innovation promotion activities conducted over an initial 5-year period. The evaluation will assess the implementation of KidneyX, measure the near-term outcomes of innovation efforts of prizewinners compared to other applicants and non-applicants in the kidney technology domain, and observe any changes in the environment for kidney technology innovation (which may or may not be attributable to KidneyX). Innovation prizes are used by a number of federal agencies (for an overview, see Office of Science & Technology Policy, 2019). We expect that this evaluation will be useful to other agencies that seek to evaluate their own innovation prize initiatives.

The data collection will measure near-term changes in KidneyX prize competition participants, such as changes in access to external investment, because longer-term outcomes relating to commercialization or public health are less likely to be observed within our observation window. Furthermore, these potential longer-term outcomes, if observed, would be rare, further inhibiting causal inference. The assessment of outcomes conducted as part of this evaluation will necessarily be primarily, if not exclusively, descriptive in nature, due to a number of limitations:

1. KidneyX competitions do not select winners based on the achievement of well-defined a priori objectives;

2. the innovation domains vary from one KidneyX prize competition to the next; and

3. the parameters defining the structure, format, and rules may also vary across KidneyX prize competitions.

2. . Study Design and Evaluation Questions

This evaluation will provide an external, independent assessment of the implementation of KidneyX, and analyze the near-term outcomes potentially influenced by the initiative. The evaluation is conducted by RTI International, a nonprofit contract research institute, under contract to and at the direction of HHS.

Numerous federal agencies conduct prize competitions pursuant to the authority granted under the America COMPETES Act Reauthorization of 2011 (PL-111-358) and its amendments. However, evaluations of prize competitions are relatively rare. Such evaluations face two key challenges: the difficulty in establishing a counterfactual (the outcomes that would take place absent the competition), and the extent to which outcomes vary with the specific rules and structure of each competition (Galasso et al., 2018; RTI International, 2017; Murray, Stern, Campbell & MacCormack, 2012). Also, since KidneyX is a new initiative and its parameters are subject to change over time, our findings may influence shifts in the design and implementation of KidneyX. This makes our project a formative evaluation, in that it provides an ongoing but necessarily incomplete analysis of the initiative’s outcomes, many of which many not be fully apparent for several years. The RTI team also approaches this evaluation study as part of a learning agenda for HHS, such that early results can provide useful feedback and information that can shape future changes to the initiative (Smith Nightingale et al., 2018).

As a result, this evaluation uses qualitative techniques, such as document analysis and stakeholder interviews, to capture the details and effects of processes and changes within the KidneyX initiative. Quantitative analysis using survey data will complement the qualitative analysis by measuring specified variables in different time periods. We will also use secondary data sources to provide independent validation of observations gathered from program participants and stakeholders, and which will be particularly helpful for gathering outcomes of non-participants.

Econometric analysis may be applied in discrete situations in which there exists a clear definition of an “intervention,” and a setting enabling quasi-experimental designs such as difference-in-differences, regression discontinuity, or synthetic control. An opportunity like this could arise in a situation where KidneyX decided to test promotional strategies to increase the quality and/or quantity of participation and was willing to test these alternative strategies by randomly varying which audiences received which promotion strategy over time. As KidneyX evolves, our analytical methods can adapt to those changes, even though the data collection methods will not change over the course of the 5-year evaluation period.

The KidneyX initiative seeks to improve the environment for innovation in kidney technologies by pursing five inter-related goals:

• Establishing of a kidney technology innovation program that integrates the priorities and perspectives of multiple stakeholder groups.

• Increasing the volume and variety of high-quality ideas and innovations in the kidney technology domain.

• Mitigating the risks of innovation in kidney technology through financial leverage, expert assistance for innovators, and coordinating reforms to regulatory and reimbursement processes in kidney treatment.

• Integrating patient and caregiver needs into innovations that improve the quality of life for those suffering from kidney disease.

• Increasing awareness of the challenges and opportunities in kidney technology innovation in a way that attracts more funding, talent, and public support to the domain.

This yields five evaluation questions and attendant indicators and data sources, shown in Exhibit 1.

Exhibit 1. Evaluation Questions and Data Sources

Evaluation Questions	Indicators		Data Source
		Document Review*	Stakeholder Interviews				Surveys		Secondary Data Sources**
			Applicants	Awardees	Non-awardees	Other Stakeholders	Applicants	Awardees
1.How did KidneyX attempt to establish a multi-stakeholder program in kidney technology innovation, and did it satisfy stakeholder needs and concerns?	Number of stakeholders recruited and retained for involvement in KidneyX	X				X
	Stakeholders’ satisfaction with KidneyX processes, strategy, and activities					X
	Adherence of KidneyX to its stated mission and program goals	X	X	X	X	X	X	X
2.To what extent, if any, did KidneyX increase the volume and variety of new approaches to address kidney disease and disorders?	Number of proposed innovations and ideas elicited	X	X	X		X	X	X	X
	Number of new innovators who entered the kidney domain		X	X			X	X	X
	Variety of proposed innovations and ideas elicited from KidneyX participants	X	X	X		X	X	X	X
	Change in diversity of innovators in kidney technology due to participation in KidneyX	X	X	X	X	X	X	X
3.How, and to what extent, did KidneyX help to de-risk promising innovations and facilitate their development and deployment?	Change in the number of ventures qualifying for investment due to participation in KidneyX?			X	X	X	X	X	X
	Degree to which KidneyX helped innovators understand the parameters of innovation in kidney technology		X	X			X	X
	Influence of KidneyX on emergence of new business models in kidney technology	X	X	X		X	X	X	X
	Degree to which KidneyX facilitated regulatory approval for participants		X	X		X	X	X

Exhibit 1 (cont). Evaluation Questions and Data Sources

Evaluation Questions	Indicators		Data Source
		Document Review*	Stakeholder Interviews				Surveys		Secondary Data Sources**
			Applicants	Awardees	Non-awardees	Other Stakeholders	Applicants	Awardees
4. To what extent, if any, did KidneyX support innovations that integrate patient and caregiver needs and concerns?	Nature and degree of patient and caregiver influence on innovations from KidneyX participants	X				X
	Degree to which KidneyX participants increased their understanding of patient and caregiver needs and concerns	X	X	X			X	X
	Degree to which KidneyX participants continued to involve patients and caregivers in the innovation process	X	X	X			X	X
	Degree to which solutions from KidneyX participants aligned with patient and caregiver needs and priorities	X
5.To what extent, if any, did KidneyX generate interest and awareness in kidney technology and innovation from those outside the kidney domain?	Additional funding invested in kidney innovation		X		X	X	X		X
	Change in the scale and speed of innovations in kidney technology from nontraditional innovators		X	X	X	X	X	X	X
	Level of awareness about KidneyX among innovators outside the traditional kidney technology community, and participation by such innovators in KidneyX activities.	X		X		X		X	X

* Program documents included documents related to the prize competitions and general program documents (e.g., strategy, planning documents).

** Secondary data sources included Web of Science, U.S. Patent & Trademark Office, D&B Hoovers, Crunchbase, and similar sources.

The evaluation will attempt to estimate the change in the rate and direction of innovation in kidney technology during the evaluation period through qualitative and descriptive analyses based conceptually on the logic of difference-in-difference. Using this logical framework, we will measure the changes in innovation outcomes at different points in time between entrants in KidneyX competitions who are named as prizewinners with the changes at the same points in time for two primary comparison groups:

• Innovators in the kidney technology domain who do not enter a KidneyX competition.

• Innovators in the kidney technology domain who enter a KidneyX competition but are not awarded a prize.

Although the comparison groups are meant to provide evidence of the counterfactual (i.e., how innovation in kidney technology, treatments, and solutions would progress if the KidneyX initiative did not exist), we note that even the first comparison group may be affected by the activities and influences of KidneyX. For example, the publicity and attention that KidneyX brings to the challenges and significance of kidney disease detection and treatment may induce increased interest and investment in innovations unrelated to the KidneyX initiative. Our study is intended to detect differential changes in outcome trends across the groups, but substantial data and careful analysis will be needed to attribute any such differences to the KidneyX initiative. Due to the complex innovation environment and the expected dynamism in how KidneyX is implemented, we intend to probe relationships between KidneyX activities and observed changes in kidney technology innovation, even though the study parameters preclude any determination of causality between KidneyX and the observed changes.

2. Purpose and use of information collection

The purpose of this data collection is to support an independent evaluation of the Kidney Innovation Accelerator (KidneyX) initiative organized by the U.S. Department of Health and Human Services (HHS). Section 401 of the Pandemic and All-Hazards Preparedness Act of 2006 authorized the Secretary of Health and Human Services to use prize payments to support and encourage biomedical innovation. The America COMPETES Reauthorization Act, as amended, requires reports to Congress every two years on prize competitions administered by HHS and other agencies. The data collected under this submission will help HHS inform the required reports to Congress and also support departmental evidence-building activities required under the Foundations of Evidence-Based Policymaking Act of 2018.

This evaluation will assess the effectiveness of the KidneyX initiative in promoting innovation in kidney technology, based on how HHS implements the initiative and the experiences of KidneyX participants and non-participants, pre- and post-competition. For this evaluation, participants include applicants awarded prizes in KidneyX competitions and non-awardee applicants. Non-participants are individuals and organizations engaged in kidney technology development who do not submit entries into KidneyX competitions. We will gather data from awardee applicants, non-awardees, non-participants, and other stakeholders through Web-based surveys and telephone interviews. The surveys and interviews will serve to both validate the data collected through each method and to provide more specific detail on the conditions and context affecting these groups. Each proposed data collection is described below.

1. . Survey Data Collection

Pre-award survey of all KidneyX applicants: The pre-award survey instrument will be administered to all applicants to the KidneyX prize competitions. The pre-award survey will be launched after a competition’s submission deadline and will close before winners are announced. These data will establish the baseline conditions for applicants prior to the award announcement. The pre-award survey will focus on how applicants became interested in KidneyX, their experiences with the application process, their broader initial perceptions about KidneyX, and their forward-looking expectations and goals. Data gathered using the pre-award survey will assist in answering Evaluation Questions 2 and 5. The pre-award survey instrument is provided in Attachment A.

Post-award surveys of all KidneyX awardees and non-awardees: The post-award survey of non-awardees will request responses from applicants who, after the prize competition announcements, were not selected for an award. Meanwhile, the post-award survey of awardees will gather data from applicants who were ultimately selected as a prize competition winner. These two surveys will identify and probe outcomes experienced by these two groups of applicants. Non-awardees in particular will be queried about their level of satisfaction with the submission process, and suggestions on methods to improve that process. The post-award surveys will be launched synchronously to both groups 6 months after each prize announcement so that the two groups are temporally consistent, enabling more direct comparisons of their activities and experiences pre- and post-award. Surveys will remain open for approximately 2 months. Although many questions will be identical for these two groups, the two post-award surveys will necessarily vary somewhat to probe whether an award, distinct from simply participating, had any tangible or intangible effect on an applicant. The post-award surveys will generate data on how winners spent the prize money, KidneyX’s facilitation of regulatory or commercial pathways and outcomes, and the concrete interactions with HHS, its partners, and other stakeholders that may have been enabling factors. Data gathered using the post-award survey will assist in answering Evaluation Questions 2, 3 and 4. The post-award survey instrument is provided in Attachment B.

The projected number of respondents for the pre- and post-award surveys is unclear because of the uncertainty about the number of KidneyX applicants. We plan to survey the population of applicants to maximize the validity of the analytical results because of the potential of a rather small number of applicants, survey non-response, and an expected high degree of heterogeneity among applicants.

2. . Qualitative Data Collection

As part of the qualitative data collection, the RTI team will review documents related to KidneyX and to each competition and code any content that is likely to affect the outcomes from each competition. This content could include competition rules, required applicant qualifications, and the innovations targeted by the competition. We will store the coded content for analysis concurrent with the analysis of data collected through interviews.

RTI International will conduct up to 30 semistructured telephone interviews for each year of the evaluation period. Interviews will be up to 45 minutes long and will provide contextual insights that build on findings from our document review. For each competition under KidneyX, we will interview selected applicants immediately after the end of the submission period, primarily to elicit views on their motivation for entering the competition, impressions of the submission process, and experience with engaging patients and care partners while preparing submissions. We will then interview awardees and non-awardee applicants 8 months after announcement of the competition’s prizewinners to examine the difference in outcomes between awardee and non-awardee applicants. Non-awardees in particular will be queried about their level of satisfaction with the submission process, and suggestions on methods to improve that process. We will interview other stakeholders 8 months after awards are made to solicit their views on how the competition was structured and conducted.

As the Redesign Dialysis Phase 1 competition awards have been made, we will only conduct post-award interviews for that competition. For future KidneyX competitions, we will conduct both pre-award and post-award interviews. An overview of the annual schedule for qualitative data collection is provided in Exhibit 2.

Exhibit 2. Overview of Annual Qualitative Data Collection

	Document Review		Stakeholder Interviews
	General Program Documents	Prize Competition Document	Applicants	Awardees	Other Stakeholders
Redesign Dialysis Phase 1 Competition (Awarded April 2019)
Submission period		X
8 mos. after award			X	X	X
Subsequent KidneyX Competitions
Submission period		X
Immediately after submission		X	X
8 mos. after award			X	X	X
Ongoing
As documents are available	X

Applicant Interviews: For each prize competition, we will interview a sample of applicants before competition winners are announced. These interviews will gather information about respondents’ perspectives on the challenge application process. Data gathered during the applicant interviews will inform responses to Evaluation 2, 4 and 5. Up to 12 respondents will be interviewed per year, each taking approximately 45 minutes to complete. The applicant interview guide is provided in Attachment C.

Non-awardee Interviews: For each prize competition, we will interview selected non-awardee applicants. These interviews will gather information about non-awardee perspectives on KidneyX, including whether their work to develop a submission has been beneficial (e.g., judging feedback useful) and whether they plan to resubmit to KidneyX and/or seek other investors. Data gathered during the non-awardee interviews interview will inform responses to Evaluation Questions 2, 3, 4, and 5. This interview will be administered to up to 6 respondents per year and take approximately 45 minutes to complete. The non-awardee instrument is provided in Attachment D.

Awardee Interviews: For each prize competition, we will interview a selection of awardees. These interviews will gather information about awardee experience with KidneyX after the award, for example, the extent to which KidneyX facilitated access to investors and business experts and provided resources and guidance to help with commercialization and experiences since award. Data gathered during the awardee interviews will inform responses to Evaluation Questions 2, 3, 4, and 5. Interviews will involve up to 6 respondents per year, and each interview will take approximately 45 minutes to complete. The awardee interview guide is provided in Attachment E.

Other Stakeholder Interviews: For each prize competition, we will interview a sample of non-applicant stakeholders, including non-government partners such as ASN executives, patient advocates, and other entities involved in funding or implementing KidneyX competitions. These interviews will gather information about stakeholder perspectives on the KidneyX operations and progress toward achieving its objectives. Data gathered during the other stakeholder interviews will inform responses to Evaluation Questions 1, 2, 4, and 5. Up to 6 respondents per year will be interviewed, and each will take approximately 45 minutes to complete. The other stakeholder interview guide is provided in Attachment F.

In all interviews, we will ask for recommendations about how KidneyX could be strengthened, to inform our findings on how HHS may improve the outcomes from KidneyX through changes to the initiative itself.

3. Use of improved information technology and burden reduction

Web-based survey data will be collected electronically through SurveyGizmo software to eliminate the burden of completing and transmitting a paper-based format. The surveys will be compatible with smartphones, tablets, and traditional laptop and desktop computers. The surveys will allow respondents to pause and return to complete them at a later time and will be programmed with skip patterns to reduce respondent burden (i.e., respondents will see only the items they are eligible to respond to).

To maximize response rates, we will use procedures to coordinate closely with HHS and to communicate timely, appropriate reminders emphasizing the importance of applicants’ participation in the surveys. For participants who do not complete the Web-based survey within 1 week of the initial invitation, we will send an e‑mail reminder 1 week after the initial invitation. Similarly, for participants who have not completed the survey within 2 weeks of the initial invitation, we will send an e-mail reminder 2 weeks after the initial invitation. A final e-mail reminder will be sent 3 weeks after the initial invitation to those that have not yet completed the survey. E-mails will come from an e-mail address easily associated with KidneyX, such as KidneyX_Survey@rti.org, so that respondents are more likely to open the e-mail. The surveys will be closed upon reaching 4 weeks after the initial invitation. These procedures will help maximize the response rate.

Telephone interviews will collect qualitative data at lower cost and respondent burden than in-person interviews. To reduce the burden on respondents further, we will review interim survey data and program documents before conducting interviews so that we can focus the discussions.

4. Efforts to identify duplication and use of similar information

In formulating the evaluation plan, the research evaluation team has carefully considered how to minimize burden by supplementing existing administrative and secondary data sources with targeted primary data collection. To this end, the evaluation will incorporate the following approaches to reducing duplication to employ similar information.

1. . Use of Administrative Data

We will use administrative data submitted by applicants before, during, and after each competition to supplant data that would otherwise be obtained through primary data collections. Administrative data collections will include data generated through the mandated use of the Compete.gov platform and any other administrative documents submitted to KidneyX, including informational queries and correspondence that KidneyX may have documented.

2. . Use of Secondary Data

We will use secondary data maintained by U.S. federal agencies and private companies and organizations to measure outcomes from innovation-related activities of awardee and non-awardee applicants from baseline (prior to competition entry) to post-intervention. Exhibit 3 lists data elements that we will obtain from secondary sources. Each wave of secondary data collection will measure change over time in key outcomes that are directly related to the aims of KidneyX.

We will construct a monitoring data system that will identify both participant and non-participant firms and researchers relevant to the kidney technology domain in these databases, and extract records mentioning those entities on an intermittent basis. These databases have all been used in prior academic studies of innovation activity in the US economy (Dalle, den Besten, and Menon, 2017; Corredoira, Goldfarb, and Shi, 2018; Fortunato et al., 2018). Each dataset describes particular aspects of innovation; therefore, we will integrate records from all sources to construct a more complete view of the level of innovation activity over time in this domain.

Exhibit 3. Data Elements and Secondary Sources

Data Sources	Data Elements
U.S. Patent and Trademark Office databases and similar third-party sources such as PatSnap and Google Patents	Patent applications Patent approvals
Web of Science database from Clarivate Analytics	Published academic literature
The Small Business Administration’s TechNet database	SBIR/STTR applications SBIR/STTR awards
Crunchbase Pitchbook	Funding information about startups Significant news about startups
D&B Hoovers	Status as an active company Company revenues
	Company job creation
USASpending.gov	Funding raised from U.S. federal government
Social media sites including AngelList, LinkedIn	Significant news about startup companies

However, several evaluation questions cannot be answered with administrative and secondary data sources, particularly those related to the experiences of applicants and the specific design elements of KidneyX that contributed to or detracted from applicants’ ability to achieve success in developing and introducing innovations in kidney technology. As a result, the primary data collection instruments included in this submission have been developed to obtain complementary and augmented information.

5. Impact on small businesses or other small entities

Applicants to the KidneyX prize competitions are likely to be entrepreneurs who have or aspire to have startup companies or small businesses focused on commercializing the projects proposed in their applications. Therefore, we will use administrative and secondary data collections to minimize the time burden on applicants from whom we plan to collect primary data.

The small firms surveyed and interviewed are likely to be privately-held enterprises that are not subject to extensive disclosure requirements. It is likely that information collected from such applicants via the primary data collection instruments included in this application may be business sensitive and not publicly available. To protect the proprietary interests of those firms, and to provide assurance that survey and interview respondents can share opinions with candor, all information collected under this request will be stored on servers at RTI’s computing facilities and accessible only to members of the RTI project team.

6. Consequences of collecting the information less frequently

Each of the primary data collection instruments described above will obtain information necessary to answer evaluation research questions. Primary data collections have been designed to gather those data that are not obtainable by administrative or secondary means. Efforts also have been made to reduce the burden of primary data collection on participants through revision of the structured client interview based on consultation with HHS.

7. Special circumstances relating to the guidelines of 5 CFR 1320.5

The information collection complies with 5 CFR 1320.5(d)(2). In particular, confidential information collected under this request will be stored on RTI’s IT infrastructure located at RTI’s physical facilities. To ensure compliance with all applicable information security laws, statutes, and agency directives, RTI has voluntarily and enthusiastically implemented the IT security guidelines and principles published by the National Institute for Standards and Technology (NIST). In particular, RTI’s IT facilities and processes comply with applicable elements of Federal Information Processing Standard (FIPS) 199 published by NIST. RTI’s network meets all National Institute of Standards and Technology (NIST) confidentiality, integrity, and availability security standards, allowing RTI to provide the appropriate level of security for the information, including personally identifiable information (PII). Any protected or confidential information collected either intentionally or inadvertently will remain protected under these standards.

8. Comments in response to the Federal Register Notice and efforts to consult outside the agency

On June 6, 2019, a 60-day Federal Register Notice was published at Register Volume 84, Number 109, pg. #26418. No public comments were received.

The evaluation design has been informed by the HHS KidneyX Technical Working Group and non-government partners such as KidneyX partners at ASN. The RTI project team also maintains active membership in the American Evaluation Association, particularly the Research and Technology Development Topical Interest Group, and consults regularly with peers involved in similar evaluations. The design is also informed by discussions and deliberations conducted at the National Academies of Science, Engineering and Medicine workshop on the Role of Inducement Prizes on May 24, 2019.

9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees

We will not provide any remuneration or incentive to respondents.

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy

Before the start of both telephone and site visit interviews, we will remind all respondents that the information gained will be used for evaluation purposes only and will not be attributable to any individual. Responses should not contain private information but will be aggregated to the extent possible so that individual answers will not be identifiable. For each respondent, we will collect name, professional affiliation, and title, but not Social Security numbers, home contact information, and similar information that could identify the respondent directly.

RTI information processing staff are trained on the current requirements of relevant federal legislation and regulation, including the Federal Information Security Modernization Act (FISMA) of 2014 and the 2002 Confidential Information Protection and Statistical Efficiency Act (CIPSEA). Any information collected under this request will be managed and stored under procedures that comply with these and other applicable statutes. RTI’s information systems and processes are audited on a regular basis to ensure conformance with government and industry standards for information protection and security.

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

The Web-based surveys and semistructured telephone interviews will collect information about respondent experiences with KidneyX, perspectives about KidneyX operations and progress toward achieving its objectives, and recommendations for strengthening KidneyX. Neither the Web-based surveys nor the telephone interviews will have questions of a sensitive nature, such as criminal behavior, sexual behavior and attitudes, alcohol or drug use, religious beliefs, and other matters that are commonly considered private. We are not collecting individuals’ race or ethnicity data or diagnoses of medical conditions from interviewees or survey respondents. Respondents may provide professional judgments and opinions, as well as facts, during data collection. Some of the information relates to organizational effectiveness and could therefore be considered sensitive by a portion of respondents; however, disclosure of this information is unlikely to result in liability or competitive disadvantage to the organization.

12. Estimates of annualized burden hours

The following table provides estimates of the average annual burden for collecting the proposed information.

Table 12a. Estimated Annualized Burden Hours to Respondents

Type of Respondent	Form Name	Number of Respondents (Expected)	Number of Responses per Respondent	Average Burden per Response (in min)	Total Burden (in hr)
Prize competition applicants	Applicant Interview Guide	12	1	50/60	10
Prize competition awardees	Awardee Interview Guide	6	1	50/60	5
Prize competition non-awardees	Non-awardee Interview Guide	6	1	50/60	5
Other Stakeholders	Other Stakeholder Interview Guide	6	1	50/60	5
Prize competition applicants	Pre-award Survey Instrument	300	1	30/60	150
Prize competition awardees and non-awardees	Post-award Survey Instrument	300	1	30/60	150
TOTAL					325

Table 12b. Estimated Annualized Cost to the Respondent

Type of Respondent	Total Burden (in hr)	Hourly Wage Rate	Total Respondent Costs
Prize competition applicants	10	$75.08*	$750.80
Prize competition awardees	5	$75.08*	$375.40
Prize competition non-awardees	5	$75.08*	$375.40
Other Stakeholders	5	$75.08*	$375.40
Prize competition applicants	150	$75.08*	$11,262.00
Prize competition awardees and non-awardees	150	$75.08*	$11,262.00
TOTAL	325	$75.08*	$24,401.00

*Average hourly wage rate is estimated from the Annual Average Weekly Wage for Research and Development in Biotechnology (NAICS 541714) in 2017 according to the Bureau of Labor Statistics (Quarterly Census of Employment and Wages, 2017, Annual Averages, All Establishment Sizes) and converting to an hourly wage using the Bureau of Labor Statistics estimate of 42.5 Average Hours per Week Worked by Persons Who Usually Work Full Time in nonagricultural industries (Labor Force Statistics, Current Population Survey, Household Data, Annual Averages, 21. Persons at Work in Nonagricultural Industries by class of worker and usual full- or part-time Status [February 20, 2018]).

13. Estimates of other total annual cost burden to respondents or record keepers

There will be no direct costs to the respondents or record keepers.

14. Annualized cost to the government

The evaluation will use data collected by HHS for programmatic purposes as part of the dataset. Therefore, no additional staff time is required for data collection. The estimated value of the contract to perform this evaluation with options is $727,115 over 5 years. Therefore, the estimated annualized cost to the government is $145,423.

15. Explain the reasons for any program changes or adjustments reported in items 13 or 14 of the OMB form 83-I

This is a new data collection.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

We will incorporate results from the evaluation in the following documents:

• Final base period Evaluation Report to be submitted to HHS in May 2020 providing a summary of evaluation status and preliminary findings.

• Option period 1 Annual Evaluation Report to be submitted to HHS in February 2021 providing a summary of evaluation status and preliminary findings.

• Option period 2 Annual Evaluation Report to be submitted to HHS in February 2022 providing a summary of evaluation status and preliminary findings.

• Option period 3 Annual Evaluation Report to be submitted to HHS in February 2023 providing a summary of evaluation status and preliminary findings.

• Final Evaluation Report to be submitted to HHS in August 2023 providing a summary of evaluation status and preliminary findings.

The research team may also incorporate the aggregate results from the cross-site evaluation into journal articles, scholarly presentations, and briefings for HHS and other government staff and stakeholders related to the outcomes of the KidneyX evaluation.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate

We are requesting no exemption.

18. Explain each exception to the certification statement identified in Item 19, "Certification for Paperwork Reduction Act Submissions," of OMB form 83-I

There are no exceptions to the certification. These activities comply with the requirements in 5 CFR 1320.9.

REFERENCES

Corredoira, RA, Goldfarb, BD, Shi, Y. 2018. Federal funding and the rate and direction of inventive activity. Research Policy, 47(9), 1777-1800.

Dalle, JM, den Besten, M, Menon, C. 2017. Using Crunchbase for Economic and Managerial Research. OECD Science, Technology and Industry Working Paper 2017/08, Paris: Organisation for Economic Cooperation and Development.

Fortunato, S, Bergstron, CT, Borner, K, Evans, JA, Helbing, D, Milojevic, S. Petersen, AM, Radicchi, F, Sinatra, R, Uzzi, B, Vespignani, A, Waltman, L, Wang, D, Barabase, A-L. 2018. Science of science. Science, 359(6379).

Galasso, A, Mitchell, M, Virag, G. 2018. A theory of grand innovation prizes. Research Policy, 47, 343-362.

Murray, F, Stern, S, Campbell G, MacCormack, A. 2012. Grand Innovation Prizes: A theoretical, normative, and empirical evaluation. Research Policy, 41, 1779-1792.

Office of Science & Technology Policy, Executive Office of the President. 2019. Implementation of Federal Prize and Citizen Science Authority: Fiscal Years 2017-18. A Report by the Office of Science and Technology Policy in Response to the Requirements of the America COMPETES Reauthorization Act of 2010 and the Crowdsourcing and Citizen Science Act. June.

RTI International. 2017. Costs and Benefits of Selected Policy Tools to Promote Drug Development. Report prepared for the Office of Science and Data Policy of the U.S. Department of Health and Human Services. October.

Smith Nightingale, D, Fudge, K, Schupmann, W. 2018. Evidence Toolkit: Learning Agendas—A Report of the Evidence Based Policymaking Collaborative Washington, DC: The Urban Institute.

LIST OF ATTACHMENTS – Section A

Note: Attachments are included as separate files as instructed.

1. Pre-award Survey Instrument (Attachment A)

2. Post-award Survey Instrument (Attachment B)

3. Applicant Interview Guide (Attachment C)

4. Non-awardee Interview Guide (Attachment D)

5. Awardee Interview Guide (Attachment E)

6. Other Stakeholder Interview Guide (Attachment F)