Attachment 2 PS-OC Evaluation Plan

to: Jerry Lee, Jim Corrigan , NCI

From: Brian Zuckerman, STPI

CC: Christina Viola Srivastava, Mary Beth Hughes

Subject: Draft Plan for an Outcome Evaluation of the NCI’s Physical Science in Oncology Centers (PS-OC) Program

Date: March 4, 2010

The National Cancer Institute (NCI) of the National Institutes of Health (NIH) recently launched the Physical Sciences in Oncology Centers (PS-OC), a program aimed at bridging the gap between the physical sciences and oncology/cancer biology. The Science and Technology Policy Institute (STPI) has been tasked with proposing an evaluation plan for the Centers that will meet the needs of the NCI Executive Committee at Program Year 5 while informing ongoing program management. In preparation for the design task, STPI engaged in the following activities:

1. Discussions with PS-OC program staff members;

2. Review of available program documents including the pre-announcement workshop materials, RFA, summary statements, applications, and selected progress reports from the newly awarded centers;

3. Interviews with five researchers who are currently conducting cancer research involving the physical sciences in order to gain insight on the current state of the field;

4. Interviews with eight of the 12 PS-OC senior management teams (PI and co-PI);

5. Developing a draft program logic model for the PS-OCs (attached as Appendix A).

Based on what STPI has learned to date about the program, we recommend that NCI’s evaluation needs for PS-OC can best be met through three separate but interdependent evaluation components. Specifically, STPI recommends the following:

1. Prospective data collection on supported activities and key outputs to inform program management as well as future evaluation efforts;

2. Structured evaluation of program design, implementation, and preliminary outcomes by an expert panel at program year 4-5 (prior to concept renewal);

3. Summative evaluation of program outcomes at year 10 or later.

Preliminary design recommendations for each component are described below.

Component 1: Prospective Data Collection on Supported Activities and Key Outputs/Outcomes

The first recommended evaluation component is prospective data collection on activities and key outputs/outcomes. Collecting data on program activities and outputs prospectively serves several purposes: 1) activities and outputs can be monitored by program managers so that changes can be made as needed; and 2) any errors or inadequacies that are detected in the data can be addressed sooner rather than later. These advantages must be balanced against the inefficiency of collecting and analyzing information as it becomes available relative to a single retrospective data collection effort. For this reason, STPI is recommending prospective data collection for the PS-OC activities and outputs/outcomes for which data are most readily available.

Specifically, STPI recommends that information on key variables be extracted from the semi-annual Center progress reports into a more structured format as they become available. Supplementary and correlative information should also be extracted from supplementary sources (e.g. bibliometric databases) at regular intervals. The variables for which STPI recommends that data be collected prospectively are summarized in Table 1, as are the sources from which data should be extracted. These activity and output variables fall into four broad categories (knowledge generation; practice of research and collaboration; training; broader impacts) derived from the program goals and program logic (see Appendix A).

Table 1: Activity/Output/Outcome Variables for which data should be collected prospectively.

Category	Activity/Output/Outcome Variable	Data Source(s)
1. Knowledge Generation	Funded research projects by type and objectives	Progress reports
	Center publications	Progress reports; SPIRES
	Bibliometric data on center publications	Bibliometric databases
	Key discoveries/key findings	Progress reports
	Invention disclosures, patent applications, and patents	Progress reports; IP databases
	New datasets developed	Progress reports
	Other notable research outputs (e.g. datasets, software, protocols, models, etc.)	Progress reports
	Clinical studies (if any) building upon advances	Progress reports; clinicaltrials.gov
2. Practice of research and collaboration	Center activities aimed at promoting collaboration between physical scienctists and cancer researchers	Progress reports
	Participating institutions	Progress reports
	Participating investigators, by institution, research project, and discipline/training	Progress reports
	Formal collaborations between Center investigators	Derive from project lists and publications
	Informal collaborations between Center investigators	Progress reports
	Cross-Center collaborations	Progress reports
	Collaborations between Center investigators and others	Progress reports
	Core resources supported, by type	Progress reports
	Use of core resources	Progress reports
3. Training and Outreach	Students and fellows supported	Progress reports
	Participants in cross-center exchanges	Progress reports
	Course materials/training modules created	Progress reports
	Seminars/workshops conducted by centers	Progress reports
	Other dissemination activities	Progress reports
4. Broader impacts	New solicitations for cancer research proposals using physical sciences approaches, with links to Centers (if any)	NIH administrative databases; Other funder databases
	New applications to NCI for cancer research using physical sciences approaches from Center investigators, others	NIH administrative databases
	New workshops/conferences/ other efforts involving physical sciences and cancer with links to Centers (if any)	NIH administrative databases; Web searches

NCI may wish to employ an independent contractor to coordinate extraction of information from progress reports and collection of supplementary data. This model has been used successfully by various Centers programs at the National Science Foundation (e.g., the Science and Technology Centers).

Component 2: Expert Panel Review at Year 4-5

The NCI Executive Committee requires evaluation prior to concept renewal at program year 5, which typically requires that evaluation efforts begin during program year 4. STPI recommends a methodologically rigorous expert panel review to assess program design, implementation, progress to date by the Centers, and potential for future success. Specifically, STPI recommends that NCI should convene a panel of experts to address the following questions:

1. Are the program’s objectives and priorities being met by progress to date?

2. How should the program’s priorities be changed based upon the program’s experience to date?

3. Did the program design facilitate achieving the objectives?

4. Are Centers the right way to achieve the program’s priorities and objectives?

5. Have there been issues with the implementation of the Centers to date?

6. Is the science being done at the Centers more innovative/multidisciplinary than most science happening elsewhere?

7. Do the relationships between physical scientists and cancer researchers appear to be appropriately collaborative, with both groups making substantive contributions to the research? Are such collaborations occurring at institutions not participating in the PS-OC program?

8. Are the training opportunities available through the Center different from training opportunities available elsewhere?

9. Overall, does the panel believe the PS-OC program is on track to meet its goals

10. Does the panel believe that the potential for future progress merits continued investment by NCI in the PS-OCs?

In order for the expert panel to render credible judgments, the selection of panel members is critical. Expert panel members would ideally be selected and recruited by the NCI Board of Scientific Advisors to assure scientific credibility and neutrality, though program staff could provide suggestions regarding reviewers.

In order to facilitate the panel members becoming sufficiently familiar with the PS-OCs without making excessive demands on their time, STPI recommends that most of the Year 4 Network Meeting should be devoted to the expert panel review. With the expert panel members in attendance, the Program staff and each Center should make a presentation summarizing progress to date. Following the presentations, the expert panel should meet individually with the leadership from each of the Centers to ask questions and interact one-on-one. Prior to the Network Meeting, expert panel members should also have had an opportunity to review all of the activity/output data collected as part of evaluation component 1 and the Center progress reports (as there are 12 Centers, one or two expert panel members should focus on each individual Center).

Component 3: Full Outcome Evaluation at Year 10 or Later

A full Outcome Evaluation would involve more extensive and resource-intensive collection of data on program performance and outcomes, possibly in comparison with similar efforts (although, as STPI’s initial efforts to identify examples of true collaboration between physical scientists and cancer researchers were not successful, it is difficult to imagine an appropriate comparative design). If it is decided that a full Outcome Evaluation of the PS-OCs is feasible and warranted, it should not be initiated until the program is stable in terms of design and sufficiently mature for outcomes to be fully developed. Ten years is frequently used as a rule-of-thumb for maturity of Centers programs, but in reality every program is unique. In the case of the PS-OCs, it seems prudent to wait at least until after the expert panel review and concept renewal to plan an outcome evaluation, as the program design may continue to evolve.

Appendix A: Draft Logic Model for PS-OCs

4