Attachment 4: STAR METRICS Participation Guide

STAR METRICS

Science and Technology in America’s Reinvestment:

Measuring the EffecTs of Research on Innovation, Competitiveness and Science

Research Institution Participation Guide

Last Updated: 10/27/10

ABOUT STAR METRICS
Welcome to the STAR METRICS project. This STAR METRICS project guide will provide an overview of
the STAR METRICS project and outline how research institutions can get involved with this project. This guide
is complemented by further documentation on the STAR METRICS website that outlines the technical
specifications for the project and provides guidance to research institutions wishing to participate in STAR
METRICS.

STAR METRICS Overview
STAR METRICS is a federal and research institution collaboration to create a repository of data and tools that will
be useful to assess the impact of federal R&D investments. The National Institutes of Health (NIH) and the National
Science Foundation (NSF), under the auspices of Office of Science and Technology Policy (OSTP), are leading this project.
This project has been developed after a successful pilot project was conducted with several research institutions in the
Federal Demonstration Partnership (FDP). The STAR METRICS project consists of two implementation phases:
•

Phase I: Developing uniform, auditable and standardized measures of the impact of science spending
(ARRA and non-ARRA) on job creation, using data from research institutions’ existing database records.
No personally identifiable information (PII) is collected in Phase I.
Phase II: Developing measures of the impact of federal science investment on scientific knowledge (using
metrics such as publications and citations), social outcomes (e.g. health outcomes measures and
environmental impact factors), workforce outcomes (e.g. student mobility and employment), and
economic growth (e.g. tracing patents, new company start-ups and other measures). Phase II will require
the collection of PII.

STAR METRICS Project Beginnings

People have asked important questions about the impact of federal investments in science, particularly
with respect to job creation and economic growth. It is important to collect and analyze data so that such
questions can be answered in a credible fashion. There is currently no data infrastructure that systematically
couples science funding with outcomes. There are also no mechanisms that currently exist to engage the
public with scientific funding.
However, there are obvious ways to leverage existing data. Federal agencies already collect data on
federal investments at the individual, award, and institutional levels for the purposes of managing awards.
Research institutions collect data on all individuals working on projects in their financial and human resources
systems. Academic researchers have collected large bodies of data on such scientific and innovation
outcomes as citations, patents, business startups and IPOs. And there is a large body of knowledge about
creating measures of job creation and the associated earnings drawn from the experience of Longitudinal
Employer-Household Dynamics program at the Census Bureau. Finally, there has been substantial investment
in visualization and other tools that convey complex information about science to a lay audience. The
existence of these and other resources motivates the STAR METRICS approach to studying the impact of
science funding and disseminating the information to the public.
The STAR METRICS project is anticipated to be a broad collaboration of federal science and technology
funding agencies with a shared vision of deveoping data infrastructures and products to support evidence2

based analyses of the impact of science and technology investment. The goal of the STAR METRICS project is
to utilize existing administrative data from federal agencies and their grantee institutions, and match them
with existing research databases on economic, scientific and social outcomes.
STAR METRICS is being created in direct response to the White House Office of Management and
Budget (OMB) and Office of Science and Technology Policy (OSTP) request that federal agencies develop
outcome-oriented goals for their science and technology activities. It is also in direct response to the reporting
requirements of the American Recovery and Reinvestment Act (ARRA), and aims to provide American
taxpayers with precise information on the value of their investments.

Getting Started
Partnering with Research institutions
As a voluntary program, the STAR METRICS team relies on research institutions for the project to be
successful. Therefore, the STAR METRICS approach was designed to be as easy as possible for participating
institutions. The STAR METRICS team works with research institutions during each step to provide
information, guidance and support. The process for Phase I and development of Phase II is outlined in this
section. The diagram below outlines the steps for Phase I.

Step 1: Initial Meeting with Research Institution
The first step in the STAR METRICS data gathering process is to schedule an introductory meeting,
usually in the form of a webinar/videoconference, to introduce the research institution to the basic approach
of STAR METRICS and to the participation requirements. In this first meeting, which lasts about one hour, we
review the details of the project and discuss the information that will be collected, the process for collection
and the reports that will be generated from the data.

3

Research institution representatives in the meeting should include a primary IT person, primary
contract finance person and a primary contact HR person. It is often useful to have someone from the Office
of the Vice President for Research in attendance as well.
Step 2: Decision to Participate
After the initial meeting with the STAR METRICS team, a Memorandum of Understanding (MOUParticipation Agreement) needs to be signed between the research institution and the National Institutes of
Health (NIH) – the data custodians for the STAR METRICS project. A proposed sample of the MOU can be
downloaded on the left hand side of the Participate page on the STAR METRICS website. Note that data
cannot be accepted until the MOU has been signed and sent to the NIH.
Additional technical meetings can be set up to provide additional information and support on
collecting the requested data elements before the MOU is received.
Step 3: Research Institution Sends Sample Data
As part of phase I of the STAR METRICS project, research institutions are asked to provide information
drawn from their administrative systems housing data on human resources, indirect costs, vendors, and subawards. The table below outlines the data elements we are requesting in Phase I.
The STAR METRICS team works with the research institution to send baseline quarterly data. Depending on
the research institution, data can be accurately pulled from as far back as seven years or as few as 2 quarters. ,
Multiple years of data permit long trend lines to be generated in the analytical reports.
The Data Dictionary and Technical Specification Guide provides further explanation of the data
elements, the requested formats, and details on how to submit data. Typically, sample data represents the
last quarter’s worth of research institution data.
Table 1- Data Elements Requested in Phase I
Data Element

Definition

Unique Award Number

The unique federal number as defined by concatenating the 6 position CFDA code
with the assigned award id from the awarding Federal Agency (such as the federal
grant number, federal contract number or the federal loan number) with a space in
between the two numbers. Format example: 47.074 AGS-0120950 where "47.074"
is the CFDA and "AGS-0120950" is the Federal Award ID.

Recipient Account Number

Research institution's internal number for the award.

Overhead Charged

Actual Overhead dollars charged to the award in the specified period.

Unique Award Number

The unique federal number as defined by concatenating the 6 position CFDA code
with the assigned award id from the awarding Federal Agency (such as the federal
grant number, federal contract number or the federal loan number) with a space in
between the two numbers. Format example: 47.074 AGS-0120950 where "47.074"
is the CFDA and "AGS-0120950" is the Federal Award ID.

Recipient Account Number

Research institution's internal number for the award.

De-identified Employee ID Number

Unique Employee ID (not Social Security number) of grant funded personnel

Occupational Classification

Occupational classification / Job description of the funded personnel (ex. Faculty,
Undergrad Student, Grad Student, Post Graduate Researcher, Research Support,
Technician/Staff Scientist, Research Analyst/Coordinator, Clinicians)

FTE Status

Designation of the status of the funded personnel (full time = 1.0, half time = .5)

Information on Awards

Information on
Individuals

4

Proportion of Earnings Allocated to award

Information on Indirect
Costs

Payments to Vendors

Sub-Awards

Overhead Salary +Fringe to Total Ratio

Calculated portion of earnings charged by funded personnel to the award in the
specified period.
Proportion of overhead dollars that goes to pay salaries and fringe benefits. The
calculation used is (Salary $ + Fringe $) ÷ Total Overhead Dollars. For more
information, see the Indirect Cost Proposal Example.

Unique Award Number

The unique federal number as defined by concatenating the 6 position CFDA code
with the assigned award id from the awarding Federal Agency (such as the federal
grant number, federal contract number or the federal loan number) with a space in
between the two numbers. Format example: 47.074 AGS-0120950 where "47.074"
is the CFDA and "AGS-0120950" is the Federal Award ID.

Recipient Account Number

Research institution's internal number for the award.

Vendor DUNS Number

The Vendor's 9 digit DUNS number. If DUNS is unavailable then substitute zip code
with "Z" prefix so as to distinguish it from the DUNs number.

Vendor Payment Amount

The funds charged to the award by the vendor in the specified period.

Unique Award Number

The unique federal number as defined by concatenating the 6 position CFDA code
with the assigned award id from the awarding Federal Agency (such as the federal
grant number, federal contract number or the federal loan number) with a space in
between the two numbers. Format example: 47.074 AGS-0120950 where "47.074"
is the CFDA and "AGS-0120950" is the Federal Award ID.

Recipient Account Number

Research institution's internal number for the award.

Sub-Award Recipient DUNS Number

The sub-recipient organization’s 9- digit DUNS number. If DUNS is unavailable then
substitute zip code with "Z" prefix so as to distinguish it from the DUNS number.

Sub-Award Payment Amount

The charged to the award by the sub-awardee in the specified period.

Step 4: STAR METRICS Team Provides Feedback and Reports
Once initial sample data is received, the STAR METRICS team will analyze the data and provide reports
on job creation, retention, and changes from the previous quarter. In addition, the team assesses the general
structure of the files and the specific data elements and provides recommendations on any changes needed to
match the data formatting specifications. Appendix A provides more details on the job calculations the STAR
METRICS team uses to generate reports.
Step 5: Research Institution Sends Recent and Historical Data
After the research institution receives confirmation that their sample submissions are in the correct
format and reports can be generated from their data, research institutions are asked to send data as far back
as is feasible without any additional burden. In order for the STAR METRICS team to provide a detailed analysis
of historical data, it is preferred that historical data is submitted at the transactional level.
Step 6: STAR METRICS Team Creates Quarterly Reports and Maps
The final step is to process the information that has been loaded into the STAR METRICS database,
produce and review the reports and return them to the research institution. The STAR METRICS team will run
the processes that will create both quarterly reports and maps. These reports include one set of tables which
report direct jobs and positions supported by federal science funding. Additional reports provide estimated
jobs generated from funds sent to sub-awardees, purchases from vendors, and research support provided by
5

overhead expenditures. These reports and maps will be delivered to the research institution for their review
and comment.
Phase II
The STAR METRICS team will be engaging in a consultation process with all stakeholders and Phase Iparticipating institutions during the fourth quarter of 2010 to begin the formal first step of conversations on
for Phase II. All research institutions in Phase I will have the opportunity to provide input into the structure
and design of Phase II.

Questions?
Please look at the FAQ’s, Data Dictionary, Technical Specifications Guide, and Data Samples on the
STAR METRICS website for further information.

6

Appendix A: Employment Calculations
STAR METRICS can be used to help support the employment calculations associated with ARRA spending. The
calculations are consistent with ARRA reporting requirements in Section 1512 of Public Law 111-5, which
requires:
A. “A narrative description of the employment impact of the Recovery Act funded work. This narrative
is for each calendar quarter and, at a minimum, will address the impact on the recipient’s or federal
contractor’s workforce (for grants and loans, recipients shall also include the impact on the
workforces of sub recipients and vendors).
B. Provide a brief narrative description of the types of jobs created and jobs retained in the United
States and outlying areas. This description may rely on job titles, broader labor categories, or the
recipient’s existing practice for describing jobs as long as the terms used are widely understood and
describe the general nature of the work.”
Source: Recipient Reporting Data Model - for quarter ending 12/31/2009 (www.recovery.gov)
Direct Jobs Calculated from Individuals Employed
Information Requested

Required for element calculation

De-identified Employee ID Number
Unique Award Number
Recipient Account Number
Occupational Classification
Proportion of Earnings Allocated to Award
FTE status
Calculation for Part A:

Parts A and B below
Verification and data quality check
Verification and data quality check
Part B below
Parts A and B below
Parts A and B below

Calculation for Part B is the same as Part A

This approach is identical to that discussed in Peter Orszag’s memo of Dec 18 2009.
Estimates of Employment Resulting from Purchases made from Vendors
Information Requested
Unique Award Number
Recipient Account Number
Vendor DUNS Number
Vendor Payment Amount

Required for element calculation
Validation
Validation
Calculation
Calculation

The vendor DUNS Number will be used to derive an industry code and geographic location 1. Economic
Census data are used to generate estimates of the amount of salaries paid as a result of the grant

1

The vendor zip code can be used (with the prefix “Z”) if the DUNS number is unavailable.

7

revenues earned by the vendor 2. This is in turn divided by the average earnings of workers in that
industry and geographic location to generate employment estimates derived from Economic Census
data. As the project expands, these calculations can be directly generated from the administrative
records of the respondents themselves.
Estimates of Employment Resulting from Sub Awards
Information Requested
Unique Award Number
Recipient Account Number
Sub-Award Recipient DUNS Number

Required for element calculation
Validation
Validation
Calculation
Calculation

Sub-Award Payment Amount

The vendor DUNS Number will be used to derive an industry code and geographic location 3. Economic
Census data are used to generate estimates of the amount of salaries paid as a result of the grant
revenues earned by the vendor 4. This is in turn divided by the average earnings of workers in that
industry and geographic location to generate employment estimates derived from Economic Census
data. As the project expands, these calculations can be directly generated from the administrative
records of the respondents themselves.
Estimates of Employment Resulting from Overhead
Estimates will be calculated using information from each research institution’s Indirect Cost Proposal.
Please see the Indirect Cost Proposal Example on the STAR METRICS website for further information.

2

http://www.census.gov/econ/census07/
The vendor zip code can be used (with a prefix of “Z”) if the DUNS number is unavailable.
4
http://www.census.gov/econ/census07/w
3

8