Supporting Statement A For:
Evaluation of the Science Education Partnership Award (SEPA) Program
November 17, 2015
Tony Beck, Ph.D.
Office of Science Education/SEPA
Office of Research Infrastructure Programs (ORIP)
Division of Program Coordination, Planning, and Strategic Initiatives (DPCPSI)
Office of the Director
National Institutes of Health
6701 Democracy Blvd, Room 206
Bethesda, MD 20892
Telephone: 301-435-0805
E-mail: beckl@mail.nih.gov
Supporting Statement A. Justification
Introduction
Overview of the SEPA Program
A.1. Circumstances Making the Collection of Information Necessary
A.2. Purpose and Use of the Information Collection
A.2.1. Research Questions
A.2.2. Overview of Data Collection Modes
A.3. Use of Information Technology and Burden Reduction
A.4. Efforts to Identify Duplication and Use of Similar Information
A.5. Impact on Small Businesses and Other Small Entities
A.6. Consequences of Collecting the Information Less Frequently
A.7. Special Circumstances Relating to the Guidelines in 5 CFR 1320.5
A.8. Comments in Response to Federal Register Notice and Efforts to Consult Outside Agency
A.9. Explanation of Any Payment of Gift to Respondents
A.10. Assurance of Confidentiality Provided to Respondents
A.11. Justification of Sensitive Questions
A.12. Estimates of Hour Burden Including Annualized Hourly Costs
A.13. Estimate of Other Total Annual Cost Burden to Respondents or Record Keepers
A.14. Annualized Cost to the Federal Government
A.15. Explanation for Program Changes or Adjustments
A.16. Plans for Tabulation and Publication and Project Time Schedule
A.16.1. Analysis of the Study Data
A.16.2. Products of the Study
A.16.3. Methods of Dissemination
A.16.4. Use of Results
A.16.5. Project Time Schedule
A.17. Reason(s) Display of OMB Expiration Date is Inappropriate
A.18. Exceptions to Certification for Paperwork Reduction Act Submissions
Exhibit A.12-1. Estimates of Annualized Burden Hours
Exhibit A.12-2. Estimates of Annualized Cost to Respondents
Exhibit A.14-1. Annual Cost to the Federal Government
Exhibit A.16-1. Project Time Table
Attachments
1. PI Web Survey
2. Partner Web Survey
3. PI Telephone Interview Protocol
4. PI Site Visit Interview Protocol
5. Partner Telephone Interview Protocol
6. Partner Site Visit Interview Protocol
7. Key Staff Interview Protocol
8. Key Evaluator Interview Protocol
9. E-Mail Announcement to PIs
10. E-Mail Announcement to Partners
11. E-Mail Announcement to Site Visit Participants
12. E-Mail Invitation Letter - PI Web Survey
13. E-Mail Invitation Letter - Partner Survey
14. E-Mail Reminder - PI Web Survey
15. E-Mail Reminder - Partner Web Survey
16. Telephone Reminder - PI Web Survey
17. Telephone Reminder - Partner Web Survey
18. Site Visit Interview Consent Form
19. IRB Approval Letter
20. E-Mail Invitation Letter - PI Telephone Interview
21. E-Mail Invitation Letter - Partner Telephone Interview
22. E-Mail Reminder - PI Interview
23. E-Mail Reminder - Partner Interview
24. Telephone Reminder - PI Interview
25. Telephone Reminder - Partner Interview
26. Telephone Script to Schedule the Telephone Interview
27. Telephone Script to Schedule the Site Visit Interview
28. NIH Privacy Act Systems of Record
29. E-Mail Follow-Up Reminder to Schedule Telephone Interview
30. E-Mail Confirmation for Telephone Interview
Request for OMB Clearance
Evaluation of the Science Education Partnership Award (SEPA) Program
Supporting Statement for Paperwork Reduction Act Submission
This is a request for clearance of a new collection titled “Evaluation of the Science Education Partnership Award (SEPA) Program.” The Science Education Partnership Award Program of the National Institutes of Health (NIH) resides within the Office of Research Infrastructure Program of the Division of Program Coordination, Planning, and Strategic Initiatives. The program provides 5-year grants for PK-12 educational projects, science centers, and museum exhibits to increase students’ interest in pursuing science-related careers, deliver topical and interactive information about NIH-funded medical research, and cultivate an understanding about healthy living habits among the general public. SEPA is undertaking an evaluation to examine the extent to which SEPA grants awarded from 2004 through 2014 have met goals related to project structure, partnership formation, and evaluation quality. The evaluation will utilize archival grant project data (e.g., SEPA solicitations, project proposals, annual and final reports, and summative evaluations). The evaluation will also collect new data to 1) determine the extent to which the SEPA portfolio is aligned with the program’s overall goals; (2) assess how the SEPA Program has contributed to the creation and/or enrichment of beneficial productive partnerships; and (3) determine the extent to which the SEPA Program is generating a rigorous evidence-based system that provides high-quality evaluations to inform the knowledge base. The goal of this process evaluation is to provide SEPA, program staff, NIH, and other interested stakeholders with information about how the program is operating, the extent to which projects address the program’s multiple goals, and the extent to which project-level evaluations are informing and enhancing the quality of work in the field.
The NIH SEPA Program provides 5-year grants for PK-12 educational projects, science centers, and museum exhibits to increase students’ interest in pursuing science-related careers, deliver topical and interactive information about NIH-funded medical research, and cultivate an understanding about healthy living habits among the general public. The program was founded in 1991 in response to the six National Education Goals set forth by President George H.W. Bush. These goals challenged educators to improve the quality of education in the United States based on the philosophy that every child can learn and that learning is a lifelong process. The SEPA Program is currently NIH’s premier investment in PK-12 education and provides a pathway for translating medical and biomedical advancements into practice.
The overarching goals of the SEPA Program are as follows:
Support development of innovative and inquiry-based science education curricula and materials for PK-12 students from underserved, rural, and disadvantaged communities to encourage these students to become interested in science and pursue careers in basic and medical research;
Educate the PK-12 community and the general public on topical and health-related issues such as stem cells and regenerative medicine, ethics of medical research and vaccine development, emerging infectious diseases, and the clinical trials process and patient safeguards; and
Provide public education and community outreach to students, teachers, and families about NIH-funded research and the clinical trials process.
Key features of the SEPA Program include addressing issues in the medical and biomedical fields and encouraging traditionally underserved populations to participate in the sciences. Although many organizations have answered the call to improve science education, understanding, and engagement, SEPA stands apart due to its unique focus on medicine and biomedicine. Within these fields, SEPA encourages and supports partnerships among scientists, clinicians, and educators to increase and diversify the population in the STEM and health career pipelines. Indeed, this emphasis on partnerships is a key feature of the program. In addition, the program aims to improve PK-12 STEM education and public understanding of, and support for, NIH-funded medical research.
SEPA draws on research and opportunities across NIH to achieve its goals. Three programs in particular have shared a unique relationship with SEPA grants: the Institutional Development Award (IDeA), Research Center in Minority Institutions (RCMI), and the Clinical and Translational Science Award (CTSA). The IDeA program promotes health-related research and enhances the competitiveness of investigators at institutions located in states in which the aggregate success rate for grant applications to NIH has historically been low. The RCMI program enhances the research capacity and infrastructure at minority colleges and universities that offer doctorates in health. The CTSA program is designed to strengthen and support the entire spectrum of translational research from scientific discovery to improved patient care.
The Office of Research Infrastructure Programs (ORIP), within the Division of Program Coordination, Planning, and Strategic Initiatives (DPCPSI) and the Office of the Director of the National Institutes of Health, was formed in 2011 under the Consolidated Appropriations Act of 2012 (H.R. 2055; Report 112-74). The ORIP is authorized to provide direct funds to various research and evaluation efforts in order to support its mission of supporting research and other activities to advance medical research in all disease areas and across the basic, translational, and clinical research continuum. ORIP has requested that SEPA conduct an evaluation of its program and the portfolio of projects and initiatives funded by the program. Westat will conduct the evaluation via a review of grant documents and through administration of two surveys and collection of interview data from a subset of projects. Findings will be used to assess the annual NIH investment in the SEPA program and provide NIH with evidence and data that will help it make effective decisions regarding program improvements and funding activities.
To determine whether and how best to evaluate the program, in 2009, Westat conducted a SEPA feasibility study. That study showed that it would be feasible to evaluate the SEPA Program and that the findings of a SEPA evaluation would be extremely valuable for both the program and the field. The current evaluation draws heavily on the feasibility study, which created a solid foundation for it. The evaluation proposes to use a range of approaches—document review, case studies, review of the literature, and discussions with experts—to examine the status of the program and to provide a multi-faceted examination of program implementation in light of the design, strategies, and key features of SEPA projects funded between 2004 and 2014.
The primary purpose of this data collection is to provide SEPA with information about the extent to which SEPA grants awarded since 2004 have met program goals related to project structure, partnership formation, and evaluation quality. SEPA will use the results to better understand its portfolio of projects, particularly the extent to which SEPA grantees are fulfilling the mission and goals of the program, how well the program has fulfilled its goal of enhancing and creating partnerships among SEPA projects, and whether the program has encouraged the use of more rigorous evidence-based evaluations. Information collected will be used to provide analytical and policy support to SEPA, and assist NIH in making decisions about future SEPA programming, funding, and other initiatives designed to improve science education partnerships. Evaluation data may also provide information for NIH’s Government Performance and Results Act (GPRA) report.
The evaluation addresses the following questions:
Is the SEPA portfolio aligned with the program’s overall goals?
Specifically, questions regarding the portfolio directly address the extent to which the grants awarded under SEPA are aligned with its intended designs and goals.
Has the SEPA Program contributed to the creation and/or enrichment of beneficial and productive partnerships?
The examination of the SEPA-facilitated partnerships addresses the extent to which partnerships and collaborations are functioning as key resources that advance the field of STEM education and are integrating the dissemination of findings to the STEM education and biomedical research fields.
Is the SEPA Program generating a rigorous evidence-based system that provides high-quality evaluations to inform the knowledge base?
The examination of rigor of individual project evaluations will assess the extent to which the SEPA Program evaluations are generating evidence that meet the standards of scientific rigor.
Westat will collect information about program implementation associated with SEPA awards using a range of data collection methods, including two surveys and four interviews conducted during site visits, which are subject to this clearance request. The application will focus on these components, but the full range of data collection activities are briefly described below:
Review of existing documents (e.g., SEPA solicitations, proposals, annual and final reports, and summative evaluations) to provide a description of the SEPA projects funded during the past 10 years in terms of their purpose, activities, target population, delivery mechanisms, and outcomes;
Semi-structured interviews with NIH leadership and staff involved in SEPA to obtain their perspectives on how the purpose and implementation of SEPA projects have evolved over time;
Principal investigator (PI) survey to obtain information about the contributions of individual partners and the extent to which partnerships are being/have been sustained, issues associated with the project’s evaluation component, and the range of outcomes associated with the project;
Partner survey to obtain partners’ perspectives on their SEPA experiences and contributions;
Case studies to engage a broader population of stakeholders and beneficiaries in deeper discussions about the extent to which projects are aligned with the goals of the SEPA Program, the outcomes associated with SEPA projects, the types of contributions made by partners, issues impinging on the quality of the evaluation, and the extent to which projects and their partnerships are being sustained; and
Rigor review of eligible project evaluations to assess the extent to which the SEPA portfolio has sufficiently high quality evaluation designs capable of generating evidence for the field.
The purpose of the PI and partner surveys is to allow for a broad-based assessment of the implementation and contributions of the projects funded during the past 10 years (2004 through 2014). The surveys will also allow for an examination of the range of factors that facilitated and hindered the efforts of the overall project and individual partner organizations. The online surveys will be administered to PIs and their formally named partners for 156 projects funded during the last 10 years.
The development of these surveys was informed by existing construct maps, as well as feedback from the SEPA management team and the members of the SEPA Advisory Committee. The PI survey collects information on (1) the characteristics of the PI, (2) the experiences of PIs, (3) the contributions of individual partners and the extent to which partnerships are being sustained, (4) issues associated with the project’s evaluation component, and (5) the range of outcomes associated with the project. The partner survey collects information on (1) the characteristics of the individual responding to the survey, (2) the characteristics of the partner organizations, (3) the roles and contributions of partner organizations, (4) the management of partnerships; (5) benefits of partnerships, and (6) the extent to which partnerships are being sustained following the conclusion of the SEPA grant. Both surveys will include a combination of close-ended and open-ended items.
Screen shots of the online PI and partner survey items are provided in Attachments 1-2.
We plan to conduct case studies of select SEPA projects to obtain more detailed information on partnerships, the rigor of study designs, and project outcomes. Information for case studies will be provided primarily through telephone interviews with PIs and partners as well as through three site visits. Descriptions of the telephone interviews and site visits follow.
While the surveys will obtain broad-based data from all SEPA projects funded in the past 10 years, the telephone interviews will be used to obtain a more detailed understanding of the implementation and contributions of approximately 40 of the 156 projects funded in the last 10 years. The scope and purpose of the interviews have been discussed with both the SEPA management team and members of the SEPA Advisory Committee. Of particular interest will be examining how partners were selected, the role(s) of project partners, partner contributions, and the range of factors that facilitated or hindered partners’ work on their SEPA projects. The PI interviews will also delve more deeply into projects’ experiences in conducting their evidence-based evaluation and what they learned from participating in the evaluation. The partner interviews will probe for their views about various aspects of the project partnership and its impacts, including information about their implementation experiences, contributions to the project, and whether and how partnerships have been sustained. Although telephone interviews will cover the same topics across projects, some items will be tailored to specific respondents and/or types of grant (e.g., curriculum projects versus projects that are developing museum exhibits) to delve more deeply into areas that emerged from the document review and/or PI/partner surveys.
Copies of the PI and Partner Interview Protocols are provided in Attachments 3 and 5.
For six to nine SEPA projects, we will conduct three site visits (in lieu of the telephone interviews) and will schedule time to spend with two to three projects per visit. Site visits will allow us to conduct observations of specific activities and conduct face-to-face interviews with PIs, partners, evaluators, and other project stakeholders. The purpose of these site visits will be to engage a broader group of stakeholders in deeper discussions about the extent to which projects are aligned with the goals of the SEPA Program, the outcomes associated with SEPA projects, the types of contributions made by partners, issues impinging on the rigor of the evaluation, and the extent to which projects and their partnerships are being sustained.
Copies of the site visit interview protocols are provided in Attachments 4, 6, 7, and 8.
The evaluation study will utilize three different modes of data collection: web-based surveys, telephone interviews, and site visit interviews. The following sections make reference to the following three stakeholder groups from which data will be collected:
Principal Investigators (PIs): The SEPA researchers comprise the PIs of all SEPA projects funded from 2004 through 2014. Based on our current document review of SEPA project materials, we have determined that there have been 156 SEPA projects funded between 2004 and 2014. Thus, the number of PIs that will be invited to participate for the PI survey will be 156. Additionally, a subsample of 40 project PIs will be invited to participate in individual telephone/in-person interviews to gain more in-depth information about their SEPA projects as part of the plan to conduct case studies.
SEPA Project Partners: All partners of the 156 SEPA projects from 2004 through 2014 will be invited to participate in the partner survey. It is expected that SEPA projects will have on average two partners associated with the project for a total of approximately 312 prospective partners eligible to participate in the partner survey. Additionally, partners of the 40 projects selected for case studies will be invited to participate in the partner interviews. It is expect that SEPA projects will have an average of two partners per project for a total of approximately 80 prospective partners eligible for the partner interview.
Key Project Staff: We will visit two to three SEPA projects during each of the three planned site visits. The site visits will include interviews with the PI and partners but will also include interviews with other key project staff such as the evaluator and other individuals identified as key by the project PI. The number of key project staff at each site who will participate in the interviews is expected to be between 5 and 10 individuals.
We have developed online versions for the PI and partner survey (Attachments 1-2). Once OMB clearance has been obtained, we will launch the PI and partner surveys by sending respondents an e-mail with information about the evaluation (Attachments 9-10) and a generic letter from NIH (to be drafted by Westat and revised by NIH as necessary) requesting participation in the survey, instructions about how to log into the survey, and respondent-specific passwords for use to access their surveys (Attachments 12-13). Once a project’s PI survey has been completed, we will administer the partner survey on a rolling basis to each of the identified partners. Staggering the administration of the partner survey until the PI survey is complete will enable us to use the PI survey responses to confirm the correct names and e-mail addresses for individuals from partner organizations that will be used to send partners the survey.
Since we need detailed information on all SEPA projects funded during the period covered by the process evaluation, PIs who have led multiple SEPA projects in the past 10 years will be asked to complete a separate survey for each project. Data from the document review and PI survey will provide information about which partners were associated with each SEPA project. In cases where projects use the same partners, we will discuss with SEPA staff steps to minimize respondent burden.
We are proposing that PIs and project partners be given 30 days to complete their surveys, although significant time will likely be needed for follow-up with nonrespondents. Westat will monitor the collection status of individual respondents and send generic e-mail reminders to remind nonrespondents of the need to complete the survey in a timely manner (Attachments 14-15). At the end of the collection cycle, Westat will generate thank-you e-mails to acknowledge that respondents have successfully submitted a survey.
Throughout the data collection period, survey respondents will have direct phone and e-mail access to Westat staff who are familiar with the surveys to ask questions or request support. The surveys will be designed to comply with the requirements outlined in the Section 508 Amendment to the Rehabilitation Act of 1973 (e.g., each survey and help document will be tested to ensure that the items are compatible with text reading browsers). The web surveys will be designed to allow users to save partially completed surveys so that respondents can continue entering data where they left off, without loss of already submitted information. We will safeguard data that have been entered using a schedule that combines incremental daily and full weekly backups. To assure data are kept confidential, Westat will maintain a unique login and password for each user. Once data are submitted, they will be stored in a database to facilitate retrieval by either standard report formats or ad hoc database queries.
Based on our experience, we anticipate three potential challenges with the PI and partner surveys. The first will be locating respondents for some projects that ended 5 or more years ago. We are anticipating the contact information for at least some PIs and partners will be outdated and that Internet searches and conversations with SEPA staff will be required to locate those individuals. Second, given the amount of time that has passed, respondents for some older SEPA projects may have difficulty recalling some information. Finally, for the partner surveys, additional follow-up may be required to identify the most appropriate respondents and/or to encourage their participation.
Westat will use SurveyBuilder, proprietary Westat software, which supports creating customized surveys. The web survey data will be downloaded directly from the platform in Excel; the database will be titled “SEPA Survey Database.” The web survey database will allow monitoring of the data and creating reports on completed responses. The software package SPPS will be used for descriptive analyses, while the software programs UCINET and NodeXL will be used for social networking analyses.
We will conduct semistructured interviews with the PIs (Attachment 3) and partners (Attachment 5) of 40 of the 156 projects funded over the last 10 years. Projects will be selected using findings from the document review and the surveys, and on recommendations from the SEPA program officer. Our goal will be to identify 20 projects that established successful partnerships and 20 projects where partnerships were less successful. As specified in the solicitation, the sample will also include partnerships among SEPA projects, among SEPA projects and CTSAs, IDeA state-based IDeA Networks for Biomedical Research Excellence (INBRE) or Centers of Biomedical Research (COBRE) projects, and with STEM education entities within and outside of NIH. The final selection of case study sites will be made in collaboration with the SEPA program officer.
A Telephone Interview Guide, labeled with the respondent’s ID number, will be used to conduct each interview. While the interviewers may record notes and comments on the Telephone Interview Guide itself, the interview data will be collected via digital audio tape. Individuals participating in the telephone interview will be asked to provide verbal consent to have the interview recorded. Each interview will be recorded on a digital file labeled with the respondent’s ID number; each audio file will have a standardized unique file name: “SEPA –I##” where “I” stands for Interview and “##” is the unique ID number for the interview respondent (e.g., SEPA-I99). Interview transcripts will be stored as Word documents and will use the same naming convention used for the telephone interview audio file; i.e., “SEPA-I##.” The file name extension will distinguish between the audio file (.wav) and the Word document (.docx). Management and analysis of the interview data, including monitoring activity and response rates, will be performed utilizing Excel.
We will conduct site visits (in lieu of the telephone interviews) for six to nine SEPA projects to allow for observations of specific activities and to conduct face-to-face interviews with PIs, partners, evaluators, and other project stakeholders. SEPA sites will be selected based on similar section criteria as the PI/partner interviews (i.e., balance of projects with successful and less successful partnerships; projects with partnerships with other SEPA projects; CTSA, IDeA, IMBRE, or COBRE projects; organizations within and outside of NIH; and recommendations from the SEPA Program Officer).
Site visits will begin in April 2016 once a preliminary analysis of key elements from the PI and partner surveys has been conducted (to inform the selection of the projects that will be visited). Prior to each site visit, a member of the SEPA evaluation team will contact the project PI to request any additional background materials that can be shared with members of the site visit team. The purpose will be to enhance site visit team members’ understanding of a project’s characteristics, partners’ contributions, approach, anticipated outcomes, and unique challenges prior to the actual visit.
The agenda for site visits, list of staff who will be interviewed, and topics that will be addressed during site visits will be developed in consultation with the SEPA Program Officer and members of the SEPA Advisory Committee. During the site visits, it will be important to adequately capture relevant differences in perspectives by selecting persons in pivotal roles to be interviewees. However, the exact number and composition of the interviewees may vary somewhat from project to project. At a minimum, the list of interviewees will likely include (1) the project PI, (2) representatives from each partner organization, (3) other key project staff in the lead organization, and (4) the project evaluator. Individual interviews will be conducted with the PI, project partner(s), and the evaluator. The interviews with these respective cohorts will be conducted by Westat researchers using the PI Site Visit Interview Protocol (Attachment 4), Partner Site Visit Interview Protocol (Attachment 6), and the Evaluator Interview Protocol (Attachment 8). We will conduct a focus group interview with project staff identified as key by the PI.1 2 The focus groups will be conducted by Westat researchers using the Key Staff Interview Guide (Attachment 7). As with the telephone interviews, the site visit sessions will be recorded and data will be captured via digital audio tape. Individuals who choose to participate in interviews conducted during site visits will sign an informed consent form prior to participation (Attachment 18). Each site visit will have a unique digital file, titled “SEPA-SV01.wav” where “SV” stands for site visit. The site visit audio files will be stored following the same process as for the telephone interviews; the files will be stored as Word documents and follow the same naming convention. The transcribed Word documents will be titled “SEPA-SV01.docx.” Also, as with the telephone interview data, management and analysis of the site visit data will be performed using Excel.
SEPA submitted a Privacy Impact Assessment (PIA) describing the system that will be used to store, process, and transmit information related to the study including surveys and interviews for which clearance is requested.
This information is a one-time collection for each of the respondent groups. No respondents will be asked the same questions twice at any point in the study, and the study will not be repeated at any point in the future. However, an estimated 40 PIs and 80 project partners will be asked to respond to a web survey as well as a telephone interview or site visit interview. PIs and project partners from SEPA projects participating in the site visits will be interviewed as part of the site visit and will not be asked to participate in the PI and partner telephone interviews. The goals of each data collection methodology employed are significantly different, and no piece of information will be collected twice. Moreover, the contractor and SEPA are maximizing use of archival and administrative records to avoid burdening study participants and keeping instruments as short as possible.
No small businesses or small entities will be involved in this study.
If the data for this evaluation are not collected, SEPA will be unable to assess the extent to which SEPA grants awarded since 2004 have met program goals related to project structure, partnership formation, and evaluation quality. SEPA will not be able to collect information that could potentially be used to provide analytical and policy support to SEPA, nor could it assist NIH in making decisions about current SEPA programming, future funding, and other initiatives to improve science education partnerships.
The study is consistent with the information collection guidelines in 5 CFR 1320.5.
The 60-Day Federal Register notice soliciting comments on this study prior to initial submission to OMB was published on June 3, 2015, Vol. 80, P. 31610-31611. We did not receive any comments to the notice.
All protocols underwent multiple rounds of review by the SEPA management team as well as review by members of the SEPA Advisory Committee. Advisory Committee members are experts in the field of survey design, statistical analyses, formal and informal STEM educational programs, and formal and informal educational evaluation. Their names and contact information are provided in Supporting Statement B, Section B.5.
Westat plans to conduct pilot tests of the web surveys and interview protocols with participants at the NIH Sci Ed meeting that will take place from May 20 to 23, 2015. Pilot test informants will be a convenience sample composed of NIH Sci Ed meeting attendees (comprising SEPA project PIs and partners). Pilot test informants will be asked to assist in reviewing the protocols in the following areas: the clarity of language used (e.g., eliminating double-barreled questions), consistency across protocols, and reduction of response burden. A full report of the pilot test results will be written up and be used to revise the web surveys and interview protocols accordingly.
This information collection does not involve payment or gifts to respondents.
Prospective
participants (PIs, project partners, evaluators, and key project
staff) will receive
e-mail notifications, invitations, and
reminders announcing the evaluation, explaining its purpose,
detailing the evaluation topics, and describing both the voluntary
nature of participation and assurance that the data will be kept
private to the extent required by law. The following steps outline
the procedures for contacting prospective participants to ensure
compliance and maximize response rates.
All proposed e-mails will include language stating that all collected information will be kept private and not disclosed in any identifiable form to anyone but the researchers conducting the study, except as otherwise required by law. Individuals who choose to participate in the web-based survey and the telephone interview will be providing implicit consent by their participation. Individuals participating in the telephone interview will be asked to provide verbal consent to have the interview audio recorded (Attachments 3 and 5). Individuals who choose to participate in the site visits will sign an informed consent form prior to participation (Attachment 18). This information collection is covered by the NIH Privacy Act Systems of Record 09-25-0156, “Records of Participants in Programs and Respondents in Surveys Used to Evaluate Programs of the Public Health Service, HHS/PHS/NIH/OD” published in the Federal Register on 9/26/2002, vol. 67, p. 60743. All study personnel will adhere to the provisions stipulated within that announcement (Attachment 28).
As the target population is not vulnerable and the questions are not personal or intrusive, risks to participants are expected to be minimal. The Westat IRB designated the SEPA Evaluation Study as exempt from IRB review (Attachment 19).
Study personnel have obtained proper security clearances and are required to adhere to strict professional survey standards and have signed a nondisclosure agreement as a condition of their employment. Web-based, audio-based, computer-based, and any hard copy data collection forms will be maintained in a secure area for receipt and processing. All data files on multi-user systems will be under the control of a database manager and will be subject to controlled access only by authorized personnel. Personal identifying information (PII) will be maintained separately from completed data collection forms and from computerized data files used for analysis. Final reports will be based on aggregate data in which individuals are not identified.
After the data collection is completed, all hard copy collected information and study materials will be stored in a locked, secure facility for 2 years and then will be shredded. Electronic data will be password protected and stored by the data management contractor, and also will be destroyed after 2 years.
Personal identifiable information is collected in the form of the participant’s name, race, ethnicity, gender, level of education, discipline of highest degree, prior funding history, professional affiliation, e-mail address, and phone number, which is needed to contact prospective participants. This information will be obtained primarily from archival sources. In the PI survey, we will collect the names of additional partners not listed in the archival data. The purpose of collecting partner names will be to use them to contact partners for participation in the partner survey as well as the partner interview. No other sensitive information will be collected as part of the data collection.
Data collection activities will occur over a 1-year period for an estimated 558 unique participants and includes completion of a web-based survey, and participation in a telephone interview or site visit interview in a subset of 40 projects. The total population of 558 participants is comprised of 156 PIs, approximately 312 project partners (assuming an average of two partners per project), and up to a total of 90 key staff from the subset of projects we will study more closely (evaluators and other key project staff are included in this group). The PIs and project partners from all 156 SEPA projects (156 PIs and 312 project partners) will be invited to participate in the web survey (Attachments 12-13). Of the 156 SEPA projects, a subsample of 40 SEPA projects will be selected for telephone or in-person onsite interviews. PIs and their project partners from this sample of 40 SEPA projects (40 PIs and 80 project partners) will be invited to participate in the telephone/in-person interviews (Attachments 20-21). We will conduct up to three site visits during which we will travel to six to nine project sites. During these site visits, we anticipate interviewing a total of 112 participants (6 PIs, 6 project partners, and up to 90 key staff personel and evaluators) who will be invited to participate in onsite face-to-face interviews or focus groups (Attachments 27). The estimated time for completing the PI and partner web surveys is 30 minutes apiece, each PI and Partner telephone interview is estimated to take 60 minutes. To allow for more in-depth discussion, interviews conducted during site visits are estimated at 90 minutes for PIs and project partners and 90 minutes for other key staff (60 minutes for key staff identified by PIs and 30 minutes for evaluators). Administration of the Site Visit Interview Consent Form (Attachment 18), the Telephone Script to Schedule the Telephone Interview (Attachment 27), and the Telephone Script to Schedule the Site Visit Interview are estimated at 5 minutes each. The estimate of the total and annualized burden is 523 hours, as summarized in Exhibit A.12-1.3
Exhibit A.12-1. Estimates of Annualized Burden Hours
Type of respondent |
Data collection type |
Number of respondents |
Number of responses per respondent |
Average burden per response (in minutes) |
Total annual burden hours |
PI |
Web survey |
156 |
1 |
30 |
78 |
Telephone script to schedule interview |
34 |
1 |
5 |
3 |
|
Telephone interview |
34 |
1 |
60 |
34 |
|
Telephone script to schedule site visit |
34 |
1 |
5 |
3 |
|
Site visit interview |
6 |
1 |
90 |
9 |
|
Project partner
|
Web survey |
312 |
1 |
30 |
156 |
Telephone script to schedule interview |
74 |
1 |
5 |
7 |
|
Telephone interview |
74 |
1 |
60 |
74 |
|
Telephone script to schedule site visit |
74 |
1 |
5 |
7 |
|
Site visit interview |
6 |
1 |
90 |
9 |
|
Other key staff |
Telephone script to schedule site visit |
90 |
1 |
5 |
8 |
Site visit interview |
90 |
1 |
90 |
135 |
|
Total |
|
|
558 |
|
523 |
The cost burden to web survey respondents is essentially the time required to read the instructions and complete the survey. The cost burden to participants in the telephone interview is the time to schedule and participate in the interview. The cost burden to the site visit participants is essentially the time to read and sign the Informed Consent Form and to participate in the site visit interview. The total annualized cost to the respondents is estimated to be $19,843, calculated at $37.94 per hour, the hourly rate for a medical scientist (U.S., Department of Labor, Bureau of Labor Statistics, 20144). The costs are summarized in Exhibit A.12-2.
Exhibit A.12-2. Estimates of Annualized Cost to Respondents
Type of respondent |
Data collection type |
Number of respondents |
Total |
Total respondent cost |
PI
|
Web survey |
156 |
78 |
$2,959 |
Telephone script to schedule interview |
34 |
3 |
$114 |
|
Telephone interview |
34 |
34 |
$1,289 |
|
Telephone script to schedule site visit |
34 |
3 |
$114 |
|
Site visit interview |
6 |
9 |
$341 |
|
Project partner
|
Web survey |
312 |
156 |
$5,919 |
Telephone script to schedule interview |
74 |
7 |
$266 |
|
Telephone interview |
74 |
74 |
$2,808 |
|
Telephone script to schedule site visit |
74 |
7 |
$266 |
|
Site visit interview |
6 |
9 |
$341 |
|
Other key staff |
Telephone script to schedule site visit |
90 |
8 |
$304 |
Site visit interview |
90 |
135 |
$5,122 |
|
Total |
558 |
523 |
$19,843 |
|
There are no direct costs to respondents other than their time to participate in the study.
The costs to the federal government include SEPA FTE costs and contractor costs to develop the PI and partner web surveys and four case study interview protocols, costs to conduct the data collection and analysis, and costs to prepare presentations and a summary report. These activities are spread over 30 months and cost approximately $733,081. Federal employee hourly rate was calculated from OPM’s Salary Table 2015-DCB for salaries effective January 2015. The hourly wage for a GS-14-1 was used. Westat’s costs are based on labor spread over six staff: Project Director/Principal Investigator, Senior Advisor, Senior Programmer, Quantitative Analyst Manager, Survey Task Leader, and Case Study Task Leader. It is estimated that the fully loaded 30-month (length of project period) amount for these six staff will be (1) $270, 603 for the Project Director/Principal Investigator at approximately 0.25 FTE; (2) $128,503 for the Senior Advisor at approximately 0.12 FTE; $12,096 for the Senior Programmer at approximately 0.01 FTE; $71,925 for the Quantitative Analyst Manager at approximately 0.10 FTE; $128,288 for the Survey Task Leader at approximately 0.21 FTE; and $89,468 for the Case Study Task Leader at approximately 0.15 FTE.
It is therefore estimated that the study will require about .84 FTE total per year spread over six staff totaling approximately $320,352 per year, as summarized in Exhibit A.14-1. These expenses are related to directing contractors, overseeing and solving problems as they arise, developing materials, supervising data collection, performing data analysis, and preparing manuscripts and presentations.
Exhibit A.14-1. Annualized Cost to the Federal Government
Staff |
Grade/Step |
Salary |
% of Effort |
Fringe (if applicable) |
Total Cost to Gov’t-30 Month Total |
Federal Oversight |
14/1 |
$107,325 |
.10 FTE |
|
$26,831 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Contractor Cost |
|
|
|
|
|
Project Director/Principal Investigator |
|
|
.25 FTE |
|
$270,603 |
Senior Advisor |
|
|
.12 FTE |
|
$128,503 |
Senior Programmer |
|
|
.01 FTE |
|
$12,096 |
Quantitative Analyst Manager |
|
|
.10 FTE |
|
$71,925 |
Survey Task Leader |
|
|
.21 FTE |
|
$128,288 |
Case Study Task Leader |
|
|
.15 FTE |
|
$89,468 |
Travel |
|
|
|
|
|
Other Cost |
|
|
|
|
|
|
|
|
|
|
|
Total |
|
|
|
|
$733,081 |
This is a new information collection.
Most of the data analyses will consist of descriptive statistics (i.e., percentages, means, medians, and standard deviations, as appropriate), cross-tabulations, and graphical summaries. The web-based survey analyses will be performed using SPSS version 20. The telephone interview and site visit analyses will be performed using Excel. Social network analysis will be conducted using NodeXL and UCINET software to analyze the extent to which partners have formed formal or informal relationships that extend beyond those established through their SEPA project, and analysis of interview transcripts will be conducted using NVivo to generate themes related to central topics of interest to the evaluation.
Products of the evaluation include a final report detailing the methodology, key findings, conclusions, and recommendations. The report will include an executive summary, be structured around the three evaluation questions, and include a discussion of the following topics: (1) the study questions, (2) the process used to develop and test data collection instruments, (3) data collection methods and analyses, (4) findings, (5) implications of the findings, and (6) recommendations for how to evaluate programs that are similar to SEPA. Data in the report will be presented via user-friendly graphs and tables to summarize textual analysis. Other products produced from the evaluation will include (1) a PowerPoint slide deck to present key findings at the 2017 Sci Ed meeting, which will include graphs, tables, and other relevant information; and (2) a searchable database of project characteristics and program activities that will be made available to SEPA. The database will assist SEPA staff in generating information about specific SEPA projects or program trends in response to questions from NIH and/or Congress.
Following preparation of a draft of the final report, we will convene a meeting of the SEPA Advisory Committee to solicit their input and then conduct a briefing for the Office of Research Infrastructure Programs/Division of Coordination, Planning, and Strategic Initiatives (ORIP/DPCPSI) officials, other federal government STEM officials, and invited parties. The briefing will present an overview of the study, a description of the evaluation design and data collection protocols, major findings, implications of the findings for public policy, and a brief description of recommendations related to evaluating programs with designs that are similar to SEPA.
Intended audiences for the results of the evaluation include the SEPA Advisory Committee and ORIP/DPCPSI officials, other federal government STEM officials, and invited parties. Evaluation researchers and other entities conducting evaluations may also be interested in the methods and instruments developed for this study.
Findings from the evaluation will be disseminated via the final report and an executive summary. In addition, SEPA staff may prepare presentations for national conferences and publish articles in peer-reviewed journals. The SEPA staff will work within NIH to disseminate the results.
As stated earlier, the primary purpose of this data collection is to provide SEPA with information on the extent to which grants awarded from 2004 through 2014 have met goals related to project activities, partnership formation, and evaluation quality. SEPA will use the results to better understand its SEPA portfolio, particularly the extent to which SEPA grantees are fulfilling the missions and goals of the SEPA Program, how well the program has fulfilled its goal of enhancing and creating partnerships among SEPA projects, and whether the program has encouraged the use of more rigorous evidence-based evaluations. All information collected will be used to provide analytical and policy support to SEPA, assisting NIH in making decisions about current SEPA programming, future funding, and other initiatives to improve science education partnerships.
The study announcement will be sent from SEPA to PIs in early January 2016 and to partners in early March 2016. Data collection will occur in 2016, beginning with the PI web survey on January 22, the partner web survey on March 25, the PI and partner telephone interviews on August 7, and the site visits on August 28. The contract period will include fielding, analyzing, and disseminating findings from these studies. Westat will be responsible for preparing the analytic databases and reports resulting from the study. The timetable for the data collection is shown in Exhibit A.16-1.
Exhibit A.16-1. Project Time Table
Activity |
Timeline |
E-mail announcement sent to PIs |
Early January 2016 |
E-mail invitation letter for PI survey to prospective participants and opening of online PI survey |
January 22, 2016 |
Reminder e-mail notice sent to survey nonrespondents |
Early February, 2016 |
Telephone follow-up call to survey nonrespondents |
Mid-February, 2016 |
E-mail announcement sent to partners |
Early March 2016 |
E-mail invitation letter for Partner survey to prospective participants and opening of online Partner survey |
March 25, 2016 |
Reminder e-mail notice sent to survey nonrespondents |
Early April 2016 |
Telephone follow-up call to survey nonrespondents |
Mid-April 2016 |
Select PI/partner and site visit samples |
Mid-April 2016 |
E-mail invitation letter for telephone interview to prospective participants |
Mid-July 2016 |
Conduct telephone interviews |
August 7, 2016 |
Reminder e-mail notices sent to telephone interview nonrespondents |
Mid-August, 2016 |
Telephone follow-up call to telephone interview nonrespondents |
Late August 2016 |
E-mail announcement sent to site visit participants |
Early July 2016 |
E-mail invitation letter for site visit to prospective participants |
Early August 2016 |
Conduct site visits |
August 28, 2016 |
Data entry and cleaning |
January-June 2016 |
Data preparation and analyses |
February-June 2016 |
Preliminary report of findings |
December 7, 2016 |
Draft of full report and executive summary |
December 21, 2016 |
Present to SEPA Advisory Committee and NIH officials |
February 10, 2017 |
Final report (including full report and executive summary) |
March 30, 2017 |
The SEPA Evaluation Study will not require exemption from displaying the expiration date of OMB approval. Any reproduction of the data collection instruments will prominently display the OMB approval number and expiration date.
The SEPA Evaluation Study does not require any exceptions to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).
1Alternatively, we will explore the possibility for the site visits to coincide with regularly scheduled SEPA project director meetings to avoid additional travel burden on participants. This will depend on the timing of the meetings and whether a critical number of project staff will be attending the meeting.
2For participants who are not available during the site visit, we will conduct a separate telephone interview with each.
3 Please note that in the table, the total number of unique respondents for the proposed data collection is 558 (156 PIs, 312 project partners, and 90 key staff personel). Since each cohort of respondents (e.g., PIs, partners) may be involved in more than one data collection, are listed under multiple data collection types (interview, survey, etc.) but are ee counted only once. Please note the total number of respondents is correct as indicated.
4 Bureau of Labor Statistics (2014, May). May 2014 National Occupational Employment and Wage Estimates United States. Retrieved from: http://www.bls.gov/oes/current/oes_nat.htm#19-0000
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Sylvie Warren |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |