Supporting Statement B For:
Evaluation of the Science Education Partnership Award (SEPA) Program
November 17, 2015
Tony Beck, Ph.D.
Office of Science Education/SEPA
Office of Research Infrastructure Programs (ORIP)
Division of Program Coordination, Planning, and Strategic Initiatives (DPCPSI)
Office of the Director
National Institutes of Health
6701 Democracy Blvd, Room 206
Bethesda, MD 20892
Telephone: 301-435-0805
E-mail: beckl@mail.nih.gov
Table of Contents
Supporting Statement B. Collections of Information Employing Statistical Methods
B.1. Respondent Universe and Sampling Methods
B.2. Procedures for the Collection of Information
B.2.1. Quality Control
B.3. Methods to Maximize Response Rates and Deal With Nonresponse
B.3.1. Follow-Up
B.3.2. Unit Nonresponse
B.3.3. Item Nonresponse
B.4. Tests of Procedures or Methods to Be Undertaken
B.4.1. Review of Web-Based Surveys and Interview Protocols
B.4.2. Pilot Tests of Web-Based Surveys and Interview Protocols
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
Exhibit B.1-1. Number of Potential and Expected Participants by Data Collection Type
Exhibit B.2-1. Procedures for PI and Partner Surveys
Exhibit B.2-2. Procedures for Telephone Interviews
Exhibit B.2-3. Procedures for Site Visits
1. PI Web Survey
2. Partner Web Survey
3. PI Telephone Interview Protocol
4. PI Site Visit Interview Protocol
5. Partner Telephone Interview Protocol
6. Partner Site Visit Interview Protocol
7. Key Staff Interview Protocol
8. Key Evaluator Interview Protocol
9. E-Mail Announcement to PIs
10. E-Mail Announcement to Partners
11. E-Mail Announcement to Site Visit Participants
12. E-Mail Invitation Letter - PI Web Survey
13. E-Mail Invitation Letter - Partner Survey
14. E-Mail Reminder - PI Web Survey
15. E-Mail Reminder - Partner Web Survey
16. Telephone Reminder - PI Web Survey
17. Telephone Reminder - Partner Web Survey
18. Site Visit Interview Consent Form
19. IRB Approval Letter
20. E-Mail Invitation Letter - PI Telephone Interview
21. E-Mail Invitation Letter - Partner Telephone Interview
22. E-Mail Reminder - PI Interview
23. E-Mail Reminder - Partner Interview
24. Telephone Reminder - PI Interview
25. Telephone Reminder - Partner Interview
26. Telephone Script to Schedule the Telephone Interview
27. Telephone Script to Schedule the Site Visit Interview
28. NIH Privacy Act Systems of Record
29. E-Mail Follow-Up Reminder to Schedule Telephone Interview
30. E-Mail Confirmation for Telephone Interview
The Science Education Partnership Award (SEPA) Program of the National Institutes of Health (NIH) provides 5-year grants for PK-12 educational projects, science centers, and museum exhibits to increase students’ interest in pursuing science-related careers, deliver topical and interactive information about NIH-funded medical research, and cultivate an understanding about healthy living habits among the general public. The goal of this process evaluation is to provide SEPA with information on the extent to which grants awarded from 2004 through 2014 have met goals related to project activities, partnership formation, and evaluation quality.
We will collect outcome data from our study population of SEPA principal investigators (PIs), project partners, and key project staff. There are three modes of data collection: web-based surveys (Attachments 1-2), telephone interviews (Attachments 3 and 5), and site visit interviews (Attachments 4, 6, 7, and 8). Statistical methods used to analyze web-based survey data will consist of descriptive statistics (e.g., percentages, means, medians, standard deviations, as appropriate), cross-tabulations, and graphical summaries. The web-based survey analyses will be performed using SPSS version 20. The telephone interview and site visit analyses will be performed using Excel.
The target populations for this process evaluation represent three groups of project stakeholders: PIs, project partners, and key staff personnel from 156 SEPA projects funded between 2004 and 2014. All PIs and project partners from these 156 SEPA projects constitute the PI and partner samples. The key project staff consists of evaluators and individuals identified as key by the PI.
The target population for the web-based surveys (Attachments 1-2) will include all PIs and project partners for a total of 468 potential participants. A total of 156 PIs and approximately 312 project partners (assuming two partners per project) will be invited to participate in the surveys with an expected number of 125 PI participants and 250 project partner participants. These numbers are based on an expected response rates of about 80 percent for the PI survey and 80 percent for the partner survey.1
The target population for the telephone interviews (Attachments 3-4) will include PIs and project partners of 40 select SEPA projects for a total of 120 participants. The respondent population will comprise 40 PIs and approximately 80 project partners (assuming two partners per project). The number of participants is expected to be 30 PIs and 60 partners. These numbers are based on an expected response rates of about 80 percent each for the PI interview and for the partner interview. The 40 SEPA projects selected for the PI and partner interviews will be based on findings from the document review, the surveys, and interviews with NIH program officers. Our goal will be to identify 20 projects that established successful partnerships and 20 projects where partnerships were less successful. As specified in the solicitation, the sample will also include partnerships among SEPA projects and among SEPA projects and CTSAs, IDeA state-based INBRE, or COBRE projects, and with STEM education entities within and outside of NIH.
The target population for the three site visits (Attachments 4, 6, 7, and 8) will include PIs, project partners, and evaluators for two to three projects for each of the three site visits or a total of six to nine project sites and up to 90 participants (approximately 10 per site). We expect that all project personnel from the six to nine project sites will participate in the site visit interviews. SEPA sites will be selected based on similar section criteria as the PI/partner interviews (i.e., balance of projects with successful and less successful partnerships; projects with partnerships with other SEPA projects, CTSA, IDeA, IMBRE, or COBRE projects; and with organizations within and outside of NIH). Though different factors may be responsible for nonresponse or noncooperation, we anticipate 80 percent of respondents will reply. The number of potential and expected participants by type of data collection is presented in Exhibit B.1-1.
Exhibit B1-1. Number of Potential and Expected Participants by Data Collection Type
Data collection type |
Number of potential participants |
Number of expected participants |
Response rate |
PI survey |
156 |
125 |
80% |
Partner survey |
312 |
250 |
80 |
PI interview |
34 |
28 |
80 |
Partner interview |
74 |
60 |
80 |
Site visits |
90 |
90 |
100 |
Data collection procedures for the web-based survey, the telephone interview, and the site visits are outlined below. In early January 2016, PIs with a valid e-mail address will be sent an e-mail announcing the study (Attachment 9). Project partners, following confirmation of partner names by PIs in the PI survey, will be sent the study announcement in early March 2016 (Attachment 10). Finally, PIs, partners, evaluators, and key staff for the three site visits will be sent the announcement in early July 2016 (Attachment 11). We anticipate that while some nonresponders may be contacted several times in the course of the study, the majority will be contacted only once or twice for each survey completed or interview administered.
Data collection procedures for the PI and partner surveys are described below and presented in Exhibit B.2-1:
Three weeks after the SEPA e-mail announcement, prospective web-based survey participants will be sent an e-mail invitation to participate. The invitation letter introduces the purpose of the survey, includes instructions on how to participate, and provides contact information in case of queries (Attachments 12-13).
One week after the e-mail invitation, a reminder e-mail will be sent to nonresponders to encourage them to participate. The reminder e-mail requests the individual’s participation and provides contact information in case of queries (Attachments 14-15).
One week after the first reminder e-mail, a second reminder e-mail will be sent to remaining nonresponders to encourage their participation. The reminder e-mail requests the individual’s participation and provides contact information in case of queries (Attachments 14-15).
One week after the second reminder e-mail, telephone follow-up will be conducted with remaining nonresponders for whom there is a telephone number available (Attachments 16-17). If there is no telephone number available, a staff member will investigate (via Google and other databases) for correct contact information. Section B.3.1 presents the study methods to address nonresponse and maximize response rates.
Exhibit B.2-1. Procedures for PI and Partner Surveys
Timeline |
Activity |
Early January 2016 |
SEPA e-mail study announcement sent to all potential PIs |
Early March 2016 |
SEPA e-mail study announcement sent to all potential project partners |
3 weeks later |
E-mail invitation sent to all potential participants for survey |
1 week later |
Reminder e-mail notice sent only to survey nonrespondents |
1 week later |
Second reminder e-mail sent only to nonresponders |
1 week later |
Telephone follow-up with nonresponders with telephone number available; nonresponders without telephone numbers will be traced |
Data collection for the telephone interviews will begin after the survey data collection is complete; the telephone interviews are currently slated to begin in August 2016. PIs, partners, and key staff for the three site visits will be sent the announcement in early July 2016. As described above, reminder and follow-up contacts will be sent to nonresponders. Data collection procedures for the telephone interview are described below and presented in Exhibit B.2-2:
In
mid-July 2016, prospective participants in telephone interviews will
be sent an
e-mail invitation to participate. The invitation
letter will also introduce the purpose of the evaluation, request an
e-mail reply with suggestions of two or three convenient times to be
contacted for the interview, and provide contact information in case
of queries (Attachments 20-21).
Two weeks after the e-mail invitation, a reminder e-mail will be sent to nonresponders to encourage them to participate. The reminder e-mail provides the same information as the invitation e-mail (Attachments 22-23).
One week after the reminder e-mail, telephone follow-up will be conducted with remaining nonresponders for whom there is a telephone number available (Attachments 24-25). If there is no telephone number available, a second reminder e-mail will be sent (Attachments 22-23). If there is no response received within 2 weeks from either the telephone follow-up or the second reminder e-mail, the prospective participant will be considered a nonrespondent and removed from the sample. Section B.3.1 presents the study methods to address nonresponse and noncooperation.
When
a prospective participant replies with interest in participating, a
follow-up
e-mail will be sent stating that someone will call
to schedule the interview (Attachment 29). If a prospective
participant does not reply to schedule the interview, the scheduler
will call the individual to schedule the interview using a telephone
script developed for this purpose (Attachment 26).
When a prospective participant accepts the invitation for an interview, an e-mail confirmation will be sent within two business days confirming the interview date and time, and the best telephone number at which to reach the interviewee (Attachment 30).
Exhibit B.2-2. Procedures for Telephone Interviews
Timeline |
Activity |
Mid-July 2016 |
E-mail invitation sent to all prospective participants for telephone interview |
August 7, 2016 |
Conduct telephone interviews |
2 week later |
Reminder e-mail notice sent to nonrespondents |
1 week later |
Telephone follow-up with remaining nonresponders with telephone number available; if a telephone number is not available, a second reminder e-mail is sent |
2 weeks later |
Nonresponders
to either telephone follow-up or second reminder |
1-4
weeks after |
If prospective participant replies with interest, a follow-up e-mail is sent that someone will call to schedule interview |
1-4
weeks after |
If a prospective participant does not reply, scheduler will call to schedule interview |
2 business days after agreement to participate |
E-mail confirmation is sent |
Data collection procedures for the site visits are described below and presented in Exhibit B.2-3.
For the three site visits, we will conduct individual interviews with the PI, project partners, and evaluators and a focus group interview with individuals identified as key staff involved in up to two to three SEPA projects per visit. A telephone call will made with the site visit participants to schedule an appropriate date to conduct the site visit. (Attachment 27).
Exhibit B.2-3. Procedures for Site Visits
Timeline |
Activity |
Early July 2016 |
SEPA study announcement sent via e-mail to all prospective participants |
Late August 2016 |
Contact prospective participants to schedule site visit |
August 28, 2016 |
Site visits |
Data will be gathered using three different modes of data collection: data resulting from the administration of web-based surveys; telephone interview audio files; and site visit audio files. The information collected will 1) determine the extent to which the SEPA portfolio is aligned with the program’s overall goals, (2) assess how the SEPA Program has contributed to the creation and/or enrichment of beneficial productive partnerships within SEPA projects, and (3) determine the extent to which the SEPA Program is generating a rigorous evidence-based system that provides high-quality evaluations to inform the knowledge base.
The contractor for this study will establish and maintain quality control procedures to ensure standardization and high standards of data collection and data processing. The contractor will maintain a log of all decisions that affect sample enrollment and data collection. The contractor will monitor response rates and completeness of acquired data, and provide SEPA with progress reports in agreed-upon intervals.
All data sets will be housed on Westat’s secure servers. Any data set containing confidential information will be further protected using passwords. We will create codebooks with definitions and explanations of all variables for each data set.
To improve response rates and cooperation, follow-up efforts will be used to encourage survey and telephone interview completion. The efforts to reduce the number of nonrespondents apply only to nonrespondents and consist of the following:
Reminder e-mails will be sent to all nonrespondents beginning 1 week after initiation of the surveys and telephone interview. For the surveys, a second reminder e-mail will be sent 1 week after the first reminder e-mail and, for the telephone interview, a second reminder e-mail will be sent to those individuals for whom there is no phone number available for a telephone follow-up call (Attachments 14-15, 22-23).
One week after the second reminder e-mail, telephone follow-up for nonresponse will begin for those individuals with a telephone number available. Experienced telephone interviewers, who are trained in nonresponse conversion, will make up to three attempts at a follow-up call. The call is a prompting call and encourages the prospective participant to accept. The telephone interviewers will use an appropriate Telephone Follow-up Script for the call corresponding to the data collection mode— either survey or telephone interview (Attachments 14-15, 24-25).
For the telephone interviews, if a participant breaks off the interview early, the interview will be considered as a nonresponse. If a participant breaks off the interview after the interview is midway through, the interview data will be retained.
As noted, to improve nonresponse, experienced interviewers who are skilled in nonresponse conversion will contact nonrespondents. If a prospective participant is unable or unwilling to participate, refusal conversion will not be attempted. Tracing, using online searches, will be conducted for survey nonrespondents for whom a telephone number is not available.
For the web survey, although our procedures are designed to maximize item response rates, the analysis will need to confront the issue of missing data. Experience with previous surveys indicates that some respondents will omit responses to some items (e.g., more burdensome items), although they may have provided most of the requested data. If item nonresponse is unexpectedly high for any of the key analytic variables, hot-deck imputation and reconstruction imputation techniques will be used to estimate missing-item values.
Two methods were used to inform development of the web surveys and the telephone interview protocols and to refine the collection of information in order to minimize burden and to improve utility: (1) a review by an advisory committee, and (2) plans to pilot test the protocols at the NIH Sci Ed meeting that will occur from May 20 through 23, 2015. Additionally, the data collection instruments have been reviewed internally by Westat researchers and also by the SEPA Program Officer.
After several rounds of internal review, the web-based surveys were reviewed by an Advisory Committee comprising a group of STEM scientists and stakeholders and SEPA representatives. Their names and contact information are provided in Section B.5. Based on feedback from the Advisory Committee, it was determined that it would be useful to add an item on the PI survey that asks PIs about the activities (e.g., developing curricula, developing professional development workshops) that their SEPA projects engaged and the audiences that the activities targeted.
Westat plans on conducting pilot tests of the web surveys and interview protocols with participants at the NIH Sci Ed meeting that will take place from May 20 to 23, 2015. Pilot test informants will be a convenience sample comprising NIH Sci Ed meeting attendees (SEPA project PIs and partners). Pilot test informants will be asked to assist in reviewing the protocols in the following areas: the clarity of language used (e.g., eliminating double-barreled questions), consistency across protocols, and reduction of response burden. A full report of the pilot test results will be written up and be used to revise the web surveys accordingly.
The following individuals make up the Advisory Committee and were consulted on the study protocols.
Vence Bonham, Associate Investigator, Social and Behavioral Research Branch, National Human Genome Research Institute, National Institutes of Health, 301-594-3973
Louisa Koch, Director of Education, National Oceanic and Atmospheric Administration, 202-482-3384
Daniel Laughlin, Research Professor, School of Education and Urban Studies, Morgan State University, 410-212-3781
Helena Mishoe, Associate Director of Research Training & Diversity, National Heart, Lung, and Blood Institute, National Institutes of Health, 301-451-5081
Luci Roberts, Director of Planning and Evaluation, Office of Extramural Research, National Institutes of Health, 301-594-1841
Robert Russell, Senior Education Associate, Space Science Institute, 703-292-2995
Heidi Schweingruber, Director, Board on Science Education, The National Research Council, 202-334-2009
Robert Tai, Associate Professor of Education, Curry School of Education, University of Virginia, 434-924-0840
David Ucko, President, Museums+More LLC, 202-270-1873
David Wilson, Director of American Indian Affairs and Science Policy, SACNAS, 202-452-8909
Westat
will be responsible for data collection and analysis under the
direction of
Dr. Jill Feldman, 240-453-2905 or
jillfeldman@westat.com.
The following individuals were critical in developing the research plan, the conceptual framework, survey questions, and sampling strategies underlying the SEPA Process Evaluation.
Joy Frechtling, Corporate Officer, 301-517-4006
Gary Silverstein, Senior Consultant, 301-251-2244
The following individuals will conduct data collection activities and analysis.
Edward Mann, Senior Systems Analyst, 301-294-4434
Michael Steketee, Senior Study Director, 240-453-2603
Henry Tran, Research Analyst, 240-314-2357
1We expect the response rates for the PI and partner surveys to be at least 80 percent. But there are number of potential factors that may suppress the response rates that should be kept in mind. First, it will be difficult to locate respondents for some projects that ended 5 or more years ago. We are anticipating the contact information for at least some PIs and partners will be outdated and that Internet searches and conversations with SEPA staff will be required to locate these individuals. Second, given the amount of time that has passed, respondents for some older SEPA projects may have difficulty recalling some information. Finally, for the partner surveys, additional follow-up may be required to identify the most appropriate respondent and/or to encourage their participation in the survey.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Sylvie Warren |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |