Supporting Statement A For:
State and Community Tobacco Control (SCTC) Research
Initiative Evaluation (NCI)
September 2, 2014
Elizabeth M. Ginexi, Ph.D.
Tobacco Control Research Branch
Behavioral Research Program
Division of Cancer Control and Population Sciences
National Cancer Institute
9609 Medical Center Drive, Room 3E564, MSC 9761
Bethesda, MD 20892-9761
Telephone: 240-276-6765
Fax: 240-276-7907
E-mail: LGinexi@mail.nih.gov
A. Justification 1
A.1 Circumstances Making the Collection of Information Necessary 1
A.2. Purpose and Use of the Information 4
A.3 Use of Improved Information Technology and Burden Reduction 8
A.4 Efforts to Identify Duplication and Use of Similar Information 12
A.5 Impact on Small Businesses or Other Small Entities 12
A.6 Consequences of Collecting the Information Less Frequently 12
A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 13
A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency 13
A.9 Explanation of Any Payment or Gift to Respondents 13
A.10 Assurance of Confidentiality Provided to Respondents 13
A.11 Justification for Sensitive Questions 15
A.12 Estimates of Annualized Burden Hours And Costs 15
A.13 Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers 17
A.14 Annualized Cost to the Federal Government 17
A.15 Explanation for Program Changes or Adjustments 18
A.16 Plans for Tabulation and Publication and Project Time Schedule 18
A.17 Reason(s) Display of OMB Expiration Date is Inappropriate 23
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 23
Attachments
1. SCTC Scientists Web Survey
2. Affiliated Partners Web Survey
3. PI/Co-PI Telephone Interview Guide
4. Affiliated Partner Telephone Interview Guide
5. Pilot Project Telephone Interview Guide
6. Working Group Telephone Interview Guide
7. Coordinating Center Telephone Interview Guide
8. PI/Co-PI Expert Panel Moderator Guide
9. Expert Panel Informed consent
10. Privacy Impact Assessment
11. Individuals Consulted in Development of Conceptual Model
12. Privacy Act Memo
13. IRB Approval Letters
14. Email Announcement
15. Email Invitation Letter - Web Survey - SCTC Scientists
16. Email Invitation Letter - Web Survey - Affiliated Partners
17. Email Invitation Letter - Telephone interview - SCTC Scientists
18. Email Invitation Letter - Telephone interview - Affiliated Partners
19. Email Invitation Letter - Expert Panel
20. Email Reminder- Web Survey - SCTC Scientists
21. Email Reminder- Web Survey - Affiliated Partners
22. Email Reminder- Telephone interview - SCTC Scientists
23. Email Reminder- Telephone interview - Affiliated Partners
24. Telephone Follow-up Reminder- Web Survey
25. Telephone Follow-up Reminder - Telephone Interview
26. Email Follow-up Reminder to Schedule Telephone Interview
27. Telephone Script to Schedule Telephone Interview
28. Email Confirmation for telephone Interview
This is a request to approve a new collection titled “State and Community Tobacco Control Research Initiative Evaluation” (NCI) for 2 years. This 5-year (FY 2012-2016), $46 million ($9.2 million/year) program resides within the Tobacco Control Research Branch in the Behavioral Research Program of the Division of Cancer Control and Population Sciences. The program targets four high-priority tobacco control research areas at the state and community level in the United States: (1) Secondhand smoke policies, (2) Tobacco tax and pricing policies, (3) Mass media countermeasures and community and social norms, and (4) Tobacco industry practices. The initiative supports innovative research to yield rapid and actionable findings for state and community tobacco control programs. The NCI is undertaking an evaluation to assess the dissemination, implementation, and community collaboration processes of the SCTC grantees and their respective state and community partners and stakeholders. The evaluation will utilize archival grant project data and archival data collected from the scientists in the first two years of the initiative. The evaluation will also collect new data to: 1) determine relationships, interactions, and connectedness among network partnerships over time and with policy makers; 2) assess the utility of research tools, interventions, products, and findings from the perspective of key tobacco control stakeholders; and 3) determine key indicators for broad adoption of research products. Results will address research-to-practice gaps by providing a critical window into the process of disseminating evidence-based research tools, products, and science findings in community public health settings. Intended audiences include staff at NIH Institutes and Centers interested in supporting translation/dissemination and implementation science.
A.1 Circumstances Making the Collection of Information Necessary
The Public Health Service Act, Section 412 (42 USC § 285a-1) and Section 412 (42 USC § 285a-1) authorizes the National Cancer Institute (NCI) to establish and support programs for the demonstration of new methods for the dissemination of information to the general public concerning the prevention, early detection, diagnosis, and treatment and control of cancer and information concerning unapproved and ineffective methods, drugs, and devices for the diagnosis, prevention, treatment, and control of cancer. The mission of the NCI Tobacco Control Research Branch (TCRB) is to lead and collaborate on research, and to disseminate evidence-based findings to prevent, treat, and control tobacco use for the purposes of cancer control. As part of the Division of Cancer Control and Population Sciences (DCCPS) TCRB aims to reduce risk, incidence, and deaths from tobacco use related cancers. Central to these activities is the process of synthesis and evaluation of what has been learned. Program evaluation can help the NCI in its decision making process by identifying new priority areas and appropriate strategies for improvements. By undertaking program evaluation research, the NCI aims to understand how to support and improve the dissemination efforts of NCI funded scientists.
A 2004 Institute of Medicine report concluded that the NIH “does not have formal regular procedures or criteria for evaluating center programs” despite the considerable financial commitments involved. The NCI subsequently created the Evaluation of Large Initiatives (ELI) project, which examines innovative evaluation approaches and methodologies to assess the challenges and scientific progress of collaborative team processes for transdisciplinary research. Significant methodological advances have been achieved in areas of assessing the functioning of research teams, network analysis, concept mapping, and quality metrics to better understand teams in healthcare and how to improve them (e.g., Hall et al, 2008; Trochim et al., 2008). These contributions led to emerging areas of research known as the Science of Team Science (SciTS) and Implementation Science. The NCI SciTS team developed innovative methods and metrics to assess the added value of transdisciplinary team science, including: quasi-experimental designs for bibliometric analyses, science mapping and visualizations of collaboration networks, and other performance measures related to measuring interdisciplinarity. The team tested a number of these metrics for reliability and validity (Hall et al, 2008; Trochim et al., 2008).
However, SciTS metrics and methods have been developed to apply to research teams, not to health policy and community health teams. More relevant assessment is needed to understand how similar SciTS metrics and methods may be applied to multi-center research-community collaborative initiatives and to study how such teams play a role in effective dissemination of research tools, products, and findings into real-world settings. The proposed evaluation not only provides more rigorous assessments of investments in tobacco control research but also contributes to the growing field of implementation science in practice settings.
Efforts have been made to develop comprehensive conceptual frameworks to guide the assessment of complex team science initiatives (Holmes, et al., 2008; Stokols, 2003; Stokols, 2010; Trochim, 2008) and to develop multi-method approaches for assessing the collaborative and cross-disciplinary processes and outcomes of team science (Hall et al. 2012, Hall et al., 2008; Masse, et al. 2008; Provan, et al. 2008; Stokols, 2003; Stokols 2010; Trochim, 2008). However, there is little theory available to guide an evaluation of team science initiatives that are intended to yield rapid and actionable findings for state and community public health programs.
Translational research is a critical scientific gap area for the NIH. The NIH Office of Behavioral and Social Sciences Research (OBSSR) has listed translation among its top scientific priority areas and it has indicated that innovative behavioral and social sciences research is needed to identify approaches to close the gap between research and practice. Little is known, however, about how best to ensure that scientific discovery is used to inform and improve public health. Research-to-practice and policy gaps are pervasive across all fields of behavioral health. This evaluation will provide a window into the process of disseminating evidence-based research tools, products, and science findings in community settings.
The overarching approach for this process evaluation draws upon models that relate to dissemination and implementation science such as the “Evidence Integration Triangle” (EIT) (Glasgow et al. 2012) and the System Antecedents for Innovation (Greenhalgh, Robert, Macfarlane, Bate & Kyriakidou, 2004). Central to these approaches is the importance of studying the iterative interactions between scientists and public health practitioners, and also documenting organizational characteristics of the practice partners in order to assess the degree of readiness for science to practice dissemination.
The announcement for the NCI State and Community Tobacco Control (SCTC) initiative (RFA-CA-10-008) required applicants to identify their plans to collaborate with state or local tobacco control programs or other public health organizations for the purposes of research and dissemination. While collaboration with community practice partners and plans for science to practice dissemination are often encouraged in NIH grant research, the process and successes in these efforts are rarely studied.
The purpose of the proposed process evaluation is to help us assess the dissemination, implementation, and community collaboration processes of the SCTC Research Initiative grantees and their respective state and community partners and stakeholders and to develop the capacity to - at a later stage – assess the products of these processes.
A.2 Purpose and Use of the Information
The SCTC Research Initiative is aimed at addressing under-studied aspects of tobacco control policy and media interventions. It targets four high-priority research areas at the state and community level and is intended to support innovative research that will yield rapid and actionable findings for state and community tobacco control programs and practitioners. The grantees are charged with developing effective strategies to translate and disseminate research findings to a wide array of audiences, including tobacco control programs, public health practitioners, researchers, parents and teachers, youth and youth-serving organizations, Federal, State, and local policy makers, and to the general public.
This project includes two phases. Phase I — already completed — relied on archival project data from the grant records and initiative administrative data as well as survey and interview data collection. Phase II will analyze the data collected in Phase I and collect new data via surveys and semi-structured interviews conducted with the SCTC scientists and their nominated affiliated partners.
The affiliated tobacco control partners are comprised of 3-5 individuals nominated by each PI at the participating research centers. Affiliated partners refer to SCTC collaborators who are not SCTC grantees and are based outside of funded research centers. Affiliated partners can include representatives at state or local tobacco control programs, public health and community based organizations, Federal or State agencies, organizations hired to consult on SCTC projects or workgroups, as well as other academic institutions.
Phase I of the evaluation has been completed. This work focused on the collection of baseline data 1to identify: (1) key network partners and stakeholders; (2) SCTC research tools, interventions, products and findings developed for broad dissemination; and (3) the range of dissemination processes employed. Data collected in Phase I of the evaluation included archival project data and data collected from the scientists via surveys and interviews. The archival grant project data collected consisted of: the original Principal Investigator (PI) dissemination plans, a publications/presentations database, a database of pilot projects, a Workgroup membership and meetings database, annual site progress reports, annual Workgroup progress reports, SCTC meeting minutes (including steering committee and workgroup meetings), and the Manual of Operations for the Initiative.
Phase II will study and analyze the archival data collected during Phase I and collect new data to: (1) determine relationships, interactions, and connectedness among different network partnerships over time and with policy makers; (2) assess the utility of research tools, interventions, products, and findings from the perspective of tobacco control partners and stakeholders; and (3) determine indicators for broad adoption of research products. This data will also be used to complete the mapping of the collaborative nature of the NCI-SCTC Research Initiative scientist-community network; and assess the collaborative activities and dissemination processes within the scientist-stakeholder network.
Phase II of the evaluation will include a web survey, telephone interviews, and expert panels. The target populations are the SCTC Research Initiative science network, and their affiliated tobacco control partners. The sample will include the SCTC site scientists, the principal investigators (PIs), co-investigators (Co-PIs) and other selected site/project scientists, for an approximate sample size of 60, and important affiliated partners nominated by the scientists comprising an approximate sample size of 71.
Data collected in Phase II will be used to complete the mapping of the collaborative nature of the NCI-SCTC Research Initiative scientist-community network; and assess the collaborative activities and dissemination processes within the scientist-stakeholder network. Data will be collected from approximately 131 individuals via a web-based survey (Attachments 1 and 2), from 58 individuals via a telephone interview (Attachments 3 through 7), and from up to 18 individuals participating in an expert panel (Attachment 8). Data will be collected to (1) determine relationships, interactions and connectedness among different network partnerships over time and with policy makers; (2) assess the utility of research tools, interventions, products, and findings from the perspective of tobacco control partners and stakeholders; and (3) determine indicators for broad adoption of research products.
Domain 1 – SCTC Collaboration Network
What “levels” of collaboration exist in the network?
What is the perceived value of the SCTC collaboration network?
What is the perceived value of collaboration within Working Groups?
What non-SCTC program partners are in the network?
Domain 2 – Processes and Activities Utilized in the Development of Dissemination Products
What processes and activities were used in the development of SCTC dissemination products?
What is the perceived value of allowing projects to branch out into new or emerging areas of research?
What role did collaboration between the projects and Coordinating Center play in producing products?
Domain 3 – Dissemination Products
How were products tailored to different audiences? Which aspects got tailored? When did this take place?
Which Working Groups had the most substantive perceived impact on the dissemination products and why?
Did the Steering Committee have a substantive impact on dissemination products? And if so, why and how?
Results from this evaluation will help address the NIH research-to-practice gap by providing a critical window into the process of disseminating evidence-based research tools, products, and science findings in community public health settings. Intended audiences for results of this evaluation include staff at NCI and other NIH institutes who are interested in supporting translation/dissemination and implementation science. These include the SCTC Steering Committee and Evaluation Subcommittee, the NCI Tobacco Control Research Branch and research community, the NCI Implementation Science Program, the NIH Working Group on Dissemination and Implementation, and NIH program evaluation experts. Other program evaluators may find the results and methodologies employed in this study informative in designing other program evaluations.
The evaluation study will utilize three different modes of data collection: web-based surveys, telephone interviews, and expert panels. The following sections make reference to the following six specific scientist and stakeholder groups from which data will be collected:
(1) Principal Investigators (PIs)/Co-Principal Investigators (Co-PIs): The SCTC scientists are comprised of the PIs, Co-PIs, and other scientists from the seven research centers located at: Emory University; University of Illinois at Chicago, Institute for Health Research and Policy; Mayo Clinic Arizona; University of North Carolina Chapel Hill; University of California San Diego; the University of California San Francisco; and the SCTC Coordination Center located at Research Triangle Institute (RTI).
(2) Affiliated Partners: The affiliated tobacco control partners are comprised of 3-5 individuals nominated by each PI at the research centers noted above. Affiliated partners refer to SCTC collaborators who are not SCTC grantees and are based outside of funded research centers. Affiliated partners can include representatives at state or local tobacco control programs, public health and community based organizations, Federal or State agencies, organizations hired to consult on SCTC projects or workgroups, as well as other academic institutions.
(3) Pilot Project Leads: These are PIs and Co-PIs who have the additional role of leading SCTC initiative pilot (developmental) projects. These are smaller projects being conducted parallel with the main projects that may be related or unrelated in subject matter. They involve collaboration between at least two of the main project teams or sites.
(4) Working Group Leads: There are PIs or Co-PIs that have the additional role of leading SCTC working groups. Working groups meet on a regular basis to exchange ideas about a particular topic of joint interest among two or more of the initiative teams.
(5) Coordinating Center Staff: Research Triangle International (RTI) serves as the coordinating center for the SCTC initiative.
(6) NCI Representatives: NCI staff involved in the development of SCTC and in its current administration.
Web-based Surveys
For the web-based surveys, respondents will provide responses to a computerized questionnaire, which is accessed through the internet. There are two versions of the survey; one to be completed by PIs and Co-PIs, SCTC Scientists Web Survey (Attachment 1) and one to be completed by the affiliated partners, Affiliated Partners Web Survey (Attachment 2). The advantages of this format lie in that (1) data may be collected for all respondents using this standardized method allowing for uniform data collection; (2) respondents can complete the survey at their convenience; (3) the system navigates through the survey skip patterns based on responses; (4) respondents do not need to return the survey by mail; (5) the dissemination of the survey to participants is easy and inexpensive; (6) reminders to study participants are also easy and inexpensive; and (7) resulting data is automatically entered in an electronic database in the course of administration and is thus ready for analysis as soon as the field is closed. In addition, information may be obtained regarding the number of survey respondents who initiated the survey but who did not complete and submit the survey. Challenges include potential lack of familiarity with online surveys for some individuals (expected to be very low in this very highly educated population).
Westat will use SurveyBuilder, proprietary Westat software, which supports customized survey requirements. The web survey data will be downloaded directly from the platform in Excel; the database will be titled “SCTC Phase II Survey Database.” The web survey database will allow monitoring of the data and creating reports on completed responses. Software packages, SPSS, UCINET, and NodeXL, will all be used for analysis; SPSS for descriptive statistics and UCINET and NodeXL for social network analysis.
Telephone Interviews
The telephone interviews will be conducted by trained interviewers using one of six Telephone Interviewer Guides, customized for each of six specific stakeholder groups: (1) PI/Co-PI Telephone Interview Guide (Attachment 3); (2) Affiliated Partner Telephone Interview Guide (Attachment 4); (3) Pilot Project Telephone Interview Guide (Attachment 5); (4) Working Group Telephone Interview Guide (Attachment 6); and (5) Coordinating Center Telephone Interview Guide (Attachment 7). A target group specific Telephone Interview Guide, labeled with the respondent’s ID number, will be used to conduct each interview. While the interviewers may record notes and comments on the Telephone Interview Guide itself, the interview data will be collected via digital audio tape. Individuals participating in the telephone interview will be asked to provide verbal consent to have the interview recorded (Attachments 3-7). Each interview will be recorded on a digital file labeled with the respondent’s ID number; each audio file will have a standardized unique file name: “SCTC –I##” where “I” stands for Interview and “##” is the unique ID number for the interview respondent (i.e. SCTC-I99). All audio files will be sent via a secure File Transfer Protocol (FTP) to a transcription service, Casting Words, for transcription. Interview transcripts will be stored as Word documents and will use the same naming convention used for the telephone interview audio files; “SCTC-I##”. The file name extension will distinguish between the audio file (.wav) and the Word document (.docx). Management and analysis of the interview data, including monitoring activity and response rates, will be performed utilizing Excel.
The advantage of using telephone interviews is that they allow for cost-effective data collection from respondents who reside across the United States and also allows respondents to participate in the study at a time most convenient to them. As with other data collection modes, some individuals may be too busy to participate. The target population consists of 60 PIs and Co-PIs and 71 affiliated partners. The researchers aim to sample and recruit a total of 58 participants for the interviews: 35 PIs and Co-PIs, 21 affiliated partners and two NCI representatives.
Expert Panel
For the expert panels, there will be at least one expert panel group but no more than two groups based on PI/Co-PI availability and interest. Each group will consist of up to nine PIs and Co-PIs. The expert panels will be conducted by Westat researchers using the PI/Co-PI Expert Panel Moderator Guide (Attachment 8). As with the telephone interviews, the expert panel sessions will be recorded and data will be captured via digital audio tape. Individuals who choose to participate in the expert panel will sign an informed consent form prior to participation (Attachment 9). Each expert panel session will have a unique digital file; titled “SCTC-EP1.wav” and “SCTC-EP2.wav” respectively, where “EP” stands for Expert Panel. The expert panel audio files will be sent for transcription following the same process as for the telephone interviews; the files will be stored as Word documents and follow the same naming convention. The transcribed Word documents will be titled “SCTC-EP1.docx” and “SCTC-EP2.docx”. Also, as with the telephone interview data, management and analysis of the expert panel data will be performed using Excel.
The main advantage of expert panels is that they allow discussions between participants to take place and consensus to be built on key overarching themes. In this particular study, the panels will occur during already scheduled SCTC Initiative meetings (April 2015) to avoid additional travel burden on participants. The target population consists of 60 PIs and Co-PIs. The researchers aim to recruit up to 18 participants for the two proposed expert panels. The recruitment strategy will ensure that—to minimize burden—no more than nine PIs/Co-PIs will participate in all three data collection modes: the survey, telephone interview, and expert panel.
A Privacy Impact Assessment (PIA) has been drafted and is currently under review at HHS. (Attachment 10).
The SCTC Research Initiative, established as part of NCI’s coordinated efforts to reduce tobacco use and its associated health burdens, is intended to support innovative research that will yield rapid and actionable findings for state and community tobacco control programs and practitioners; develop effective strategies to translate and disseminate SCTC research findings; and develop collaborative research and dissemination partnerships. There are many established methods to assess the scientific progress of this program; however, there is less ability to assess progress of two key program goals: collaboration with tobacco control partners and translation and dissemination of SCTC research findings.
A.5 Impact on Small Businesses or Other Small Entities
No small entities will be involved in this study.
The study is consistent with the information collection guidelines in 5 CFR 1320.5.
The 60-Day Federal Register notice soliciting comments on this study prior to initial submission to OMB was published on June 6, 2014, Vol. 79, P. 32742. One public comment was received on June 5 which requested copies of the data collection plans and instruments. An email response was sent on June 5 requesting the mailing address where to send the documents. The requester responded with the mailing address on June 5 and a package containing the data collection plans and instruments was sent to the requester on July 17.
The web-based surveys, telephone interview guides and expert panel moderator guide were developed through NCI’s collaboration with Westat, Inc. NCI worked previously with the Evaluation Advisory Committee in developing the conceptual model that will be used to guide the study (Attachment 11).
This information collection does not involve payment or gifts to respondents.
Potential participants (PIs, Co-PIs, affiliated partners, and stakeholders) will receive email notifications, invitations and reminders announcing the evaluation, explaining its purpose, detailing the evaluation topics, and describing both the voluntary nature of participation and assurance that the data will be kept private to the extent provided by law. The following steps outline the procedures for contacting potential participants to ensure compliance and maximize response rates.
All proposed emails include language stating that all collected information will be kept private and not disclosed in any identifiable form to anyone but the researcher conducting the study, except as otherwise required by law. Individuals who choose to participate in the web-based survey and the telephone interview will be providing implicit consent by their participation. Individuals participating in the telephone interview will be asked to provide verbal consent to have the interview recorded (Attachments 3-7). Individuals who choose to participate in the expert panel will sign an informed consent form prior to participation (Attachment 9). This information collection is covered by the NIH Privacy Act Systems of Record 09-25-0156, “Records of Participants in Programs and Respondents in Surveys Used to Evaluate Programs of the Public Health Service, HHS/PHS/NIH/OD” published in the Federal Register on 9/26/2002, Vol. 67, P. 60743. All study personnel will adhere to the provisions stipulated within that announcement (Attachment 12).
As the target population is not vulnerable and the questions are not personal or intrusive, risks to participants are expected to be minimal. The NIH Office of Human Subjects Research (OHSR) and the Westat IRB designated the SCTC Evaluation Study as exempt from IRB review. (Attachment 13).
Study personnel have obtained proper security clearances and are required to adhere to strict professional survey standards and have signed a non-disclosure agreement as a condition of their employment. Web-based, audio-based, computer-based and any hard copy data collection forms will be maintained in a secure area for receipt and processing. All data files on multi-user systems will be under the control of a database manager and will be subject to controlled access only by authorized personnel. Personal identifying information (PII) will be maintained separately from completed data collection forms, and from computerized data files used for analysis. Final reports will be based on aggregate data in which individuals are not identified.
After the data collection is completed, all hard copy collected information and study materials will be stored in a locked, secure facility for two years, and then will be shredded. Electronic data will be password protected and stored by the data management contractor, and will also be destroyed after two years.
A.11 Justification for Sensitive Questions
Personally identifiable information (PII) is collected in the form of the participant’s name, professional affiliation, email address, and phone number, which is needed to contact potential participants. This information will be obtained primarily from archival sources. No sensitive information will be collected as part of the data collection.
Data collection activities will occur over a one year period for a total of 131 participants that include completion of a web-based survey, participation in a telephone interview and/or participation in an expert panel. The total population of 131 participants is comprised of 60 PIs/Co-PIs and 71 Affiliated Partners. All 60 PIs/Co-PIs (SCTC Scientists) and all 71 Affiliated Partners will be invited to participate in the web survey (Attachments 1 and 2). For the telephone interview, 21 PIs/Co-PIs, 21 Affiliated Partners, 6 Pilot Project PI/Co-PIs, 6 Working Group PIs/Co-PIs, and 2 NCI Representatives will be selected for participation (Attachments 3 through 7). There will be 18 PIs/ Co-PIs invited to participate in the Expert Panel; however, only up to nine may participate in all three modes of data collection; the survey, telephone interview and expert panel (Attachment 8). All data collection will be completed in one year. While two federal government employees will be invited to participate in the telephone interview, they have not been included below in the calculation of burden since their participation in the telephone interview is part of their job description and they will be participating during their normal work hours. Participants will be selected based on the criteria stated previously. The estimated time for completing the web survey is 20 minutes, while the telephone interview is estimated at 40 minutes and the expert panel at 90 minutes. Administration of the Expert Panel Informed Consent (Attachment 9) and administration of the Telephone Script to Schedule the Interview (Attachment 27) is estimated at 5 minutes each. The estimate of the total and annualized burden is 112 hours as summarized in Table A.12-1.
Table A.12 – 1. Estimates of Annualized Burden Hours
Type of Respondent |
Data Collection Type |
Number of Respondents |
Number of Responses Per Respondent |
Average Burden per Response (in hours) |
Total Annual Burden Hours |
SCTC Scientist |
Web Survey |
60 |
1 |
20/60 |
20 |
Affiliated Partner |
Web Survey |
71 |
1 |
20/60 |
24 |
Telephone Interview |
21 |
1 |
40/60 |
14 |
|
Script to Schedule Telephone Interview |
7 |
1 |
5/60 |
1 |
|
Pilot Project |
Telephone Interview |
6 |
1 |
40/60 |
4 |
Working Group |
Telephone Interview |
6 |
1 |
40/60 |
4 |
Coordinating Center |
Telephone Interview |
2 |
1 |
40/60 |
1 |
PI/Co-PI
|
Expert Panel |
18 |
1 |
90/60 |
27 |
Consent Form |
18 |
1 |
5/60 |
2 |
|
Telephone Script to Schedule Interview |
6 |
1 |
5/60 |
1 |
|
Telephone Interview |
21 |
1 |
40/60 |
14 |
|
Total |
112 |
||||
The cost burden to web survey respondents is essentially the time required to read the instructions and complete the survey. The cost burden to participants in the telephone interview is the time to schedule and participate in the interview. The cost burden to the expert panel participants is essentially the time to read and sign the Informed Consent and to participate in the panel. The total annualized cost to the respondents is estimated to be $6,076.00, calculated at $54.25 per hour; an average of the hourly wage rate for epidemiologists at $34, medical scientists at $42, college professors at $50 and physicians at $91 (U.S., Department of Labor, Bureau of Labor Statistics, 2012). The costs are summarized in Table A.12-2.
Table A.12 – 2. Estimated Annualized Cost to Respondents
Type of Respondents |
Data Collection Type |
Number of Respondents |
Total Annual Burden Hours |
Hourly Wage Rate |
Total Respondent Cost |
SCTC Scientist |
Web Survey
|
60 |
20 |
$54.25 |
$1,085.00 |
Affiliated Partner |
71 |
24 |
$54.25 |
$1,302.00 |
|
PI/Co-PI |
Telephone Interview |
21 |
14 |
$54.25 |
$759.50 |
Affiliated Partner |
21 |
14 |
$54.25 |
$759.50 |
|
Stakeholder |
14 |
9 |
$54.25 |
$488.25 |
|
PI/Co-PI |
Expert Panel |
18 |
27 |
$54.25 |
$1,464.75 |
PI/Co-PI |
Informed Consent |
18 |
2 |
$54.25 |
$108.50 |
PI/Co-PI |
Telephone Script to Schedule Interview |
6 |
1 |
$54.25 |
$54.25 |
Affiliated Partner |
7 |
1 |
$54.25 |
$54.25 |
|
Total |
112 |
$6,076.00 |
|||
There are no direct costs to respondents other than their time to participate in the study.
The estimated annualized cost to the Federal Government is $229,930, summarized in Table 14-1. The costs to the federal government are minimal and include contractor costs and NCI FTE costs. Contractors are utilized in this project to conduct the study and deliver data files spread over two years. This cost is approximately $370,000. NCI costs are based entirely on labor spread over three scientists: Health Scientist Administrator, Public Health Advisor, and Cancer Research Training Award (CRTA) Postdoctoral Fellow. It is estimated that the annual amount for the three scientists will be: (1) $5,040 for the CRTA Postdoctoral Fellow at 0.10 FTE of the annual salary of $50,400; (2) $14,388 for the Public Health Advisor (GS13 step 3) at 0.15 FTE of the annual salary of $95,919; and (3) $25,502 for the Health Scientist Administrator (GS 14 step 7) at 0.20 FTE of the annual salary of $127,512. It is therefore estimated that the study will require about 0.15 FTE total per year spread over three scientists totaling $44,930 per year. These expenses are related to directing contractors, overseeing and solving problems as they arise, developing materials, supervising data collection, data analysis, and preparation of manuscripts and presentations.
Table A.14-1 Annual Cost to the Federal Government
|
ANNUAL AVERAGE |
2 YEAR TOTAL |
Contractor Costs |
$185,000 |
$370,000 |
CRTA Postdoctoral Fellow ($50,400 x .10 FTE) |
$5,040 |
$10,080 |
Public Health Advisor GS 13/3 ($95,919 x .15) |
$14,387.85 |
$28,776 |
Health Scientist Administrator GS 14/7 ($127,512 x .2 FTE) |
$25,502.40 |
$51,005 |
NCI Personnel Subtotal |
$44,930 |
$89,860 |
Grand Total |
$229,930 |
$459,860 |
This is a new information collection.
Most of the data collection analyses will consist of descriptive statistics (e.g. percentages, means, medians, and standard deviations, as appropriate), cross-tabulations, and graphical summaries. The web-based survey analyses will be performed in SPSS for descriptive statistics, and in UCINET and NodeXL for social network analysis. The telephone interview and expert panel analyses will be performed using Excel. See Table A.16.1 for an illustrative list of research questions and associated measures.
Table A.16-1. Research Questions and Measures for the SCTC Evaluation Study
Research Domain |
Selected Research Questions |
Measures |
Data Collection Method |
Collaboration |
What levels of collaboration exist in the network? |
Quantitatively assessed using social network analysis; Subjectively assessed via Interviews by PIs, Co-PIs, Affiliated Partners, and other Stakeholders |
Web Survey Interviews |
What is the perceived value of the SCTC collaboration network? |
Subjectively assessed by PIs, Co-PIs, Affiliated Partners and other Stakeholders |
Interviews Expert panel |
|
What is the perceived value of collaboration within Work Groups? |
Subjectively assessed by PIs |
Interviews |
|
What non-SCTC program partners are in the network? |
Quantitatively assessed using social network analysis; Delineated by PIs, Co-PIs, Affiliated Partners, and other Stakeholders |
Web Survey Interviews |
Processes |
What processes and activities were used in the development of SCTC dissemination products? |
Delineated by PIs, Co-PIs, Affiliated Partners, and other Stakeholders |
Interviews Expert Panel |
What is the perceived value of allowing projects to branch out into new or emerging areas of research? |
Subjectively assessed by PIs, Co-PIs, Affiliated Partners, and other Stakeholders |
Interviews Expert Panel |
|
What role did collaboration between the projects and Coordinating Center play in producing products? |
Subjectively assessed by PIs, Co-PIs, Affiliated Partners, and other Stakeholders |
Interviews |
|
Products |
How were products tailored to different audiences? Which aspects got tailored? When did this take place? |
Subjectively assessed by PIs, Co-PIs, Affiliated Partners, and other Stakeholders |
Interviews Expert Panel |
Which Work Groups had the most substantive perceived impact on the dissemination products and why? |
Subjectively assessed by PIs, Co-PIs, Affiliated Partners, and other Stakeholders |
Interviews |
|
Did the Steering Committee have a substantive impact on dissemination products? Why/How? |
Subjectively assessed by PIs, Co-PIs, Affiliated Partners, and other Stakeholders |
A.16.2 Products of the Study
Products of the evaluation include a full final report detailing the methodology, key findings, conclusions, and recommendations. Data in the report will be presented in user-friendly graphs and tables with textual analysis. An executive summary will be developed to concisely convey the salient findings to a general audience. Additionally, a Power Point slide deck will be prepared for presentation purposes to highlight the key findings and will include graphs, tables, and other relevant information.
A.16.3 Methods of Dissemination
Intended audiences for the results of the evaluation include staff at NCI and other NIH Institutes and other centers who are interested in supporting translation/dissemination and implementation science. These include the SCTC Steering Committee and Evaluation Sub Committee, the NCI Tobacco Control Research Branch and research community, the NCI Implementation Science Program, the NIH Working Group on Dissemination and Implementation, NIH program evaluation experts, and the external program evaluators. Evaluation researchers and other entities conducting evaluations may also be interested in the methods and instruments developed for this study.
Findings from the evaluation will be disseminated through multiple methods, including a full report, a technical report, and an executive summary. In addition, NCI staff may prepare presentations for national conferences and publish articles in peer-reviewed journals. The NCI staff will work within NIH to disseminate the results.
A.16.4 Use of Results
The proposed evaluation is an integral part of the SCTC initiative and will help the NCI to change and adapt in its focus as the initiative evolves. Timely and continuous feedback of evaluation findings to NCI, center investigators, and community stakeholders will be essential to inform rapid response collaborative development projects and maintain commitment and buy-in for cross-center and workgroup projects. These findings are expected to be used by NCI and NIH leadership to inform the decision making process around the funding and design of large dissemination and implementation research initiatives and influence the field of implementation science. The SCTC initiative is funded under a cooperative agreement and therefore can more readily respond to the key findings and recommendations identified by this evaluation than may be possible through other types of grant funding. An example of potential areas for recommended changes may include where and how to devote core resources in order to accelerate key advances and innovative dissemination activities.
Translational research is a critical scientific gap area for the NIH and the NIH Office of Behavioral and Social Sciences Research (OBSSR) has listed translation among its top scientific priority areas. This evaluation will provide a window into the process of disseminating evidence-based research tools, products, and science findings in community settings, and will allow the evaluation of whether, which, and how SCTC collaborative network connections are facilitating science to practice dissemination. The findings from this proposed process evaluation will help to uncover effective strategies that may inform further implementation of SCTC network’s research activities. Additionally, this information may be used by leadership at the NCI and by other NIH Institutes to inform planning and decision-making with respect to designing initiatives that promote successful state and community dissemination and implementation partnerships in the context of large multi-center health promotion initiatives. This evaluation will also enable the dissemination of these metrics and methods to a broader audience including NIH center grants PIs, program managers, scientists, and the evaluation community in general.
A.16.5 Project Time Schedule
The study announcement will be sent from NCI within three weeks of OMB approval. Data collection will begin with the web survey 5-6 weeks after OMB approval, followed by the telephone interview information collection beginning approximately 13 weeks after OMB approval. The expert panel will take place during the SCTC Annual Meeting in April 2015. The contract period will include fielding, analyzing, and disseminating findings from these studies. Westat, Inc. will be responsible for preparing the analytic databases and reports resulting from the study. The timetable for the data collection is shown below, in Table A.16-2.
Table A.16-2. Project Time Table
Activity |
Timeline (after OMB Approval) |
Email announcement sent to potential participants |
Within 3 weeks |
Email invitation letter sent for survey to potential participants |
Within 5-6 weeks |
Reminder email notice sent to survey non-respondents |
Within 6-7 weeks |
Telephone follow-up call to survey non-respondents |
Within 7-8 weeks |
Email invitation letter sent for telephone interview to potential participants |
Within 13 weeks |
Reminder email notices sent to telephone interview non-respondents |
Within 14 weeks |
Telephone follow-up call to telephone interview non-respondents |
Within 15 weeks |
Invitation to participate in Expert Panel during SCTC Annual Meeting |
1 month prior to April SCTC Annual Meeting |
Completed field work |
4-5 months |
Analyses (Web Survey, Interview and Expert Panel) |
5-7 months |
Produce global map of SCTC network analyses |
7 months |
Preliminary report of findings |
7-8 months |
Draft of full report and executive summary |
8 months |
Present to SCTC Steering Committee |
9 months |
Final Reports (including full report, technical report and executive summary) |
9 months |
The SCTC Evaluation Study will not require exemption from displaying the expiration date of OMB approval. Any reproduction of the data collection instruments will prominently display the OMB approval number and expiration date.
The CPFP Evaluation Study does not require any exceptions to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).
1 Phase I of the evaluation was found to be in violation of the Paperwork Reduction Act. NCI agrees that PRA clearance should have been sought prior to the collection of this survey data. Both the contractor and NCI have been instructed to contact either the NCI PRA Liaison or the NIH Project Clearance Branch should similar activities be conducted in the future so that a determination can be made prior to information being collected. Data were collected in 2013 from 46 grantees, who provided information regarding: respondent information (name, role, title, degree, site affiliation), collaboration partners, workgroups, pilot projects, level of collaboration, and non-SCTC collaboration partners. NCI was unaware that the collection of this information required PRA clearance.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | PLCO SCREENING TRIAL |
| Author | Linda Suit |
| File Modified | 0000-00-00 |
| File Created | 2021-01-26 |