Supporting Statement A

Purpose of the data collection: The purpose of this data collection is to gather feedback from state, tribal, local, and territorial health department program directors who submitted an application to an FY 2017 or FY 2018 CDC non-research, domestic Notice of Funding Opportunity, or NOFO, on the clarity and consistency of the application language and project language, and the overall structure of the NOFO.

Intended use of the resulting data: The data will be used to identify common barriers applicants face when responding to NOFOs and areas of opportunities for improving the current NOFO development process. The data will also assist identifying NOFOs that are rated highly for clarity and consistency, and these NOFOs will be analyzed to identify common traits and shared as examples of high-quality NOFOs to CDC programs developing a NOFO. The results from this information collection will provide valuable input into the development of a redesigned NOFO template and revised guidance language, resources, and training materials for CDC programs. The results will also be used for the development of new resources for STLT health departments interested in funding on how to apply to a CDC NOFO.

Methods to be used to collect data: Both quantitative and qualitative methods will be used to obtain feedback. Respondents will be asked to complete a web-based data collection instrument and telephone-based semi-structured interviews will be conducted with a sample of those respondents.

Respondent Universe: State, tribal, local, and territorial health department program directors who submitted an application to a new FY 2017 or FY 2018 CDC non-research, domestic NOFO.

How data will be analyzed: Results will be analyzed using quantitative and qualitative methods. Quantitative analysis of the results from the web-based instrument will use descriptive statistics to determine frequency distributions. Qualitative and thematic analysis will be performed on the open-ended questions from the web-based instrument and the notes from the semi-structured interviews. Responses will be cross-tabulated to compare similarities and differences in clarity, consistency, and organization between different NOFOs.

Section A – Justification

Circumstances Making the Collection of Information Necessary

Background

This information collection is being conducted using the Generic Information Collection mechanism of the OSTLTS OMB Clearance Center (O2C2) – OMB No. 0920-0879. The respondent universe for this information collection aligns with that of the O2C2. Data will be collected from a total of 755 respondents across 212 state, tribal, local, and territorial (STLT) health departments/jurisdictions. Please see Attachment A – Respondent Breakdown for a breakdown of respondents and jurisdictions by type of health department.

This information collection is authorized by Section 301 of the Public Health Service Act (42 U.S.C. 241). This information collection falls under the essential public health service(s) of

1. Monitoring health status to identify community health problems

2. Diagnosing and investigating health problems and health hazards in the community

3. Informing, educating, and empowering people about health issues

4. Mobilizing community partnerships to identify and solve health problems

5. Development of policies and plans that support individual and community health efforts

6. Enforcement of laws and regulations that protect health and ensure safety

7. Linking people to needed personal health services and assure the provision of health care

when otherwise unavailable

8. Assuring a competent public health and personal health care workforce

9. Evaluating effectiveness, accessibility, and quality of personal and population-based

health services

10. Research for new insights and innovative solutions to health problems¹

More than half² of CDC’s budget is awarded through research and non-research cooperative agreements and grants. The Notice of Funding Opportunity (NOFO) is an awarding CDC program’s formally issued announcement of the availability of Federal funding. A CDC NOFO invites applications and provides information such as eligibility and review criteria, funding preferences and priorities, how to structure applications, and submission deadlines. Entities that successfully receive funding through a NOFO’s application process are known as recipients.

A CDC NOFO provides a foundation for STLT health department program planning by identifying public health issues and guiding the strategies and activities recipients must implement to achieve short, mid, and long-term health outcomes. Supported by a NOFO’s funding and guidance, a STLT health department program can plan and conduct activities that will improve the overall health and well-being of its community. A NOFO also helps STLT health department programs develop performance measures to assess the effectiveness of their program and improve its services to its population. Through a NOFO, CDC helps STLT health departments advance health, safety, and awareness in their communities.

Domestic NOFOs fund activities conducted in the United States or its territories. These NOFOs can only be awarded to a public (including a State or governmental agency) or private non-profit or for-profit organization that is located in the United States or its territories. A NOFO can also be categorized as research or non-research based on its activities. Research NOFOs are systematic investigations, including research development, testing, and assessment, that are designed to develop or contribute to generalizable knowledge³. Non-research NOFOs are designed to prevent or control disease or injury and improve health, or to improve a public health program or service. Non-research NOFO activities include public health surveillance, intervention implementation, capacity building, emergency response, and other activities that engage the public and improve health and well-being⁴.

A non-research NOFO lays the groundwork for the intended strategies and activities the recipient is expected to implement, the evaluation and performance measures for which the recipient is expected to collect information, and the recipient’s expected organizational capacity. The NOFO describes the project to be implemented and how the project will advance CDC priorities and achieve specific health outcomes. A well-written NOFO is crucial to the future operations of both the recipient and the CDC program. Funding decisions are based on applications submitted in response to a NOFO, therefore it is critical that the NOFO is written clearly and applicants fully understand what is expected of them. A total of 76 non-research, domestic NOFOs were published and awarded in FY 2017-2018, representing over $10 billion in total funding.

There are several sections to a non-research NOFO. An important section is the CDC Project Description, where the CDC program responsible for funding describes the strategies and activities of the project, the expected outcomes, examples of performance measures, the expected organizational capacity, and how applicants should format their work plan in the application. Another important section for applicants is the Application and Submission Information, which details the section headers and relevant information (such as requirements or additional documentation) an applicant is expected to include in their application narrative and package. The Review and Selection Process section describes how review criteria will be applied to applications, and how recipients will be selected.

In 2012, CDC initiated a NOFO redesign project for all new, non-research, domestic NOFOs (including both grants and cooperative agreements) across the agency. This effort was in response to the CDC Advisory Committee to the Director recommendations for improving grants, which focused on: engaging the STLT community, increasing financial flexibility, creating standardized approaches, and investing in quality improvements. In response to these recommendations, the Program Performance and Evaluation Office (PPEO), in collaboration with the Office of Grants Services (OGS) began redesigning the non-research NOFO development process to improve the quality of NOFOs and ensure consistency among CDC programs. Changes included implementation of a NOFO planning process that begins 12-18 months prior to NOFO publication, new NOFO writing templates with guidance to streamline and clarify language, and a comprehensive grants management system.

Since 2012, PPEO and OGS have been working with CDC programs to ensure that new, non-research NOFOs reflect strong program strategy designed to achieve the greatest health impact as well as strong accountability, performance, and evaluation for CDC programs and recipients.

Today, PPEO provides critical support to CDC programs who are developing new, non-research NOFOs. PPEO leads an average of 55 kickoff meetings per fiscal year with CDC programs planning to publish a NOFO. During these meetings, PPEO ensures CDC programs are made aware of any changes to the development process and provides guidance for writing different sections of the NOFO. During the drafting phase, PPEO provides technical assistance to CDC programs for developing logic models, performance measures, and the overall writing of the NOFO. PPEO reviews each NOFO at least twice through official review channels prior to publication. During these reviews, PPEO assesses the draft NOFO for clarity, consistency, and alignment with new grant policies or updates. PPEO does not review research NOFOs or provide technical assistance during development of research NOFOs.

PPEO collects continuous and regular feedback from CDC programs during the NOFO development process, and uses that feedback to improve PPEO’s products and services. However, PPEO does not know what effects the NOFO redesign project and current NOFO development efforts have had on applicants responding to a NOFO. PPEO sometimes receives anecdotal evidence from recipients through their CDC Project Officers. Depending on the feedback, PPEO will incorporate it into the guidance provided to CDC programs developing NOFOs. However, there has not been a formal and systematic attempt to solicit feedback from both funded and unfunded applicants relating to their experiences with reading a NOFO and developing their application.

Over half of the applications to FY 2017 and FY 2018 non-research, domestic NOFOs were from STLT health departments; other types of applicants include non-profit organizations and academic institutions. Additionally, several non-research, domestic NOFOs can only be implemented by STLT health departments (e.g. state surveillance, emergency preparedness programs). The funding available through these NOFOs represents a majority of the total amount of funding for FY 2017 and FY 2018 non-research, domestic NOFOs.

It is critical that a NOFO is written clearly and applicants fully understand how to respond to a NOFO and what is expected of them if they are funded. As an “end-user” of the NOFO development process, applicants are a key stakeholder whose opinions on the changes to the NOFO process have not been collected. Applicants are highly impacted by the NOFO template and guidance CDC programs use, and without collecting their feedback, PPEO does not know how the NOFO template and guidance affects applicants. CDC cannot make fully informed decisions if only a few stakeholder perspectives are considered.

The purpose of this data collection is to gather feedback from STLT health department program directors who submitted an application to a FY 2017 or FY 2018 CDC non-research, domestic NOFO on the clarity and consistency of the application language and project language, and the overall structure of the NOFO.

The data will be used to identify common barriers applicants face when responding to NOFOs and areas of opportunities for improving the current NOFO development process. The data will also assist in identifying NOFOs that are rated highly for clarity and consistency, and these NOFOs will be analyzed to identify common traits and shared as examples of high-quality NOFOs to CDC programs developing a NOFO. The results from this information collection will provide valuable input into the development of a redesigned NOFO template and revised guidance language, resources, and training materials for CDC programs. The results will also be used for the development of new resources for STLT health departments interested in funding on how to apply to a CDC NOFO.

By improving the existing guidance and resources, CDC can produce more clear and well-written NOFOs that allow a STLT health department to quickly understand and develop an application for funding. Qualified entities who may not have understood a NOFO or believed they did not have the capacity to write a NOFO application will be better positioned to produce high-quality applications. By incorporating applicant feedback, PPEO can improve and expand the NOFO development efforts to better meet all stakeholders’ needs.

Overview of the Information Collection System

Using a web-based data collection instrument and telephone based semi-structured interviews, information will be collected from 755 STLT health department program or project directors who submitted an application to a FY 2017 or FY 2018 CDC non-research, domestic NOFO. The instruments will be used to gather information from applicants regarding the clarity, consistency, and organization of application instructions and project information in their NOFO. The instruments were pilot tested by 7 public health professionals. Of this group, 4 participated in the web-based data collection instrument pilot and all 7 participated in the telephone-based semi-structured interview pilot. Feedback from each group was used to refine questions as needed, ensure accurate programming and skip patterns and establish the estimated time required to complete the information collection instrument.

Web-Based Data Collection Instrument

Data will be collected from 755 STLT health department program or project directors via a web-based data collection instrument (see Attachment B — Web-Based Data Collection Instrument Word Version, Attachment C – Web-Based Data Collection Instrument Web Version). If a respondent applied to more than one CDC NOFO in FY 2017-2018, they will be asked to complete the web-based data collection instrument based on their latest application, in order to ensure one response is received from one respondent. This data collection instrument will provide CDC with quantitative feedback that identifies and prioritizes areas of a NOFO for improvement.

Telephone-based semi-structured interviews

Telephone-based semi-structured interviews will be conducted with a sample of 20 respondents who completed the web-based data collection instrument (see Attachment D – Telephone Interview Guide). Respondents who have already completed the web-based data collection instrument were chosen because they have demonstrated a willingness to provide feedback and their feedback will provide greater context to data analysis for a specific NOFO. The interviews will provide CDC with detailed examples in a NOFO that respondents felt affected the clarity, consistency, and organization of the application instructions and project information.

Items of Information to be Collected

The web-based data collection instrument consists of 15 main questions of various types, including dichotomous (yes/no), multiple response, interval (rating scales), and open-ended text entry. The instrument will collect data on the following:

Applicant characteristics and eligibility for the data collection (e.g., which NOFO the respondent applied to, whether respondent was substantially involved in the development and writing of their agency’s application for a NOFO)
NOFO-specific feedback on the clarity and consistency of the application instructions throughout the NOFO and alignment with review criteria
NOFO-specific feedback on the clarity and consistency of the project information throughout the NOFO and alignment with review criteria
Feedback on the overall structure of the NOFO

The telephone-based semi-structured interview guide consists of 18 open-ended main questions and additional probing questions. The instrument will collect data on the following:

Applicant characteristics (e.g., how many NOFOs the respondent has applied to, role of the respondent in the health department)
Application development business processes respondents use, including any specific methods for organizing application requirements
NOFO-specific feedback on the clarity and consistency of the application instructions for each required section of an application
NOFO-specific feedback on the clarity and consistency of project information (e.g., strategies and activities, evaluation and performance measurements, organizational capacity)
Feedback on the overall structure of the NOFO

Purpose and Use of the Information Collection

The data will be used to identify common barriers applicants face when responding to NOFOs and areas of opportunities for improving the current NOFO development process. The data will also assist identifying NOFOs that are rated highly for clarity and consistency, and these NOFOs will be analyzed to identify common traits and shared as examples of high-quality NOFOs to CDC programs developing a NOFO. The results from this information collection will provide valuable input into the development of a redesigned NOFO template and revised guidance language, resources, and training materials for CDC programs. The results will also be used for the development of new resources for STLT health departments interested in funding on how to apply to a CDC NOFO.

Use of Improved Information Technology and Burden Reduction

Data will be collected via a web-based data collection instrument. This method was chosen to reduce the overall burden on respondents because it allows respondents to complete and submit their responses electronically. By using a web-based instrument, PPEO can collect and analyze feedback on the NOFOs from applicants throughout the country and territories. The web-based data collection instrument was designed to collect the minimum information necessary for the purposes of this project (i.e., limited to 15 questions).

In addition to the web-based data collection instrument, data will be collected using telephone-based semi-structured interviews. This method was chosen to provide more insight into the feedback collected through the web-based instrument, and allow for more detailed information that could not be effectively obtained in a web-based instrument. The interview guide for the telephone-based semi-structured interviews was designed to collect the minimum information necessary for the purposes of this project (i.e., limited to 18 questions).

Efforts to Identify Duplication and Use of Similar Information

To our knowledge, no prior comprehensive assessment on CDC NOFOs has been conducted among STLT health departments nationwide. Relevant work completed in this area includes four non-research NOFO-related questions shared with a limited number of funded recipients in a larger data collection instrument in August through October 2012 (Assessment of State, Tribal, Local and Territorial Grantees’ Technical Assistance Needs and Support from CDC Project Officers, OMB No. 0920-0879). However this information reflected only the sampled recipients’ feedback, and feedback on NOFOs published prior to the redesign project. A qualitative assessment from select applicants for research NOFOs managed by one office in CDC was conducted in March 2015 (Assessing Customer Satisfaction with the Extramural Research Program Services Office Serving NCCDPHP and NCBDDD, OMB No. 0920-0919). Research NOFOs are fundamentally different in structure, application, and review process from non-research NOFOs. PPEO is one of few offices at CDC that is significantly involved in the development of non-research NOFOs agency-wide. Feedback on NOFOs collected through this assessment and from this audience is not available from other data sources.

Prior to developing this information collection, PPEO consulted with CDC’s OGS and Center for State, Tribal, Local, and Territorial Support (CSTLTS), to confirm that this effort is not duplicative. PPEO also consulted with the Association for State and Territorial Health Officials (ASTHO), a national nonprofit organization representing public health agencies in the United States, the U.S. Territories, and the District of Columbia.

Impact on Small Businesses or Other Small Entities

No small businesses will be involved in this information collection.

Consequences of Collecting the Information Less Frequently

This request is for a one time data collection. There are no legal obstacles to reduce the burden. If no data are collected, CDC will be unable to:

Assess how well applicants understand the program’s expectations as determined in the NOFO and the appropriate information and requirements an application is expected to address
Assess the usefulness of PPEO’s NOFO writing guidance to CDC programs
Understand common barriers applicants face when responding to NOFOs
Identify areas of opportunities for improving existing NOFO content
Share feedback and examples of high-quality NOFOs with CDC programs developing NOFOs
Make fully informed changes and adjustments to the NOFO template and NOFO development process that includes all stakeholder perspectives

Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances with this data collection package. This request fully complies with the regulation 5 CFR 1320.5 and will be voluntary.

Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

This data collection is being conducted using the Generic Information Collection mechanism of the OSTLTS OMB Clearance Center (O2C2) – OMB No. 0920-0879. A 60-day Federal Register Notice was published in the Federal Register on April 27, 2017, Vol. 82, No. 80, pp 19371-19373. One non-substantive comment was received. CDC sent forward the standard CDC response.

CDC partners with professional STLT organizations, such as the Association of State and Territorial Health Officials (ASTHO), the National Association of County and City Health Officials (NACCHO), and the National Association of Local Boards of Health (NALBOH) along with the National Center for Health Statistics (NCHS) to ensure that the collection requests under individual ICs are not in conflict with collections they have or will have in the field within the same timeframe.

Explanation of Any Payment or Gift to Respondents

CDC will not provide payments or gifts to respondents.

Protection of the Privacy and Confidentiality of Information Provided by Respondents

The Privacy Act does not apply to this data collection. STLT governmental staff and / or delegates will be speaking from their official roles.

Institutional Review Board (IRB) and Justification for Sensitive Questions

No information will be collected that are of personal or sensitive nature. This data collection is not research involving human subjects.

Estimates of Annualized Burden Hours and Costs

The estimate for burden hours for both data collection instruments is based on pilot tests by 7 public health professionals. Of this group, 4 participated in web-based data collection instrument pilot and all 7 participated in the telephone-based semi-structured interview pilot.

In the pilot test of the web assessment, the average time to complete the instrument including time for reviewing instructions, gathering needed information and completing the instrument, was approximately 20 minutes (range: 10 –25). For the purposes of estimating burden hours, the upper limit of this range (i.e., 25 minutes) is used.

In the pilot test of the semi-structured interview, the average time to complete the instrument including time for reviewing instructions, gathering needed information and completing the instrument, was approximately 55 minutes (range: 50 – 60). For the purposes of estimating burden hours, the upper limit of this range (i.e., 60 minutes) is used.

Estimates for the average hourly wage for respondents are based on the Department of Labor (DOL) Bureau of Labor Statistics for occupational employment for Medical and Health Services Managers (http://www.bls.gov/oes/current/oes_nat.htm). Based on DOL data, an average hourly wage of $53.69 is estimated for all 755 respondents. Table A-12 shows estimated burden and cost information.

There will be a total of 755 respondents and 775 responses.

Table A-12: Estimated Annualized Burden Hours and Costs to Respondents

Data collection Instrument: Form Name	Type of Respondent	No. of Respondents	No. of Responses per Respondent	Average Burden per Response (in hours)	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
Web-Based Data Collection Instrument	STLT health department program directors	755	1	25 / 60	315	$53.69	$16,912
Telephone-Based Semi-Structured Interviews	STLT health department program directors	20 (of the 755 total number of respondents)	1	60 / 60	20	$53.69	$1,074
	TOTALS	775	1		335		$17,986

Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers

There will be no direct costs to the respondents other than their time to participate in each data collection.

Annualized Cost to the Government

There are no equipment or overhead costs. The only cost to the federal government would be the salary of CDC staff to develop the data collection instrument, collect data, and perform data analysis. The total estimated cost to the federal government is $16,005.00. Table A-14 describes how this cost estimate was calculated.

Table A-14: Estimated Annualized Cost to the Federal Government

Staff (FTE)	Average Hours per Collection	Average Hourly Rate	Total Average Cost
Public Health Analyst – GS-12, Step 4; Instrument development, pilot testing, OMB package preparation, data collection, data coding and entry, quality control, data analysis, and report preparation	300	$40.62 /hour	$12,186.00
ORISE Fellow – (GS-9, equivalent); Assist with data collection, perform data analysis, support summary reports/presentation writing	150	$25.46 / hour	$3,819.00
Estimated Total Cost of Information Collection			$16,005.00

Explanation for Program Changes or Adjustments

This is a new data collection.

Plans for Tabulation and Publication and Project Time Schedule

Data from the web-based data collection instrument will be exported from SurveyMonkey into an Excel spreadsheet and stored on a secure drive on the CDC network that is only accessible to project members. Notes from the interviews will also be stored on a secure drive on the CDC network that is only accessible to project members.

Results will be analyzed using a mixture of quantitative and qualitative methods. Quantitative analysis of the results from the web-based instrument will use descriptive statistics to determine frequency distributions. Qualitative and thematic analysis will be performed on the open-ended questions from the web-based instrument and the notes from the semi-structured interviews. Responses will be cross-tabulated to compare similarities and differences in clarity, consistency, and organization between different NOFOs. Results may be cross-tabulated in other ways to identify response similarities and differences among sub-groups of respondents, such as funded and unfunded applicants.

Resulting data will be reported in aggregate form only. Reports will be shared with CDC leadership and staff across the agency that work on non-research, domestic NOFO development. The results will identify common barriers applicants face when responding to NOFOs and areas of opportunities for improving the current NOFO development process. The results will provide valuable input into the development of a redesigned NOFO template and revised guidance language, resources, and training materials for CDC programs and applicants.

Project Time Schedule

Design instruments (COMPLETE)

Develop protocol, instructions, and analysis plan (COMPLETE)

Pilot test instruments (COMPLETE)
Prepare OMB package (COMPLETE)
Submit OMB package (COMPLETE)

OMB approval (TBD)
Conduct data collection (Open 6 weeks)
Code data, conduct quality control, and analyze data (2 weeks)
Prepare summary report(s) (2 weeks)
Disseminate results/reports (1 week)

Reason(s) Display of OMB Expiration Date is Inappropriate

We are requesting no exemption.

Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification. These activities comply with the requirements in 5 CFR 1320.9.

LIST OF ATTACHMENTS – Section A

Attachment A – Respondent Breakdown

Attachment B – Web-Based Data Collection Instrument Word Version

Attachment C – Web-Based Data Collection Instrument PDF Version

Attachment D – Telephone Interview Guide

REFERENCE LIST

Centers for Disease Control and Prevention (CDC). "National Public Health Performance Standards Program (NPHPSP): 10 Essential Public Health Services." Available at http://www.cdc.gov/nphpsp/essentialservices.html. Accessed on 8/14/14.
Based on an analysis of data from the CDC financial system of record (UFMS).
Protection of Human Subjects, 45 CFR §46.102 (2017).
Centers for Disease Control and Prevention (CDC). "Distinguishing Public Health Research and Public Health Nonresearch." Available at https://www.cdc.gov/od/science/integrity/docs/cdc-policy-distinguishing-public-health-research-nonresearch.pdf. Accessed on 10/25/18.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	0000-00-00