Justification for Submission

DI-4010 (Rev. 07/2018)
U.S. Department of the Interior

OMB Control. No. 1040-0001
Expiration Date 10/31/2021

JUSTIFICATION FOR SUBMISSION UNDER THE
“DOI PROGRAMMATIC CLEARANCE FOR
CUSTOMER SATISFACTION SURVEYS”

See Page 5 for Instructions on Completing This Form
1. Bureau/Office

2. Date Submitted

U.S. Department of the Interior – U.S. Fish and Wildlife Service

May 5, 2020

3. Survey Title

John Heinz National Wildlife Refuge at Tinicum "Youth Crew Program Survey"
4. Abstract (Not to exceed 150 words)

The purpose of this survey is to better understand the impacts of the Youth Crew Program (“the Program”) at John
Heinz National Wildlife Refuge at Tinicum (“the Refuge”) on program participants, and to use this information to
improve the Program for future sessions. The Program survey will assess: 1) students’ interest in and reasons for
participating in the program; 2) students’ overall satisfaction with the program experience; and 3) students’ awareness
and interest in further engagement in Refuge events and activities.
The survey will provide Refuge managers and planners with scientifically sound data that can be used to: 1) Improve
the design of the Program to increase student participation and engagement; 2) Tailor the Program to match the wants
and interests of high school student participants; and 3) Inform strategies to better engage students in events and
programs offered at the Refuge.

5. Bureau/Office Point-of-Contact Information
First Name

Last Name

Lamar

Gore

Title

Bureau/Office

Refuge Manager, John Heinz NWR at Tinicum

U.S. Fish and Wildlife Service, John Heinz NWR at Tinicum

Mailing Address

City

8601 Lindbergh Boulevard

Philadelphia

Phone

■

215-365-3118; Ex: 3201

Work
Cell

Fax
215-365-2846

State
PA

Last Name
Gallagher, Ph.D.

Title
Director of Research & Evaluation

Bureau/Office
ImpactED at the University of Pennsylvania

Mailing Address
3440 Market Street, Suite 450

City
Philadelphia
Work
Cell

Fax

19153

Email
lamar_gore@fws.gov

6. Principal Investigation (PI) Point-of-Contact Information
First Name
Nina

Phone
215-410-0960

Zip Code

State
PA

Zip Code
19104

Email
ninahoe@sas.upenn.edu
Page 1 of 6

DI-4010 (Rev. 07/2018)
U.S. Department of the Interior

OMB Control. No. 1040-0001
Expiration Date 10/31/2021

7. Name of Program or Office Conducting Survey

John Heinz NWR at Tinicum
8. Description of Customers and Services Provided
The Youth Crew Program provides an opportunity for high school students to learn more about outdoor conservation work and education, collaborating in teams to complete
conservation projects around the Refuge, including improvements to hiking trails, wildlife habitat, and visitor infrastructure, and to participate in environmental education and outdoor
recreation activities led by Youth Crew team leaders with Mobilize Green and U.S. Fish and Wildlife Service Refuge staff. Youth Crew Program participants include currently enrolled
high school students (between the ages of 15 and 18 during the program dates) living in the Philadelphia region. Participants have applied for and successfully been accepted to the
Youth Crew Program.

9. Survey Dates
(mm/dd/yyyy)

July 2020

(mm/dd/yyyy)
August 2021

to

10. Type of Information Collection Instrument (Check ALL that Apply)
Intercept

Telephone

Mail

Web-based

Comment Cards

Other: (Explain)
Focus Groups
11. Survey Development (Who assisted in survey content development statistics? Was the survey pretested? How did you
integrate improvements? Which of the six topic areas did you address?)
The pre- and post-surveys were developed by researchers at the University of Pennsylvania, Nina Gallagher and Alexandra Bird Becker, in consultation with U.S. Fish and Wildlife staff at the John Heinz National Wildlife Refuge. Team members include: Lamar Gore Refuge Manager, Brianna Amingwa - Environmental Education Supervisor, and Kelly Kemmerle - Environmental Education Specialist.
The team convened numerous meetings in order to develop the survey method, discuss requirements, and to design and edit the survey instruments. The surveys were designed to align to topic area 1: Delivery, quality, and value of products, information, and
services. The questions focus on customer (i.e., program participant) satisfaction with their experience in the Refuge Youth Crew Program. Specifically, questions are aimed at collecting information on the quality and usefulness of the program.
The team used feedback from survey pre-testing efforts as well as input from the U.S. Fish and Wildlife Service staff at the Refuge to improve the survey design and questions included. The survey was tested with a group of Youth Crew Program participants in
December 2019.

12. Survey Methodology (Use as much space as needed; if necessary, include additional explanation on separate page.)
12A. Respondent Universe
The respondent universe includes Youth Crew Program participants, who are 15 to 18-year-old currently-enrolled high school students
participating in the Youth Crew Program at the John Heinz National Wildlife Refuge, which runs in the Fall, Spring, and Summer.

12B. Sampling Plan/Procedure
The sampling method will be a census, in that the team aims to collect survey responses from all Youth Crew Program Participants. The survey will be administered
in-person to program participants. All program participants who are present at the time the survey is administered will have an equal opportunity to participate.

12C. Instrument Administration
For the on-site survey administration, the Principal Investigator and trained interns will introduce the survey to Program participants on the first and last day of each Youth Crew Program session, and distribute physical copies of the survey to each participant to be completed by hand.
Survey interns will administer physical copies of the Youth Crew Program Survey to participants on-site at the Refuge. All Youth Crew Program participants present at the time will be given the in-person survey to complete.
Survey interns will be trained to ensure that they are using the same methods and understand the project objectives in administering the Youth Crew Program Survey. In addition, training will address any questions that the survey aides may have. Project leads will also be available to answer questions
that arise, arrange for unannounced spot checks on the surveying procedure, and in general ensure survey aides are following protocol.

12D. Expected Response Rate and Confidence Levels
The expected response rate on surveys administered to participants is 95-100%. While the goal is to survey responses from all program participants during each data collection period, survey response rates will be dependent upon the
number of participants involved in each Youth Crew Program session, as well as the number of participants who are present at the time of survey administration. The expected response rate from the total population (~60) is 90% for each
Youth Crew session, given the administration of the survey on-site at the Refuge, at a time when Program participants are already present for a Youth Crew Program session. This will give us a confidence level of 95%.

12E. Strategies for dealing with potential non-response bias
While administering the survey on-site, the Principal Investigator and/or survey interns will have refusal conversations with any participant who verbally refuses to complete the
survey. The conversations will be brief, and will consist of an inquiry as to why the participant is refusing and 1-2 of the survey questions focused specifically on participant
satisfaction. Interns will also plan to administer the survey to any participants absent at the next available opportunity.

12F. Description of any pre-testing and peer review of the methods and/or instrument (recommended)
The survey was developed by researchers at the University of Pennsylvania in consultation with the U.S. Fish and Wildlife Service staff at the John Heinz National Wildlife Refuge. The team convened numerous meetings to discuss data requirements and priorities, develop the survey method, and to
design the survey instrument. The University of Pennsylvania researchers provided a rigorous review process, and the survey underwent multiple rounds of revisions and iterations based upon input received.
The team used Feedback from survey pre-testing efforts, as well as input from the U.S. Fish and Wildlife Service staff at the Refuge to improve the survey design and questions included. The survey was tested with a group of Youth Crew Program participants in December 2019. On average, it took the
students who pretested the survey between 5 and 8 minutes to complete it. The pretested surveys were inspected by the research team to ensure that pre-testers answered all relevant questions and recorded their answers correctly. Such findings validate the survey design and response questions
included in the survey instrument submission.

13. Burden Hours Calculations
Category of Respondent

Number of Annual
Respondents

Number of
Responses Each

Total Annual
Responses

Time per
Response

Total Burden
Hours

Initial Contact
Completion of Survey Instrument

54

Totals: 54

2

108
108

10 mins

18 hrs
18 hrs

Page 2 of 6

DI-4010 (Rev. 07/2018)
U.S. Department of the Interior

OMB Control. No. 1040-0001
Expiration Date 10/31/2021

14. Federal Enterprise Architecture (FEA) Business Reference Model (Check only one “Line of Business” and one
“Subfunction.” Refer to OMB guidance “FEA Consolidated Reference Model Document Version 2.3”)
Line of Business
Subfunction
Line of Business
Subfunction
Community and
Social Services
Defense and
National Security
Economic
Development

(Select One)

Correctional
Activities
Disaster
Management

(Select One)

(Select One)

Education

(Select One)

Energy

(Select One)

Environmental
Management

(Select One)

General Science
and Innovation

(Select One)

Health

(Select One)

Homeland Security

(Select One)

(Select One)

(Select One)

Income Security

(Select One)

Intelligence
Operations

(Select One)

(Select One)

Law Enforcement

(Select One)

International Affairs
and Commerce
Litigation and
Judicial Activities

Natural Resources

Conservation, Marine and Land ManagementTransportation

Workforce
Management
15. Reporting Plan

(Select One)

(Select One)

(Select One)

A presentation will be made to explain the survey findings and their implications to the U.S. Fish and Wildlife Service Staff at the John Heinz National Wildlife Refuge. A separate report detailing the key findings of the
survey and associated recommendations for improvement will be developed and submitted to the same group.
Refuge staff will share results with others in the FWS Urban Wildlife Conservation Program, regional and national leadership, external partners and collaborator, and others interested FWS programs and offices.

16. Justification, Purpose, and Use
16A. Survey Justification and Purpose
The survey will provide Refuge managers and planners with scientifically sound data that can be used to: 1) Improve the design of the Program to increase
student participation and engagement; 2) Tailor the Program to match the wants and interests of high school student participants; and 3) Inform strategies to
better engage students in events and programs offered at the Refuge.

16B. Survey Goals
The goal of the pre-/post surveys is to collect information from Youth Crew Program participants at the John Heinz National Wildlife Refuge, specifically impacts of
the Youth Crew Program on: 1) students’ interest in and reasons for participating in the program; 2) students’ overall satisfaction with the program experience;
and 3) students’ awareness and interest in further engagement in Refuge events and activities.

16C. Utility to Managers
Survey data will provide Refuge managers, planners, and U.S. Fish and Wildlife Service staff with information that can be used to: 1) Prepare Youth Crew
Program planning and evaluation documents; 2) Determine if Program goals are being met; and 3) Improve the design of the Program for future sessions.

16D. How will the results of the survey be analyzed and used?
All data will be collected through in-person survey administration will be entered into the Qualtrics survey platform and then downloaded into a data summary report for analysis. All data will be stored in both electronic and hard copy, and will adhere to data
management procedures as required by the Federal Government. Overall frequencies of survey results will be analyzed, consisting of frequency of distributions of responses, and means of responses as appropriate. For instance, a question asking respondents to
select which Refuge activities they have previously participated in will result in categorical data, and will be analyzed as a percentage of respondents selecting each activity. A question that asks participants to rate their interest in or awareness of a conservation topic
area on a 4-point scale can utilize parametric statistics such as means. Frequencies or means will be reported for all questions in the survey. The paired-samples T-test will be used to determine if there are significant differences between pre- and post-survey results.
Relationships between variables will also be examined as appropriate, including satisfaction with the program, awareness of Refuge events and activities, and conservation career interests with facilities by demographic variables such as age or zip code.

16E. How will the data be tabulated? How What Statistical Techniques will be used to generalize the results to the entire
customer population? How will limitations on use of data be handled? If the survey results in a lower than anticipated
response rate, how will you address this when reporting the results? (Use as much space as needed; if necessary, include
additional explanation on separate page.)
Frequencies of survey data will be tabulated, as described in the previous question. Statistical tests used to generalize the results to the entire population include
ANOVA and t-tests when the variable being compared is interval or ratio data. Confidence intervals will be reported. Any limitations on use of the data will be
noted in the data codebook and in the survey report. Appropriate confidence intervals will be reported, based on sample size.

16F. Is this survey intended to measure a Government Performance and Results Act (GPRA) performance measure? If
yes, please include an excerpt from the appropriate document. (Use as much space as needed; if necessary, include additional
explanation on separate page.)
No, this survey is not intended to measure a GPRA performance measure.

Page 3 of 6

DI-4010 (Rev. 07/2018)
U.S. Department of the Interior

OMB Control. No. 1040-0001
Expiration Date 10/31/2021

17. Federal Cost: (Consult your Bureau/Office Information Collection Clearance Officer for assistance, if necessary)
The estimated annual cost to the Federal government is $ 13,185.00

, based on: (provide details below)

The annual cost to the Federal government is based on the contracted amount with ImpactED for data collection and analysis.

Sample Response:
“If we receive 20 submissions and it takes 30 minutes to process and implement each one, then the total burden is $322.40
assuming a GS-7 step 5 is processing the submissions. This custom form is a tool meant to accept submissions in a standard format
rather than through the freeform submissions that would otherwise come in by personal email. The existence of this form actually
saves the government money by standardizing submissions and decreasing the workload of processing each one.”
18. The survey methodology presented in form DI-4010 includes a specific description of:
The respondent universe,
The sampling plan and all sampling procedures, including how respondents will be selected,
How the instrument will be administered,
Expected response rate and confidence levels,
Strategies for dealing with potential non-response bias,
A description of any pre-testing and peer review of the methods and/or the instrument is highly recommended,
The burden hours reported in the Justification include the number of burden hours associated with the initial contact of all
individuals in the sample (i.e., including refusals), if applicable, and the number of burden hours associated with individuals
expected to complete the survey instrument, and
The package is properly formatted (using MS Word) and submitted to the Office of Policy Analysis (through the Bureau/Office
Information Collection Clearance Officer).
19. The approval package includes:
A completed and signed Form DI-4010, Generic Clearance for Customer Satisfaction Surveys.
A copy of the survey instrument.
Other supporting materials, such as:
• Cover letters to accompany mail-back questionnaires,
• Introductory scripts for initial contact of respondents,
• Necessary Paperwork Reduction Act and Estimated Burden compliance language, and/or
• Follow-up letters/reminders sent to respondents.
20. Checklist for Submitting a Request to Use DOI Programmatic Clearance for Customer Satisfaction Surveys
All questions in the survey instrument are within the scope of one of the DOI Programmatic Clearance for Customer
Satisfaction Surveys topic areas.
A qualified statistician has reviewed and approved your request (see question 21A).
Your Bureau/Office Information Collection Clearance Officer receives your package for review/approval at least 75 days
prior to the first day the PI wishes to administer the survey to the public.
21. Required Certifications for Submission Under OMB Control Number 1040-0001
Certification is required to submit a collection of information for approval under the DOI Programmatic Clearance for Customer
Satisfaction Surveys. If the collection does not satisfy the requirements of the Programmatic Clearance, you should follow the
regular PRA clearance procedures described in 5 CFR 1320.
21A. Bureau/Office Statistician
Signature
Date
■

Recommend

Not Recommended

DANIELLE ROSS-WINSLOW

21B. Bureau/Office Program or Subgroup

Bureau/Office Point-of-Contact

USFWS

John Heinz NWR at Tinicum

Title (Please be specific)

Digitally signed by DANIELLE ROSS-WINSLOW
Date: 2020.04.30 15:13:49 -06'00'

Signature

LAMAR GORE

Refuge Manager

Date
Digitally signed by LAMAR GORE
Date: 2020.05.05 11:25:55 -04'00'

FOR PROGRAM USE ONLY
Required certifications: The information collection requested by this submission meets the requirements of OMB Control No. 1040-0001

Bureau/Office ICCO
Recommend

Signature
Not Recommended

DOI Office of Policy Analysis
Recommend
Approved

Signature

Not Recommended

DOI PRA Program Lead
Not Approved

MADONNA BAUCUM

Date
Digitally signed by MADONNA BAUCUM
Date: 2020.05.06 08:51:43 -04'00'

DOI Tracking No.

BENJAMIN SIMON

Date
Digitally signed by BENJAMIN SIMON
Date: 2020.05.21 09:41:43 -04'00'

Signature

2020-FWS-CS7 JEFFREY PARRILLO

Date
Digitally signed by JEFFREY PARRILLO
Date: 2020.05.27 12:37:59 -04'00'

Page 4 of 6

DI-4010 (Rev. 07/2018)
U.S. Department of the Interior

OMB Control. No. 1040-0001
Expiration Date 10/31/2021

Instructions for Completing Form DI-4010,
Justification for Submission Under the
“DOI Programmatic Clearance for Customer Satisfaction Surveys”
OMB Control Number 1040-0001
1.

Bureau/Office: Insert the name of the bureau/office conducting the survey.

2.

Date Submitted: Date you submit the package to the Bureau/Office Information Collection Clearance Officer (ICCO) for review.

3.

Survey Title: Insert title for the proposed survey.

4.

Abstract: Summarize the proposed study with an abstract not to exceed 150 words.

5.

Bureau/Office Point of Contact Information: Complete the bureau/office contact information. PPA will communicate with the
point of contact listed here throughout the entire approval process.

6.

Principal Investigator (PI) Conducting the Survey: Complete information about the PI who will be conducting the survey, if
different from Point of Contact listed in #4. Otherwise note: Same as #4.

7.

Name of Program Office Conducting Survey: Provide the name of the bureau program, office, or organizational unit conducting
the survey.

8.

Description of Customers and Services Provided: Provide a brief description of the customers you will survey, the services
provided by the program conducting the survey, and customers receive these services.

9.

Survey Dates: List the time-period in which you will conduct the survey, including specific starting and ending dates. The starting
date should be at least 75 days after the date you submit the package to your bureau/office Information Collection Clearance
Officer (ICCO).

10. Type of Information Collection Instrument: Check the type(s) of information collection instrument(s) you will use. If other,
please explain.
11. Survey Development: Explain how the survey was developed. With whom did you consult during the development of the survey
on content? On statistics? Did you pretest the survey? What actions did you take to improve the survey? What suggestions did
you receive for improving the survey? Which of the six topic areas will the collection address? (Note: A description of any pretesting and peer review of the methods and/or instrument is highly recommended.)
12. Survey Methodology: Explain how you will conduct the survey. Provide a description of the survey methodology including:
•
Question 12A – The respondent universe,
•
Question 12B – The sampling plan and all sampling procedures;
•
Question 12C – How the instrument will be administered;
•
Question 12D – Expected response rate and confidence levels;
•
Question 12E – Strategies for dealing with potential non-response bias; and,
•
Question 12A – Description of any pre-testing and peer review of the methods and/or instrument (recommended, but not
required).
Note: Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to
services provided by the web site.
13. Burden Hours Calculations: Provide an estimated total of the following for each category – initial contact and completion of
survey instrument:
•
Number of annual respondents – Enter the number of unique respondents who will complete the information collection;
•
Number of responses per respondent – Enter the total number of responses per unique respondent;
•
Total annual responses – Enter the number of unique respondents multiplied by the total number of responses each;
•
Time per response – Estimate the time to complete the initial contact and the time to complete the survey instrument (in
minutes), and
•
Total burden hours –The total burden hours should account for the amount of time required to instruct the respondents in
completing the survey, and the amount of time required for the respondent to complete the survey.
14. Federal Enterprise Architecture (FEA) Business Reference Model: Using the drop-down menus provided, select ONE “Line of
Business” and ONE corresponding Subfunction that most accurately describes your information collection.
15. Reporting Plan: Provide a brief description of the reporting plan for the data you will collect.
Page 5 of 6

DI-4010 (Rev. 07/2018)
U.S. Department of the Interior

OMB Control. No. 1040-0001
Expiration Date 10/31/2021

16. Justification, Purpose and Use: For questions 16A through 16F, provide a brief justification for the survey, its purpose, goals,
and utility to managers. Specifically, describe how you will tabulate the data and what the statistical techniques you will use to
generalize the results to the entire customer population. Describe how you will use the data from the survey. Describe how you
will acknowledge any limitations related to the data, particularly in cases where we obtain a lower than anticipated response rate.
Note whether you intend the survey to measure a Government Performance and Results Act (GPRA) performance measure.
17. Federal Cost: Provide the cost estimate for the Federal government to administer the information collection, along with a
description of how you calculated the cost estimate (sample response provided). Contact your bureau/office ICCO for more
information or for assistance.
18. Survey Methodology Checklist: Carefully review each item and check each box to indicate your submission provides
the required description of each item.
19. Checklist for Submitting a Request to Use DOI Programmatic Clearance for Customer Satisfaction Surveys: Carefully
review each item and check each box to indicate your understanding and concurrence of each requirement.
20. Approval Package Content: Carefully review each item and check each box to indicate your package contains each of the
requirement elements listed.
NOTE: Your survey instrument document must show the OMB Control Number 1040-0001 and Expiration Date ##/##/###, and it
MUST include the following Statements somewhere on the instrument document (preferably at the bottom of page 1 or at the end
of the document):
Paperwork Reduction Act Statement: We are collecting this information subject to the Paperwork Reduction Act
(44 U.S.C. 3501) to [insert brief justification for collection of information]. Your response is voluntary and results we
will not share them publicly. We may not conduct or sponsor and you are not required to respond to a collection of
information unless it displays a currently valid OMB Control Number. OMB has reviewed and approved this survey
and assigned OMB Control Number 1040-0001, which expires ##/##/####.
Estimated Burden Statement: We estimate the [insert type of instrument] will take you ## minutes to complete,
including time to read instructions, gather information, and complete and submit your response. You may submit
comments on any aspect of this information collection to the Information Collection Clearance Officer, [Insert
Bureau/Office], [Insert mailing address].
21. Required Certifications: Completion of all information in this section is required before forwarding your approval
package to your bureau/office ICCO for review and processing.
Question 21A – Ensure the bureau/office statistician reviewing your information collection certifies the request satisfies
the requirements of the DOI Programmatic Clearance for Customer Satisfaction Surveys under OMB Control No. 10400001.
Question 21B – Ensure the requestor provides the requested contact information needed by the bureau/office and/or
Departmental ICCO to resolve questions or concerns.

Page 6 of 6