LSAMP Cognitive Testing - Fast-track Administrator, Faculty, Staff

NOTICE OF OFFICE OF MANAGEMENT AND BUDGET ACTION
Date

04/17/2024

National Science Foundation

FOR CERTIFYING OFFICIAL:

George Strawn

FOR CLEARANCE OFFICER:

Suzanne Plimpton

In accordance with the Paperwork Reduction Act, OMB has taken action on your request received
04/09/2024
ACTION REQUESTED: Generic IC
Louis Stokes Alliances for Minority Participation (LSAMP) – Cognitive Pilot Testing of LSAMP
IC TITLE:
Administrator, Faculty, Staff Interview Protocol
ICR REFERENCE NUMBER:
202307-3145-001
AGENCY ICR TRACKING NUMBER:
TITLE:

Generic Clearance for the Collection of Qualitative Feedback on Agency Service Delivery

LIST OF INFORMATION COLLECTIONS: See next page
OMB ACTION: Approved without change
OMB CONTROL NUMBER:

3145-0215

The agency is required to display the OMB Control Number and inform respondents of its legal significance in
accordance with 5 CFR 1320.5(b).
EXPIRATION DATE: 08/31/2026

BURDEN:

DISCONTINUE DATE:

RESPONSES

HOURS

COSTS

Previous

100,000

25,000

75,000

New

100,000

25,000

75,000

Change due to New Statute

0

0

0

Change due to Agency Discretion

0

0

0

Change due to Agency Adjustment

0

0

0

Change due to PRA Violation

0

0

0

Difference

TERMS OF CLEARANCE:

Terms of the Generic ICR remain in effect.

OMB Authorizing Official:

Dominic J. Mancini
Deputy Administrator,
Office Of Information And Regulatory Affairs

Request for Approval under the “Generic Clearance for the Collection of Routine Customer
Feedback” (OMB Control Number: 3145-0215)

TITLE OF INFORMATION COLLECTION:
Louis Stokes Alliances for Minority Participation (LSAMP) – Cognitive Pilot Testing of LSAMP
Administrator, Faculty, Staff Interview Protocol
PURPOSE:

The LSAMP program, a Congressionally mandated program established in 1991 to assist the nation in
diversifying its STEM workforce, began conducting a program evaluation in 2023. In addition to the
interviews discussed in this request, the evaluation includes a document review of annual reports from
LSAMP alliances as well as quantitative analysis of LSAMP program monitoring data and student
outcome data on the institutional (DoED IPEDS) and individual level (National Student Clearinghouse).
The interviews discussed in this request will involve LSAMP alliance staff (i.e., university faculty, staff,
and high-level administrators such as provosts/presidents) at grantee institutions. The focus of this request
is to conduct cognitive testing of the interview protocol with up to 18 interviews in 6 alliances.
Cognitive testing of administrators and staff involves interviewing individuals who have led or
implemented LSAMP supported activities on university campuses. The purpose of the cognitive testing is
to gather feedback on the content, flow, instructions, and length of time to respond to the interview
questions. Cognitive testing will be conducted virtually to provide feedback on question understanding
and appropriateness of questions. Virtual facilitators will verbally guide respondents through the protocol
to receive real-time feedback on the instrument. Additionally, the protocol will be tested for flow and
appropriateness of question wording about creating/implementing or leading LSAMP alliance activities.
The results will be used to improve the set-up and format of the administrator/staff interview protocol.
Data from the cognitive testing will not be used for analysis purposes in the overall evaluation.
DESCRIPTION OF RESPONDENTS:
A maximum of 18 administrative, faculty, or staff level respondents will be invited from six LSAMP
alliances to participate in the cognitive interview pilot testing. All administrators, faculty, or staff have
experience designing and implementing the LSAMP alliance activities.
TYPE OF COLLECTION: (Check one)
[] Customer Comment Card/Complaint Form
[] Customer Satisfaction Survey
[X] Cognitive Testing (e.g., Website or Software)[] Small Discussion Group
[] Focus Group
[] Other:
CERTIFICATION:
I certify the following to be true:
1. The collection is voluntary.
2. The collection is low-burden for respondents and low-cost for the Federal Government.
3. The collection is non-controversial and does not raise issues of concern to other federal agencies.
4. The results are not intended to be disseminated to the public.
5. Information gathered will not be used for the purpose of substantially informing influential policy
decisions.
6. The collection is targeted to the solicitation of opinions from respondents who have experience with
the program or may have experience with the program in the future.
Name: Suzanne H. Plimpton, NSF Reports Clearance Officer
To assist review, please provide answers to the following question:

1

Personally Identifiable Information:
1. Is personally identifiable information (PII) collected? [] Yes [X] No
2. If Yes, is the information that will be collected included in records that are subject to the Privacy Act
of 1974? [] Yes [X] No
3. If Applicable, has a System or Records Notice been published? [] Yes [X] No
Gifts or Payments:
Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to
participants? [] Yes [X] No
BURDEN HOURS
Category of Respondent
LSAMP Alliance University Level
Administrators (e.g., Provost, Dean or
Faculty PIs)
LSAMP Alliance Staff (e.g., Program
Directors, Project Managers)
Totals

No. of Respondents

Participation Time

Burden

9

60 minutes

9 hours

9

60 minutes

9 hours

18

60 minutes

18 hours

FEDERAL COST: It is incorporated in the formal agreement with the contractor.
If you are conducting a focus group, survey, or plan to employ statistical methods, please provide
answers to the following questions:
The selection of your targeted respondents
1. Do you have a customer list or something similar that defines the universe of potential respondents
and do you have a sampling plan for selecting from this universe?
[] Yes
[X] No
The respondents in this cognitive testing are LSAMP administrators, faculty, and staff at LSAMP
alliances. Information on administrators, faculty, and staff associated with LSAMP is available in both
NSF’s internal data systems as well as LSAMP’s program monitoring systems (WebAMP and LSAMPBD). Respondents will be selected by alliance and institution and their contact information extracted from
WebAMP, LSAMP BD, or NSF internal data systems. Alliances will be selected to ensure programmatic
and geographical diversity amongst the LSAMP community.
Administration of the Instrument
1. How will you collect the information? (Check all that apply)
[X] Web-based or other forms of Social Media
[X] Telephone (Interviews)
[] In-person
[] Mail
[] Other
2. Will interviewers or facilitators be used? [X] Yes [] No

2

Instructions for completing Request for Approval under the “Generic Clearance for the
Collection of Routine Customer Feedback”
TITLE OF INFORMATION COLLECTION: Provide the name of the collection that is the subject of
the request. (e.g. Comment card for soliciting feedback on xxxx)
PURPOSE: Provide a brief description of the purpose of this collection and how it will be used. If this
is part of a larger study or effort, please include this in your explanation.
DESCRIPTION OF RESPONDENTS: Provide a brief description of the targeted group or groups for
this collection of information. These groups must have experience with the program.
TYPE OF COLLECTION: Check one box. If you are requesting approval of other instruments under
the generic, you must complete a form for each instrument.
CERTIFICATION: Please read the certification carefully. If you incorrectly certify, the collection will
be returned as improperly submitted or it will be disapproved.
Personally Identifiable Information: Provide answers to the questions.
Gifts or Payments: If you answer yes to the question, please describe the incentive and provide a
justification for the amount.
BURDEN HOURS:
Category of Respondents: Identify who you expect the respondents to be in terms of the following
categories: (1) Individuals or Households;(2) Private Sector; (3) State, local, or tribal governments; or (4)
Federal Government. Only one type of respondent can be selected.
No. of Respondents: Provide an estimate of the Number of respondents.
Participation Time: Provide an estimate of the amount of time required for a respondent to participate
(e.g. fill out a survey or participate in a focus group)
Burden: Provide the Annual burden hours: Multiply the Number of responses and the participation time
and divide by 60.
FEDERAL COST: Provide an estimate of the annual cost to the Federal government.
If you are conducting a focus group, survey, or plan to employ statistical methods, please provide
answers to the following questions:
The selection of your targeted respondents. Please provide a description of how you plan to identify
your potential group of respondents and how you will select them. If the answer is yes, to the first
question, you may provide the sampling plan in an attachment.
Administration of the Instrument: Identify how the information will be collected. More than one box
may be checked. Indicate whether there will be interviewers (e.g. for surveys) or facilitators (e.g., for
focus groups) used.
Please make sure that all instruments, instructions, and scripts are submitted with the request.

3

LSAMP Program Evaluation1
NORC at the University of Chicago has been selected to conduct an evaluation of the
Louis Stokes Alliances for Minority Participation Program (LSAMP) on behalf of the
NSF. The data for this evaluation will be collected during the 2024-2025 academic year
and will occur in multiple phases.
Phase 1: Preparation and Testing
 Multiple one-hour cognitive interviews with program administrators and
students to evaluate our data collection tools (e.g., interview or survey
questions).
 NSF has selected the administrative individuals for participation to ensure
representation in feedback. These individuals will also be asked to refer
students for student-focused testing and feedback.

Phase 2: Data Collection
 Collect direct feedback (e.g., interviews, focus groups) from those engaged
with LSAMP programs (e.g., administrators, staff, and students).
 Participation by all selected individuals will be crucial to fully capture the
successes and opportunities of LSAMP alliances and institutions.
 Review information already available, including annual reports.

Phase 3: Analysis and Reporting
 Analyze all data to provide insights on the LSAMP programs.
 Provide reports with summary findings at the alliance level to program
administrators.
 Provide actionable insights and ‘lessons learned’ that may be helpful across
alliances.

For more information or to participate, contact [NSF Contact; NORC Contact].

This document will be shared with LSAMP alliances and institutions beginning in Spring 2024 to
provide initial information about the evaluation efforts. It will continue to be distributed as a quickreference FAQ sheet throughout data collection efforts.

1

Email/language inviting administrators, faculty, and staff for interview testing
Dear [insert name],
We are writing to ask for your help with an important study to help us better understand how
staff like you think about your work with the NSF-funded Louis Stokes Alliances for Minority
Participation (LSAMP) program. NORC at the University of Chicago is collaborating with the
NSF for this important work. We are currently developing our interview protocol for site visits to
select alliance sites, planned for the 2024-2025 academic year. We would like to speak with you
to hear your perspective on whether we are asking the right questions, and doing so in a way
that provides the best possible information for supporting LSAMP programming moving forward.
These conversations will take about one hour to complete and will occur via Zoom. The OMB
approval number for this activity is 3145-0215. You do not need to do anything to prepare for
our conversation; we are simply interested in your feedback and opinions given your role as
[insert role]. If you are interested and able to participate, please either email Justine BulgarMedina bulgarmedina-justine@norc.org, or you can self-schedule a one-hour time that is most
convenient for you [insert link].
If you have any questions or concerns, please do not hesitate to reach out.
Sincerely,
[NORC Contacts, NSF Contact]

OMB Data Collection Number 3145-0215, expires August 31, 2026.

Name:
The National Science Foundation (NSF) is leading this voluntary study of LSAMP activities at
participating alliance sites. NORC at the University of Chicago, a social science research
organization, is conducting this study on behalf of NSF. Today, we will be discussing our data
collection instruments with you. They are currently in development and your participation will
provide valuable feedback for finalizing our instruments. Your participation in this work is
confidential, and your responses and personal information will not be shared. There are no risks
in participating in this research beyond those experienced in everyday life. Participation is
voluntary, and you will be able to skip any question I ask you.
As a participant of the cognitive testing, you will provide feedback via a phone interview on the
following: content, flow, instructions, and length of time to complete the interview. The main
emphasis of the testing is to gauge your understanding of each question and assess your ability to
provide accurate responses. The results will be used to improve instructions, wording of
questions, and flow. Data from the pilot testing will not be used for analysis purposes.
PARTICIPATION AND WITHDRAWAL: You are under no obligation to participate, but
your feedback will be invaluable to the study. We strongly encourage you to consent. There are
no consequences or risks for participating. Refusal to participate will have no impact on current
NSF awards you may have or on your ability to obtain awards in the future.
AUDIO RECORDING: With your permission, your feedback will be recorded to assist in the
accurate presentation of your responses. Audio recordings will be protected on a secure server to
prevent inappropriate use.
CONTACTS: For additional information about the NSF study, you may contact Shelitita
Holland (sholland@nsf.gov), Martha James (mjames@nsf.gov) or LeRoy Jones
(ljones@nsf.gov).
If you have any questions about this interview, you may contact the Principal Investigator at
kim-debbie@norc.org.
If you have any questions about your rights as a research participant, you may contact the NORC
Institutional Review Board toll-free at 866-309-0542.

1
LSAMP – STAFF & ADMINISTRATOR INTERVIEW PROTOCOL – DRAFT
Relevant Study Research Questions
•
•

What pathways, structures, polices, and mechanisms (PSPMs) have LSAMP institutions implemented to recruit, support, and
retain highly competitive STEM undergraduate and graduate students?
To what extent have the PSPMs implemented by alliances influenced increases in the number of STEM degrees award to
LSAMP participants (Level 1 students) from underrepresented minority populations at LSAMP institutions?

Opening
Thanks for taking the time today to talk with me. My name is [your name]. I am with NORC at the University of Chicago. We are a
nonprofit social science research organization. We have been contracted by NSF to evaluate the LSAMP Program. We believe this
will take about an hour, and the OMB approval number for this is 3145-0215.
We are working to learn more about how LSAMP alliances institutionalize their programs and practices within their partner colleges
and universities. I have some questions for you today about ongoing work conducted by the [name of alliance] at [name of
institution].
Our conversation will be confidential. I have a consent form her for you to sign. To summarize, the consent form provides contact
information should you have any questions later and says that you should feel free to skip any questions you do not want to answer
and that you can end the interview at any time.
Is it ok if I audio record? It’ll help me make sure I accurately capture everything that you say. Can you (verbally) confirm that it’s okay
for us to proceed? Let’s begin.
[PSPM: policies structures programs mechanisms including [will update with real examples from the document review].
Section A: Introduction
[Goal: orient to person, so we know what questions to ask; how Respondent understands their role in LSAMP.]
1. Can you tell me a bit about your current position? How long have you been in it? What are your primary job responsibilities?
2. [If they did not describe LSAMP] What are your responsibilities directly related to LSAMP [at institution]? What does your
involvement with LSAMP look like? Can you provide an example?
3. [If they only describe LSAMP] What are your responsibilities outside of the LSAMP alliance?
Section B: Description of LSAMP Activities

2
Activities
1. (Must) Please describe your current LSAMP-related activities. What are the activities you do?
a. Have these changed over time? If so, how and why?
2. Please describe how your current LSAMP-related activities are executed.
a. Who is in charge? Who maintains? Etc.?
3. How did you pick these particular activities?
4. [If not mentioned] Tell us about the professional development component of your alliance?
How works get done
5. (Must) What types of meetings or other engagements do you attend as a member of LSAMP?
6. How do you interact with other institutions within your alliance?
a. Probe. Is communication by email, Teams, f2f? What frequency? Others?
7. How were the other institutions in your alliance selected?
8. Who is responsible for executing the LSAMP program here?
Institutionalization
9. (Must) How does the LSAMP work get done?
a. Note: This is not about the activities themselves, but more about the administrative work that supports it. We will want
to get at routines, both formal and informal.
b. Probe: Who is doing the work? What are their positions?
c. Probe: Are there any on-campus champions for the work? Who are they? How do they champion the work?
d. Have you received any guidance from NSF? If so, what is that guidance?
10. (Must: ask of PI) Are there particular positions at your institution dedicated to LSAMP-related work?
a. If so, what is the position and when was it created?
b. If not, how is the overall work of the LSAMP distributed across your staff?
c. How are they funded? (e.g, through LSAMP grant)
d. If the funding went away, would these positions still exist?
Student recruitment/retention
11. (Must) What does your student recruitment process look like?
a. Probe: how do you ensure you are receiving quality candidates?
12. (Must) What do you do to support student retention?

3
Section C: Contextual Understanding of LSAMP goals
[Goal: identify R’s understanding about LSAMP and the goals of the program.]
1. What was your institution’s motivation for creating an LSAMP program?
a. Probe: What types of students does LSAMP cater to?
i. What characteristics/elements do you think position students to succeed at your institution?
ii. What elements do you think are necessary for a student to succeed in STEM at this institution?
b. Probe: Were you involved in proposal planning for the project?
i. [If yes] What were your or the institution’s reasons for applying to the LSAMP program? Thinking back, what
were your expectations for having an LSAMP alliance at the time?
ii. Note: Get institution specific, alliance specific and national related reasons if they have them.
2. How does your LSAMP program contribute to your institution’s goals of enhancing access to STEM for all students?
a. Probe: How do you think this institution works for students of color in general? In other words, who does not participat
in LSAMP programming?
b. Note: in other words who does not participate in LSAMP program?
3. What kinds of additional supports do you feel your students need overall?
4. How does your LSAMP program work with other institutional initiatives?
a. Note: is this isolated effort or connected with other efforts on campus?

Section D: Impact of LSAMP Activities
1. What are some achievements you have had in LSAMP?
a. [If students not discussed, then] what have been some successes with students?
b. Probe: which activities have been most helpful in supporting students?
c. Are there students that these activities work better for than others? How do you know?
d. Has LSAMP increased the number of students completing a STEM degree?
2. (Optional) What successes are you most proud of?
3. What have been some of the challenges? How have you addressed them?
4. (Must) How do you obtain feedback on your LSAMP programming?
a. Probe: From students? From faculty?
b. Note: If they mention data/survey, request copies.
5. (Optional) What LSAMP-related activities do you think have been less helpful for supporting students? How do you know?
a. Note: Probe for differences between recruitment, support, and retention

4
b. Are there students that these activities are less helpful for than others? How do you know?
6. (Optional) If you could redo any part of the LSAMP design now, with the benefit of hindsight and experience, what would it
be? Why? What difference do you think this change would make and why?
Section E: Influence of the Alliance
1. (Must) How do you engage with LSAMP members at other institutions, if at all?
o Distinguish between students, staff, administration.
2. (Optional) If LSAMP funding goes away, what pieces of the program do you think would be sustained?
Section F: Wrap-up
1. Based on what we’ve talked about today and the types of questions we’ve been discussing, is there anything else you think I
should have asked that I didn’t?