LSAMP Cognitive Testing - Fast-track Administrators

NOTICE OF OFFICE OF MANAGEMENT AND BUDGET ACTION
Date

04/17/2024

National Science Foundation

FOR CERTIFYING OFFICIAL:

George Strawn

FOR CLEARANCE OFFICER:

Suzanne Plimpton

In accordance with the Paperwork Reduction Act, OMB has taken action on your request received
04/09/2024
ACTION REQUESTED: Generic IC
Louis Stokes Alliances for Minority Participation (LSAMP) – Cognitive Pilot Testing of LSAMP
IC TITLE:
Administrator Survey
ICR REFERENCE NUMBER:
202307-3145-001
AGENCY ICR TRACKING NUMBER:
TITLE:

Generic Clearance for the Collection of Qualitative Feedback on Agency Service Delivery

LIST OF INFORMATION COLLECTIONS: See next page
OMB ACTION: Approved without change
OMB CONTROL NUMBER:

3145-0215

The agency is required to display the OMB Control Number and inform respondents of its legal significance in
accordance with 5 CFR 1320.5(b).
EXPIRATION DATE: 08/31/2026

BURDEN:

DISCONTINUE DATE:

RESPONSES

HOURS

COSTS

Previous

100,000

25,000

75,000

New

100,000

25,000

75,000

Change due to New Statute

0

0

0

Change due to Agency Discretion

0

0

0

Change due to Agency Adjustment

0

0

0

Change due to PRA Violation

0

0

0

Difference

TERMS OF CLEARANCE:

Terms of the Generic ICR remain in effect.

OMB Authorizing Official:

Dominic J. Mancini
Deputy Administrator,
Office Of Information And Regulatory Affairs

Request for Approval under the “Generic Clearance for the Collection of Routine Customer
Feedback” (OMB Control Number: 3145-0215

TITLE OF INFORMATION COLLECTION:
Louis Stokes Alliances for Minority Participation (LSAMP) – Cognitive Pilot Testing of LSAMP
Administrator Survey
PURPOSE:

The LSAMP program, a Congressionally mandated program established in 1991 to assist the nation in
diversifying its STEM workforce, began conducting a program evaluation in 2023. In addition to the
surveys discussed in this request, the evaluation includes a document review of annual reports from
LSAMP alliances as well as quantitative analysis of LSAMP program monitoring data and student
outcome data on the institutional (DoED IPEDS) and individual level (National Student Clearinghouse).
The surveys discussed in this request will involve LSAMP alliance staff (i.e., university faculty, staff, and
high-level administrators such as provosts/presidents) at grantee institutions. The focus of this request is
to conduct cognitive testing of the survey with up to 18 surveys in 6 alliances.
Cognitive testing of administrators and staff involves surveying individuals who have led or implemented
LSAMP supported activities on university campuses. The purpose of the cognitive testing is to gather
feedback on the content, flow, instructions, and length of time to respond to the survey questions.
Cognitive testing will be conducted virtually to provide feedback on question understanding and
appropriateness of questions. Virtual facilitators will verbally guide respondents through the protocol to
receive real-time feedback on the instrument. Additionally, the protocol will be tested for flow and
appropriateness of question wording about creating/implementing or leading LSAMP alliance activities.
The results will be used to improve the set-up and format of the administrator/staff survey. Data from the
cognitive testing will not be used for analysis purposes in the overall evaluation.
DESCRIPTION OF RESPONDENTS:
A maximum of 18 faculty or staff level respondents will be invited from six LSAMP alliances to
participate in the cognitive survey pilot testing. All administrators, faculty, or staff have experience
designing and implementing the LSAMP alliance activities.
TYPE OF COLLECTION: (Check one)
[] Customer Comment Card/Complaint Form
[] Customer Satisfaction Survey
[X] Cognitive Testing (e.g., Website or Software)[] Small Discussion Group
[] Focus Group
[] Other:
CERTIFICATION:
I certify the following to be true:
1. The collection is voluntary.
2. The collection is low-burden for respondents and low-cost for the Federal Government.
3. The collection is non-controversial and does not raise issues of concern to other federal agencies.
4. The results are not intended to be disseminated to the public.
5. Information gathered will not be used for the purpose of substantially informing influential policy
decisions.
6. The collection is targeted to the solicitation of opinions from respondents who have experience with
the program or may have experience with the program in the future.
Name: Suzanne H. Plimpton, NSF Reports Clearance Officer
To assist review, please provide answers to the following question:

1

Personally Identifiable Information:
1. Is personally identifiable information (PII) collected? [] Yes [X] No
2. If Yes, is the information that will be collected included in records that are subject to the Privacy Act
of 1974? [] Yes [X] No
3. If Applicable, has a System or Records Notice been published? [] Yes [X] No
Gifts or Payments:
Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to
participants? [] Yes [X] No
BURDEN HOURS
Category of Respondent
LSAMP Alliance Staff (e.g., Program
Directors, Project Managers)
Totals

No. of Respondents

Participation Time

Burden

18

10 minutes

3 hours
3 hours

FEDERAL COST: It is incorporated in the formal agreement with the contractor.
If you are conducting a focus group, survey, or plan to employ statistical methods, please provide
answers to the following questions:
The selection of your targeted respondents
1. Do you have a customer list or something similar that defines the universe of potential respondents
and do you have a sampling plan for selecting from this universe?
[] Yes
[X] No
The respondents in this cognitive testing are LSAMP administrators, faculty, and staff at LSAMP
alliances. Information on administrators, faculty, and staff associated with LSAMP is available in both
NSF’s internal data systems as well as LSAMP’s program monitoring systems (WebAMP and LSAMPBD). Respondents will be selected by alliance and institution and their contact information extracted from
WebAMP, LSAMP BD, or NSF internal data systems. Alliances will be selected to ensure programmatic
and geographical diversity amongst the LSAMP community.
Administration of the Instrument
1. How will you collect the information? (Check all that apply)
[X] Web-based or other forms of Social Media
[X] Telephone (Surveys)
[] In-person
[] Mail
[] Other
2. Will surveyers or facilitators be used? [X] Yes [] No

2

LSAMP Program Evaluation1
NORC at the University of Chicago has been selected to conduct an evaluation of the
Louis Stokes Alliances for Minority Participation Program (LSAMP) on behalf of the
NSF. The data for this evaluation will be collected during the 2024-2025 academic year
and will occur in multiple phases.
Phase 1: Preparation and Testing
 Multiple one-hour cognitive interviews with program administrators and
students to evaluate our data collection tools (e.g., interview or survey
questions).
 NSF has selected the administrative individuals for participation to ensure
representation in feedback. These individuals will also be asked to refer
students for student-focused testing and feedback.

Phase 2: Data Collection
 Collect direct feedback (e.g., interviews, focus groups) from those engaged
with LSAMP programs (e.g., administrators, staff, and students).
 Participation by all selected individuals will be crucial to fully capture the
successes and opportunities of LSAMP alliances and institutions.
 Review information already available, including annual reports.

Phase 3: Analysis and Reporting
 Analyze all data to provide insights on the LSAMP programs.
 Provide reports with summary findings at the alliance level to program
administrators.
 Provide actionable insights and ‘lessons learned’ that may be helpful across
alliances.

For more information or to participate, contact [NSF Contact; NORC Contact].

This document will be shared with LSAMP alliances and institutions beginning in Spring 2024 to
provide initial information about the evaluation efforts. It will continue to be distributed as a quickreference FAQ sheet throughout data collection efforts.

1

C. General Administration and Survey Development
Email/language that will be sent to main institutional contact for the survey testing
Dear [insert name],
We are writing to ask for your help with an important study to help us better understand how
staff like you think about your work with the NSF-funded Louis Stokes Alliances for Minority
Participation (LSAMP). We are currently developing our survey instruments for future
administration to alliance sites. We would like to speak with you to hear your perspective on
whether we are asking the right questions, and doing so in a way that provides the best possible
information for supporting LSAMP programming moving forward.
This survey will take about ten minutes to complete and the link will be provided to you. You do
not need to do anything to prepare for our conversation; we are simply interested in your
feedback and opinions as a [insert role]. The OMB approval number for this activity is 31450215.
If you are interested and able to participate, please either email Justine Bulgar-Medina
bulgarmedina-justine@norc.org, or you can self-schedule a one-hour time that is most
convenient for you [insert link]. If you have any questions or concerns, please do not hesitate to
reach out.
Sincerely,
[NORC Contacts, NSF Contact]

C. General Administration and Survey Development
Email/language that will be sent to main institutional contact for the survey testing
Dear [insert name],
We are writing to ask for your help with an important study to help us better understand how
staff like you think about your work with the NSF-funded Louis Stokes Alliances for Minority
Participation (LSAMP). We are currently developing our survey instruments for future
administration to alliance sites. We would like to speak with you to hear your perspective on
whether we are asking the right questions, and doing so in a way that provides the best possible
information for supporting LSAMP programming moving forward.
This survey will take about ten minutes to complete and the link will be provided to you. You do
not need to do anything to prepare for our conversation; we are simply interested in your
feedback and opinions as a [insert role]. The OMB approval number for this activity is 31450215.
If you are interested and able to participate, please either email Justine Bulgar-Medina
bulgarmedina-justine@norc.org, or you can self-schedule a one-hour time that is most
convenient for you [insert link]. If you have any questions or concerns, please do not hesitate to
reach out.
Sincerely,
[NORC Contacts, NSF Contact]

Proposed Survey: LSAMP: 2008-2018
Methodological Notes:
The proposed sampling strategy utilized a combined approach of WebAMP data, annual
reports, and referral (“snowball”) sampling. All individuals named specifically in WebAMP data
and/or annual reports (e.g., in the costing reports) will be included in the sampling plan.
Acknowledging that these individuals may be difficult to contact, or may not include all
individuals who were involved in LSAMP programming in the evaluation years (e.g., mentors
who were unpaid/completed mentorship as a part of university service requirements), we also
propose including all currently identified program officers or other known staff both as
respondents and as a component of referral sampling. We propose requesting referrals for
relevant respondents who we may not have in the WebAMP data, by allowing the identified
respondent (e.g., program officer) to forward on the survey request to anyone they think would
be able to provide valuable information. This approach also allows a greater degree of inference
and generalizability as it reflects all institutions, allowing greater understanding of LSAMP
operations generally.
Acknowledging the burden on programs in evaluation efforts, the survey will not exceed 10
minutes for any individual. The OMB Control Number for this activity is 3145-0215. Additionally,
we will only present relevant information (e.g., faculty will not be asked for enrollment data that
is better answered by the administration). This will be accomplished by using a module
approach, where only select modules are presented to any one respondent, reflecting where
their experiences will be most valuable and relevant.
This approach also allows for inclusion of students and graduates, as possible and desired by
NSF, where only select questions regarding their time as a student and graduate of the program
would be requested. The inclusion of faculty, affiliated staff (who may not be named in
WebAMP data), and students—who are rarely included in current data sources, are unique
contributions offered by a survey in the present evaluation efforts.
Cognitive Testing Notes:
The relevant modules from the survey will be cognitively tested with the individuals identified by
NSF for protocol testing alongside the case study site interview and focus group protocols. This
will allow the evaluation team to have a more complete understanding of how questions and
topics may be thought about by different individuals, and in different formats. Not all topics will
be relevant to all participants, however we will strive to have no fewer than five respondents for
each item tested across these three data collection instruments. Additionally, while closedended options will be sought and utilized in instruments for full data collection, many are left
open-ended here solely for the purposes of testing. This will allow a broader range of
unexpected or unpredicted responses to be collected during testing, and considered for use in
final instrumentation.

Page 1 of 10

Introduction:
[Introduction to NSF, LSAMP, and the purpose of the evaluation, as well as consent language.]
The following questions will ask you to think back to the years 2008-2018 as you answer them.
To help you think about this period of time, a few notable moments from history: the 2008
election saw Barack Obama run against John McCain; 2009 saw the ‘Great Recession’, 2012
saw Barack Obama run against Mitt Romney and the Olympics hosted in London, 2014 saw the
Olympics hosted in Sochi, 2016 saw Hilary Clinton run against Donald Trump, 2018 saw South
Korea host the Olympics, the passing of John McCain, and there was a government shutdown.
We know this may be difficult, so best estimates are acceptable, as is conferring with a
colleague or referencing notes or documents as you answer questions. The information you
provide is an invaluable contribution to understanding the many facets of programming
supporting students in these fields, and ensuring continued funding for future students.
It should take you approximately 10 minutes to complete. If you have any questions, please
contact [program contact]. Thank you for your support of this important evaluation!
Demographics:
D.1 What was your role within this institution from 2008-2018?
a) Administration—Executive/Academic (e.g., president, dean, chancellor)
b) Administration—Professional (e.g., advising, admissions, area directors)
c) Faculty—Tenure Track (e.g., assistant, associate, professor)
d) Faculty—Non-Tenure Track (e.g., lecturer, adjunct)
e) Undergraduate student
f) Graduate student
g) Something else (please specify):___________
D.1.2 What is your present role within this institution?
a) Administration—Executive/Academic (e.g., president, dean, chancellor)
b) Administration—Professional (e.g., advising, admissions, area directors)
c) Faculty—Tenure Track (e.g., assistant, associate, professor)
d) Faculty—Non-Tenure Track (e.g., lecturer, adjunct)
e) Undergraduate student
f) Graduate student
g) I am no longer affiliated with this institution.
h) Something else (please specify):___________

Page 2 of 10

D.2 (non-students) In what year did you begin working at this institution?
(*Note, if they did not work there during the 2008-2018 reference period, they will not be asked
to continue.)
D.2.2 (students) In what years did you study at this institution?
(*Note, if they did not study there during the 2008-2018 reference period, they will not be asked
to continue.)
Year Began: _____
Year Ended: _____
D.2.3 (students) Did you graduate from this institution? (Yes/No)
D.3 During 2008-2018, were you involved in any aspects of STEM or LSAMP programming?
a) Yes, I was involved in STEM programming
b) Yes, I was involved in LSAMP programming
c) Yes, I’m involved in both
d) No, I’m not involved in either STEM or LSAMP programming at my institution
Module 1: Ensuring Comparable Outcomes
Q1.1 When thinking about programming and goals for ensuring all students at your institution
can access resources and ensure comparable outcomes, which phrases does your institution
use either in writing or colloquially?
Please select all that apply.
a) Equity
b) Equal access and outcomes
c) Removing barriers
d) Opportunity for all
e) Something else (please specify): ____________________

Q1.2 What are the processes in place at the institution level for enacting these [RESPONSE
from Q1.1] frameworks?
________________________________________________________________
Q1.3 What are the processes in place at the department level for enacting these [RESPONSE
from Q1.1] frameworks?
________________________________________________________________

Page 3 of 10

Q1.4 Which of the following activities, programs, or actions best supported [RESPONSE from
Q1.1]. Please select all that apply.
a) Encourage students to seek out mentors with shared values and identities who are
committed to the mentoring
b) Implement policy that fosters the connection to service and communal responsibility
afforded by STEM careers as a means of attracting more students from diverse
backgrounds
c) Foster collaboration that between community colleges and four-year institutions that
advocates for greater articulation in state-level transfer policies
d) Strengthen formal channels for providing students with information about crossenrollment policies to decrease disparities
e) Include accessible mapping of all programs, exact course alignments and time-to-degree
completion for future employment and transfer opportunities
f) Provide targeted supports including tutoring, cohort classes, tailored counseling and
timely alerts for not meeting requirements
g) Gather feedback from students on a regular basis
h) Instructors intentionally build community within classrooms by providing culturally
relevant curriculum
i) Establish more cohesive classroom and instructional practices between two- and fouryear institutions
j) Support underrepresented students’ STEM identity development
k) Support underrepresented students’ STEM identity development in ways designed to
offset the adjustment to fitting the “mold” of a traditional STEM student.
l) Provide summer research experiences to increase the likelihood of students from
underrepresented racial/ethnic minority groups applying to STEM PhD programs
m) Provide students with support to manage their coursework
n) Provide students with financial support
o) Provide advising to support alternative pathways for non-STEM undergraduate majors to
matriculate into STEM PhD programs
p) Support development of student’s scientific identities through mentoring relationships
Encourage students to have multiple mentors as a means of increasing support with
navigating STEM pathways
Module 2: ROI
Q2.1 Which of the following activities, programs, or actions do you feel was the most impactful
for retention rates? Please select all that apply.

Page 4 of 10

a) Encourage students to seek out mentors with shared values and identities who are
committed to the mentoring
b) Implement policy that fosters the connection to service and communal responsibility
afforded by STEM careers as a means of attracting more students from diverse
backgrounds
c) Foster collaboration that between community colleges and four-year institutions that
advocates for greater articulation in state-level transfer policies
d) Strengthen formal channels for providing students with information about crossenrollment policies to decrease disparities
e) Include accessible mapping of all programs, exact course alignments and time-to-degree
completion for future employment and transfer opportunities
f) Provide targeted supports including tutoring, cohort classes, tailored counseling and
timely alerts for not meeting requirements
g) Gather feedback from students on a regular basis
h) Instructors intentionally build community within classrooms by providing culturally
relevant curriculum
i) Establish more cohesive classroom and instructional practices between two- and fouryear institutions
j) Support underrepresented students’ STEM identity development
k) Support underrepresented students’ STEM identity development in ways designed to
offset the adjustment to fitting the “mold” of a traditional STEM student.
l) Provide summer research experiences to increase the likelihood of students from
underrepresented racial/ethnic minority groups applying to STEM PhD programs
m) Provide students with support to manage their coursework
n) Provide students with financial support
o) Provide advising to support alternative pathways for non-STEM undergraduate majors to
matriculate into STEM PhD programs
p) Support development of student’s scientific identities through mentoring relationships
Encourage students to have multiple mentors as a means of increasing support with
navigating STEM pathways
Q2.2 Which of the following activities, programs, or actions do you feel your institution
contributed the most to (i.e., matching NSF funding) in terms of direct funding? Please select all
that apply.

Page 5 of 10

a) Encourage students to seek out mentors with shared values and identities who are
committed to the mentoring
b) Implement policy that fosters the connection to service and communal responsibility
afforded by STEM careers as a means of attracting more students from diverse
backgrounds
c) Foster collaboration that between community colleges and four-year institutions that
advocates for greater articulation in state-level transfer policies
d) Strengthen formal channels for providing students with information about crossenrollment policies to decrease disparities
e) Include accessible mapping of all programs, exact course alignments and time-to-degree
completion for future employment and transfer opportunities
f) Provide targeted supports including tutoring, cohort classes, tailored counseling and
timely alerts for not meeting requirements
g) Gather feedback from students on a regular basis
h) Instructors intentionally build community within classrooms by providing culturally
relevant curriculum
i) Establish more cohesive classroom and instructional practices between two- and fouryear institutions
j) Support underrepresented students’ STEM identity development
k) Support underrepresented students’ STEM identity development in ways designed to
offset the adjustment to fitting the “mold” of a traditional STEM student.
l) Provide summer research experiences to increase the likelihood of students from
underrepresented racial/ethnic minority groups applying to STEM PhD programs
m) Provide students with support to manage their coursework
n) Provide students with financial support
o) Provide advising to support alternative pathways for non-STEM undergraduate majors to
matriculate into STEM PhD programs
p) Support development of student’s scientific identities through mentoring relationships
Encourage students to have multiple mentors as a means of increasing support with
navigating STEM pathways
Q2.3 Which of the following activities, programs, or actions do you feel your institution
contributed the most to in terms of indirect funding (i.e., unofficial mentorship, unpaid supports,
tangential opportunities)? Please select all that apply.

Page 6 of 10

a) Encourage students to seek out mentors with shared values and identities who are
committed to the mentoring
b) Implement policy that fosters the connection to service and communal responsibility
afforded by STEM careers as a means of attracting more students from diverse
backgrounds
c) Foster collaboration that between community colleges and four-year institutions that
advocates for greater articulation in state-level transfer policies
d) Strengthen formal channels for providing students with information about crossenrollment policies to decrease disparities
e) Include accessible mapping of all programs, exact course alignments and time-to-degree
completion for future employment and transfer opportunities
f) Provide targeted supports including tutoring, cohort classes, tailored counseling and
timely alerts for not meeting requirements
g) Gather feedback from students on a regular basis
h) Instructors intentionally build community within classrooms by providing culturally
relevant curriculum
i) Establish more cohesive classroom and instructional practices between two- and fouryear institutions
j) Support underrepresented students’ STEM identity development
k) Support underrepresented students’ STEM identity development in ways designed to
offset the adjustment to fitting the “mold” of a traditional STEM student.
l) Provide summer research experiences to increase the likelihood of students from
underrepresented racial/ethnic minority groups applying to STEM PhD programs
m) Provide students with support to manage their coursework
n) Provide students with financial support
o) Provide advising to support alternative pathways for non-STEM undergraduate majors to
matriculate into STEM PhD programs
p) Support development of student’s scientific identities through mentoring relationships
Encourage students to have multiple mentors as a means of increasing support with
navigating STEM pathways

Q2.4 Which of the following activities, programs, or actions do you feel your institution saw the
greatest return on investment (either direct or indirect)? Please select all that apply.
a) Providing hands-on research experience in lab facilities

Page 7 of 10

b)
c)
d)
e)

Completing math coursework early in community college
Completing a higher number of overall STEM courses
Understand pathways using an intersectional lens
Support pre-enrollment (Align student academic readiness in high school more closely
with enrollment in early college course to improve retention on STEM pathways)
f) Target academic major planning to occur during second semester of second year or first
semester of third year
g) Provide 1-1 or small group advising support
h) Differentiate advising support based on students’ expressed desire to switch majors or
leave STEM major altogether
i) Provide opportunities for students to learn about careers and earning potential
j) Facilitate a diverse and inclusive curriculum that actively acknowledges and appreciates
students’ backgrounds

Q2.4 What other activities, programs, or actions do you think were important to the return on the
investment (either direct or indirect investments) made by your institution that we haven’t asked
about yet?
________________________________________________________________
Q2.5 Of all the programming and investments made by your institution, in your opinion, which
three had the highest return on investment? [PIPE DROPDOWN OPTIONS]
(1) __________________________________________________
(2) __________________________________________________
(3) __________________________________________________

Q2.6 Of all the programming and investments made by your institution, in your opinion, which
three had the lowest return on investment? [PIPE DROPDOWN OPTIONS]
(1) __________________________________________________
(2) __________________________________________________
(3) __________________________________________________

Module 3: Barriers and Opportunities
Q3.1 What were the top three barriers faced by your institution or department?
(1) __________________________________________________
(2) __________________________________________________
(3) __________________________________________________

Page 8 of 10

Q3.2 In your own words, please share anything about this activity that you think is a 'lesson
learned', either for your organization or that may be helpful to similar organizations or future
programs.
________________________________________________________________
Q3.3 Thinking back to all the activities and programming you offered from 2008-2018, how often
did you hire or assign a designated full-time staff member to oversee them?
a) Never
b) Sometimes
c) About half the time
d) Most of the time
e) Always
Q3.4 What other staff models did you use to facilitate these activities and programs?
________________________________________________________________
Q3.5 Who are the primary contributors to curriculum development? [PIPE RESPONSE
OPTIONS]
Q3.6 What supports would you like to see in the future to benefit LSAMP programming and
activities?
Q3.7 How does your institution support your department and LSAMP programming? [PIPE
RESPONSE OPTIONS]
Module 4: Collaboration Across LSAMP Programs/Awardees
Q4.1 How much collaboration is there with other LSAMP programs and your institution?
a) None at all
b) A little
c) A moderate amount
d) A lot
e) A great deal
Q4.1 Do you engage in any data sharing or information sharing with other awardees?
a) Yes
b) No
c) Unsure
Q4.2 How do you engage in community building, leadership opportunities, and lessons learned
sharing with other awardees?

Page 9 of 10

________________________________________________________________
Q4.3 What opportunities for collaboration across the alliance(s) do you regularly engage in?
_______________________________________________________________
Q4.4 What are the successes of such collaboration?
________________________________________________________________
Q4.5 What are the obstacles of such collaborations?
________________________________________________________________
Q4.5 How frequently, and for what purpose, do you have intra-institution meetings and
collaborations?
________________________________________________________________
Module 5: Wrap-Up
Q5.1 How have things changed at your institution, within your department, or in regards to
LSAMP programming from 2018 to today?

Page 10 of 10