LSAMP Cognitive Testing - Fast-track for students

NOTICE OF OFFICE OF MANAGEMENT AND BUDGET ACTION
Date

04/17/2024

National Science Foundation

FOR CERTIFYING OFFICIAL:

George Strawn

FOR CLEARANCE OFFICER:

Suzanne Plimpton

In accordance with the Paperwork Reduction Act, OMB has taken action on your request received
04/09/2024
ACTION REQUESTED: Generic IC
Louis Stokes Alliances for Minority Participation (LSAMP) – Cognitive Pilot Testing of LSAMP
IC TITLE:
Student Focus Group Protocol
ICR REFERENCE NUMBER:
202307-3145-001
AGENCY ICR TRACKING NUMBER:
TITLE:

Generic Clearance for the Collection of Qualitative Feedback on Agency Service Delivery

LIST OF INFORMATION COLLECTIONS: See next page
OMB ACTION: Approved without change
OMB CONTROL NUMBER:

3145-0215

The agency is required to display the OMB Control Number and inform respondents of its legal significance in
accordance with 5 CFR 1320.5(b).
EXPIRATION DATE: 08/31/2026

BURDEN:

DISCONTINUE DATE:

RESPONSES

HOURS

COSTS

Previous

100,000

25,000

75,000

New

100,000

25,000

75,000

Change due to New Statute

0

0

0

Change due to Agency Discretion

0

0

0

Change due to Agency Adjustment

0

0

0

Change due to PRA Violation

0

0

0

Difference

TERMS OF CLEARANCE:

Terms of the Generic ICR remain in effect.

OMB Authorizing Official:

Dominic J. Mancini
Deputy Administrator,
Office Of Information And Regulatory Affairs

Request for Approval under the “Generic Clearance for the Collection of Routine Customer
Feedback” (OMB Control Number: 3145-0215)

TITLE OF INFORMATION COLLECTION:
Louis Stokes Alliances for Minority Participation (LSAMP) – Cognitive Pilot Testing of LSAMP
Student Focus Group Protocol
PURPOSE:

The LSAMP program, a Congressionally mandated program established in 1991 to assist the nation in
diversifying its STEM workforce, began conducting a program evaluation in 2023. In addition to the
focus groups discussed in this request, the evaluation includes document review of annual reports from
LSAMP alliances as well as quantitative analysis of LSAMP program monitoring data and student
outcome data on the institutional (DoED IPEDS) and individual level (National Student Clearinghouse).
The focus groups discussed in this request will involve LSAMP student participants (i.e., undergraduate
and graduate students). The focus of this request is to conduct cognitive testing of the focus group
protocols with student participants at institutions in 6 alliances.
Cognitive testing of student participants involves interviewing individuals who have participated in
LSAMP supported activities on university campuses. The purpose of the cognitive testing is to gather
feedback on the content, flow, instructions, and length of time to respond to the focus group questions.
Cognitive testing will be conducted virtually and involve providing individuals with a link to the protocol
to verbal provide feedback on question understanding and appropriateness of questions for student
participants. Facilitators will virtually guide respondents through the protocol to receive real-time
feedback on the instrument. Additionally, the protocol will be tested for flow and appropriateness of
question wording for the different types of student participants in LSAMP activities. The results will be
used to improve the set-up and format of the focus group protocol. Data from the cognitive test surveys
will not be used for analysis purposes in the overall evaluation.
DESCRIPTION OF RESPONDENTS:
Three students per alliance from 6 different LSAMP alliances will be invited to participate in virtual
interviews (18 students total). The 18 student participants will be invited with designations as follows:
• Level 1 (students who receive any form of direct financial support in LSAMP)
• Level 2 (students who attend LSAMP activities but do not receive funds)
• Level 3 (spillover students; secondary impact. Underrepresented STEM students at

institutions in the LSAMP alliances who were not Level 1 or Level 2 students)

TYPE OF COLLECTION: (Check one)
[] Customer Comment Card/Complaint Form
[] Customer Satisfaction Survey
[X] Cognitive Testing (e.g., Website or Software)[] Small Discussion Group
[] Focus Group
[] Other:
CERTIFICATION:
I certify the following to be true:
1. The collection is voluntary.
2. The collection is low-burden for respondents and low-cost for the Federal Government.
3. The collection is non-controversial and does not raise issues of concern to other federal agencies.
4. The results are not intended to be disseminated to the public.
5. Information gathered will not be used for the purpose of substantially informing influential policy
decisions.

1

6. The collection is targeted to the solicitation of opinions from respondents who have experience with
the program or may have experience with the program in the future.
Name: Suzanne H. Plimpton, NSF Reports Clearance Officer
To assist review, please provide answers to the following question:
Personally Identifiable Information:
1. Is personally identifiable information (PII) collected? [] Yes [X] No
2. If Yes, is the information that will be collected included in records that are subject to the Privacy Act
of 1974? [] Yes [X] No
3. If Applicable, has a System or Records Notice been published? [] Yes [X] No
Gifts or Payments:
Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to
participants? [X] Yes [] No
BURDEN HOURS
Category of Respondent
LSAMP Student participants
Totals

No. of Respondents
18
18

Participation Time
60 minutes
60 minutes

Burden
18 hours
18 hours

FEDERAL COST: It is incorporated in the formal agreement with the contractor.
If you are conducting a focus group, survey, or plan to employ statistical methods, please provide
answers to the following questions:
The selection of your targeted respondents
1. Do you have a customer list or something similar that defines the universe of potential respondents,
and do you have a sampling plan for selecting from this universe?
[] Yes
[X] No
For the LSAMP student cognitive interviews, the LSAMP administrators or staff (who are also included
in the cognitive testing for staff interviews) will be requested to recommend students to participate in the
interviews per the following designations:
•
•
•

Level 1 (students who receive any form of direct financial support in LSAMP)
Level 2 (students who attend LSAMP activities but do not receive funds)
Level 3 (spillover students; secondary impact. Underrepresented STEM students at

institutions in the LSAMP alliances who were not Level 1 or Level 2 students)

Each alliance institution will be asked to provide name and contact information for students, taking into
account institutional diversity for institutions within the alliance. Only 3 students per alliance will be
included in the cognitive testing.
Administration of the Instrument
1. How will you collect the information? (Check all that apply)
[X] Web-based or other forms of Social Media
[] Telephone (Interviews)
2

[] In-person
[] Mail
[] Other
2. Will interviewers or facilitators be used? [X] Yes [] No
Please make sure that all instruments, instructions, and scripts are submitted with the request.

3

Instructions for completing Request for Approval under the “Generic Clearance for the
Collection of Routine Customer Feedback”
TITLE OF INFORMATION COLLECTION: Provide the name of the collection that is the subject of
the request. (e.g. Comment card for soliciting feedback on xxxx)
PURPOSE: Provide a brief description of the purpose of this collection and how it will be used. If this
is part of a larger study or effort, please include this in your explanation.
DESCRIPTION OF RESPONDENTS: Provide a brief description of the targeted group or groups for
this collection of information. These groups must have experience with the program.
TYPE OF COLLECTION: Check one box. If you are requesting approval of other instruments under
the generic, you must complete a form for each instrument.
CERTIFICATION: Please read the certification carefully. If you incorrectly certify, the collection will
be returned as improperly submitted or it will be disapproved.
Personally Identifiable Information: Provide answers to the questions.
Gifts or Payments: If you answer yes to the question, please describe the incentive and provide a
justification for the amount.
BURDEN HOURS:
Category of Respondents: Identify who you expect the respondents to be in terms of the following
categories: (1) Individuals or Households;(2) Private Sector; (3) State, local, or tribal governments; or (4)
Federal Government. Only one type of respondent can be selected.
No. of Respondents: Provide an estimate of the Number of respondents.
Participation Time: Provide an estimate of the amount of time required for a respondent to participate
(e.g. fill out a survey or participate in a focus group)
Burden: Provide the Annual burden hours: Multiply the Number of responses and the participation time
and divide by 60.
FEDERAL COST: Provide an estimate of the annual cost to the Federal government.
If you are conducting a focus group, survey, or plan to employ statistical methods, please provide
answers to the following questions:
The selection of your targeted respondents. Please provide a description of how you plan to identify
your potential group of respondents and how you will select them. If the answer is yes, to the first
question, you may provide the sampling plan in an attachment.
Administration of the Instrument: Identify how the information will be collected. More than one box
may be checked. Indicate whether there will be interviewers (e.g. for surveys) or facilitators (e.g., for
focus groups) used.
Please make sure that all instruments, instructions, and scripts are submitted with the request.

4

LSAMP Program Evaluation1
NORC at the University of Chicago has been selected to conduct an evaluation of the
Louis Stokes Alliances for Minority Participation Program (LSAMP) on behalf of the
NSF. The data for this evaluation will be collected during the 2024-2025 academic year
and will occur in multiple phases.
Phase 1: Preparation and Testing
 Multiple one-hour cognitive interviews with program administrators and
students to evaluate our data collection tools (e.g., interview or survey
questions).
 NSF has selected the administrative individuals for participation to ensure
representation in feedback. These individuals will also be asked to refer
students for student-focused testing and feedback.

Phase 2: Data Collection
 Collect direct feedback (e.g., interviews, focus groups) from those engaged
with LSAMP programs (e.g., administrators, staff, and students).
 Participation by all selected individuals will be crucial to fully capture the
successes and opportunities of LSAMP alliances and institutions.
 Review information already available, including annual reports.

Phase 3: Analysis and Reporting
 Analyze all data to provide insights on the LSAMP programs.
 Provide reports with summary findings at the alliance level to program
administrators.
 Provide actionable insights and ‘lessons learned’ that may be helpful across
alliances.

For more information or to participate, contact [NSF Contact; NORC Contact].

This document will be shared with LSAMP alliances and institutions beginning in Spring 2024 to
provide initial information about the evaluation efforts. It will continue to be distributed as a quickreference FAQ sheet throughout data collection efforts.

1

Email/language inviting LSAMP students for focus group protocol testing
We are writing to ask for your help with an important study to help us better understand how
students like you think about your participation in STEM education enrichment activities on your
campus through the [Insert Alliance name] in your [Insert university name]. This alliance is an
NSF-funded alliance through the Louis Stokes Alliances for Minority Participation (LSAMP)
program at the National Science Foundation. We are currently developing our student focus
group protocol for site visits to alliance sites during the 2024-2025 academic year. We would
like to speak with you to hear your perspective on whether we are asking the right questions,
and doing so in a way that gets at the best possible information for supporting LSAMP
programming moving forward.
These conversations will take about one hour to complete and will occur via Zoom. The OMB
approval number for this activity is 3145-0215. You do not need to do anything to prepare for
our conversation; we are simply interested in your feedback and opinions as a student. As a
token of our appreciation, we will offer you $50 for your time.
If you are interested and able to participate, please either email Justine Bulgar-Medina
bulgarmedina-justine@norc.org, or you can self-schedule a one-hour time that is most
convenient for you [insert link]. If you have any questions or concerns, please do not hesitate to
reach out.
Sincerely,
[NORC Contacts, NSF Contact]

OMB Data Collection Number 3145-0215, expires August 31, 2026.

Name:
The National Science Foundation (NSF) is leading this voluntary study of LSAMP activities at
participating alliance sites. NORC at the University of Chicago, a social science research
organization, is conducting this study on behalf of NSF. Today, we will be discussing our data
collection instruments with you. They are currently in development and your participation will
provide valuable feedback for finalizing our instruments. Your participation in this work is
confidential, and your responses and personal information will not be shared. There are no risks
in participating in this research beyond those experienced in everyday life. Participation is
voluntary, and you will be able to skip any question I ask you.
As a participant of the cognitive testing, you will provide feedback via a phone interview on the
following: content, flow, instructions, and length of time to complete the interview. The main
emphasis of the testing is to gauge your understanding of each question and assess your ability to
provide accurate responses. The results will be used to improve instructions, wording of
questions, and flow. Data from the pilot testing will not be used for analysis purposes.
PARTICIPATION AND WITHDRAWAL: You are under no obligation to participate, but
your feedback will be invaluable to the study. We strongly encourage you to consent. There are
no consequences or risks for participating. Refusal to participate will have no impact on current
NSF awards you may have or on your ability to obtain awards in the future.
AUDIO RECORDING: With your permission, your feedback will be recorded to assist in the
accurate presentation of your responses. Audio recordings will be protected on a secure server to
prevent inappropriate use.
CONTACTS: For additional information about the NSF study, you may contact Shelitita
Holland (sholland@nsf.gov), Martha James (mjames@nsf.gov) or LeRoy Jones
(ljones@nsf.gov).
If you have any questions about this interview, you may contact the Principal Investigator at
kim-debbie@norc.org.
If you have any questions about your rights as a research participant, you may contact the NORC
Institutional Review Board toll-free at 866-309-0542.

LSAMP CASE STUDY PROTOCOL: STUDENT FOCUS GROUP – DRAFT

Student Levels:
Level 1: Students who receive any form of direct financial support in LSAMP. The
alliance, home institution, and the LSAMP program are expected to collect and maintain
demographic, academic, and program activity data for students who receive direct
financial support for activities such as undergraduate research experiences.
Level 2: Students who attend LSAMP activities and do not receive funds. General
attendance counts for students who attend events, workshops and other activities
funded using LSAMP funds.
Level 3: Spillover students; secondary impact. Underrepresented STEM students at
institutions in the LSAMP alliances who were not Level 1 or Level 2 students. These
students potentially represent students who may have benefited from LSAMP
programming via spillover effects, but no evidence is expected to be collected or
maintained on whether these students did or did not receive direct funding or support
(attending events) from LSAMP. Aggregate level data on these students are included in
the enrollment and degree completion counts by race and ethnicity provided by LSAMP
Alliances in WebAMP.
Student Focus Group Protocols
Interviewer: The focus group will take approximately one hour and will be conducted in
person. There should be 6-8 participants in attendance.
[IF ZOOM NEEDED: You will help the participants get Zoom set up and troubleshoot
any connectivity issues.]
After set-up is completed, proceed with introductions, study background, and
instructions on what to expect from the session. Obtain consent to participate, then
proceed with the focus groups.
Red is script meant to be read aloud.
Step 1A: Introduction and Study Description
1. Introduce yourself, and anyone else on the call.
2. Describe the current study.
As you may know, today’s focus group is meant to be an opportunity for students like
yourselves to offer direct feedback on your experiences with the program. Specifically,
we will focus on how you learn about [LSAMP] activities, how you engage with these
activities (or reasons why you do not), what supports or hinders your engagement, your
perceptions of LSAMP-related success, your perceptions of ways that LSAMP has
shaped your postsecondary experiences.
1

This feedback is vital to supporting the National Science Foundation’s (NSF) efforts to
evaluate these programs, and NORC is assisting in these data collection efforts by
hosting these focus groups today. Your open and honest feedback is exactly what we’re
looking for, this will help us truly understand the successes and opportunities for
improvement of these programs.
Remember, your participation is private. We will not report anything attributed to your
name, identify you as a participant today, and ask that you do the same. That is, please
do not discuss anything you hear in the room today with anyone outside of this group
and do not discuss who was in attendance today.
3. Move on complete informed consent and get permission to record.
1.

Read consent statement to participants.

Participation in this focus group is entirely voluntary, and you are free to withdraw at any time
without penalty. Your responses will be kept confidential and will only be used for research
purposes.
Please know that what you say during this Focus Group will be kept secure. Only deidentified
findings will be shared with anyone outside of the research team.
If you have questions about the evaluation, please contact our Principal Investigator, Debbie
Kim, kim-debbie@norc.org. If you have questions about your rights as a participant, you may
call the NORC Institutional Review Board Manager toll-free at 1-866-309-0542.
In order to have a complete record of your comments, with your permission, the focus group
session will be video recorded. This recording will be confidential and will not be shared with
anyone outside the research team.  If you are not comfortable with this, please feel free to end
your participation.

2.
3.
4.

Answer any questions.
Obtain consent and record response.
Once consent is obtained, begin focus group guided discussion.

[Turn on recording.]
Step 1B: Set ground rules for discussion.
1. Set expectations and ground rules for the discussion
-

No right or wrong answers, only differing points of view
We're recording, one person speaking at a time
We're on a first name basis
You don't need to agree with others, but you must listen respectfully as
others share their views
My role as moderator will be to guide the discussion
2

-

Talk to each other
If someone said what you were going to say, you don’t need to repeat it.

2. As we present each question for discussion, we’ll let the group discuss, but may
ask follow-up questions.
Step 2: Conduct Focus Group
Section A: Opening Discussion
1. What are your names, your programs of study, and your year of school?
2. (Must) We'd like to start by asking you about how you came to be a part of
LSAMP. What was the enrollment process like? How did you learn about it?
a. Probe: Were you approached? Did you do reach out?
3. (Must) What supports has the LSAMP program provided you?
4. (Can) Can you describe the LSAMP program? What does it look like for you? Is
that the way it looks/works for everyone?
5. (Must) What relationships have you developed with staff members through
LSAMP?
Section B: Successes
1. (Must) What have you found to be most supportive in the program?
a. Probe: What has been the impact of LSAMP staff on your experiences
with LSAMP? How have they contributed to your successes, if at all?
2. (Must) Which programs or events have you participated in that were particularly
helpful?
3. (Can) In what ways would you say LSAMP has supported you in your time at
[institution]? Probe for specific examples.
4. (Must) (Level 1) How do you use the financial support you receive through
LSAMP? How did you come to decide how to use your financial support?
a. In what ways might these activities shape your future plans?
5. (Must) (Level 2) Of the LSAMP-related activities you’ve attended or participated
in, what have you found to be the most helpful and why?
6. (Must) (Level 2) Why did you decide to attend these LSAMP-related activities?
What were you hoping to learn or gain?
Section C: Gaps/Opportunities
1. (Can) What have you found to be the least supportive aspects of the program?
Probe: How have LSAMP staff contributed to or lessened these aspects?
2. (Can) Which programs do you wish were being offered (that aren’t/haven’t
been)?
3. (Must) (Level 1) Is there anything (e.g., supports, guidance, activities) you feel is
missing from the opportunities available to you through your LSAMP?
a. What would change for you if these supports, guidance, activities, etc.
were offered?
4. (Must) (Level 2) Is there anything (e.g., supports, guidance, activities) you feel is
missing from the LSAMP-related activities that are available to you?
3

a. What would change for you if these supports, guidance, activities, etc.
were offered?
Section D: Level 3 Students Only
1. (Must) What are your academic goals for your time at [institution]?
a. Probe: What was your motivation for pursuing a degree in [major]?
2. (Must) What activities do you participate in that are designed to support you in
reaching your goals?
3. (Must) What, if anything, have you heard about LSAMP? (Probe for awareness of
available activities or supports.)
4. (Must) Are there reasons why you have not participated or engaged in (available
activities or supports)?
Section E: Future Outlook
1. (Must) (If non-doctorate holders/students) Do you see yourself enrolling in any
educational programs in the future? Like what?
2. (Must) What advice would you give current decision makers developing funding
and programming for the next generations of students like yourself?
3. (Can) Where do you see yourself in 1, 5, 10, or 15 years?
4. (Can) What LSAMP-specific supports are you getting that will help you attain the
goals you just described?
Section F: Step 3: Wrap-Up
Thank you for your time today.
If you have any questions or concerns after this discussion, please contact
[MODERATOR] and [PI].
[End recording, provide incentives.]

4