0704-XXXX_SSB_High Risk Installation Evaluations_5.18.2021

Supporting Statement Outline – Sample
NOTE: Complete Part B for Survey ICR Requests
SUPPORTING STATEMENT – PART B
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
If the collection of information employs statistical methods, the following information
should be provided in this Supporting Statement:

1. Description of the Activity
These information collections support an emergent, high-visibility Secretary of Defense
requirement directed in February 2021 to conduct evaluations of 20 DoD installations where the
military community is at increased or decreased risk for destructive behaviors as evidenced by
measures of unhealthy command climate. Site visits will take place June-August 2021, report
development in August 2021, Military Department coordination in September 2021 and delivery
of the report to the Secretary in Oct 2021. Given the aggressive timeline of this requirement, the
purpose of the initial high risk installation evaluations is to pilot an evaluation process and
metrics in order to develop an enduring evaluation method to support future evaluations
(expected to be conducted on biennial basis).
DoD Office of Force Resiliency (OFR) will identify 20 DoD installations to take part in
the assessment. At each location, a handful of service members who either have direct
responsibility for prevention activities or their superiors will participate. There will be three data
sources: 1) responding to a “request for information”; 2) participating in discussions during a
three day site visit; and 3) completing a survey.
The request for information asks a series of questions about experiences conducting
prevention activities targeting a variety of negative behaviors at each location, including sexual
assault and harassment, substance abuse, suicide, and domestic violence. The request for
information also asks what kind of prevention activities service members conduct, what their
relationships are like with their colleagues at their location who also are involved in prevention,
and any relevant documents about the prevention activities conducted. The request for
information will be sent before the site visit.
The site visit discussions will involve staff from the RAND Corporation and DoD asking
questions to follow-up on data elements in the request for information, along with additional
questions about prevention at each location. The site visit will be three days. Depending upon
roles of each service member at the different locations, they may be asked to participate in all
three days, or for some amount of time less than three days.

1

The survey asks questions about individual level of knowledge and skills conducting
prevention activities. Depending on their role, they may be asked to participate all three
activities, two activities, or just one. Participation will be determined in consultation with RAND
and OFR staff. The survey will be collected on paper at the site visit. RAND staff will hand out
the surveys to be completed in meetings, collect them back, and then transport them back to
RAND for data entry.
All three data sources will have identifiers, including the name of the respondent, email
addresses, and phone numbers. None of the information is sensitive, but mostly describes
activities that while not well known, are public (existence of prevention programs at an
installation). Some of the information will be about topics that are not publicly known—e.g., the
individual prevention competencies of an individual respondent.
The respondents to the site visit discussion and the survey (all respondents will complete
both) will be a cross section of personnel at the participating installations that fall into two
general categories. First, are personnel that are specific to prevention and intervention activities
relating to a variety of negative behaviors (e.g., alcohol use, suicide, sexual harassment). These
include: Sexual assault response coordinators, victim advocates, Family Advocacy Program
staff, MEO/EEO staff, Mental Health Professionals, Enlisted medical personnel (e.g., medics,
corpsmen), Inspectors general and misdemeanor and felony-level law enforcement
representatives, Chaplains, MWR and community/ support services staff, and Physical health
professionals. We are collecting data from these individuals because they have first-hand
knowledge of how prevention activities are carried out at the installation. The second category
are general personnel that will be important to talk to for their perceptions of how prevention is
prioritized and experienced at the installation and will include (the target sample is in
parentheses): Installation commander, E1-E4 Men (10), E1-E4 Women (10), O1-O3 Men (10),
O1-O3 Women (10), E5-E6 Men (10), E5-E6 Women (10), O4-O5 Men (10), O4-O5 Women
(10), E7-E9 Men (10), E7-E9 Women (10), O6 Men (10), O6 Women(10), First sergeants (5 or
more). In total, each installation will have about 220 respondents for a total of about 4400
respondents across the 20 installations.
2. Procedures for the Collection of Information
There will be three data sources: 1) responding to a “request for information”; 2) participating in
discussions during a three day site visit; and 3) completing a survey. Data source 1 and 2 will
actually be the same questions. The request for information will be sent ahead via an emailed
Word document to a point of contact (POC) at each installation. The POC will arrange to have
the relevant individual(s) answer the questions in the Word document and send back to RAND
by email. In our experience, these types of request for information are often returned incomplete
or unclear. Thus, at the site visits, site visit staff will follow up to confirm the information and
fill in any missing data. During the three day site visit, various groups of individuals specified
above will be organized into a series of discussion groups in one hours slots (see Table 1).
Although RAND will provide general guidance, each installation will determine the best
individuals to participate.

2

Table 1. Hypothetical schedule of data collection at each installation
Tuesday

09001000

Wednesday

Team 1

Team 2

Installation
commander/
command
team in-brief
and
interviews

Tour of
installation
housing
and work
areas

Thursday

Team 1

Team 2

Team 1

E1-E4 Men
(10)1

E1-E4
Women
(10)

MWR and
community/
support services
staff
(10)

10301130

O1-O3 Men
(10)

O1-O3
Women
(10)

E5-E6
Women
(10)

E5-E6 Men
(10)

FRG/SFRG/FRP/
Key Spouse staff
(10)

Friday
Team 2

Chaplains
(5 or more)

Team 1
Farewell
meeting
with
installation
commander,
if s/he would
like

Team 2
Any other
type of
prevention
staff (e.g.,
AF Violence
Prevention
Integrators)

Sexual assault
response
coordinators,
victim
advocates,
Family
Advocacy
Program staff
(10)

1130-1300 Lunch on the installation
13001400
O4-O5 Men
(10)

O4-O5
Women
(10)

14301530
O6 Men
(10)

O6
Women
(10)

E7-E9 Men
(10)

E7-E9
Women
(10)

MEO/EEO staff
(5 or more?)

Mental Health
Professionals*
(10)

First sergeants
(Army/AF/MC,
any equiv.
Navy would
like to
provide)
(5 or more?)

Enlisted
medical
personnel
(e.g.,
medics,
corpsmen)
(10)

Inspectors general
AND both
misdemeanor and
felony-level law
enforcement
representatives
(10)

Physical health
professionals**
(10)

1

Parentheses indicate project sample size
*Mental health professional group should include at least two of each of the following: (a) psychiatrists, (b) psychologists or social workers, and
(c) MFLC counselors (Military and Family Life Counselors) AND at least one staff member from the substance/drug and alcohol abuse program
**Physical health professionals should include at least two each of the following: (a) primary care physicians (b) emergency room nurses (c)
physician assistants
FRG = Family Readiness Group (Navy), SFRG = Soldier and Family Readiness Group (Army), FRP = Family Readiness Program (Marine Corps), Key
Spouse program is Air Force; MWR = morale, welfare, recreation programs; MEO = Military Equal Opportunity program for military personnel,
EEO = Equal Employment Opportunity program for civilian personnel

The questions for 1 and 2 will be open-ended questions about the prevention activities. The
discussions will NOT be recorded, but a notetaker will be part of the site visit team and will take
detailed notes. Those notes will be hand carried back to RAND to be organized and used for
analysis. Data source 3 will be a survey of individual competencies to carry out prevention
activities. Individuals will complete the survey at their respective discussion group slot during
the three day site visit. The survey will be paper and pencil and collected by site visit staff who
will hand carry the completed forms back to RAND for data entry and analysis.
3. Maximization of Response Rates, Non-response, and Reliability
High response rates are expected given this data collection has been ordered by the
Secretary of Defense. The request for information will likely be the most challenging to have
completed. However, we specifically are employing the site visit mechanism to ensure that the
all the questions will have complete responses. Regarding the survey, we will administer it at the
3

beginning of each discussion group time slot and immediately collect the surveys back. Thus, we
expect the response rates will be very high.
We will use multiple procedures to ensure high reliability. Each site visit will
comprise two government individuals (team lead and analyst), two RAND staff who will lead the
discussion (accompanied by a RAND notetaker). There will be 16 RAND individuals in total.
All 16 will be trained to use the site visit protocol using hypothetical data. training will continue
until at least an interrater reliability score of .8 is achieved across all 16 raters. After each visit,
the raters from each site visit team will independently make their ratings. Inter-rater reliability
will be calculated using these separate scores. Then, the raters will discuss their ratings and come
to a census when they disagree. It is this consensus score that will be used in analysis.
The data collection is not intended to be generalizable to a larger universe. That is
because the data collection is intended to be an installation-level evaluation of that installation’s
activities. The data will be used to help each installation improve as well as provide an overview
of progress to DoD leadership.
4. Tests of Procedures
RAND is convening an expert panel of three subject matter experts to review the
measures to be used in the data collection. Each panel member will receive the instruments, a
background paper, and instructions to rate each item of each measure for its validity and
importance. Only the highest rated items will be retained. In addition, RAND will ask each
respondent at the end of each group discussion slot about their perceptions of the process (e.g.,
what worked well; what should be improved). In addition, DoD will convene a panel of three
representative end users to review the measures and provide feedback about perceived relevance,
clarity, and utility. Given that this data collection is a pilot, the feedback will be used to refine
the data collection for future use.
5. Statistical Consultation and Information Analysis
a. Provide names and telephone number of individual(s) consulted on statistical
aspects of the design.
Matthew Chinman, Ph.D., RAND Corporation, 4570 5th Ave, Pittsburgh, PA
15213, 412 683-2300 x 4287
b. Provide name and organization of person(s) who will actually collect and analyze
the collected information.
Team Members

Organization/Role

Dr. Andra Tharp

DoD/Team Lead

Mr. Travis Bartholomew

DoD/Team Lead

Col Tony Haught

DoD/Team Lead

4

Lt Col Thomas Maguire

DoD/Team Lead

Dr. Suzanne Holroyd

DoD/Team Lead

Dr. Beverly Fortson

DoD/Analyst

Dr. Tracy Hipp

DoD/Analyst

LCDR Evette Pinder

DoD/Analyst

LtCol Ric Cloninger

DoD/Analyst

LCDR Rich Yates

DoD/Analyst

LTC Gennelle Conway

DoD/Analyst

Team Members
Christine LaCoste
Deborah Zajdman
Mallika Bhandarkar
Max Steiner
Sarita Lee
Shoshana Shelton
Varun Chandorkar
Gregory Schumacher
Laura Miller
Peter Whitehead
Sarah Meadows
Stephanie Holliday
Susan Bush-Mecenas
Abe Wandersman
Amanda Meyer
Amber Watson 
Amy Kerr
Jack Baker
Jenny Kolodny-Goetz
Katelyn Wargel
Katie Zenger
Pam Imm
Paul Flashpohler

Organization/Role
RAND – Notetaker
RAND - Notetaker
RAND - Notetaker
RAND - Notetaker
RAND - Notetaker
RAND - Notetaker
RAND - Notetaker
RAND – Lead data collector
RAND – Lead data collector
RAND – Lead data collector
RAND – Lead data collector
RAND – Lead data collector
RAND – Lead data collector
RAND – Lead data collector
RAND – Lead data collector
RAND – Lead data collector
RAND – Lead data collector
RAND – Lead data collector
RAND – Lead data collector
RAND – Lead data collector
RAND – Lead data collector
RAND – Lead data collector
RAND – Lead data collector

5