Supporting Statement B

Office of Management and Budget
Supporting Statement Part B
for the Healthy Weight Collaborative Evaluation:
Data Collection Procedures and Statistical Methods

December 28, 2012

This page has been left blank for double-sided copying.

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
In September 2010, the U.S. Department of Health and Human Services’ Health Resources and
Services Administration (HRSA) awarded $5 million to the National Initiative for Children’s
Healthcare Quality (NICHQ) to implement the “Collaborate for Healthy Weight” (C4HW)
initiative. Funded by the Prevention and Public Health Fund created by the Affordable Care Act of
2010, this 30-month cooperative agreement was designed to “create partnerships between primary
care, public health, and community-based organizations to discover sustainable ways to promote
healthy weight and eliminate health disparities in communities across the United States” (National
Initiative for Children’s Healthcare Quality 2012). As part of C4HW, NICHQ created the Healthy
Weight Collaborative (HWC), a national quality improvement effort to share and spread promising,
evidence-based practices to prevent and treat obesity among children and families.
In this two-phased learning collaborative, NICHQ is working with about 50 community teams
of primary care, public health, and community-based organizations to implement, test, and
disseminate an integrated change package of promising, evidence-based healthy weight strategies. In
Phase 1 (July 2011 through July 2012), NICHQ selected 10 teams from a field of 83 applicants that
responded to a general call for learning collaborative applications. In Phase 2 (March 2012 through
March 2013), the project is working with about 40 more teams, recruited through the same process.
HRSA has hired a contractor, Mathematica Policy Research, to conduct the HWC project’s
external evaluation. This document provides supporting statements for each of the five points
outlined in Part B of the Office of Management and Budget (OMB) guidelines, to collect
information for the evaluation of the HWC project. This submission requests clearance for the
HWC evaluation’s data collection instruments, plans for data analysis, and reporting of findings.
1. Respondent Universe and Sampling Methods
Statistical sampling methods were not used to select the respondent universe. The NICHQ
purposively selected community teams from among groups that applied to take part in the HWC
project. The HWC evaluation will collect and analyze site visit and interview information from three
groups of respondents: the four Phase 1 teams that will continue to participate in Phase 2; seven of
the approximately 40 teams that are participating in Phase 2; and the NICHQ leadership, faculty
members, and program managers that are implementing Phase 2. Table B.1 depicts our sampling
approach for community teams and NICHQ staff.
Phase 1 HWC Community Team Site Visits
Of the 10 community teams that participated in Phase 1, four continued to participate as
mentors to the Phase 2 teams. The evaluation team will conduct one-time site visits with each of the
four. Each visit is expected to include interviews with eight community team members. Interviews
will be conducted with the team’s coordinator, its data manager, and six other members (on average,
the Phase 1 teams had 15 members).
Team coordinators will be asked to identify six members who have regularly been involved in
team activities. The six will be selected purposively with input from the team coordinator to ensure
that we get a cross-section of members representing each sector targeted by the project (primary
care, public health, and community). The team coordinators will also provide insights and
information about member roles, activity on the collaborative, and availability.
3

Table B.1 Summary of Sampling Approach
Data Collection Type

Instrument Type

Sampling Approach

Continuing Phase 1 Team
Site Visits:

No sampling: a site visit will be
conducted with all four Phase 1 teams
that are participating in Phase 2.

Team Coordinator

Interview Guide

No sampling

Team Data Manager

Interview Guide

No sampling

Team Members

Group Interview Guide

Purposeful identification within team:
the evaluation team will select the team
members based on team lists from
NICHQ and input from team
coordinators to ensure all three sectors
are represented and active team
members participate in the interview.

Phase 2 Community Team
Site Visits:

Purposeful Identification of teams: the
evaluation team will select teams based
on team documentation from NICHQ
and input from HRSA on key team
characteristics, to ensure that a diverse
set of teams are selected.

Team Coordinator

Interview Guide

No sampling

Team Data Manager

Interview Guide

No sampling

Team Members

Group Interview Guide

Purposeful identification within team:
the evaluation team will select the team
members based on team lists from
NICHQ and input from team
coordinators to ensure that all three
sectors are represented and active
team members participate in the
interview.

NICHQ Interviews:

Purposeful Identification: the evaluation
team will interview NICHQ leadership,
staff, and faculty that have the most
involvement in the project’s design and
implementation.

Leadership

Interview Guide

Purposeful identification within group:
the evaluation team will select the
leaders most involved in the HWC
project.

Faculty

Interview Guide

Purposeful identification within group:
the evaluation team will select the
faculty most involved in the HWC
project.

Program Managers

Interview Guide

Purposeful identification within group:
the evaluation team will select the
program managers most involved in
the HWC project.

4

Phase 2 HWC Community Team Site Visits
The seven Phase 2 community teams will be purposefully selected using several criteria to
ensure that site visits are conducted with a diverse set of teams. The teams will be selected with
input from HRSA and NICHQ staff based on selection criteria, including the team’s HRSA region,
the age and size the team’s target population, and the team’s community setting (urban, rural, or
tribal). The evaluation team will also factor in whether the team coordinator is from the public
health, primary care or community sector, the size of the team, and whether it had an existing
coalition before participating in the HWC project.
As with the Phase 1 site visits, each Phase 2 visit will include interviews with eight team
members. These include the coordinator and data manager, as well as six others selected purposively
with input from the team coordinator to ensure a cross-section representing each sector targeted by
the project (primary care, public health, and community). The team coordinators will also provide
insights and information about member roles, activity on the collaborative, and availability.
Interviews with NICHQ Leaders, Faculty, and Project Managers
The evaluation team will conduct interviews with the NICHQ leaders, faculty, and program
managers who have been most involved in Phase 2. The universe of NICHQ staff includes 8
NICHQ leaders, 17 faculty members, and 9 NICHQ project managers (staff roles are described in
Supporting Statement A.1). We will conduct individual or group interviews with a purposive sample
of the NICHQ leaders, faculty, and project managers who have played key roles in Phase 2. This will
include up to 4 of the 8 NICHQ leadership staff and up to 5 of the 9 HWC project managers. We
will also conduct one group interview with up to 6 of the 17 HWC faculty members.
2. Procedures for the Collection of Information
Site Visits with Phase 1 and Phase 2 Healthy Weight Collaborative Community Teams
The NICHQ evaluation director and program managers will first inform community teams that
they have been selected for the site visits. The evaluation team will then contact the community
team coordinators by phone or email to introduce the study and schedule the onsite interviews to
take place over one or two days and within regular work hours. A two-person interview team will
include a senior person to lead the interviews and a junior person to help schedule, facilitate, and
take detailed notes. All community team member interviews will follow interview protocols
(Attachments C and D in Supporting Statement A).
The protocols will permit a range of issues to be discussed in a consistent and thorough manner
across all interviews and sites while also allowing the flexibility for interesting issues to be considered
as they arise. The evaluation team will audio-record the interviews if respondents agree. Recordings
will not be transcribed, but will be available to clarify team notes or extract verbatim quotes (not
attributed to specific respondents). To foster consistency in documentation and create efficiencies
with data synthesis, the interview notes will be summarized in a thematic reporting template.
Content from the template will be synthesized in a final report to HRSA. Findings from team site
visits will be supplemented by review of team documents, including teams’ project applications,
work plans and progress reports, and descriptive secondary data analyses of teams’ performance
measures and other feedback submitted to NICHQ.
5

Interviews with NICHQ Leaders, Faculty Members, and Project Managers
The evaluation team will conduct NICHQ interviews in person when possible, as the contractor
and NICHQ headquarters are both located in the Boston area. Telephone interviews will be
conducted if there are barriers to in-person interviews for some of the NICHQ staff. The evaluation
team will coordinate with a NICHQ liaison to schedule the interviews. A two-person interview team
will include a senior person to lead the interviews and a junior person to work with the NICHQ
liaison and take detailed notes. All interviews will follow protocols (Attachment E in Supporting
Statement A).
The protocols will permit a range of issues to be discussed in a consistent and thorough manner
across all interviews and sites while also allowing the flexibility for interesting issues to be considered
as they arise. The evaluation team will audio-record the interviews if respondents agree. Recordings
will not be transcribed, but will be available to clarify team notes or extract verbatim quotes (not
attributed to specific respondents). To foster consistency in documentation and create efficiencies
with data synthesis, the interview notes will be summarized in a thematic reporting template.
Content from the template will be synthesized in a final report to HRSA.
3. Methods to Maximize Response Rates and Deal with Nonresponse
The HWC evaluation team expects to obtain a very high response rate (close to 100 percent)
for all site visits and interviews. This expectation is based on previous experience with the
community team pre-test site visit and with the NICHQ staff interviews conducted in Phase 1, and
on the high level of motivation from the community teams and NICHQ personnel involved in the
project. The NICHQ evaluation director and program managers will serve as links to work with the
community teams, if needed, to address response issues. Strategies for maximizing response in the
data collection efforts are described below.
Phase 1 Community Team Site Visits. All interviews conducted with the four Phase 1 teams
will occur during site visits. In Phase 1, the evaluation team met and developed a rapport with the
Phase 1 teams during the learning sessions. It is anticipated that all four community teams will agree
to participate in these visits. To help ensure high participation, we will coordinate with the teams to
schedule convenient dates.
Phase 2 Community Team Site Visits. All interviews with seven Phase 2 teams will also
occur during site visits. It is anticipated that all teams selected and recruited for the site visits will
agree to participate. The Phase 2 teams are aware that the contractor is conducting an external
evaluation for the HWC project, and team participation in the external evaluation is encouraged by
NICHQ and HRSA. In addition, the Phase 2 site visits will be scheduled at the convenience of the
Phase 2 teams.
NICHQ Interviews. To ensure a high participation rate, the contractor team will coordinate
with a NICHQ liaison to determine convenient dates for the NICHQ interviews and to arrange
telephone interviews for respondents not near NICHQ’s Boston-area office.
4. Tests of Procedures or Methods to Be Undertaken
As described in Supporting Statement A, the evaluation team pilot-tested the community team
interview protocol on a site visit to one of the Phase 1 community teams in March 2012; there were
seven respondents. The team used this experience to revise the content and reduce the length of the
6

interview guides for the four continuing Phase 1 community team site visits, the seven Phase 2 visits,
and the interviews with NICHQ leaders, faculty, and program managers. Those guides were
included in the attachments to OMB Supporting Statement A. In addition, the evaluation team
reviewed several site visit interview protocols used in previous similar studies under contract with
HRSA and used that information to guide the general format and length of the HWC protocols.
Following the submission of Supporting Statement A on November 28, 2012, HRSA provided
OMB with changes to the evaluation’s informed-consent form based on a requested revision by the
New England Institutional Review Board. The revised form added more information about (1) the
length of the interviews, (2) the fact that respondents could participate in the interview if it was not
recorded, and (3) the small risk of the loss of respondent confidentiality for interviewees.
5. Individuals Consulted on Statistical Aspects and Individuals Collecting
and/or Analyzing Data
The evaluation’s design, primary data collection activities, and data analyses are being conducted
for HRSA by the contractor, Mathematica Policy Research. There are no statistical aspects of these
design, data collection, or analysis activities. The person responsible for receiving and approving the
contract deliverables is Dr. Sylvia Fisher, the HRSA project officer.

7