SSB - Formative GenIC Research: FSSDD GES

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Family Self-Sufficiency Demonstration Development Grants Evaluation Support

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part B

December 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Emily Ross, Amelia Popham, and Sarita Barton

Part B

B1. Objectives

Study Objectives

The purpose of the information collection is to conduct semi-structured discussions to identify the evaluation support needs and research priorities of each of the Administration for Children and Families’ (ACF) 20 Family Self-Sufficient Development Demonstration (FSSDD) grant recipients. Each FSSDD grant recipient applied for ACF support to conduct evaluation activities on an innovative intervention intended to improve family and economic well-being among populations with low incomes. Collectively, these evaluation activities will be used to enhance the evidence base on what works to improve family and economic well-being of populations with low incomes.

ACF awarded a contract to Mathematica and its subcontractor, The Adjacent Possible, to provide evaluation support to each grant recipient. The results of the information collection proposed in this request will be used to inform the provision of this support by the Contractor for each grant recipient. To best determine the type of evaluation support that would benefit each FSSDD grant recipient’s own intervention, the discussions will focus on the extent to which the grant recipient’s intervention is evidence-based or evidence-informed, the strength and quality of intervention implementation, and past experiences with and current interests in evaluation activities.

Generalizability of Results

This study is intended to present internally-valid descriptions of the design and implementation of interventions by FSSDD grant recipients, not to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

Conducting qualitative, semi-structured discussions focused on key aspects of intervention design and implementation, as well as past research and evaluation activities, will support the development and tailoring of evaluation support activities that meet the needs of each grant recipient. Results are not designed to be representative of or generalizable to a given subpopulation—the intent is to gather information specific to each grant recipient to tailor evaluation support activities to each of them.

Data collected under this information collection are not intended to be representative of broader populations. Data will not be used to assess participant outcomes. Such limitations will be included in written products associated with this data collection.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B2. Methods and Design

Target Population

The unit of analysis is the FSSDD grant recipient, of which there are 20. The types of respondents that will be included in the semi-structured discussions are staff who are employed by the organization that received the grant. Most of the staff respondents will be managerial or administrative leaders or program staff from the grant recipient organizations (typically community or social service nonprofit organizations).

Sampling and Site Selection

Respondents are staff working in organizations that received the ACF FSSDD grant. An expectation of their grant is active participation in evaluation support activities with the Contractor. Therefore, respondents will be asked to participate in these discussions as part of their grant activities.

B3. Design of Data Collection Instruments

Development of Data Collection Instrument

One instrument will be used: the FSSDD Discussion Guide. The discussion guide incorporates concepts from several bodies of literature on implementation science, evaluation capacity, evaluation capacity-building, performance management, and continuous quality improvement. It also reflects the practice wisdom and expertise of seasoned evaluators and program administrators who are now Contractor and Subcontractor staff. The discussion guide contains three modules: (1) well-defined and evidence-informed intervention; (2) implementation strength; and (3) intervention-specific staff capacity for using and building evidence.

Prior to conducting discussions with grant recipients, the Contractor will review the grant applications and publicly available documents to determine which information in the discussion guide is already available. The Contractor will not ask questions for which the answers are already known through existing documents. Additionally, only certain modules and questions in the discussion guide will be asked of each grant recipient, based on relevance to each grant recipient, as determined by review of the grant applications and other documents.

B4. Collection of Data and Quality Control

Mathematica, an ACF contractor, and its subcontractor, The Adjacent Possible, will collect the information. Contractor and subcontractor staff will reach out to the main point of contact at each grant recipient and work with that person to identify additional grant recipient staff to include in each discussion (based on the extent of their planned involvement in the grant and evaluation support activities). Contractor staff will set up a convenient time to meet virtually, either over videoconference or telephone. Contractor staff will be trained on how to implement the discussion guide by project leadership before conducting any discussions. Discussions will be audio-recorded. Respondents will be informed about the audio-recording and asked for their permission before recording occurs.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The semi-structured discussions are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.

NonResponse

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated.

B6. Production of Estimates and Projections

The data are strictly qualitative. They will not be used to generate population estimates, either for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

Each discussion will be conducted by two Contractor staff, with one staff member leading it and the other taking notes. Discussions will also be audio-recorded (if permission is given by the grant recipient staff). Data will not be coded.

Data Analysis

Data will be combined with information from grant recipients’ grant applications and other background information provided by the grant recipients to inform the development and tailoring of evaluation support activities. Contractor staff will extract key themes from interview notes and audio recordings to identify evaluation support activities that would address gaps or challenges relevant to each grant recipient. For example, if a grant recipient respondent reports that her intervention does not have a logic model, the Contractor staff might recommend an evaluation support activity to develop a logic model.

Data Use

This information will be used to develop tailored evaluation support activities for each of the 20 FSSDD grant recipients. These support activities will be delivered over a two-year period. Project briefs and a final report summarizing all of the evaluation support activities undertaken by the FSSDD grant recipients, their accomplishments, and lessons learned will be published in approximately 2026. Although the information collected through the proposed discussion guide will not be the primary subject of a public report or briefs, the briefs and report may include summarized findings from these interviews to contextualize the information presented.

B8. Contact Person(s)

The OPRE representative is Emily Ross, Social Science Research Analyst, Division of Economic Independence, OPRE, ACF, HHS. Her email is Emily.Ross@acf.hhs.gov.

The Contractor representative is Annalisa Mastri, Principal Researcher, Mathematica. Her email is AMastri@mathematica-mpr.com.

Attachments

Instrument 1_FSSDD Discussion Guide

4