Supporting Statement A: SIMR

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Strengthening the Implementation of Marriage and Relationship Programs (SIMR)

Formative Data Collections for ACF Research

0970-0356

Supporting Statement

Part A

October 2020

Submitted by:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4th Floor, Mary E. Switzer Building

330 C St., SW

Washington, D.C. 20201

Project Officers:

Samantha Illangasekare

Shirley Adelstein

Part A

Executive Summary

• Type of request: This information collection request is for a generic information collection under the umbrella generic, Formative Data Collections for ACF Research (0970-0356).

• Description of request: For this phase of the project, we are seeking clearance to conduct telephone interviews and a series of virtual meetings with healthy marriage and relationship education (HMRE) programs to collect information relevant to understanding HMRE programs and planning program-specific evaluations to be implemented in the Strengthening the Implementation of Marriage and Relationship Programs (SIMR) study. We do not intend for this information to be used as the principal basis for public policy decisions.

• Time sensitivity: We would like to begin phone interviews by late October 2020.

A1. Necessity for the data collection

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) requests permission to contact grantees funded by the Office of Family Assistance (OFA) for the purpose of gathering preliminary information about the grantees and their OFA-funded healthy marriage and relationship education (HMRE) programming for the Strengthening the Implementation of Marriage and Relationship Programs (SIMR) study. Permission to contact organizations for this purpose is requested under ACF’s generic clearance for formative data collections. The study team will collect information relevant to understanding the grantee’s HMRE program, which will support planning for the grantee-specific evaluation to be implemented by the SIMR team.

Study background

Since 2005, Congress has authorized dedicated funding for discretionary grants from the OFA to promote HMRE programs. ACF’s Office of Planning, Research, and Evaluation (OPRE) launched the SIMR study to understand implementation challenges faced by HMRE programs and test strategies to address these challenges. Using a mix of research methods, this study plans to partner with HMRE programs to test and build evidence for strategies to address implementation challenges and improve HMRE program implementation. The ultimate goal is to strengthen programs’ effectiveness and ability to participate in a potential, future summative evaluation. The study will include rapid-learning cycles—grounded in careful analysis of underlying problems and promising solutions—to test new approaches to common recruitment, retention, and content engagement challenges. The influence of SIMR will extend beyond this project as participating grantees become better prepared to evaluate the effectiveness of subsequent programs. ACF contracted with Mathematica and Public Strategies to implement the SIMR project.

The current information collection request is for gathering preliminary information about HMRE grantees to inform the development of SIMR.

Legal or administrative requirements that necessitate the collection

There are no requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose of survey and data collection procedures

Overview of purpose and approach

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for research and evaluation (0970-0356):

• Inform the development of ACF research

• Inform the provision of technical assistance

The purpose of the current information collection request is to seek approval to gather preliminary information about HMRE programs and explore with the programs the implementation challenges and solutions to those challenges that are most relevant to the programs. Once evaluation plans are finalized, the study team will submit another information collection request in order to seek approval for rapid-cycle testing activities to strengthen programs in preparation for summative evaluation. This future request will include surveys and guides for qualitative data collection, such as focus groups or interviews.

For over 10 years, OPRE has led a sustained effort within the federal government to develop, document, and evaluate HMRE programs, particularly with low-income populations. SIMR seeks to generate evidence about how to improve the effectiveness of HMRE programs through testing and refining strategies to overcome common implementation challenges related to recruitment, retention, and engagement. To do this, the study team will partner with grantees to implement solutions tailored to the context and specific challenges of each grantee and conduct multiple learning cycle assessments of the solutions. Early cycles will seek to understand the feasibility of implementing a strategy, while later cycles will examine the effectiveness of the solutions through more rigorous designs (for example, quasi-experimental).

The study team is requesting permission to contact programs from the 2020 round of OFA-funded HMRE grantees to gather information about their program plans, experiences, and the feasibility of implementing specific solutions to common implementation challenges and engaging in the rapid-learning activities. The information collection will take place from October 2020 through January 2021. The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information. The remainder of this section describes the study team’s plans for contacting programs and how the information will be used.

Research questions

1.) What is the range of characteristics of HMRE programs currently operating in the field, including the programs’ administrative structures, services offered, populations served, community service contexts, and capacities?

2.) What feedback do HMRE programs have in terms of a) the relevance of the identified implementation challenges to the field; b) their interest in the proposed interventions to test; and c) the feasibility of the potential design options?

Study design

This data collection effort serves as a preliminary step to gather information that will inform the SIMR study. First, the study team will conduct a review of HMRE grant applications from awarded grantees. ACF will provide the submitted grant applications for the study team to review. After this review, the study team will work with ACF to select up to 20 sites to contact for further information gathering. The criteria for selecting up to 20 sites is described in Supporting Statement Part B, Section B2.

The second step will involve a phone interview with program leadership in the 20 sites (up to two leaders per grantee). Upon OMB approval, the study team will send a request via email for a one-hour telephone call to the program leader(s) specified in their grant application from each grantee (see Appendix A, SIMR Phone Meeting Email Template). The email will introduce the study, including the goals and funders, describe the purpose of the phone interview, and offer suggested times. The SIMR study team will also attach the SIMR project description to this email (see Appendix B, SIMR Project Description) and a list of questions they hope to collect during the phone call (see Appendix C, Topics for SIMR Phone Meeting). Prior to the meeting, the SIMR team will send a reminder email to the program leader(s) (see Appendix A, SIMR Phone Meeting Email Template). The study team will lead the telephone interview using a semi-structured protocol, tailored based on the review of the applications (see Appendix D, SIMR Phone Interview Protocol). This conversation will include an overview of the SIMR study and its goals, and a series of questions aimed to further probe and confirm information from the grantee application. The SIMR team will also preview implementation solutions identified through a literature review and expert and stakeholder consultation to get input on the relevance and applicability of these to grantees. The protocol is designed to collect the minimum information necessary to allow us to understand the grantee’s organization and HMRE program and assess the feasibility of implementing and testing various solutions to implementation challenges through SIMR. The study team will contact up to 20 programs for a phone interview. The SIMR team will then review the information collected during these calls, which will serve to fill in gaps in the information gathered through the grant application review process, and select up to 12 sites for further information gathering. The criteria and process for selecting up to 12 sites is described in Supporting Statement Part B, Section B2.

The study team will follow up the phone interview with a request for further discussion with up to 12 sites via a series of three interactive, 90-minute virtual meetings. An agenda will be used to guide the three discussions (see Attachment E, SIMR Virtual Meeting Series Agenda). These meetings will include the program leader(s) and up to four additional staff members who are knowledgeable about the grantee’s organization and HMRE program (for example, staff supervisors or frontline staff such as a case manager). The goal of these meetings is to solicit input on a specific plan for an evaluation of a potential implementation solution or set of solutions by: (1) identifying challenges the grantee is likely to face while implementing the HMRE program; (2) identifying solutions that are of interest to the grantee that may address these challenges; (3) developing a primary evaluation plan; and (4) identifying potential needs and barriers to implementing that plan. Activities during the first meeting will focus on identifying current and/or anticipated implementation challenges. The second meeting will focus on pinpointing solutions that could address the challenges identified during the first meeting and that are most relevant and of interest to the grantee. The SIMR study team will also use the second meeting to introduce and describe the rapid-learning cycle process and receive input and feedback on this approach. The third meeting will delve further into the specifics of implementing the solutions identified during the second meeting and solicit input on a preliminary evaluation plan.

There are no quantitative components for the current phase of the SIMR project and this request.

Universe of data collection efforts

There are two main data collection efforts involved with the current request for approval summarized in the table below).

(1) SIMR Phone Interview Protocol (Appendix D): An initial one-hour phone interview that will use a semi-structured protocol, tailored based on the SIMR team’s review of the grantee applications submitted to ACF. To schedule this interview, the study team will also utilize an email template, the project description, and a handout that lists topics for the discussion (Appendices A–C).

(2) SIMR Virtual Meeting Series Agenda (Appendix E): A series of three 90-minute virtual meetings, outlined in a detailed agenda. For this collection, the study team will use Mural, an online whiteboard platform, to facilitate interactive activities during these meetings.

The phone protocol is designed to share information about the SIMR study and solicit information about the grantee’s organization and HMRE program and grantee’s thoughts on the feasibility of implementing and testing various solutions to implementation challenges. The virtual meeting agenda is designed to gather more detailed information and feedback on specific rapid learning cycle plans and implementation approaches from each grantee.

Data collection activity	Instrument(s)	Respondent, content, purpose of collection	Mode and duration
Phone interview	Phone interview protocol	Respondents: Program leaders Content: Overview of SIMR study and questions related to grantee’s organization and HMRE program. Purpose: This conversation will include an overview of the SIMR study and its goals as well as the benefits of participation, and a series of questions aimed to further probe and confirm information from the grantee application. The SIMR team will also preview implementation challenges and solutions identified through a literature review and expert and stakeholder consultation to get input on the relevance and applicability of these to grantees.	Mode: Phone Duration: 1 hour
Virtual meetings	Virtual meeting series agenda	Respondents: Program leaders and up to four additional staff Content: Interactive activities to identify implementation challenges and potential solutions to those challenges and solicit input on potential research designs. Purpose: The goal of these meetings is to solicit input on a specific plan for an evaluation of a potential implementation solution or set of solutions by: (1) identifying challenges the grantee is likely to face while implementing the HMRE program; (2) identifying solutions that are of interest to the grantee that may address these challenges; (3) developing a primary evaluation plan; and (4) identifying potential needs and barriers to implementing that plan.	Mode: Virtual meeting Duration: Three 90-minute meetings

A3. Improved information technology to reduce burden

The burden on grantees is minimal and the study team plans to use improved information technology wherever possible. The SIMR study team will collect as much information as they can from the initial grantee application review so that the phone interview can focus only on questions that are not addressed in applications. In response to COVID-19, we have altered original plans for in-person site visits to three virtual meetings. This virtual format should be less burdensome to grantees, since they do not have to host study team members for in-person site visits.

A4. Use of existing data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The study team will obtain as much information as possible from HMRE grantee applications. The phone interview and virtual meetings will focus only on obtaining information that cannot be extracted from applications, such as detailed information about anticipated challenges and feasibility of potential solutions.

A5. Impact on small businesses

We expect most of the programs in the study will be small, non-profit organizations. The burden for respondents will be minimized by restricting the interview length to the minimum required, by conducting telephone and virtual meeting discussions at times convenient for the respondents, and by not requiring record-keeping or written responses on the part of the programs.

A6. Consequences of less frequent data collection

The study team proposes a multi-staged process for gathering information from grantees. Without the information requested for this phase of the study, it would be difficult to move forward with the next stages of the study, including site recruitment, development of learning cycle study designs, and survey development. The approach attempts to limit the scope of conversations to just the information needed for the current phase of the project.

A7. Special circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal register notice and consultation

Federal register notice and comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. This notice was published on October 11, 2017, volume 82, number 195, page 47212, and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received.

Consultation with experts outside of the study

Several experts in HMRE programming and research provided consultation to the study team and ACF on multiple occasions throughout 2020. These experts have helped identify common implementation challenges facing HMRE programs and the solutions that may address these challenges. Experts also provided input on priority solutions we will test as part of SIMR.

A9. Tokens of appreciation

No tokens of appreciation for respondents are proposed for this information collection.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally identifiable information

This data collection effort does not include collecting personally identifiable information.

Assurances of privacy

As specified in the contract, the contractor shall protect respondent privacy to the extent permitted by law and will comply with all federal and departmental regulations for private information. The contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. Respondents are not considered human subjects, but they will still be informed of all planned uses of data, that their participation is voluntary, and that some of the information they provide may be shared with OPRE to help us design the study.

Data security and monitoring

No information will be given to anyone outside of the SIMR study team and ACF. Typed notes and audio recordings will be stored on Mathematica’s network, which is accessible only to the study team.

A11. Sensitive questions

There are no sensitive questions in this data collection.

A12. Burden

Grantee program leadership will review materials and complete a phone interview with SIMR staff about their organization and HMRE program. Program leadership and other staff will participate in a series of virtual meeting with the study team.

Estimated annualized cost to respondents

Instrument	Total/annual number of respondents	Number of responses per respondent	Average burden hours per response	Annual burden hours	Average hourly wage	Total annual cost
SIMR semi-structured phone interview	40	1	1	40	$34.20	$1,368
SIMR virtual meetings	72	3	1.5	324	$34.20	$11,080.80
Estimated annual burden total				364		$12,448.80

Total annual cost

To compute the total estimated annual cost, the total burden hours were multiplied by the estimated average hourly wage for local program directors (see table above). According to the Bureau of Labor Statistics’ Current Population Survey 2019, the median hourly wage for full-time employees over age 25 with a bachelor’s degree or higher is $34.20.

A13. Costs

There are no additional costs to respondents.

A14. Estimate annualized costs to the federal government

The total estimated cost for the federal government for the data collection activities under this current request will be $310,171.72. This includes personnel effort plus other direct and indirect costs.

Cost category	Estimated costs
Instrument development and OMB clearance	$20,238.00
Phone interviews	$75,056.19
Virtual meetings	$214,877.53
Total costs over the request period	$310,171.72

A15. Reasons for changes in burden

This is an individual information collection request under generic clearance 0970-0356.

A16. Timeline

The information collected under this request will be used to inform study design and development of the SIMR project. Upon OMB approval, initial calls to program leadership and virtual meetings for the purpose of information gathering will take place from October 2020 through January 2021. There are no plans to publish the information collected under this request.

A17. Exceptions

All instruments will display the expiration date for OMB approval. No exceptions are necessary for this information collection.

Attachments

Appendix A, SIMR Phone Meeting Email Template

Appendix B, SIMR Project Description

Appendix C, Topics for SIMR Phone Meeting

Appendix D, SIMR Phone Interview Protocol

Appendix E, SIMR Virtual Meeting Series Agenda