Supporting Statement A - R3-Impact 02.22.24_clean

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Replication of Recovery and Reunification Interventions for

Families-Impact Study (R3-Impact)

OMB Information Collection Request

New Collection

Supporting Statement

Part A

February 2024

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Calonie Gray

Kelly Jedd McKenzie

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a new collection. We are requesting three years of approval.

• Description of Request: The Replication of Recovery and Reunification Interventions for Families-Impact Study (R3-Impact) is a large multi-site study that will provide evidence about the effectiveness of two programs that use recovery coaches and what it takes to implement them in multiple settings with diverse populations. Data collection for the R3-Impact study under this information collection request includes (1) a parent survey measuring parent well-being at study enrollment, (2) quarterly contact forms to update participant contact information, (3) in-person and phone interview topic guides for site staff involved in program referral and implementation, (4) in-person and phone interview topic guide for parents participating in the study, and (5) a participant interview information form for parents interviewed. We do not intend for this information to be used as the principal basis for public policy decisions.

• Time Sensitivity: The goal is to begin the study pilot in April 2024 and full study enrollment in June/July 2024. It will be important to maintain this timeline so we can build the sample needed to detect impacts and carry out the five-year follow-up required by the 2018 Substance Use-Disorder Prevention that Promotes Opioid Recovery and Treatment (SUPPORT) for Patients and Communities Act before the expiration of federal funds. We are seeking OMB approval by November 2023 to ensure that data collected from the pilot can be included in the study sample.

A1. Necessity for Collection

The Replication of Recovery and Reunification Interventions for Families-Impact Study (R3-Impact) is being conducted in accordance with the 2018 Substance Use-Disorder Prevention that Promotes Opioid Recovery and Treatment (SUPPORT) for Patients and Communities Act (Public Law 115-271; see Attachment A). The SUPPORT Act authorized $15 million for the replication and evaluation of an intervention utilizing coaches for families engaged in the child welfare system due to parental substance use disorder (SUD). In response, the Office of Planning, Research, and Evaluation (OPRE) partnered with the Children’s Bureau (CB), both within the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS). Because ACF had not previously undertaken work in this area, together they launched the R3 project to lay the foundation for a rigorous evaluation of the effectiveness of recovery coaches to improve child welfare outcomes and SUD recovery outcomes to fulfill the legislative mandate. OPRE and CB partnered with Abt Associates in 2019 for the first phase of this work, Expanding Evidence on Replicable Recovery and Reunification Interventions for Families (R3-Feasibility; OMB No. 0970-0356), a feasibility and design study to lay the foundation for a second phase. The second and current phase, R3-Impact, is a large multi-site study that will provide evidence about the effectiveness of two recovery coaching programs and what it takes to implement them with fidelity in multiple settings with diverse populations.

A2. Purpose

Purpose and Use

Parental SUD has become one of the most common reasons families are involved in the child welfare system, driven in part by the opioid epidemic as well as ongoing misuse of other drugs and alcohol (Radel, Baldwin, Crouse, Ghertner & Waters, 2018; HHS, 2019). Recovery coaching is a promising approach to support parents who are working toward treatment completion, recovery, family preservation, and reunification with their children when possible. Recovery coaching refers to services provided by peers with lived experience in SUD and child welfare. Recovery coaches are trained peers who work one-on-one with parents to motivate and connect them to treatment and other services to support the recovery process, build recovery capital, and help navigate systems. While recovery coaching interventions hold promise, information about their effectiveness in the child welfare context is sparse (Francis et al., 2021).

The R3-Impact Study aims to fill gaps in the evidence base with a rigorous, multisite impact study that includes existing and new program sites across multiple states and policy contexts. The evaluation will provide policymakers, practitioners, and program funders with high-quality evidence about two programs that use recovery coaches: (1) Parent Mentor Program (PMP) and (2) Sobriety Treatment and Recovery Teams (START). Both programs were designed to serve diverse families involved in the child welfare system with parental SUD as a primary risk factor.

The R3-Impact Study includes independent impact evaluations of PMP and START and an implementation evaluation of each program.

• PMP Impact Evaluation. PMP recovery coaches, called parent mentors, are individuals with lived experience with SUD and as a parent formerly engaged in the child welfare system. They use motivational interviewing to help parents identify and achieve their recovery goals in a self-directed way, while also helping them navigate the child welfare system and eliminate barriers to recovery. The impact evaluation of PMP will test the program’s effects on parent well-being (e.g., substance use, parenting stress) and child welfare outcomes (e.g., foster care placement, subsequent maltreatment) using an experimental design. To our knowledge, no evaluations of recovery coaching in child welfare have investigated impacts on proximal parent outcomes due to the high cost of collecting primary data from a treatment group and a control or comparison group of sufficient size to detect impacts. Thus, the R3-Impact evaluation has the potential to assist with narrowing the knowledge gap at the intersection of the child welfare and peer recovery coaching literatures regarding impacts on substance use recovery and adult well-being more generally. The impact evaluation will align with the design and execution standards of the Title IV-E Prevention Services Clearinghouse¹ and add to what is known about the effectiveness of interventions in helping parents achieve positive recovery outcomes and prevent foster care placement and subsequent maltreatment.

• START Impact Evaluation. START pairs Family Mentors (peers with lived experience with SUD and the child welfare system) with specialized child welfare caseworkers trained in the START model; they share a caseload and provide recovery support for parents alongside intensive case management. The START impact evaluation will test the program’s effect on foster care placement and subsequent maltreatment using a quasi-experimental design.² The START impact evaluation will measure outcomes using only child welfare administrative data³, thus no data collection activities are proposed under this information collection request. However, we discuss it here for context, as the study will include an implementation evaluation of the START program (as discussed in the following bullet).

• Implementation Evaluation of PMP and START. The implementation evaluations of PMP and START have two complementary goals to support adoption of evidence-based practices in child welfare. The first goal is to promote understanding of program implementation and how implementation varies across program sites, for use in interpreting the impact study findings. The second goal is to generate knowledge about conditions required to replicate the programs with fidelity in new locations and with different populations. This information collection request includes data collection activities for both the PMP and START implementation evaluations.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests

PMP’s logic model posits that its peer recovery coaching services will improve participants’ parenting stress, SUD recovery, and child welfare case engagement over the short, medium, and long term. We hypothesize that improvements in these proximal parent-level outcomes that are PMP’s direct targets may also contribute to the prevention of foster care entry and subsequent maltreatment.

Understanding PMP’s impacts on its intended outcomes and on child welfare outcomes will help policymakers, funders, and service providers determine where best to invest resources. Based on these considerations, we have identified the research questions listed below for a rigorous impact evaluation of PMP.

#	PMP Impact Evaluation Research Questions
Primary Question (Confirmatory)
RQ1	What is the effect of PMP on parent mental and emotional health?
RQ2	What is the effect of PMP on substance use?
RQ3	What is the effect of PMP on parent engagement with the child welfare case?
Secondary Questions (Exploratory)
RQ4	What is the effect of PMP on family functioning?
RQ5	What is the effect of PMP on economic and housing stability?
RQ6	What is the effect of the offer of PMP on substance use treatment engagement? On recovery support services engagement?
RQ7	What is the effect of PMP on foster care placement?
RQ8	What is the effect of PMP on any child permanency? On the average time to permanency?
RQ9	What is the effect of PMP on experiencing a reunification with a child who has been removed from the home? On the average time to reunification?
RQ10	What is the effect of PMP on other permanency (e.g., adoption, guardianship)? On the average time to other permanency?
RQ11	What is the effect of PMP on subsequent substantiated allegations of maltreatment?
RQ12	What is the effect of PMP on parent mental and emotional health, substance use, parent engagement with the child welfare case, family functioning, economic and housing stability, substance use treatment engagement, recovery support services engagement, foster care placement, any child permanency, reunification, and subsequent maltreatment outcomes for subgroups defined by race and ethnicity? Do these effects differ according to race/ethnicity? Are there any baseline factors, including those related to inequities and marginalization, associated with any differential impacts?
RQ13	Do the effects of PMP differ according to other baseline characteristics (such as level of previous child welfare involvement and level of risk for foster care placement)?
RQ14	Does engagement in substance use treatment or recovery support services lead to (i.e., mediate) favorable effects of PMP on parent well-being or child welfare outcomes?
RQ15	Do short-term improvements in parent well-being lead to (i.e., mediate) favorable effects of PMP on child welfare outcomes?
RQ16	Does the use of PMP result in any net savings for states, accounting for costs borne by states and the federal government?

Note: We plan to use an intent-to-treat (ITT) framework to estimate impacts. Therefore, the effect of PMP in the research questions will be estimates of the effect of the offer of PMP on outcomes. The impact analysis of RQs 8-10 will be conducted on the endogenous subgroup of children who were placed into foster care after baseline.

The implementation evaluation of the PMP and START programs will provide a framework for understanding the role of program implementation and its variation across sites in interpreting the impact study findings. It will also provide insight about what it takes to replicate the recovery coaching programs in new locations and with different populations. The table below shows the research questions that the implementation evaluation will explore.

#	PMP and START Implementation Evaluation Research Questions
RQ1	To what extent is the program implemented as intended?
RQ2	How does the organizational context in which the programs are implemented inform implementation fidelity?
RQ3	How does the community and policy context in which programs are implemented inform implementation fidelity?
RQ4	How do key partners coordinate to implement the program?
RQ5	What are the considerations associated with successful replication? What are the barriers and what strategies were used to mitigate them?
RQ6	What modifications, if any, were needed to fit local and cultural contexts?

Study Design

The R3-Impact Study has three primary components:

1. The impact evaluation of PMP will use a randomized controlled trial (RCT) design. As noted in section B1 of Supporting Statement B, an RCT design is appropriate in the study sites (Michigan, Minnesota, Virginia, and Oregon) as the program being tested is not yet the standard of care for the target population. As such, random assignment can be done without withholding longstanding services to the target population. In this design, eligible parents (those receiving services to prevent a foster care placement and who have substance use as a primary risk factor) will be randomly assigned to either an offer to receive services as usual (SAU) plus PMP or to services as usual.

The key outcomes will be self-reported measures of parent well-being and engagement with the child welfare case. The evaluation will collect information on these measures in a parent survey administered at study enrollment and then again at approximately 15 months after a parent enrolls in the study.⁴ The evaluation will also collect child welfare administrative records from the participating states to measure foster care placement, reunification, and subsequent maltreatment. Further, the administrative data will be collected across the study to assess select outcomes for a 5-year follow-up period as outlined in the SUPPORT Act.

The PMP impact evaluation has two key limitations. First, the parent well-being outcomes will be assessed at a single point in time, approximately 15 months after study enrollment. Given the changes that can accompany SUD recovery, this single wave of follow-up may not be able to capture the post-program average well-being for participants. Instead, the follow-up survey may occur at a peak or trough of adult well-being. If the program has an effect on the timing of SUD recovery, the differences in well-being at 15 months may not reflect average well-being in the post program period. Second, the PMP impact evaluation includes only four states (referred to as sites). While this set of results should be more generalizable than the results from a single site, it is quite possible that the average results of these four sites are not generalizable to certain prospective sites that have characteristics that differ from these four sites. We will note these limitations in R3-Impact Study publications.

2. The impact evaluation of START will include at least four states (sites) implementing the program. In each site, a quasi-experimental design will match START families to similar families found in administrative data. The START evaluation will estimate the pooled effect of the offer of START by combining site-specific impact estimates using meta-analytic methods. For the START evaluation, the key outcome will be foster care placement, an outcome available from administrative data. The evaluation will also examine the outcomes of subsequent maltreatment and reunification from administrative data records. We include the description of the START evaluation here for context but are not requesting any primary data collection for the START impact study under this information collection request. The START evaluation also has some key limitations. First, because the study will not include any primary data, i.e., parents’ self-reports, there will be limited available data to contextualize the study’s findings. Issues of data quality are also of concern when using administrative data, though the study team will utilize appropriate analyses to detect and ameliorate any data-quality issues to the extent possible. Finally, the START evaluation will use a quasi-experimental design requiring the identification of well-matched comparison sites. However, the degree of comparability among availability sites is beyond the study team’s control. As with the PMP evaluation and appropriate, we will note these limitations in R3-Impact Study publications. The implementation evaluation of PMP and START will document local contexts, the operational structures and partnerships that support program implementation, fidelity of implementation, implementation facilitators and barriers, and staff and parent experiences. Data collection for the implementation study will occur during two rounds of site visits to all PMP sites and up to two START sites. The first visit will be in-person and the second will be virtual. In each site, we will conduct semi-structured interviews with child welfare leadership and frontline staff, program managers, parent mentors, parent mentor supervisors, and parents.

Data Collection Activity	Instruments	Respondent, Content, Purpose of Collection	Mode and Duration
PMP Impact Study
Baseline measure of parent primary outcomes	Baseline Parent Survey (Instrument 1)	Respondents: Parents involved in the child welfare system that are identified as having substance use risk factors who agree to participate in the study. Content: Survey questions will ask about varied aspects of parents’ well-being (mental and emotional health, substance use, family functioning, economic and housing stability, and physical health), substance use treatment, and recovery support services engagement. Purpose: To describe the sample at baseline, assess baseline equivalence of the treatment and control/comparison groups, and include in impact models as baseline measures of the outcomes.	Mode: Electronic using Audio Computer-Assisted Self-Interviewing (ACASI). Completed in-person with computer-assisted personal interviewing (CAPI) or, if necessary, over the phone with a field interviewer using computer-assisted telephone interviewing (CATI) Duration: 45 minutes
Collecting Updated Contact Information	Contact Form (Instrument 2)	Respondents: Parents involved in the child welfare system that are identified as having substance use risk factors who agree to participate in the study. Content: Participant contact information Purpose: To obtain up-to-date contact information for participants to enhance response rates on the 15-month follow-up survey (which will be submitted in a future package).	Mode: email, SMS, and mail, and phone (response option) Duration: 10 minutes
Assessing the quality of data collection	Validation Interview Script (Instrument 3)	Respondents: Parents involved in the child welfare system that are identified as having substance use risk factors who agree to participate in the study. Content: Questions about the survey experience and two questions from the baseline survey. Purpose: To monitor the quality of survey data collection.	Mode: Completed over the phone with a field interviewer supervisor. Duration: 5 minutes
PMP and START Implementation Study
Site Visit	Topic Guides	Purpose: Enable the evaluation team to describe local contexts, program implementation, and staff and participant perceptions of the program. Information will be used to interpret the impact analysis results, and identify lessons learned for the purpose of program replication. Content: Implementation facilitators; Roles; Barriers; Perceptions; Coordination; Service Types; Community Needs/Gaps; Context; Adoption; Acceptability; Implementation Activities Each topic guide is described below.	Mode: In-person for the first visit and virtual for second visit.
	Topic Guide - Child Welfare Lead Staff (Instrument 4)	Respondents: Child welfare lead staff	Duration: 60 minutes per interview.
	Topic Guide - Child Welfare Frontline Staff (Instrument 5)	Respondents: Child welfare frontline staff	Duration: 60 minutes per interview
	Topic Guide – Partners (Instrument 6)	Respondents: Key collaborative partners such as treatment providers, mental health providers, and housing agencies	Duration: 60 minutes per interview
	Topic Guide – Program Managers (Instrument 7)	Respondents: Program managers	Duration: 90 minutes per interview
	Topic Guide – Mentor Supervisors (Instrument 8)	Respondents: Parent mentor supervisors	Duration: 90 minutes per interview
	Topic Guide - Parent/Family Mentors (Instrument 9)	Respondents: Parent/family mentors	Duration: 90 minutes per interview.
	Topic Guide – Parents (Instrument 10)	Respondents: Parents participating in PMP and START.	Duration: 60 minutes per interview.
Site Visit	Participant Interview Information Form (Instrument 11)	Respondents: Parents participating in PMP and START. Content: Demographic information to support analysis of parent perspectives by personal characteristics and history. Purpose: Allow the study team to use demographic information while analyzing the parent interviews.	Mode: Paper Duration: 6 minutes

Other Data Sources and Uses of Information

Impact Evaluation

The impact evaluation will examine the programs’ effect on parent outcomes as measured by the Baseline Parent Survey-Instrument 1 (for PMP only) and child welfare outcomes as measured by administrative data (for PMP and START). For child welfare outcomes, we will use data on child safety, family preservation, and permanency from each site’s Statewide Automated Child Welfare Information System (SACWIS) or Comprehensive Child Welfare Information System (CCWIS).

Implementation Study

In addition to the data obtained using the instruments described above, the implementation evaluation will use program data to measure fidelity of implementation. This includes service delivery data documented by PMP staff as part of program delivery, and aggregated fidelity scores prepared by START fidelity reviewers. These data will allow us to understand service dosage and intensity by measuring the number, length, and format of recovery coach contacts; achievement of parent goals; referrals to treatment and recovery support services; and other key aspects of program implementation.

A3. Use of Information Technology to Reduce Burden

This study will use information technology to minimize participant burden and to collect data efficiently. The baseline parent survey will be administered electronically (using a computer or tabletwhen possible. Completing the form electronically will move the participant quickly through the form. The electronic baseline survey will utilize skip pattern logic, reducing participant burden by allowing them to move quickly to the next appropriate question depending upon a participant’s previous answer. When the baseline survey is administered in person, it will utilize both CAPI and ACASI, which allows participants to listen to prerecorded survey questions through headphones, rather than having the questions read to them by field staff. When an in-person administration is not possible, field interviewers will use CATI.

The implementation study interviews will be audio-recorded, assuming the participant provides consent. This will reduce the likelihood of researchers needing to ask participants for clarification following the interview.

The contact form will be administered via email, SMS text, or mail, allowing participants to update their contact information easily, efficiently, and at a time most convenient for them. Participants can return the paper contact form, call us to provide updated information (and an Abt staff member will enter into the web form on their behalf), or access a personal, secure link to update their information.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

Primary data being collected for R3-Impact Study is not available in any other form in a consistent manner across the evaluation’s sites.

Impact Evaluation

The PMP program focuses on three core outcomes: decreasing parenting stress, improving SUD recovery, and increasing parent engagement with the child welfare case. These outcomes, in turn, are expected to contribute to the downstream goals of preventing recurrence of maltreatment, preventing foster care placement, and family reunification when possible. While we can measure longer-term child welfare outcomes using administrative data, the shorter-term parent well-being outcomes must be explored through primary data collection. To our knowledge, no evaluations of recovery coaching in child welfare have investigated impacts on proximal parent outcomes due to the high cost of collecting primary data from a treatment group and a control or comparison group of sufficient size to detect impacts. Thus, the R3 impact evaluation has the potential to begin to fill an important knowledge gap at the intersection of the child welfare and peer recovery coaching literatures regarding impacts on SUD recovery and adult well-being more generally.

Implementation Evaluation

Information that is being collected for implementation research is not expected to be available in any other form. For example, it would be impossible to collect information on staff perceptions of program facilitators and barriers without obtaining it directly from those involved in program implementation. However, the study team will verify with each site that information being requested is available only through the qualitative interviews that are proposed.

A5. Impact on Small Businesses

We anticipate that some of the implementation study respondents may be small private non-profit organizations. We will minimize the burden on program staff by scheduling data collection at times that are convenient for participants and utilizing virtual data collection options as needed to reduce the time needed for travel.

A6. Consequences of Less Frequent Collection

The R3-Impact data collection aims to collect information only as frequently as needed to achieve the aims of the study. Eliminating any proposed data collection items would compromise our ability to address key research questions.

1. Impact Evaluation – Baseline Parent Survey (Instrument 1). The baseline survey will be administered once at study enrollment prior to random assignment. Without the baseline survey, we would be unable to verify that treatment and control groups are similar in their observable background characteristics and in their baseline measures of outcomes. The baseline survey is also essential for describing the characteristics of the study sample and capturing information about baseline levels of the study’s primary parent outcomes and correlates.

2. Impact Evaluation – Contact Form (Instrument 2). To obtain up-to-date contact information for study participants, we will employ a participant tracking strategy where we will use frequent and responsive incentivized contacts designed to encourage participants to keep their contact information updated. Participants will be asked to complete the brief contact form four times (in three-month intervals) between the baseline collection and the 15-month follow-up survey. A lack of, or outdated, contact information would make it difficult to achieve the target response rates. It could also diminish the participant’s connection to the study, which could make it difficult for interviewers to gain cooperation at the 15-month follow-up even if they are successful in locating participants.

3. Implementation Evaluation Interviews (Instruments 3-8). The study team will conduct semi-structured interviews with Child Welfare Leadership and frontline staff, program managers, parent and family mentors and their supervisors, at two points in time, when possible. We will also conduct semi-structured interviews with parents at one point in time. We propose to conduct two rounds of implementation study visits to all PMP sites and up to two START sites to help understand implementation of the programs over time, thus necessitating data collection at two points. These interviews will be critical to understand the implementation of each program and its context.

4. Participant Interview Information Form (Instrument 9). For the in-depth participant interviews, participants will be asked to complete a brief form. This instrument will capture demographic information that will allow the research team to describe in aggregate the group of respondents. Lack of demographic information would hinder the evaluation’s ability to understand how parents may experience the programs differently depending on their backgrounds.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on June 6, 2023, Volume 88, Number 108, page 37067, and provided a sixty-day period for public comment. During the notice and comment period, no comments were received.

Consultation with Experts Outside of the Study

In developing the data collection instruments and study design, the study team consulted with a small number of individuals with pertinent knowledge in child welfare systems, family preservation and reunification, SUD treatment, recovery coaching programs, and other relevant areas of interest. To ensure equity and account for power differentials and marginalization across experts’ ways of knowing, we sought input from two types of experts: those with lived and frontline experience (i.e., Parent Mentors implementing PMP) and research and practitioner experts.⁵

• A group of four PMP Parent Mentors participated in nine workgroup sessions to provide feedback on the parent well-being outcomes and associated questions in the baseline survey instrument. The R3 project team solicited input on the appropriateness of the outcomes, the wording of survey questions, the response options, and the timing of survey administration. They also provided input on scripts introducing the evaluation and the survey.

• An advisory group comprised of seven research and practice experts consulted on the identification of specific candidate programs and evaluation design options, and a subset of three research experts consulted on outcome measures.

A9. Tokens of Appreciation

The R3-Impact study is designed to examine the effectiveness of two programs using recovery coaching (PMP and START) using experimental and quasi-experimental designs, with a longitudinal follow-up. The R3-Impact Study panel for the PMP evaluation is small (2,750 parents) and a high response rate is necessary to maintain statistical power to detect meaningful effects when measuring participant outcomes. In addition, the integrity of the study’s estimates requires maintaining similar response rates for the treatment and control or comparison groups and across demographic groups of central interest to the research study. Maintaining high response rates will be especially difficult in the R3-Impact Study because the study includes a target population facing SUD who may be particularly difficult to maintain contact with over time. Their circumstances often result in frequent moves, short stays in hospitals or treatment centers, short periods of time living with others, and in some instances, homelessness. Because of the complex design and study population, it is important to build respondent buy-in early in the study and retain as much of the sample as possible over time.

In addition to other efforts to recruit and retain respondents (see Supporting Statement B, section B4, for additional information), we propose including tokens of appreciation to study participants at enrollment, after completing the baseline survey, and after completing each quarterly contact form. Research has shown that tokens of appreciation are effective at increasing response rates for populations similar to participants in the R3-Impact Study who tend to respond less to surveys (i.e., young people, racial or ethnic minorities, or groups with low socio-economic status), and are financially prudent for researchers because they reduce time spent pursuing responses (Lipps et al., 2019). The tokens of appreciation are intended to build participant buy-in and maintain participation over time. The participant perspective will be a critical data source for the implementation study, and offering tokens of appreciation will promote interview participation and support their participation.

We propose the following structure and rationale for tokens of appreciation:

• Study participants will be sent a $40 Visa gift card for completing the 45-minute Baseline Parent Survey. Recent research has found that $20 to $40 is an effective range for collecting data from disadvantaged populations (D’Angelo et al., 2016). Furthermore, our experts with lived experience recommended $40 because it is the approximate cost of a package of diapers. To contextualize the proposed amounts, our strategy is similar to that approved for longitudinal survey studies in other federal information collections with target populations similar to R3-Impact. For example, the Building Evidence on Employment Strategies (BEES) Project (OMB control number 0970-0537), which targets low-income individuals with substance and opioid use disorder, is using a $25 token of appreciation for a shorter 30-minute follow-up survey. While the follow-up survey collection for BEES is not complete, early lessons are that the token of appreciation combined with accurate contact information are key to being able to locate participants 12 to 15 months after enrollment. The Family Options Study (OMB #2528-0259), whose target population is low-income families experiencing homelessness, used a $50 token of appreciation for a 60-minute follow-up survey. In the use of tokens of appreciation enhanced the ability of local interviewers to engage study participants in all aspects of participant tracking and data collection. The study ultimately achieved high response rates to the follow-up surveys; more than 80 percent for the 18-month follow-up survey, and 78 percent for the three-year follow-up survey.

• Study participants will be mailed a $5 cash token of appreciation for completing each brief contact form (up to four, or a possible total of $20). The BEES Project (OMB #0970-0537) also used a $5 token of appreciation for updating contact information quarterly post study enrollment. This token of appreciation has increased the ability of the local interviewers to maintain accurate contact information throughout the study follow-up period of 12-15 months.

• Study participants in PMP and START who complete a 60-minute semi-structured interview will be given a $50 Visa gift card. Unlike other implementation study participants who will participate in interviews within their professional capacity during their normal workday, PMP and START participants may need to take time off work, secure childcare, or participate during their free time. For context, this amount is consistent with the $50 token of appreciation used in two other studies of similar populations: the Mother and Infant Home Visiting Program Evaluation (MIHOPE) (OMB control number 0970 - 0402) and BEES (OMB control number 0970-0537). Both of these studies determined this was an appropriate amount to offset the costs of participation such as childcare or transportation and the burden associated with participation in the qualitative interviews

We believe $50 is a reasonable amount for the time and cost associated with participation in these data collection activities but is not so high as to appear coercive for potential participants.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

For the R3-Impact evaluation, the collection of personally identifiable information (PII) is necessary for participant locating for the 15-month follow-up survey and to allow us to access and match administrative records data. Participants who have agreed to be in the study will be asked to provide PII on the Baseline Parent Survey (Instrument 1) including, the name, date of birth, address, phone number, and email information for all study participants. Name and contact information are necessary to aid in the contact update and survey data collection procedures. The last four digits of social security numbers will be requested to aid in matching to the SACWIS and CCWIS administrative databases and in confirmation of respondent identity for the follow-up survey.

Assurances of Privacy

Respondents will be informed of all planned uses of data, that their participation in the impact study and in the interviews is voluntary, that audio recording of interviews is voluntary, and that their information will be kept private. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information. All data collection protocols will receive Institutional Review Board (IRB) approval before data collection begins. The study’s consent forms are included as Appendix B and C.

At least some of the information collected under this ICR will likely be retrieved by an individual’s personal identifier in a way that triggers the Privacy Act of 1974, as amended (5 U.S.C. 552a). The system of records notice (SORN) for this collection will be 09-80-0361: OPRE Research and Evaluation Project Records. Each individual will be provided with information that complies with 552a€(3) prior to requesting information that will be placed into that system of records. This means respondents will receive information about the authority, the purposes for use, the routine uses, that the request is voluntary, and any effects of not providing the requested information.

Due to the sensitive nature of this research (see A.11 for more information), the evaluation will obtain a Certificate of Confidentiality. The Certificate of Confidentiality helps to assure participants that their information will be kept private to the fullest extent permitted by law.

Data Security and Monitoring

As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ PII. The Contractor will ensure that all its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.

As specified in the contract, the Contractor will use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor will securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor will: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor will submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or PII that ensures secure storage and limits on access.

The study team plans to use the National Data Archive on Child Abuse and Neglect (NDACAN) to archive the study’s data. Data to be archived will include the baseline survey data collected under this information collection request, for those participants that have consented to sharing/archiving of their data. The NDACAN has archived adult well-being and substance use related data, in addition to child welfare related outcomes. Given the vulnerability of the study population and sensitivity of the data to be collected, the study team will request the highest level of data access restrictions allowable by the archive: the same level of security and restricted access data licensing used for the National Child Abuse and Neglect Data System (NCANDS) child data files. The study team will gather consent for archiving de-identified data from study participants as part of our study enrollment consent procedures. The study team will compile the documentation required by NDACAN such as instrument forms, IRB approval and consent forms, data collection and sampling procedures, a codebook/data dictionary, and copies of reports, in accordance with the approved archive plan.

A11. Sensitive Information ⁶

Some questions on the Baseline Parent Survey may be considered sensitive. Individuals are being asked about their use of alcohol and other drugs as well as about their physical and mental health, particularly depression symptoms and parenting stress. These questions are necessary because a goal of the study is to understand the effects of the recovery coaching programs on parents’ well-being, including mental and emotional health, substance use, quality of life, family functioning, and economic and housing stability. As noted in section A4, this information will not be available from other data sources.

Across all data collection, participants will be informed by research staff prior to the start of the interviews or surveys that their answers will be kept private, that results will only be reported in the aggregate, and that their responses will not affect any services or benefits they or their family members receive.

At the point of enrollment in the study, the informed consent forms (Appendix B and C) will provide an overview of data collection efforts to expect during the course of the study. Staff obtaining consent from participants will be trained to answer questions about what it means to participate in the study.

A12. Burden

Explanation of Burden Estimates

Burden estimates were calculated based on the following assumptions:

• Baseline Parent Survey. The survey will be administered to all study participants at the time of study enrollment. We expect to enroll 2,750 respondents across the four PMP sites. Parents will be required to complete the baseline survey as a condition of study enrollment, though they may choose not to answer questions on the survey. Because of this, we expect a 100 percent participation rate for the survey for a total of 2,750 completed surveys. The survey is estimated to take 45 minutes to complete with a field interviewer, with a portion of the survey to be self-administered if the participant chooses.

• Contact Form. Due to timing and cost considerations, only the first two-thirds of families enrolled in the study will be asked to complete the follow-up survey. As the goal of the contact form is to maintain up-to-date contact information for the follow-up survey, the contact form will only be sent to the study participants in the follow-up survey sample. We estimate that 1,843 study participants (two-thirds of the full sample) will be invited to complete the contact form. The contact form will be sent to each participant four times, 3 months, 6 months, 9 months, and 12 months after study enrollment. Participants will complete the contact form online or by mail. The study team estimates that it will take 10 minutes to complete.

• Validation Interviews. The study team plans to conduct validation interviews with 10 percent of study participants (approximately 275 participants). The study team will conduct the interviews by phone; they are expected to take approximately 5 minutes.

• Interviews with Child Welfare Lead Staff, Child Welfare Frontline Staff, Program Managers, Mentor Supervisors, and Parent / Family Mentors. The study team will conduct interviews with each respondent type during two rounds of site visits. We expect to interview five individuals in each respondent category in each site during (5 respondents * 6 sites = 30 respondents in each respondent category). Given high staff turnover rates in service organizations and the desire to capture a range of perspectives, we expect to interview different staff in each of the two site visits (30 respondents * 2 site visits = 60 in each respondent category). Interviews will last no more than 60 minutes for Child Welfare Lead and Frontline Staff and 90 minutes for Program Managers, Mentor Supervisors, and Parent / Family Mentors.

• Interviews with Program Partners. The study team will conduct 60-minute interviews with representatives from key collaborative partners such as treatment providers, mental health providers, and housing agencies. We expect to interview ten partners in each site during both site visits (10 respondents * 6 sites * 2 site visits = 120 program partners).

• Interviews with Parents. The study team will conduct 60-minute interviews with 5 parents in each site (5 parents * 6 sites = 30 parents). The study team will only conduct parent interviews during the first site visit, due to the challenges of conducting these interviews virtually.

• Participant Interview Information Form. All parents who participate in an interview will be asked to complete the Participant Interview Information Form. The form will be completed on paper and will take approximately 6 minutes to complete.

Estimated Annualized Cost to Respondents

Table A12 shows the annual burden and cost of the data collection instruments and activities described in this ICR. The assumed wage rate is based on the May 2022 employment and wages from Occupational Employment Statistics survey from the Bureau of Labor (https://www.bls.gov/oes/current/naics4_999300.htm#21-0000):

• Program participants are likely to be lower-income and working in entry level positions. The rate used for program participants is $11.55, which is the mean of the average hourly wage in each of the four PMP sites (Michigan $10.10, Minnesota $10.59, Oregon $13.50, and Virginia $12.00), according to the US Department of Labor Consolidates State Minimum Wage Update Table: Consolidated Minimum Wage Table | U.S. Department of Labor (dol.gov).

• The rate used for Child Welfare Lead Staff, $31.43, is equivalent to child, family, and School Social workers in local government under SOC code 21-1021.

• The rate used for Child Welfare Frontline Staff, $27.00, is equivalent to the Community and Social Service Specialists in local government under SOC code 21-1099.

• The rate used for Program Managers and Program Partners $46.83 is equivalent to Social and Community Service Managers under SOC code 11-9151.

• The rate used for Mentor Supervisors, $29.06, is equivalent to First-Line Supervisors of Personal Service Workers under SOC code 39-1022.

• The rate used for Parent/Family Mentors, $23.43, is equivalent to Social and Human Service Assistants under SOC code 21-1093.

Table A12: Estimated Annualized Cost to Respondents
Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total Burden (in hours)	Annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Baseline Parent Survey	2,750	1	.75	2063	688	$11.55	$7,946.40
Contact Form	1843	4	.17	1253	418	$11.55	$4,827.90
Validation Interview Script	275	1	.08	22	7	$11.55	$80.85
Topic Guide – Child Welfare Lead Staff	60	1	1	60	20	$31.43	$628.60
Topic Guide – Child Welfare Frontline Staff	60	1	1	60	20	$27.00	$540.00
Topic Guide – Partners	120	1	1	120	40	$46.83	$1,873.20
Topic Guide – Program Managers	60	1	1.5	90	30	$46.83	$1,404.90
Topic Guide – Mentor Supervisors	60	1	1.5	90	30	$29.06	$871.80
Topic Guide – Parent/Family Mentors	60	1	1.5	90	30	$23.43	$702.90
Topic Guide – Parents	30	1	1	30	10	$11.55	$115.50
Participant Interview Information Form	30	1	.1	3	1	$11.55	$11.55
Total	5,348			3,881	1,294		$19,004

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

The total cost for the data collection activities for the PMP impact study and the implementation study of PMP and START is estimated to be $5,446,408. Annual costs to the Federal government will be $1,815,469 for the proposed data collection. The table below breaks down total costs by category.

Cost Category	Estimated Costs
Field Work	$2,967,621
Analysis and Reporting	$1,337,323
Dissemination	$169,461
Total costs over the request period	$4,474,405
Annual costs	$1,491,468

A15. Reasons for changes in burden

This is a new information collection request.

A16. Timeline

We expect baseline data collection to take place over approximately a three-year period, following OMB approval. Prior to OMB approval, sites will begin a formative test period focused on refining implementation of the program, to ensure fidelity of implementation. The length of the formative period will vary by site and may begin prior to OMB approval in some sites⁷. Following OMB approval and the formative test, each PMP site will conduct a 3-month pilot test focused on testing study enrollment procedures to ensure that all aspects of the evaluation are carried out as required and any necessary adjustments are made before proceeding to the impact study full enrollment period. Burden for the pre-testing is included in this ICR. PMP sites will being their full enrollment period on a rolling basis following the pilot test.

Implementation evaluation interviews will begin at 12 and 24 months following the start of the pilot test. Pending OMB approval, we expect this data collection will continue for 3 years.

The table below summarizes the data collection timeline and the publication of implementation and impact findings through briefs, reports, and peer-reviewed journal articles.

Activity	Length of Activity	Timeframe Post OMB Approval
Formative Test	Varies (2-6 months)	Prior to OMB approval
Pilot test, including Baseline Parent Surveys / Study Enrollment	3 months	Months 1–3
Ongoing Baseline Parent Surveys / Study Enrollment	33 months	Months 4–36
Contact Information Form	48 months	Months 6–54
Implementation Study Interviews	24 months	Months 12–36
Publications/Dissemination	84 months	Months 6–90

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

• Instruments

  • Instrument 1: Baseline Parent Survey

  • Instrument 2: Contact Form

  • Instrument 3: Validation Interview Script

  • Instrument 4: Topic Guide – Child Welfare Lead Staff

  • Instrument 5: Topic Guide – Child Welfare Frontline Staff

  • Instrument 6: Topic Guide – Partners

  • Instrument 7: Topic Guide – Program Managers

  • Instrument 8: Topic Guide – Mentor Supervisors

  • Instrument 9: Topic Guide – Parent/Family Mentors

  • Instrument 10: Topic Guide – Parents

  • Instrument 11: Participant Interview Information Form

• Appendices

  • Appendix A: Text from SUPPORT Act Section C.8082

  • Appendix B: Informed Consent Form – Impact Evaluation

  • Appendix C: Informed Consent Form – Parent Interview

  • Appendix D: Crosswalk of Survey Items to Research Questions

  • Appendix E: Crosswalk of Implementation Topic Guides to Research Questions

References

Choi, S., Huang, H., & Ryan, J. (2012). Substance abuse treatment completion in child welfare: Does substance abuse treatment completion matter in the decision to reunify families? Children and Youth Services Review, 34, 1639-1645.

D’Angelo, A.V., Crowley, J., Maul, L., & Strong, D. (2016). Data Collection Technical Assistance Brief: Offering Data Collection Incentives to Adults at Risk for Substance Use Disorder. Mathematica Policy Research. https://www.mathematica.org/publications/offering-data-collection-incentives-to-adults-at-risk-for-substance-use-disorder

Green, B., Rockhill, A., & Furrer, C. (2007). Does substance abuse treatment make a difference for child welfare outcomes? A statewide longitudinal analysis. Children and Youth Services Review, 29, 460-473.

Francis, K., Thornton Walker, J., Hamadyk, J., & Wilson, S.J. (2021). Recovery Coaching Interventions for Families Involved with the Child Welfare System: Moving Toward Evidence-Based Practices (OPRE Report #2021-53). Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. https://www.acf.hhs.gov/sites/default/files/documents/opre/R3-recovery-coaching-march-2021.pdf

Radel, L., Baldwin, M., Crouse, G., Ghertner, R., & Waters, A. (2018). Substance use, the opioid epidemic, and the child welfare system: Key findings from a mixed methods study. ASPE Research Brief. Office of the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services.

Smith, B. (2003). How parental drug use and drug treatment compliance relate to family reunification. Child Welfare, 82(3), 335-365.

U.S. Department of Health and Human Services, Administration for Children and Families, Children’s Bureau. (2019). Adoption and Foster Care Analysis and Reporting System (AFCARS) report. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/afcarsreport26.pdf

1 The Title IV-E Prevention Services Clearinghouse was established by ACF in accordance with the Family First Prevention Services Act (Public Law 115-123) and includes the review and rating of programs and services based on existing evidence.

2 START has received a rating of “supported” by the Title IV-E Prevention Services Clearinghouse. This study aims to contribute evidence to aid in the attainment of a “well-supported” rating, the highest rating provided by the Clearinghouse.

3 The request for this information is to the one program and does not entail the collection of new data elements from any individuals or entities.

4 Note that the 15-month follow-up survey is not part of this information collection request (ICR); it will be included in a future ICR.

5 Through these activities we did not request the same information from more than 9 individuals and therefore the feedback requests were not subject to the Paperwork Reduction Act.

6 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

7 These activities to not include data collection and are not subject to the Paperwork Reduction Act.

14