FCL Supporting Statement A_7.9.21_clean

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Fathers and Continuous Learning in Child Welfare Project

OMB Information Collection Request

New Collection

Supporting Statement

Part A

August 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Pooja Curtin

Katie Pahigiannis

Part A

Executive Summary

• Type of Request: This Information Collection Request (ICR) is for a new data collection. We anticipate data collection to be complete within about 21 months and are requesting two years of approval.

• Description of Request:

This ICR is for an evaluation of five child welfare agencies implementing a continuous quality improvement process called the Breakthrough Series Collaborative, which aims to improve father and paternal relative engagement in the child welfare system for the Fathers and Continuous Learning in Child Welfare Project. This descriptive evaluation will explore the implementation of strategies to engage fathers and paternal relatives by examining process outcomes, such as changes in staff attitudes, knowledge, and behaviors and father and paternal relative engagement in services. The information collected will include interviews with participating agency staff, key partners, and community stakeholders (Instrument 1), focus groups or interviews with fathers and paternal relatives of children with involvement in the child welfare system (Instrument 2), and web surveys of participating agency and partner staff (Instrument 3). We do not intend for the data we collect in the study to be generalized to a broader population, and we do not intend for this information to be used as the principal basis for public policy decisions.

A1. Necessity for Collection

Research suggests that high quality father involvement is beneficial to children’s well-being and development (Lamb 2004) and helps protect against child maltreatment (Bumpass and Lu 2000; Gaudin and Dubowitz 1997). Findings from federal Child and Family Services Reviews reveal that child welfare agencies struggle to engage fathers and paternal relatives (JBS International 2016, 2019). The Fathers and Continuous Learning in Child Welfare (FCL) project attempts to address the longstanding challenge of improving fathers and paternal relatives’ engagement with children involved in the child welfare system and to contribute to the evidence base for father and paternal relative engagement strategies. This descriptive evaluation will build on the findings of a pilot study conducted under ACF umbrella generics for formative data collections¹.

There are no legal or administrative requirements that necessitate this collection. The Administration for Children and Families (ACF) is undertaking the collection at the discretion of the agency. ACF has contracted with Mathematica and their partners at the University of Denver to conduct the study.

A2. Purpose

Purpose and Use

The descriptive evaluation will collect information from agencies implementing a continuous quality improvement process called the Breakthrough Series Collaborative (BSC), which aims to improve father and paternal relative engagement in the child welfare system. ACF’s three aims for the evaluation are to (1) describe potentially promising strategies for engaging fathers and paternal relatives in the child welfare system, (2) assess the promise of the BSC as a continuous quality improvement framework for addressing challenges in the child welfare system, and (3) assess the extent to which agencies experienced a shift in their organizational culture. By examining process outcomes, the evaluation seeks to indicate the likelihood that strategies developed in the BSC will lead to placement stability and permanency outcomes (see B1).

We will prepare a public report and briefs to share the results of the study. By achieving the aims of the evaluation, the information collected will help ACF, child welfare agencies, and the field continue to develop, promote, and spread potentially promising strategies to engage fathers and paternal relatives in the child welfare system. It will also help ACF understand whether and how to apply the BSC methodology to address other challenges in child welfare practice. The information collection can be used to identify future directions for research and evaluation of strategies and approaches to engage fathers and paternal relatives.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision maker and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests

Four broad research questions address the three aims of the evaluation:

1. How did implementing the BSC contribute to the launch and potential sustainment of strategies and approaches for engaging fathers and paternal relatives?

2. Are the strategies and approaches implemented by each team linked to increased engagement among fathers, paternal relatives, or caseworkers (or other near-term outcomes, as data are available)?²

3. To what extent did BSC implementation facilitate organization-level or system-level shifts in the culture surrounding the engagement of fathers and paternal relatives?

4. Is the BSC a useful and promising tool for addressing challenges facing child welfare agencies?³

Table 1 provides a crosswalk between the study instruments and the research questions.

Table 1. Research questions and study instruments

Research questions	Instruments
	1	2	3
	Interview topic guide	Father and paternal relative focus group protocol	Staff survey
How did implementing the BSC contribute to the launch and potential sustainment of strategies and approaches for engaging fathers and paternal relatives?	X		X
Are the strategies and approaches implemented by each team linked to increased engagement among fathers, paternal relatives, or caseworkers (or other near-term outcomes, as data are available)?	X	X
To what extent did BSC implementation facilitate organization-level or system-level shifts in the culture surrounding the engagement of fathers and paternal relatives?	X	X	X
Is the BSC a useful and promising tool for addressing challenges facing child welfare agencies?	X		X

Study Design

The proposed evaluation is a descriptive process study that will build on information collected in the pilot study. The FCL project team worked with federal partners, experts, and stakeholders familiar with child welfare agencies to select six Improvement Teams representing five agencies⁴ for the pilot study. Teams comprised child welfare agency staff (such as managers, supervisors, and workers) and community partner staff (such as staff from father engagement organizations).

The FCL project team developed a Collaborative Change Framework that drove the Improvement Teams’ efforts. Improvement Teams tracked and reported on specific measures related to the framework to monitor progress and improvements at multiple points throughout the pilot study.

For this evaluation, we will work with the same five agencies from the pilot study to assess each site’s experience using the BSC methodology and their experience planning, testing, and adjusting strategies and approaches to engage fathers and paternal relatives. This IC will include, but will not be limited to, participants in the pilot study. To do this, both qualitative and quantitative data sources are necessary.

The FCL project team will conduct a two-day site visit to each of the agencies to interview and hold focus groups with key stakeholders about their experiences with the BSC methodology and the engagement strategies and approaches. (The site visit in Los Angeles will last three days, in order to speak to staff from both Improvement Teams and conduct additional focus groups; see Footnote 4). While we are currently planning for in-person site visits, we may need to administer Instruments 1 and 2 virtually given ongoing uncertainty about the COVID-19 pandemic or other extenuating circumstances. We have designed these instruments to be administered in-person or virtually.

A broad group of frontline child welfare agency and partner staff who interact directly with clients will be asked to complete a web survey at two points in time about their organization’s culture and their own practices for engaging fathers and paternal relatives. Some agency staff who participate in interviews will also respond to the survey, but the survey will be administered to a broader group of agency staff, including those whom we will not interview. The surveys and interviews provide complementary information. The survey asks about discrete aspects of organizational culture. We will be able to compare responses at the two time points to provide a quantitative assessment of changes in organizational culture. The staff who participate in interviews will also be asked their overall reflections about changes they’ve observed in organizational culture, which will provide rich qualitative details to supplement and contextualize the survey responses. The interviews will also gather information about how strategies and approaches are implemented and supported.

One notable limitation of the study design is the generalizability of the study’s findings. The limited number of agencies in the evaluation will not represent all child welfare agencies across the United States. Rather, the agencies serve as a convenience sample of child welfare agencies selected for their willingness to participate in a BSC. As a result, information we collect through this study about the process, challenges, and outcomes of BSC implementation cannot be generalized to the broader population of child welfare agencies. Observing BSC implementation with high-capacity sites will, however, directly inform ACF’s assessment of the potential of the BSC methodology as a continuous quality improvement framework for addressing challenges in the child welfare system. Limitations will be noted in materials that result from this study.

To address the research questions, we will collect data using three instruments. Table 2 provides an overview of the proposed instruments.

Table 2. Information collections

Data collection activity/Instrument	Respondents	Content and purpose of collection	Mode and duration
Instrument 1: Interview topic guide	Child welfare agency senior leaders, team managers and supervisors, and frontline staff; partner agency leaders and frontline staff; community stakeholders	Content: Perspectives on the selection, implementation, and monitoring of engagement strategies and approaches Perceived effects of strategy implementation and BSC on organizational culture and staff’s ability to engage fathers and paternal relatives Plans to continue use of and refine BSC elements Purpose: To obtain staff’s perspectives on father and paternal relative engagement, including barriers and facilitators to implementation of strategies and approaches, organizational culture shifts, and changes they observe in outcomes related to engaging fathers and paternal relatives.	Mode: In person or virtual Duration: 1.5 hours
Instrument 2: Father and paternal relative focus group protocol1	Fathers and paternal relative program participants	Content: Experiences with services and interactions with staff in the child welfare system Experiences with and perspectives on the engagement strategies and approaches and the BSC Purpose: To collect feedback from fathers and paternal relatives about their experiences with the child welfare system and child welfare agency staff.	Mode: In person or virtual Duration: 1.5 hours
Instrument 3: Staff survey	Frontline child welfare agency and partner staff	Content: Assessment of agency values, support, staff, and monitoring efforts Experience using new strategies and approaches and engaging fathers and paternal relatives Purpose: To learn how staff assess their organization’s culture and their own practices related to engaging fathers and paternal relatives and whether these assessments change over time.	Mode: Web Duration: 0.33 hours per administration, administered at two points in time

¹ The focus group protocol (Instrument 2) has been designed to be easily adapted for use with individual participants as a semi-structured interview protocol, if it is not possible to hold focus groups or if the participating agency feels that the topics are more suitable to an individual conversation.

Other Data Sources and Uses of Information

This ICR is part of a collection of ICRs for each phase of the FCL project. The first part began in March 2018 (OMB# 0970-0356) and helped ACF identify promising candidate agencies for the pilot study. During the second part, which began in November 2019 ACF recruited agencies and collected information for the pilot study. We will use the pilot study data to inform our analysis. These include semi-structured interviews with staff involved in the BSC.

In addition to the information collection detailed in Table 2, we will explore with agencies whether they can share existing program data on engagement and process outcomes and other documents related to implementing strategies. We seek to obtain only aggregate data already collected by the programs during their regular activities. We will not collect individual-level data.

A3. Use of Information Technology to Reduce Burden

The staff survey will use a secure web-based platform. To reduce burden, the survey will employ drop-down or radial response categories and logical rules for responses so that respondents can only select answers intended by the question. We anticipate that this format will provide the lowest burden on the respondent.

The information to be collected during site visits is not conducive to the use of information technology such as computerized interviewing. Site visits offer the best opportunity to tailor interviews to the specific child welfare agency (or partner) with minimal burden on the agency (or partner). For the site visit discussions, the team will use audio recorders with permission from participants to supplement notes. If site visit activities are conducted virtually, we will use a videoconferencing platform to conduct and record interviews and focus groups.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

Data collected for this study are not available anywhere else. To our knowledge, this is the first study using the BSC methodology to increase father and paternal relative engagement in child welfare. We will also use data from the pilot study in our analysis to minimize respondent burden and reduce duplication. Pilot study data cover an earlier time period; as a part of this study, we will not collect the same information already collected for the pilot.

A5. Impact on Small Businesses

The study will include state and local child welfare agencies, partner organizations, and community stakeholders, some of which might be small nonprofit organizations. The FCL project team will minimize burden for respondents by restricting the interview length to the minimum required and visiting agencies and interviewing people at times convenient for the respondents. The FCL project team will request only information required for the intended use.

A6. Consequences of Less Frequent Collection

Without collecting information from multiple stakeholders involved in the FCL project at multiple points in time, ACF risks missing information on how the implemented strategies and approaches influence father and paternal relative engagement, the feasibility of using BSC methodology in child welfare, and changes in organizational culture over time. The interviews and focus groups are one-time data collections, and the information collected will help expand the knowledge base on strategies that might be effective in increasing engagement of fathers and paternal relatives. The staff survey will be administered twice to assess changes in staff skills, knowledge, and behaviors related to engaging fathers and paternal relatives, as well as changes in organizational culture. Administering the survey twice will allow us to assess this quantitively and objectively. While some agency staff who participate in interviews will also respond to the survey the information collected through these instruments are complementary (as described in A2).

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on April 20, 2021, Volume 86, Number 74, page 20498, and provided a sixty-day period for public comment. During the notice and comment period, no comments were received.

Consultation with Experts Outside of the Study

We have not consulted with experts outside of the study on the design, data collection plan, or instruments for which we request clearance.

A9. Tokens of Appreciation

We propose offering participants in the father and paternal relative focus groups or interviews $35 gift cards intended to offset costs of participation in the study, such as arranging child care or transportation. We estimate the data collection will take 90 minutes to complete. Father and paternal relative focus group or interview data will not be representative in a statistical sense in that they will not be used to make statements about the prevalence of experiences for the entire service populations. It is important, however, to secure participants with a range of background characteristics to capture a variety of possible experiences with these programs. Without offsetting the direct costs incurred by respondents for participating in the focus groups or interviews, the research team is concerned that only those people able to overcome the financial barriers to participate will agree to an interview, which would reduce the overall quality of the qualitative data collection.

We propose no tokens of appreciation for public agency or partner staff, or community stakeholders for their participation in FCL.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information (PII)

This effort does not include collecting sensitive PII. The only PII collected for this effort are names, email addresses, and telephone numbers for the purpose of contacting child welfare agency and partner staff and community stakeholders for information collection in which they may participate. Child welfare agency staff will serve as the points of contact for the father and paternal relative focus groups or interviews, and project staff will request only the names of these participants for planning purposes.

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier. Any files containing PII are stored on Mathematica’s network in a secure project folder whose access is limited to select project team members. Only the principal investigator, project director, and key study staff have access to this folder. Furthermore, approved study team members can only access this folder after going through multiple layers of security. PII will not be kept in the same location as any data collected. Access to respondents’ contact information is restricted to only those working on the FCL project.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.

We will record interviews and focus group discussions only with permission from participants. Before the discussions begin, we will inform participants that we want to record the discussion and ask them for their permission. The recording from the discussion will be deleted as soon as information has been transcribed. We will wait to begin recording the discussion until after everyone has introduced themselves, and focus group participants will be instructed to use only their first names. Virtual discussions will take place on a secure platform. We will ask participants to keep each other’s information private.

Before starting the interviews and focus groups, we will inform all participants that their identities will be kept private to the extent permitted by law, that results will only be reported in the aggregate, that their responses will not affect any services or benefits they or their family members receive, and that they do not have to answer any questions that make them uncomfortable.

The project team has obtained IRB approval from the Health Media Lab IRB and a Certificate of Confidentiality from the National Institutes of Health. The certificate helps assure participants that their information will be kept private to the fullest extent permitted by law.

Data Security and Monitoring

As specified in the contract, the Contractor will protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ PII. The Contractor will ensure that all its employees, subcontractors (at all tiers), and employees of each subcontractor who perform work under this contract or subcontract are trained on data privacy issues and comply with the above requirements. 

The project team use will Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. They will securely generate and manage encryption keys to prevent unauthorized decryption of information in accordance with the Federal Processing Standard. They will ensure that it incorporates this standard into its property management or control system and establishes a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically, including audio recordings of discussions with key program and partner staff and participants, will be secured in accordance with the most current National Institute of Standards and Technology requirements and other applicable Federal and Departmental regulations. In addition, the project team will submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or PII that ensures secure storage and limits on access.  

A11. Sensitive Information ⁵

To evaluate strategies and approaches for engaging fathers and paternal relatives in the child welfare system, it is necessary to ask some sensitive questions of participants. The focus group guide includes questions that ask about the circumstances of participants’ involvement with the child welfare agencies. Although the guide does not specifically ask for details about why the participants or their families are involved with the agencies, it includes questions about how the participants learned about an open investigation on a child or relative or that the agency was considering placing a child or relative into foster care. It is possible that these items will be considered embarrassing by some participants. The responses to these items, however, will provide important context about the timeline and circumstances of the participants’ involvement with the agencies that is necessary to conduct an effective evaluation.

As noted in A10, we will tell respondents that they do not have to answer any questions that make them uncomfortable. We have also designed Instrument 2 to be adapted to a semi-structured individual interview protocol, as necessary.

A12. Burden

Explanation of Burden Estimates

Table 3 presents the burden estimates for the new information collection. The estimates of time to complete each instrument are based on the FCL project team’s experience collecting qualitative data from child welfare agency staff for the project’s pilot study and from social service providers for other ACF studies such as the Parents and Children Together Evaluation (OMB Control No. 0970-0403).⁶

• The interview topic guide is estimated to take 90 minutes (1.5 hours) to complete. The total burden over 21 months is estimated to be 345 hours (230 participants x 1.5), and the annual burden for this data collection is estimated to be 173 hours (345/2).

• The father and paternal relative focus group is estimated to take 90 minutes (1.5 hours) to complete. The total burden over 21 months is estimated to be 108 hours (72 participants x 1.5), and the annual burden for this data collection is estimated to be 54 hours (108/2).

• The staff survey is estimated to take 20 minutes (0.33 hours) to complete. The total burden over 21 months is estimated to be 240 hours (360 participants x 2 responses x 0.33), and the annual burden for this data collection is estimated to be 120 hours (240/2).

Estimated Annualized Cost to Respondents

The total annual cost for data collection instruments is $10,978. The hourly wage rate for staff (Instruments 1 and 3) is based on the mean hourly wage rate for social and community service managers ($36.13) (SOC code 11-9151, National Occupational Employment and Wage Estimates, Bureau of Labor Statistics, Department of Labor, May 2020, http://www.bls.gov/oes/current/oes_stru.htm).

The average hourly wage of study participants (Instrument 2) is estimated to be $7.25, the federal minimum wage.

Table 3. Burden hours requested under this information collection

Instrument	Number of respondents (total over request period)	Number of responses per respondent (total over request period)	Average burden per Response (in hours)	Total burden (in hours)	Annual burden (in hours)	Average hourly wage rate	Total annual respondent cost
Instrument 1: Interview topic guide	230	1	1.5	345	173	$36.13	$6,250.49
Instrument 2: Father and paternal relative focus group protocol	72	1	1.5	108	54	$7.25	$391.50
Instrument 3: Staff survey	360	2	0.333	240	120	$36.13	$4,335.60
Total	-	-	-		347	-	$10,977.59

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

The total cost to the Federal government for the data collection activities under this ICR will be about $1,144,494. Annualized costs to the Federal government will be about $572,247 for the proposed data collection. These estimates of costs come from Mathematica’s budgeted estimates and include labor rates, direct costs, and tokens of appreciation for respondents.

Table 4. Cost of proposed data collection

Cost category	Estimated costs
Instrument development and OMB clearance (includes IRB, study registration, and Certificate of Confidentiality)	$131,032
Conduct evaluation, collect data, and conduct analyses	$508,940
Publications and dissemination	$504,522
Total costs over the request period	$1,144,494
Annual costs	$572,247

A15. Reasons for changes in burden

This is a new information collection request.

A16. Timeline

The request for approval is for two years which will cover the total period of the study. Data collection will begin after OMB approval and continue for 15 months. Preliminary analysis of data will begin one month after data collection begins. The FCL project team expects to complete a study report six months after completing data collection.

Table 5 presents the study timeline for data collection, analysis, and reporting.

Table 5. Timeline for FCL study

Activity	2022				2023
	15 months, following OMB approval					3 months, following completion of data collection	3 months, while data analysis ends
Data collection	✓	✓	✓	✓	✓
Analysis	✓	✓	✓	✓	✓	✓	✓
Reporting							✓

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Instruments

Instrument 1. Interview topic guide

Instrument 2. Father and paternal relative focus group protocol

Instrument 3. Staff survey

Appendices

Appendix A. Informed consent form

Appendix B. Focus group recruitment flyer

Appendix C. Focus group reminders

Appendix D. Staff survey notifications

References

Bumpass, L., and H.-H. Lu. “Trends in Cohabitation and Implications for Children’s Family Contexts in the United States.” Population Studies, vol. 54, no. 1, 2000, pp. 29–41.

Gaudin, J.M., and H. Dubowitz. “Family Functioning in Neglectful Families: Recent Research.” In Child Welfare Research Review, edited by J.D. Berrick, R.P. Barth, and N. Gilbert. New York: Columbia University Press, 1997, pp. 28–62.

JBS International. “Child and Family Services Aggregate Report.” Washington, DC: Department of Health and Human Services, Administration for Children and Families, Children’s Bureau, 2016.

JBS International. “Parent Engagement – Reflections From the CFSR: 2015-2017.” Washington, DC: Department of Health and Human Services, Administration for Children and Families, Children’s Bureau, 2019.

Lamb, Michael E. (ed.). The Role of the Father in Child Development. Hoboken, NJ: John Wiley, 2004.

1 Formative Data Collections for ACF Research (OMB Control 0970-0356; GenIC approved 3/26/2018 and updates incorporated 8/31/2018) and Formative Data Collections for ACF Program Support (OMB #: 0970-0531; GenIC approved 11/5/2019).

2 The BSC is implemented by Improvement Teams at the study sites. The teams are discussed as part of the study design in A2.

3 Research Questions 1, 2, and 3 will inform conclusions related to Research Question 4.

4 The pilot study included five separate child welfare agencies, which are also expected to participate in the descriptive evaluation. At one of these agencies, there were two Improvement Teams representing different offices, for a total of six teams across the participating agencies.

5 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

6 To create these estimates, we assume that the Los Angeles child welfare agency includes twice as many respondents as other agencies, given that there were two Improvement Teams from Los Angeles that participated in the pilot study. For the interview topic guide, we assume up to 70 respondents in Los Angeles and 40 respondents in each of the other agencies. For the focus group, we assume 24 respondents in Los Angeles and 12 respondents in each of the other agencies. For the staff survey, we assume 120 respondents in Los Angeles and 60 respondents in each of the other agencies.

11