Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Evaluation of Project Connect
OMB Information Collection Request
New Collection
Supporting Statement
Part A
June 2021
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers: Kathleen Dwyer, Alysia Blandon
Part A
Executive Summary
Type of Request: This Information Collection Request is for a new information collection. We are requesting three years of approval.
Description of Request:
The Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) at the Department of Health and Human Services (HHS) proposes to collect information to assess the implementation of Project Connect, a comprehensive home visitation intervention that provides home-based services and treatment to child welfare-involved, substance-affected families with children and adolescents ages 0 to 17. The implementation study will support a planned impact evaluation that will rely on administrative data to examine the impact of the program on child welfare outcomes. Information collection would include interviews with child welfare agency and program administrators, focus groups with program and child welfare agency staff, interviews and focus groups with participants, and interviews with other program stakeholders. The implementation study will also include direct observations of program staff meetings, program delivery, and judicial hearings. We do not intend for this information to be used as the principal basis for public policy decisions. The data will not be generalized to the broader population.
A1. Necessity for Collection
Although child abuse and neglect are prevalent and serious threats to children’s well-being, the evidence base for interventions that effectively meet the needs of children, youth, and families served by our nation’s child welfare agencies is extremely limited. This shortage of evidence deeply constrains agencies’ abilities, at all levels of government, to implement evidence-based programs and practices. To contribute to the evidence base regarding programs supporting child welfare-involved families, OPRE plans to conduct an impact study and an implementation study of Project Connect, a comprehensive home visitation intervention that provides home-based services and treatment to child welfare-involved, substance-affected families with children and adolescents ages 0 to 17. The information from the two studies will complement each other. ACF has contracted with the Urban Institute to complete these studies.
The focus of this information collection request (ICR) is the implementation study, which will allow OPRE to assess the implementation of Project Connect. The impact study will rely solely on existing administrative data to examine the impact of the program on child welfare outcomes and will impose burden on a single individual1.
There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.
A2. Purpose
Purpose and Use
The purpose of this information collection is to evaluate how Project Connect is being implemented in Rhode Island, as well as the service context in which it is being implemented, from multiple perspectives including program staff, families, child welfare case workers, court judges, and others. In conjunction with findings from the impact study, this information collection will elucidate how Project Connect compares to other services that are relevant for families in the child welfare system who are affected by unhealthy substance use. ACF and the project team will use the results from the study to contribute to the research and evidence base on program models for families affected by substance use in the child welfare system. Findings from the implementation study can inform policymakers and local agencies on the elements that contribute to program success and program challenges. The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.
Research Questions or Tests
The primary research questions for the implementation study include:
How is the Project Connect program implemented in Rhode Island?
How do relevant aspects of the local demographic, political, economic, and service environment (e.g., how substance use is viewed and treated generally in Rhode Island) shape the Project Connect program in Rhode Island?
For families in Rhode Island’s child welfare system who are affected by substance use, what services and service providers are families referred to in the absence of Project Connect, and how does Project Connect differ from those services as usual?
Is Project Connect being delivered as intended, or are there modifications made based on different factors (e.g., rural versus urban geographic location)?
What infrastructure supports Project Connect’s implementation, and what are the key implementation drivers?
Study Design
ACF’s evaluation of Project Connect has two primary parts: an impact study and an implementation study. The impact study will rely on child welfare administrative records requested from the Department of Children, Youth, and Families (DCYF) of Rhode Island, spanning three years between 2021 and 2024, to compare child welfare outcomes for families referred to Project Connect to child welfare outcomes for similar families who were not referred to Project Connect and, instead, received services as usual. The implementation study, which is the focus of this ICR, uses a qualitative case study design for an in-depth examination of the Project Connect intervention through interviews and focus groups with key respondents using a purposive sampling approach. The implementation study will capture how the program operates during the period when study families would receive Project Connect services. Through five site visits over 24 months (outlined in more detail below), the implementation study will capture the substance-use treatment services available for substance-affected families in DCYF and the broader child welfare services delivery context between 2021 and 2023.
Implementation study data will be collected from staff and administrators at DCYF and Project Connect, as well as from other substance use and behavioral health program and agency staff (health providers) and family and drug court judges, attorneys and other staff (judicial stakeholders). Respondents will be selected based on their positions and knowledge that will enable the study team to answer the research questions posed above. The project team will also collect data from families (parents) enrolled in Project Connect, as well as those who were randomized to the control group in the impact study (i.e., those receiving services as usual). Each of these respondents has key insights on the Project Connect program, as well as a unique perspective on the greater service context in which Project Connect operates. Qualitative data collection with these respondents will provide critical information for addressing the research questions listed above.
Data collection will occur over the course of five site visits (which may be virtual or in-person depending on current conditions at the time – e.g., based on COVID-19 safety and travel restrictions). Site visits will take place intermittently over the two years to capture program implementation during the study period and allow for data collection with a range of respondents. The first site visit will focus on developing a better understanding of the program model and goals, while the second site visit will focus on the context of substance use services in Rhode Island (informed by information learned during the first site visit about different providers and services). The third site visit, roughly midway through the study, will focus on the judicial stakeholders for reflections on judicial practices occurring during the study. The fourth site visit will focus on family perspectives and implementation updates at a point when sufficient time has passed to include families who have participated in services for several months. The fifth and final visit will focus on outcomes from the perspectives of administrators and staff, as well as one-on-one family interviews with families who have completed treatment. A crosswalk of the data to be collected and the site visits is presented in table A1 below. The table also outlines the data collection group, timing, respondents, and type of data collection for each instrument.
Interviews with administrators, staff, health providers, judicial stakeholders and families: The study includes interviews to gather and synthesize stakeholders’ perspectives on the Project Connect program and on services as usual. Interviews with administrators and staff will focus on the referral process, agency procedures, service provision, and the service context. Interviews with health providers and judicial stakeholders will focus on court procedures, service provision, and policies and system responses to substance use. Interviews with a small number of families – including families in Project Connect and families similarly involved with DCYF but receiving services as usual – will occur toward the end of the study and focus on families’ personal experiences with DCYF and substance use treatment services, as well as how services affected their lives and their families.
Focus groups with program staff and families: The study will also include focus groups with program staff, as well as separate focus groups with families participating in Project Connect and with families receiving services as usual. Different from the goal of one-on-one interviews, the focus group design creates opportunities for group-level insights as participants reflect together about services, challenges, and recommendations, and is an efficient way to gather multiple perspectives at one time. The focus groups with program staff will focus on program elements and goals, family needs and engagement, and the policy and state service context. Unlike the one-on-one interviews with families described above, the focus groups with families will occur earlier in the study and focus on participants’ general opinions about how substance use treatment programs and services work in Rhode Island. The focus groups will also generate discussion about opportunities and challenges of the current system.
The purposive sampling design is limited in that it will not capture every potential stakeholder and each stakeholder’s participation is voluntary – meaning administrators, staff, and families may decline an interview, or may not be available to take part in a focus group. As noted in Supporting Statement B, the results are not designed to be representative of or generalizable to all families or providers, but are intended to reflect variation in stakeholders’ experiences.
Table A1.
Instruments |
Respondent, Content, Purpose of Collection |
Mode and Duration |
Site Visit |
Admin/ Staff Interview Guide
|
Respondents: Project Connect/DCYF Administrators Content:
Purpose: Program administrator knowledge and perspective |
Mode: in-person or virtually (i.e., phone, video)
Duration: 1 hour |
#1/#5 |
Admin/ Staff Focus Group Guide
|
Respondents: Project Connect/DCYF Front Line Staff Content:
Purpose: Child welfare family-facing staff processes and perspective |
Mode: in-person or virtually (i.e., phone, video) Duration: 1.5 hours |
#1 |
Interview Guide for Other Health Providers |
Respondents: Behavioral Health Providers and Judicial Stakeholders Content:
Purpose: Behavioral health provider/judicial stakeholder perspective and knowledge |
Mode: in-person or virtually (i.e., phone, video)
Duration: 1 hour |
#2/#3 |
Interview Guide for Program and Services as usual (SAU) Families |
Respondents: Project Connect/SAU families Content:
Purpose: Family perspective |
Mode: in-person or virtually (i.e., phone, video)
Duration: 1 hour |
#5 |
Focus Group Guide for Program and Services as usual (SAU) Families |
Respondents: Project Connect/SAU families Content:
Purpose: Family perspective |
Mode: in-person or virtually (i.e., phone, video) Duration: 1.5 hours |
#4 |
Other Data Sources and Uses of information
The project team will use information from published program materials like Project Connect’s program manual, logic model, and fidelity training documentation to support their understanding and interpretation of findings from the implementation study.
Data from interviews and focus groups will be used in concert with findings from the impact study and other available information to minimize collecting data on information already compiled.
A3. Use of Information Technology to Reduce Burden
With respondents’ permission, the project team will audio record the interviews and focus groups to minimize time needed for potential follow-up to clarify notes.
A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency
The data captured by the interview and focus group protocols do not duplicate any current data collection efforts with these populations. To reduce the time burden on program staff and participants, the project team will conduct the interviews and focus groups as efficiently as possible and will work with program leaders and staff to determine the most appropriate respondents for each interview and focus group. The project team has designed the data collection instruments so that different respondents (e.g., Project Connect staff and DCYF staff) may be asked the same questions in order to capture different knowledge and different perspectives. This provides a more robust description of the program model and service context. The project team will also leverage existing published materials (see A2) on Project Connect to gain a baseline understanding of the program model which will reduce participant burden as the team will not need to ask foundational questions of respondents. This material will also aid their understanding and interpretation of study findings.
A5. Impact on Small Businesses
No small businesses will be involved with this information collection.
A6. Consequences of Less Frequent Collection
The project team will collect data during five site visits over the course of 2 years. Participants will either participate in an interview OR a focus group, but not both, and will only participate in data collection once over the course of the study. Potential negative consequences of less frequent data collection would be inaccurate findings that are relevant only to a specific point in time. The purposive sample study design allows for the project team to strategically identify and interview respondents with various perspectives, at various points during the implementation study (see A2 for more information on site visits).
A7. Now subsumed under 2(b) above and 10 (below)
A8. Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on March 19, 2021, Volume 86, Number 52, page 14928, and provided a sixty-day period for public comment. During the notice and comment period, no comments were received.
The project team consulted with substance use and health policy experts to develop the interview and focus group protocols for site visits. These experts included Lisa Clemans-Cope (Health Policy Center, Urban Institute), Ian Hill (Health Policy Center, Urban Institute), and Kimá Taylor (nonresident fellow, Urban Institute). They also consulted with Teresa Derrick-Mills (Labor, Human Services, and Population, Urban Institute), an expert on implementation science, to ensure our research questions and approach aligned with its principles.
A9. Tokens of Appreciation
The project team will give parents who participate in the interviews and focus groups, which are estimated to take 1 and 1.5 hours on average respectively, a $50 token of appreciation in the form of a gift card (or e-gift card if interviews and focus groups are conducted virtually). While the qualitative data from focus groups and interviews are not intended to be statistically generalizable to the full population, the study’s findings will be most relevant for policy and practice if the project team is able to secure participation from a wide range of participants, including those with substantial financial challenges or other barriers. This token is intended to offset costs of participation in the study, such as transportation costs, childcare, or other expenses and to help ensure that individuals with more constraints on their ability to participate may take part. The project team anticipates that $50 will serve as a reasonable amount that is high enough to support participation but is not so high as to appear coercive for potential participants.
A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing
Personally Identifiable Information
The project team will obtain names, emails, and phone numbers in order to schedule focus groups with program and agency staff during the site visits. As discussed in more detail in Supporting Statement B, for interviews and focus groups with Project Connect families, Project Connect program staff will lead recruitment and share parents’ contact information with the project team only after the parent has consented to participate. This information will be used for scheduling purposes, and parent emails will also be used to send the token of appreciation if they participate in a virtual focus group or interview. To recruit parents receiving services as usual, the project team will ask DCYF caseworkers to share a study recruitment flyer (Appendix D) with families they believe may be eligible. Families who voluntarily contact the project team will be asked screening questions to verify their eligibility prior to collecting their names and telephone numbers. At the time of the study interview, once they’ve sent the parent the token of appreciation gift card, the project team will ask for an email or text message confirming receipt. Once confirmation is received, the project team will permanently delete any correspondence containing parent contact information, emails, and texts with the parent.
To maintain participants’ privacy, the project team will request verbal consent at the start of each discussion. Participants will be provided a physical copy of the consent form before the interview if it is in-person or presented with the consent form via video or email if the visit is virtual. Program staff who helped with the recruitment may be physically present at the sites if these discussions are conducted in-person but will not be permitted in the focus group itself. If conducted virtually, program staff will not be permitted on the Zoom or phone call during the discussion.
Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.
Assurances of Privacy
Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. The project team will comply with all Federal and Departmental regulations for private information. With respondents’ permission, the project team will audio record the interviews and focus groups to minimize time needed for potential follow-up to clarify notes. The project team will ask for consent to record before beginning the interview or focus group.
For interviews and focus groups with DCYF and Project Connect administrators and staff, other health providers, and judicial stakeholders, the project team will use the informed consent documents attached to each interview and focus group guide (Instruments 1-3) to obtain consent for participation in the study. This form details the risks and benefits of participating and the level of expected privacy for each participant. Agency administrators, staff, health professionals and judicial stakeholders are categories of respondents not designated as vulnerable populations, and the information the project team will collect is not highly sensitive. The project team will ask respondents for factual information about their programs and work (e.g., what the programs do, the number of people they serve, who is eligible, the outreach and referral process). Because some study participants will be local agency or organization leaders, administrators or staff members, and because the project team will name the site in its reports, individuals reading the reports may be able to attribute particular information or comments to that respondent. The project team will tell respondents about this potential risk.
For interviews and focus groups with families, the project team will use the informed consent form designed for parents (Appendix F). This consent statement details the risks and benefits of participating and the level of expected privacy for each participant. Although there are some sensitive questions that will be asked, the questions primarily revolve around the parent’s experience with the Project Connect or other services. Parents will be informed that they may choose not to answer any and all questions during the interview. Notes from parent interviews will identify parents using their child’s child welfare case ID to avoid the collection of names and other personally identifiable information. This child welfare case ID is a random string of numbers to anyone who doesn’t have access to Rhode Island Children’s Information System (RICHIST) and was approved by DCYF for use in this study as an identifier for participants. The project team will not have access to RICHIST and will therefore not have access to data that links the case ID to any personally identifiable information.
Due to the sensitive nature of this research (see A.11 for more information), the evaluation has obtained a Certificate of Confidentiality (Appendix H). The Certificate of Confidentiality helps to assure participants that their information will be kept private to the fullest extent permitted by law.
The project team has also obtained Institutional Review Board (IRB) approval for all data collection. An Urban Institute-developed and IRB-approved confidentiality pledge, agreeing to adhere to the data security procedures laid out in the approved IRB submission, will be read and signed during the project training process by all researchers working with the data.
Data Security and Monitoring
The contract with the Urban Institute explicitly requires a data security plan that outlines how the project will store, transfer, and destroy sensitive information as well as the precautions to be taken during each of those activities to ensure the security of those data. The project team will protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The project team has developed a data security plan that outlines how the project will store, transfer and destroy sensitive information as well as the precautions to be taken during each of these activities to ensure the security of those data. The Urban Institute ensures that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.
The data security plan meets the requirements of U.S. federal government agencies and is continually reviewed in the light of new government requirements. Such security is based on (1) exacting company policy promulgated by the highest corporate officers in consultation with systems staff and outside consultants, (2) a secure systems infrastructure that is continually monitored and evaluated with respect to security risks, and (3) secure work practices of an informed staff that take all necessary precautions when dealing with private data.
Information collected from all interviews and focus groups will not be maintained in a paper or electronic form that includes the individual’s personal identifier. As noted above, the project team will use the case ID and all notes will use this deidentified ID to achieve this.
The project team will archive the data at the National Data Archive on Child Abuse and Neglect (NDACAN) as required by its contract with ACF with the following provisions:
All personal identifying information will be stripped from the file.
To prevent secondary disclosure, the project team will conduct disclosure analysis and mask, suppress, or categorize any items that could lead to identification of individuals
A11. Sensitive Information
There are no sensitive questions that will be asked of program or agency staff. The only sensitive questions that will be asked as a part of the data collection are in the guides for the implementation study for families (Instruments 4-5). The goal of these interviews and focus groups is to understand how parents have experienced the program they are currently enrolled in, including what services they received and how it has affected their lives. All sensitive questions are asked purely in the context of how the families experience the program. These questions will be used to describe the program from the family’s perspective. The sensitive topics include:
Substance Use Service Receipt. There are questions in the guide that ask whether parents received substance use treatment and what their experiences were if they did.
Child Welfare Involvement. All families in the study will have been involved in the child welfare system. The guides include questions about how the program has impacted their child welfare case.
Stress and Other Challenges. The focus group guides also cover issues of material hardship and stress and ask how stress affects program participation and participants’ lives.
Before starting the interviews and focus groups, all respondents will be informed that their identities will be kept private and that they do not have to answer any question that makes them uncomfortable. All interview/focus group protocols have been approved by the Urban Institute IRB (Appendix I).
A12. Burden
Explanation of Burden Estimates
To estimate the burden for each proposed instrument, the project team piloted each instrument internally and considered the amount of time allotted for each interview or focus group per site visit. The goal of each instrument and the data collection effort overall was to maximize the efficiency of data collection activities and minimize burden on participants. Each respondent will only participate in data collection once over the course of the study; either one interview or one focus group. Although we expect data collection to be complete within 2-years, we are requesting a 3-year approval period to account for any unforeseen delays. Therefore, we have annualized burden over a 3-year request period.
The total annual respondent cost was calculated using market prices for time and effort based on the Bureau of Labor Statistics’ wage data. The total annual cost burden to respondents is approximately $1,009.67. For administrators and managers of both DCYF and Project Connect, the figure ($46.93/hr) is based on the mean wages for “Social and Community Service Managers,” job code 11-9151, as reported in the May 2020 U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment and Wages for Rhode Island. For front-line staff at Project Connect and DCYF, the figure ($ 31.46/hr) is based on the mean wages for “Child, Family, and School Social Workers,” job code 21-1021, as reported in the May 2020 U.S. Department of Labor, Bureau of Labor Statistics for Rhode Island, Occupational Employment and Wages. For Other Stakeholders, the figure ($44.75) is based on the average of Health Providers and Judicial Stakeholders wages. The average wage for Health Providers ($24.90) is based on the mean wage for “Substance Abuse, Behavioral Disorder, and Mental Health Counselors” job code 21-1018, as reported in the May 2018 U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment and Wages for Rhode Island. The average wage for Judicial Stakeholders ($64.60) is based on the mean wage for “Lawyers” job code 23-1011. The mean wage for judges was not available. For families, the $11.50 figure is based on the Rhode Island minimum wage.
Estimated Annualized Burden and Cost to Respondents
Table A2.
Instrument |
No. of Respondents (total over request period) |
No. of Responses per Respondent (total over request period) |
Avg. Burden per Response (in hours) |
Total Burden (in hours) |
Annual Burden (in hours) |
Average Hourly Wage Rate |
Total Annual Respondent Cost |
Interview Guide for Administrators (Project Connect, Child Welfare Agency, and Child Welfare Central Referral Unit) |
14 |
1 |
1 |
14 |
5 |
$46.93 |
$234.65 |
Focus Group Guide for Staff (Project Connect and Child Welfare Agency Staff) |
24 |
1 |
1.5 |
36 |
12 |
$31.46 |
$377.52 |
Interview Guide for Other Stakeholders (Behavioral Health and Judicial Stakeholders) |
12 |
1 |
1 |
12 |
4 |
$44.75 |
$179.00 |
Interview Guide for Families |
16 |
1 |
1.15 |
18 |
6 |
$11.50 |
$69.00 |
Focus Group Guide for Families |
24 |
1 |
1.65 |
40 |
13 |
$11.50 |
$149.50 |
Total Annual Burden and Cost Estimates
|
38 |
|
$1,009.67 |
||||
A13. Costs
There are no additional costs to respondents.
A14. Estimated Annualized Costs to the Federal Government
The total cost for the data collection activities under this current request will be $685,070. The annualized cost is $228,357. The estimate includes the costs of project staff time to draft the discussion guides, collect the information, analyze the responses, and write up the results. Table A3 below shows estimated costs to the federal government by cost category.
Table A3.
Cost Category |
Estimated Costs |
Instrument Development and OMB Clearance |
$14,512 |
Field Work |
$117,033 |
Analysis Total |
$392,381 |
Implementation Study Analysis |
$186,190 |
Administrative Data Impact Study |
$206,191 |
Publications/Dissemination |
$161,144 |
Total costs over the request period |
$685,070 |
Annual costs |
$228,356.67 |
A15. Reasons for changes in burden
This is a new information collection request.
A16. Timeline
Table A4 below provides a data collection schedule over the following three years. The project team will prepare a final report for public dissemination following the completion of data collection. See Supporting Statement B, section B7 for additional information about plans for dissemination.
Table A4.
Activity |
Description |
Timeframe (after OMB approval) |
Site visits (including interviews and focus groups) |
Interviews with program leaders, staff, and partners; Focus groups with program participants |
Months 1-24 |
Analysis |
Analyze interview and focus group data |
Months 24-30 |
Reporting and Disseminating findings |
Individual formative evaluation reports |
Months 30-36 |
A17. Exceptions
No exceptions are necessary for this information collection.
Attachments
Instrument 1 – Interview Guide for PC administrators, DCYF administrators and Central Referral Unit (CRU) staff
Instrument 2 – Focus Group Guide for Front Line staff
Instrument 3 – Interview Guide for Other Health Providers and Judicial Stakeholders
Instrument 4 -- Interview Guide for Families
Instrument 5 -- Focus Group Guide for Families
Appendix A. Outreach Email – for staff connected to via program/agency
Appendix B. Outreach Email - for staff not connected to via program/agency
Appendix C. Consent Form for to Release Parent/Program Participant Contact Information - Generic
Appendix D. Services as Usual Recruitment Flyer Text – Project Connect
Appendix E. Outreach Phone Call for Parents/Program Participants - Project Connect
Appendix F. Informed Consent for Parents/Program Participants - Project Connect
Appendix G. Informed Consent for Staff - Project Connect
Appendix H. Certificate of Confidentiality
Appendix I. IRB Approval Letter
Epstein, D., & Klerman, J. A. (2012). When is a program ready for rigorous impact evaluation? The role of a falsifiable logic model. Evaluation Review, 36(5), 375-401.
Olsen, L. J. (1995). Services for substance abuse-affected families: The Project Connect experience. Child and Adolescent Social Work Journal, 12(3), 183-196.
Olsen, L. J., Laprade, V., & Holmes, W. M. (2015). Supports for families affected by substance abuse. Journal of Public Child Welfare, 9(5), 551-570.
1 This request is not subject to PRA due to the number of respondents (fewer than 10).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Kuhns, Catherine |
| File Modified | 0000-00-00 |
| File Created | 2021-06-09 |