Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Evaluation of Project Connect
OMB Information Collection Request
New Collection
Supporting Statement
Part B
June 2021
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers: Kathleen Dwyer, Alysia Blandon.
Part B
B1. Objectives
Study Objectives
The study objectives are to contribute to the research and evidence base on program models for families affected by substance use in the child welfare system, and to help ACF and child welfare stakeholders, including administrators and program providers, better understand one state’s substance-use treatment service system. The project will achieve this objective through a rigorous impact and implementation study of Rhode Island’s Project Connect program. Project Connect is a comprehensive home visitation intervention that provides home-based services and treatment to child welfare-involved, substance-affected families with children and adolescents ages 0 to 17.
The focus of this information collection request (ICR) is the implementation study, as the impact study will rely solely on existing administrative data to examine the impact of the program on child welfare outcomes and will impose burden on a single individual. The implementation study will include interviews and focus groups with program staff, families, child welfare case workers, court judges and other service providers. The research questions for the implementation study aim to understand how the program is implemented in Rhode Island, the contextual factors that shape program implementation, if there are any modifications to implementation, the other services that one would receive in the absence of Project Connect, and the infrastructure that supports implementation.
Generalizability of Results
This study is intended to present an internally valid description of the Project Connect program in Rhode Island, not to promote statistical generalization to other sites or service populations.
Appropriateness of Study Design and Methods for Planned Uses
This study design has two primary parts: the impact study and the implementation study. The impact study will rely on the use of administrative data requested from the Department of Children, Youth, and Families (DCYF) of Rhode Island while the implementation study will be based on qualitative data collection with child welfare agency administrators and staff, Project Connect administrators and staff, health providers, judicial stakeholders, parents receiving Project Connect services, and parents receiving “services as usual” (i.e., services received by those referred to the control group in the impact study). The impact study will investigate the effects of Project Connect on individuals’ outcomes, while the implementation study will complement findings from the impact study by clarifying the services that are delivered to enrollees and the factors that influence their implementation. Specifically, the implementation study design identifies and recruits key stakeholders most knowledgeable about the program and services, maximizing what the project team will learn from the data collection, while it also includes strategies for minimizing potential participation burden. As such, the planned study design with qualitative interviews and focus groups is the best approach for obtaining the information OPRE and the project team need to better understand how Project Connect is delivered and the greater service context that exists for substance-affected families involved in the child welfare system in Rhode Island. The results, drawn from a subsample of families involved with DCYF and participating in Project Connect during the study period, are not designed to be representative of or generalizable to all DCYF-involved families participating in substance use treatment services or to all providers, but are intended to reflect variation in stakeholders’ experiences.
The overall study design leverages quantitative and qualitative methodology to efficiently answer key questions in a complementary, but not duplicative, way. As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information. Results from this study are not intended to be representative of all substance-affected families in a state’s child welfare system, and the study’s limitations – including that all local perspectives and opinions may not be represented - will be included in all written products associated with the study.
B2. Methods and Design
Target Population
The target population for this study includes Project Connect and Agency (DCYF) administrators and staff; substance use and behavioral health program and agency staff (health providers); family/drug court judges, attorneys and other judicial staff (judicial stakeholders); and parents enrolled in Project Connect and services as usual. For administrator and staff interviews and focus groups, the sampling frame will consist of the roster of Project Connect and DCYF staff, health providers, and judicial stakeholders. For parent interviews and focus groups, the sampling frame will consist of parents affiliated with Project Connect or other substance use treatment services, or parents involved with DCYF affected by unhealthy substance use. The unit of analysis is the individual, and the project team will use purposive sampling to select respondents in each category who are recommended by program and agency administrators. This will allow the project team to speak with respondents who have the role and experience necessary to give sufficient information per the research questions. Parents will be recruited through Project Connect and DCYF agency staff and will not be representative of the population of families that Project Connect or services as usual programs serve.
There is only one site for Project Connect under the current study, and therefore data collection will be confined to Rhode Island. The project team will interview or conduct focus groups with up to:
14 Administrators
24 Staff
12 Other stakeholders (health providers and judicial stakeholders)
40 Parents
Respondent Recruitment
Project Connect and DCYF Administrators: Sampling of Project Connect and DCYF administrators will be purposeful. For the purpose of developing the evaluation plan, the project team has been in communication with key Project Connect and DCYF administrators that have primary responsibility for overseeing the Project Connect program and DCYF services. The project team will rely on the insight of the administrators they already know and work with to identify the most suitable individuals to interview.
Project Connect and DCYF staff: Sampling of Project Connect and DCYF staff will be purposeful. The project team will aim to talk with Project Connect frontline staff that work directly with families and administer the program to them. Additionally, the project team aims to talk with DCYF staff who work directly with families. These staff may vary in their roles and responsibilities, and we will aim to recruit a group of staff with diverse perspectives on service provision and who work with families in a variety of capacities. The project team will rely on the insight of the administrators they interviewed to identify the most suitable staff to recruit for focus groups.
Health providers and judicial stakeholders: Sampling of Project Connect and DCYF staff will be purposeful. Potential respondents may include substance use counselors, behavioral health professionals, attorneys, judges, and others who interact with or provide support services to parents facing substance use issues. The project team will rely on nominations by DCYF and Project Connect administrators to identify those with the best knowledge of the context of substance use and child welfare system involvement for families with unhealthy substance use in Rhode Island. If needed, the project team will do independent research on suitable individuals to talk to and reach out to them.
Program Participants/Parents: Sampling of Project Connect parents will be purposeful. The project team will rely on Project Connect staff to administer a consent form (Appendix C) to parents that release their contact information to the project team. Of those who consent, the project team will randomly select a subset to contact to participate in an interview or a focus group. Given that Project Connect has a small staff and families in Project Connect work with multiple staff across a variety of services, we anticipate capturing a range of perspectives about what it’s like to work with staff among families who consent to the study.
Parents receiving services as usual: Sampling of parents receiving “services as usual” will be purposeful. The project team seeks variation in families DCYF-related experiences and will request that DCYF staff share a flyer (Appendix D) about participating in an interview or a focus group as part of the study with clients who meet the study’s eligibility criteria. Those criteria include having an open DCYF case and participating in a substance use treatment program in the last 6 months. Any parent interested in participating can call the study’s number. The project team will speak with the parent to confirm their interest in and eligibility to participate in an interview or a focus group.
B3. Design of Data Collection Instruments
Development of Data Collection Instruments
The project team developed data collection instruments appropriate for addressing the key research questions included in Supporting Statement A. First, the project team reviewed the research questions, identified appropriate data sources (i.e., target respondents most knowledgeable about the topic), and outlined the type of instruments (e.g., individual interview or focus group protocols) most appropriate to address the questions. The project team then created a crosswalk of the instruments, respondents, and topics and constructs to be addressed. Next, the project team developed the instruments based on the research questions each needed to answer as well as the topics and constructs, tailored to the specific respondents. After developing each of the initial protocols, the project team compared the data collection instruments and streamlined each of the protocols to ensure they were only asking necessary questions and avoiding duplication. The project team also asked internal experts in the health and substance use fields to review the protocols for content and clarity. Finally, the project team mapped the protocol questions against the study research questions to ensure no questions were unnecessary and that each contributed to understanding the implementation of the program.
The project team does not plan to pilot the discussion guides. The implementation study relies on triangulation, as stakeholders will be asked about similar topics to give a full picture of the questions the project team is attempting to answer. See table B1 below for a description of how each data collection instrument aligns with the research questions.
Table B1. Crosswalk Between Research Questions and Instruments
|
Method |
||||
|
Interviews |
Focus Groups |
|||
|
Instrument 1 – Interview Guide for PC administrators, DCYF administrators and Central Referral Unit (CRU) staff |
Instrument 3 – Interview Guide for Other Health Providers and judicial stakeholders
|
Instrument 4 – Interview Guide for Families
|
Instrument 2 – Focus group guide for staff
|
Instrument 5 – Focus Group Guide for Families
|
Research Questions |
|
|
|
|
|
|
x |
x |
x |
x |
|
|
x |
x |
x |
x |
|
|
x |
|
|
|
x |
|
x |
|
x |
|
x |
|
x |
|
x |
|
|
B4. Collection of Data and Quality Control
The Urban Institute will be collecting all necessary data. Tables A1 and A4 in Supporting Statement A summarize all the data collection that will be conducted for the study.
Recruiting DCYF and Project Connect administrators and staff: For DCYF administrators and staff and Project Connect administrators and staff, the project team will request a roster from each organization as well as solicit suggestions for whom to interview or include in focus groups based on their knowledge of Project Connect, DCYF processes, and/or the intersection of the child welfare system and substance use in Rhode Island. The project team will use an email outreach script (Appendix A). If necessary, the team will follow-up with emails or phone calls to answer any questions respondents may have. If and when an administrator or staff member agrees to participate, the project team will follow up with an email that includes logistical details of the site visit and proposed interview/focus group times.
Recruiting health providers and judicial stakeholders: For health providers and judicial stakeholders, the project team will rely on DCYF and Project Connect administrators to refer the project team to those with the best knowledge of the context of substance use and child welfare system involvement for families with unhealthy substance use in Rhode Island. The project team will use an email outreach script (Appendix B). If necessary, the team will follow-up with emails or phone calls to answer any questions respondents may have. If and when a health provider or judicial stakeholder agrees to participate, the project team will follow up with an email that includes logistical details of the site visit and proposed interview times.
Recruiting program participants/families: To recruit Project Connect parents to participate in interviews and focus groups, Project Connect staff will request that parents complete a consent form (Appendix C) to share their contact information with the project team at their initial Project Connect intake meeting. The team will train staff to administer this consent document prior to the start of the study. The team will randomly select, from among those who consented to share their information with the project team, a subset to contact to participate in an interview or a focus group. If the parent is interested in participating, the project team will schedule an interview or focus group at a time and place that is convenient for the parent.
Recruiting parents receiving services as usual: To recruit parents receiving “services as usual,” DCYF staff will share a flyer (Appendix D) about participating in an interview or a focus group as part of the study with clients who meet the study’s eligibility criteria. Any parent interested in participating can call the study’s number. The project team will speak with the parent to confirm their interest in and eligibility to participate in an interview or a focus group. If the parent is interested in participating, the project team will schedule an interview or focus group at a time and place that is convenient for the parent.
Mode of Data Collection
Data collection will consist of interviews and focus groups that will take place during site visits. Currently, due to COVID-19 travel restrictions and safety concerns these are planned to all be virtual. If there are changes to travel restrictions and safety concerns, we will begin in-person site visits.
To reduce burden and work disruption, the team will schedule virtual site visits based on interview and focus group participants’ availability. If in-person site visits become feasible, the visits will generally last two days. At least two researchers, one leading and one taking notes, will conduct the interviews and focus groups. The researcher taking notes will also follow along the interview protocol to make sure no questions are missed. Additionally, interviews and focus groups will be audio recorded (with the consent of participants) to monitor for quality and consistency.
Monitoring of Data Collection Activities for Quality and Consistency
The project team leads will train all team members who participate in site visits on consent and interview procedures prior to entering the field. All interviews and focus groups will be audio recorded to monitor for quality and to ensure that key themes are wholly and accurately captured. Additionally, during data collection, the interviewer team will meet after each site visit to debrief and identify any challenges that may need to be addressed for future site visits. These meetings will support consistency in how interviews and focus groups are conducted across site visits.
B5. Response Rates and Potential Nonresponse Bias
Response Rates
The interviews and focus groups are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.
NonResponse
As study respondents will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the data collection. Any substantial nonresponse from staff will be documented and reported as a study limitation.
B6. Production of Estimates and Projections
The data will not be used to generate population estimates, either for internal use or dissemination.
B7. Data Handling and Analysis
Data Handling
The project team will fully transcribe audio recordings of interviews and focus groups. In cases when participants did not consent to being recorded, the project team will clean the typed notes taken during the interview or focus group.
As discussed below, the interview and focus group data will be qualitatively coded. Once the coding scheme has been established, the project team will ensure inter-rater reliability by having multiple coders code several transcripts and re-code until a kappa coefficient of over 0.80 is achieved (considered a high level of agreement between raters) (McHugh, 2012). If the initial level of agreement is below 0.80, the coders will meet to discuss the definitions of each code before returning to recode the transcripts.
Data Analysis
The implementation study will employ qualitative methods to analyze all interview and focus group data. The project team will follow a deductive coding process, beginning with development of a coding scheme based on key constructs drawn from implementation study research questions. The coding scheme will be further developed by reviewing transcripts and interview notes in conjunction with the instruments and the research questions to identify key themes and topic areas that arise through different interviews. Project team members who participated in qualitative data collection will review the coding scheme to ensure that important points are not missed.
Transcripts of the interviews and focus groups will be uploaded into qualitative data analysis software. Project team members will test the coding scheme by coding one or two interviews each, before running a query to examine coder reliability. The coding team will then meet to review any questions and divergent codes. The coding team will continue coding sections of an interview until they have reached an interrater reliability of a kappa coefficient of .80 or greater.
The study is preregistered with the American Evaluation Association.
Data Use
The project team will use the collected data to inform a technical report that will describe the study design and findings. The report will contain information about both the impact study and the implementation study. The report will include a detailed study methodology that will help the public understand and properly interpret the information derived from the data collection. The methodology section will include, but not be limited to, interview and focus group discussion topics, qualitative data analysis technique, and administrative data analysis techniques. The project team will also publish the findings in a peer-reviewed journal, so they can be reviewed by evidence clearinghouses. As discussed in section B1, the study’s limitations will be included in all written products associated with the study.
B8. Contact Person(s)
The information for this study is being collected by the Urban Institute on behalf of ACF. Principal Investigator Michael Pergamit (mpergamit@urban.org) and Catherine Kuhns (ckuhns@urban.org) led development of the study design plan and data collection protocols and will oversee collection and analysis of data gathered through interviews and focus groups.
Attachments
Instrument 1 – Interview Guide for PC administrators, DCYF administrators and Central Referral Unit (CRU) staff
Instrument 2 – Focus Group Guide for staff
Instrument 3 – Interview Guide for Other Health Providers and Judicial Stakeholders
Instrument 4 -- Interview Guide for Families
Instrument 5 -- Focus Group Guide for Families
Appendix A. Outreach Email – for staff reached via program/agency
Appendix B. Outreach Email – for staff not reached via program/agency
Appendix C. Consent Form for to Release Parent/Program Participant Contact Information
Appendix D. Services as Usual Recruitment Flyer Text – Project Connect
Appendix E. Outreach Phone Call for Parents/Program Participants - Project Connect
Appendix F. Informed Consent for Parents/Program Participants - Project Connect
Appendix G. Informed Consent for Staff - Project Connect
Appendix H. Certificate of Confidentiality
Appendix I. IRB Approval Letter
References
McHugh M. L. (2012). Interrater reliability: the kappa statistic. Biochemia medica, 22(3), 276–282.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Kuhns, Catherine |
| File Modified | 0000-00-00 |
| File Created | 2023-08-27 |