Evaluation of Resources to Support the Identification and Care of Children with Prenatal Substance or Alcohol Exposure in the Child Welfare System
OMB Information Collection Request
0970 – 0608
Supporting Statement Part B –
Statistical Methods
Type of Request: Revision
October 2023
Submitted By:
Children’s Bureau
Administration for Children and Families
U.S. Department of Health and Human Services
Note: This Supporting Statement has been updated to reflect a revision request to include focus groups with participants. Only minor changes are proposed to previously approved survey activities, to remove a few items no longer aligned with toolkit content.
Respondent Universe and Sampling Methods
No statistical methodology for stratification and sample selection will be used for the information collection activities in this request. The information to be collected is descriptive and will not be used for formal hypothesis testing. A nonprobability sampling approach is the most efficient and appropriate method for generating respondent groups for the information collection activities described here.
Evaluation Sites
Child welfare (CW) agencies in up to two states are being recruited to participate in the formative evaluation. The study team began by identifying states for potential outreach and engagement. The team is focused on states that (1) participated in a preceding PAE study conducted by the study team1; (2) have been identified by project consultants or partners (including their Children’s Bureau (CB) Regional Office) as engaging in efforts to address PAE in their child welfare system; or (3) were previously approached by the study team about participating in the usability testing phase of the study (described above) and declined, but indicated they might be willing to participate in the formative evaluation. The initial set of states has been identified for outreach and engagement. The study team is working with state or county agency directors to determine up to four “sites” total within the states to implement the toolkit intervention and participate in the study. Sites are being determined in consultation with the child welfare agency but may include one or two supervisory (or otherwise grouped) teams within one local child welfare agency, an office within one local child welfare agency, a region (e.g., multiple offices within a child welfare jurisdiction) or a county.
The criteria for inclusion in the formative evaluation are as follows:
The site has interest in the project, evaluation, and desired outcomes
The values and culture of the agency align with the project objectives and activities (e.g., known interest/prior experience in contributing to building evidence for child welfare practices)
The site has adequate staff resources to test the toolkit. Indicators include:
Sufficient number of supervisors
Staff who serve in varied agency roles
Time to complete evaluation activities
There is evidence of buy-in/support (internal and external, frontline to leadership) to implement PAE/PSE resources as envisioned. Indicators include:
Enthusiastic response to the toolkit from individuals with influence at the site (e.g., a likely program champion)
Presence of state plans regarding Fetal Alcohol Syndrome Disorders or evidence of state’s awareness of the importance of addressing alcohol impacts within substance exposed newborn plans of safe care, etc.
There is capacity to engage teams/individuals within the site who are in roles aligned with or are targeted users of specific components of the toolkit (e.g., foster care/adoption workers able to assess quality and utility of resources to be provided to caregivers)
Access to adjunct partners (e.g., established referral processes with assessment and other services that will help to inform evaluation of aligned toolkit resources)
Child welfare staff and leadership will be recruited from within each implementation site to implement the toolkit intervention and serve as survey respondents. The targeted sample size is approximately 8 respondents for each site (we estimate up to 32 respondents total across four sites). Seven of the 8 respondents at each site will be recruited to participate in the focus groups (all respondents except for the Director).
The child welfare staff that will be needed to participate in implementation and data collection at each site will be similar to the anticipated end users of the toolkit and would include staff from among the following roles shown in Table B1.
Table B1. Child welfare staff roles for inclusion in the study
Child welfare agency role |
Number of participants (per site) |
Director |
1 |
Supervisor |
2-3 |
Staff from a variety of program areas including investigation/intake, ongoing case management, foster care/adoption/permanency,2 family preservation services |
2-3 |
State or local child welfare agency professionals working in specialist roles that align with toolkit resources and targeted processes. These staff roles/participants will be explored during site engagement discussions, and may include:
|
1-2 |
Recruitment
The study team and Federal Project Officer are connecting with ACF Regional Offices to discuss potential states and sites for inclusion in the study (see B-1 for more detailed information about site selection; see appendix A for the text of this letter and appendix B for a written description of the study that is being applied in outreach to the Regional Offices). Once potential states are identified, the Federal Project Officer and study director would send letters to directors of the state and/or county child welfare agencies to request a meeting to discuss potential participation in the study (see appendix A). If the state and county is interested in participating, the study team would work with the state to identify site teams that might be the best fit for implementation of the toolkit and participation in the study. The study team would then send letters (see appendix A) to the directors or identified contacts of those sites to request a discussion about this possibility.
For sites that agree to participate in the study, agreements are being developed and executed with the agencies to outline the activities and resource expectations of site participation (including desired number and type of staff to participate, estimated hours and timeframe, information collection activities), treatment of the data (including data security and privacy, analysis, and reporting), and extent and type of study team support to the site during the formative evaluation.
Procedures for the Collection of Information
Surveys
At each designated data collection timepoint (see A-2), respondents will receive an email notification from the study team, inviting them to complete one or more of the survey instruments (see appendix A). The email will include details regarding the purpose of the survey, administration timing, and consent processes. The email will include a respondent-specific web-link to the online survey. At the top of each survey there will be introductory language including the purpose of the data collection effort, instructions for completing the survey, and language to inform the respondent that their participation is voluntary and that their information will be kept private to the extent permitted by law (see attached Instruments). Respondents will be asked to indicate (by clicking a radial button) that they agree or disagree that (1) they have read the information, (2) they voluntarily agree to participate in the survey, and (3) they are 18 years of age or older. Agreement with these statements indicates active consent to data collection via that instrument and enables the respondent to begin the survey. The study team will send email reminders twice to those who receive the survey link but do not complete the survey (with the exclusion of those who received the survey but actively declined to complete it).
As data are collected, the study team will conduct quality control procedures with regard to the following aspects of survey administration and data analysis: (1) accuracy and completeness of survey distribution lists; (2) accuracy and completeness of the survey programming in Qualtrics, including internal testing of all versions of the surveys; (3) comprehensive documentation of the data structure (e.g., variable names; coding information); (4) monitoring of response rates and missing data; (5) email reminders to non-responders; (6) descriptive analysis of data; (7) completeness and accuracy in all data handling) and reporting with regard to respondent privacy (e.g., deidentification of data).
Focus Groups
Advance appointments will be made for the Focus Groups of the participating child welfare agency staff. The study team will work with designated CW agency study liaisons to identify appropriate participants. The study team will provide written descriptions of the purpose and process for the focus groups which will be used by the study liaisons to invite participants. If staff members indicate their agreement, the team will contact them with an invitation to participate in the focus group. The team will follow up with each participant to schedule the date and time of the focus group.
Quality control procedures for the focus groups will be implemented by the study team in regard to the following: (1) accuracy and completeness of inclusion of appropriate respondents; (2) accuracy and completeness of questions and training of focus group facilitators; (3) completeness and accuracy of coding processes for qualitative responses to open-ended questions; and (4) completeness and accuracy in all reporting in consideration of respondent privacy. Data codebooks will detail all decisions applied during coding and analysis. Subsets of data will be reviewed and/or coded for reliability across collectors, where appropriate.
The collection of survey and focus group data will be guided by a written data security plan, the purpose of which is to document the activities the study team will undertake to maintain the safety and security of all data collected (see A-10).
Methods to Maximize Response Rates and Deal with Nonresponse
Response Rates
Maximizing response rates is critical to the administration of these data collection efforts. The content and format of the instruments were developed in close consultation with Federal partners and expert consultants. Though these data collection activities are not designed to produce statistically generalizable findings and participation in the data collection activities is wholly at the respondents’ discretion, response rates will be collected when applicable and possible for quality improvement purposes.
Data collection strategies that emphasize flexibility, privacy, and a respect for the respondent’s time facilitate timely participation. The following strategies will be implemented to maximize participation in the data collection:
Introduction and notification: Strategies to introduce the study to state agency and site leadership (described in A-2), and to recruit staff to the data collection efforts will be used for all instruments.
Timing of data collection: Individualized discussions will be held with each site to determine optimal periods for data collection to minimize respondent burden and to facilitate recall.
Administration: For surveys, reminder emails will be sent (per discussion above) to promote participation and a high response. For focus groups, the CW agency study liaison will be asked to contact possible respondents and inquire about their interest and willingness to be contacted by the study team to learn more about participation and their preferred method of contact (phone or email). The study team will contact each of the willing respondents, introduce the study, and provide details regarding the purpose, scheduling and consent processes. The study team will schedule the focus group at a time available to the most respondents. If held in person, a location most convenient to the respondents and with the capacity for private discussion will be identified and reserved for the meeting (e.g., a library meeting room space). If held virtually, the study team will provide respondents with a link to participate in a virtual meeting via Zoom. The study team will confirm the focus group via email or phone 2 to 3 days before the focus group.
Alternate response methods: Respondents will be given the opportunity to use an alternate method for responding to surveys and focus groups, such as submitting a paper version with written responses to questions, if this method helps to increase participation.
Assurances of data privacy: Respondents to surveys and focus groups will be assured that reported data are aggregated and not attributable to individual respondents.
Non-Response
As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. The study team will, however, track refusal rates and refusal demographics (where possible), to gain an understanding of potential patterns in data collection participation and refusal. For some data collections, respondent demographics will be documented and reported in written materials associated with the data collection.
Test of Procedures or Methods to be Undertaken
All surveys for the formative evaluation were tested by members of the study team to determine the time required to complete them (i.e., burden). The survey of PAE/PSE-related knowledge (see A-2) was also tested by fewer than 10 outside consultants to aid in an item analysis. The study team analyzed those test data to determine item discrimination and made refinements to some items to adjust their difficulty level.
The focus group protocol was internally reviewed and refined by team members, the Federal Project Officer, and one evaluation subcontractor consultant from the University of Louisville listed below in 5. The estimates of burden hours for the focus group were determined based upon the proposed 90-minute administration period. Based upon the usability phase of evaluation, similar focus group protocols of between 10-12 questions conducted with 6-8 informants on similar topics (e.g., facilitators, barriers, applicability of the toolkit) were clearly understood and successfully completed in a 90-minute administration period.
Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
The contractor will collect and analyze the information for the CB.
Study Contractor |
Subcontractors |
James Bell Associates 2000 15th Street North, Suite 100 Arlington, VA 22201 |
ICF 9300 Lee Highway Fairfax, VA 22030 |
Anita Barbee, Ph.D.; Martin Hall, Ph.D.; and Becky Antle, Ph.D. Raymond A. Kent School of Social Work and Family Science Oppenheimer Hall 2217 S. Third Street University of Louisville Louisville, KY 40292 |
1 The Prenatal Alcohol and Other Drug Exposures in Child Welfare Study explored the current state of CW policies and practices related to PSE in five geographically diverse states from 2016 to 2021 (OMB 0970-0511, exp 5/31/2021). The study identified gaps as well as opportunities to enhance training, policies, and practices related to identifying and caring for children with PSE and served as the basis for the development of the toolkit that the proposed information collection will evaluate.
2 Child welfare agencies vary in the description of program areas. For example, “ongoing” workers in one agency may be “foster care” workers in another. The project team will clarify the role types and responsibilities during engagement calls.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Heidi Melz |
| File Modified | 0000-00-00 |
| File Created | 2023-10-16 |