SSA - CfS Eval Ancillary IC_fnl 6.28.23

Center for States Evaluation Ancillary Data Collection

OMB Information Collection Request

0970 - 0501

Supporting Statement Part A - Justification

July 2023

Submitted By:

Children’s Bureau

Administration for Children and Families

U.S. Department of Health and Human Services

1. Circumstances Making the Collection of Information Necessary

The Child Welfare Capacity Building Collaborative, Center for States (the Center) is sponsored by the Children’s Bureau (CB) in the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS) and seeks approval for a revision of an existing data collection. The proposed information collection is necessary to facilitate, track, monitor, and evaluate the activities of the Center for States, part of the Capacity Building Collaborative (“The Collaborative.”) The Collaborative includes three federally funded centers (Center for States, Center for Tribes, and Center for Courts) that deliver national child welfare expertise and evidence-informed training and technical assistance services to state, tribal, and territorial public child welfare agencies and Court Improvement Programs (CIP).

The Center for State’s (the Center) goal is to build the capacities of public child welfare systems to successfully undertake practice, organizational, and systemic reforms necessary to implement federal policies, meet federal standards, and achieve better outcomes for the children, youth, and families they serve. The proposed information collection is necessary to facilitate, track, monitor, and evaluate the Center’s service delivery.

Legislative Background and Purpose

Agencies that receive formula funding through the Child Abuse Prevention and Treatment Act (CAPTA), and titles IV-B and IV-E of the Social Security Act (the Act) are eligible for technical assistance from CB to support implementation of these programs, compliance with federal requirements, and improvement of outcomes for children, youth, and families. This information collection is necessary to perform routine evaluation of quality and effectiveness of the Center’s services and to inform future planning and decision making about the provision and improvement of technical assistance services authorized under multiple sections of CAPTA and title IV of the Act. This information collection also complies with the statutory requirement for projects authorized by P.L. 111-320 Section 5106 of CAPTA to be evaluated for their effectiveness.

This revised information collection will allow CB to better understand the services delivered to jurisdictions, the impact of services provided, the quality of services, and support improved infusion of equity and lived expertise into services provided. The proposed information collection includes minor changes to the previous estimates of annualized burden hours and costs to reflect updated response estimates, addition of questions focused on equity and lived expertise, and revisions to reflect updated Center services (e.g., reduced instruments/questions, added instruments/questions).

2. Purpose and Use of the Information Collection

The purpose of this evaluation is to provide CB with information about the quality and effectiveness of supports provided to jurisdictions. This information collection is part of a larger data collection effort being conducted for the evaluation of the Child Welfare Capacity Building Collaborative. This notice details a group of instruments that are specific only to the Center for States. The instruments focus on (1) evaluating an innovative approach to engaging professionals in networking and professional development through virtual expos, (2) understanding fidelity to and effectiveness of the Center’s Capacity Building Model, (3) capturing information about individualized support to jurisdictions and (4) enhancing the Center’s support focused on racial equity and lived experience. Data collection and resulting findings are used (a) to assess satisfaction with service delivery and make adjustments to improve quality, (b) to support professional development of child welfare professionals through improving the virtual expo experience, and (c) to support provision of effective and high-quality individualized support to jurisdictions.

Each of the following instruments described below will be administered annually.

Previously Approved Instruments, Edits Proposed

Child Welfare Virtual Expo Exit Survey: The purpose of this instrument is to gather feedback about the annual Child Welfare Virtual Conference. It will be administered to all participants at the conclusion of the Expo to understand participant’s experiences and support planning for the next year. The number of overall survey questions was reduced, removing items no longer applicable, and new items were added to capture topics of interest for future events and ways in which participants intend to use what they learned.

Child Welfare Virtual Expo Registration Form: The purpose of this form is to gather information from all Expo registrants to support understanding reach and inform expo planning. This registration has been revised to better reflect the background and roles of anticipated attendees.

New Instruments

Many of these new instruments use questions from previously approved instruments but instruments have been restructured after a thorough review to identify where burden could be reduced and to reflect on the continued utility of the data collected since the previous approval. Some new questions have been added to explore new topics of interest to the Children’s Bureau surrounding a more heightened focus on the integration of lived expertise, racial equity, and to better align with the Center’s three workstreams: Child and Family Service Reviews, the Family First Prevention Services Act, and workforce. See section A4 for more information about efforts to reduce burden.

Jurisdiction Lead Interview: The purpose of this instrument is to gather feedback from jurisdictional project leads about the Center’s services. The interviews will be used to enhance Center service delivery and better understand the outcomes associated with Center services.

Jurisdiction Lead Observation Debrief Protocol: The purpose of this instrument is to gather feedback from the jurisdiction lead about the Center’s services following the Center’s observation of service delivery. Trained observers will meet informally with a key jurisdiction leader who participated in the meeting, to ask a few questions aligned with the meeting topic and fidelity to the Center’s practice model. Data collected will support ongoing continuous quality improvement of services provided to jurisdictions. This is a new instrument, which was created from and replaces two previously approved observation debrief protocols.

Jurisdiction Lead Survey Related to Lived Experience: The purpose of this instrument is to gather feedback from jurisdictional project leads about the Center’s efforts to promote engagement and partnership with individuals who have lived experience and/or lived expertise. The survey will be used to enhance Center service delivery and better understand ways to meaningfully engage individuals with lived experience and/or expertise and the outcomes associated with this engagement.

Outcomes of and Satisfaction with Tailored Services Survey Appended Items (Section 4): These 10 (ten) new items to be will be administered by the cross-center evaluator by adding them to the existing ‘Outcomes of and Satisfaction with Tailored Services Survey’ (OMB #0970-0576). The purpose is to collect feedback on jurisdiction leads’ perspectives about the Center’s practice model fidelity, which will inform the Center’s outcome analyses. The survey responses will also be used to enhance Center service delivery as a part of ongoing continuous quality improvement.

Peer Group Focus Group Protocol: The purpose of these focus groups is to gather feedback about the nature of the peer groups, outcomes they achieve, and how members make and maintain connections to peers. Feedback on the technology used for peer groups and ways to improve the experience for child welfare professionals will also be collected. Results will be used for continuous quality improvement purposes related to topics such as methods, formats, and technology.

3. Use of Improved Information Technology and Burden Reduction

Wherever possible and appropriate, information technology will be used to capture information and reduce burden relative to alternative methods of data collection. Administration of any evaluation surveys will be web-based and mobile device enhanced, utilizing email notification and Internet-based survey technologies thereby creating efficiencies for survey administrators, allowing flexibility and convenience for participants, and ideally resulting in a user-friendly experience for respondents. Based on the services provided, survey respondents will receive an email notification inviting them to complete the appropriate survey instrument by accessing a web-link to an online version of the survey.

Nearly all of the targeted respondents are expected to be able to access the web-link to the surveys.¹ The majority of questions in most surveys are closed-ended response items that can be completed quickly, allowing descriptive and comparative analyses.

The Child Welfare Virtual Expo Registration Form and Child Welfare Virtual Expo Exit Survey are fully web accessible and can be completed in 5 minutes or less. To minimize the burden on Virtual Expo participants, data collection will be embedded into the Expo technology platform. At the end of the event, anyone who registered and attended any part of the virtual expo is directed to the exit survey inside the platform, so they do not have to navigate outside of the technology. The survey is also linked directly in a ‘thank you for participating’ email and will stay open for 7 days following the event.

4. Efforts to Identify Duplication and Use of Similar Information

The proposed instruments are intended to uniformly collect data that will allow for the evaluation of Center for States-specific processes and outcomes. CB has required its cross-center evaluator and center-specific evaluators to ensure data collection is necessary and complementary. The proposed information collection and evaluation activities have been coordinated to avoid potential duplication of data collection and reduce burden to respondents. Additionally, the Center reviewed existing instruments to find areas to streamline both the number and length of instruments while also incorporating critical areas of interest to CB (e.g., infusion of lived expertise and racial equity) and populations that previously did not have the opportunity to provide input (e.g., peer group participants will now have a focus group). All instruments in this information collection have been reviewed for duplication across all of the instruments for The Collaborative. These instruments have been revised to address potential overlap and the timing of data collection activities is being closely coordinated to minimize burden. Specifically, to reduce burden:

• The items in the Outcomes of and Satisfaction with Tailored Services Survey Appended Items (Section 4) will be administered by the cross-center evaluator and included in their existing survey ‘Outcomes of and Satisfaction with Tailored Services Survey’ (OMB #0970-0576) instead of creating additional burden by asking jurisdictions to complete a separate instrument. Only respondents for the Center for States will receive these questions (using survey skip logic for other Centers) Additionally, these questions will replace the previously approved Tailored Services Practice Model Survey with fewer questions.

• The questions from the previously approved Assessment and Service Delivery State Lead Interviews – Supplemental Questions will be eliminated. Additionally, the instrument CBCS – Jurisdiction Interview Protocol (OMB#0970-0576) has been eliminated (the associated OMB package will be updated during the next revision request). The instrument proposed in this package to replace these instruments - Jurisdiction Lead Interview Protocol - has incorporated critical questions from both of those instruments without increasing burden.

• The instrument Annual Assessment Update (8 systematic questions) has been removed.

• Two instruments, Assessment Observation – Group Debrief and Service Delivery and Tracking and Adjustment Observation – Group Debrief, have been combined into one instrument Jurisdiction Lead Observation Debrief Protocol.

• Three instruments, Child Welfare Virtual Conference Sessions Surveys, Child Welfare Virtual Conference Interview Guide, and Child Welfare Virtual Conference Focus Group Guide have been removed.

5. Impact on Small Businesses or Other Small Entities

A number of efforts are in place to minimize respondent burden, regardless of organizational size, for each of the data collection strategies described herein. For one survey instrument, skip patterns were included based on the respondent’s satisfaction with the service types received, and the timing of all data collection activities is being coordinated to minimize respondent burden. In addition, all instruments were subject to internal review processes to ensure that surveys included questions and items required to answer evaluation questions. Lastly, each interview instrument has a maximum burden time regardless of the number of questions to ensure that the interviewer prioritizes questions within the time allotted and does not exceed the participant’s agreed upon time and burden.

6. Consequences of Collecting the Information Less Frequently

In order to improve the Center services and collaborate effectively to provide coordinated support to state and territorial public child welfare agencies, CB and its providers need timely data on the provision of services delivered by the Center, the accessibility of services, the perceived effect and quality of the services received, and the interactions of service providers with one another. Each of the instruments described in this request will be administered annually. Less frequent data collection would inhibit the timely use of the information by CB and providers to improve service coordination and service quality and to potentially make decisions about resource allocation.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances associated with this data collection.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on May 19, 2023, Volume 88, Number 97, page 32230, and provided a sixty-day period for public comment. We did not receive any comments.

9. Explanation of Any Payment or Gift to Respondents

This data collection plan does not entail payment or gifts to respondents for participation.

10. Assurance of Confidentiality Provided to Respondents

No assurances of complete confidentiality are provided to respondents. However, all respondents are informed of the importance that the CB places on maintaining respondent privacy and that reported data are not attributed to individuals but rather are aggregated and shared with the Center for States service planning teams and CB.

Participants at the virtual expo are informed that a name and email address are required to facilitate access to the expo, and that any personally identifiable information (PII) included on the Child Welfare Virtual Expo Registration Form will not be shared externally. All registrants are informed that general demographic data captured on the registration form will be used in aggregate by the Collaborative, Center for States, and CB. To ensure that the PII being collected meets required ACF security standards, all PII contact data will be stored in a password-protected secure data system managed by ICF, under the CB’s Center for States contract. Access to PII is controlled by and restricted to system administrators. All PII will be removed from the database following the virtual conference and destroyed and unique identifiers assigned.

11. Justification for Sensitive Questions

There are no questions or requirements of a sensitive nature contained in the survey instruments and data collection forms described herein.

12. Estimates of Annualized Burden Hours and Costs

Estimated Annualized Burden to Respondents

Burden estimates were calculated based on historical data regarding estimates of time to complete and response numbers for each instrument. This included piloting by fewer than 10 Center staff for new instruments or through Center staff completion of previously approved instruments.

Burden for the Child Welfare Virtual Expo Exit Survey was calculated based on an estimate of 300 respondents annually, completing the survey one time in approximately 5 minutes.

Burden for the Child Welfare Virtual Expo Registration Form was calculated based on an estimate of 3,000 respondents annually, completing the form one time in approximately 3 minutes.

Burden for the Jurisdiction Lead Interview was calculated based on an estimate of 40 respondents annually, completing the interview one time in approximately 1 hour.

Burden for the Jurisdiction Lead Observation Debrief Protocol was calculated based on an estimate of 25 respondents annually, completing the debrief one time in approximately 15 minutes.

Burden for the Jurisdiction Lead Survey Related to Lived Experience was calculated based on an estimate of 30 respondents annually, completing the survey one time in approximately 15 minutes.

Burden for the Outcomes of and Satisfaction with Tailored Services Survey Appended Items (Section 4) was calculated based on an estimate of 40 respondents annually, completing the survey one time in approximately 3 minutes.

Burden for the Peer Group Focus Group Protocol was calculated based on an estimate of 150 respondents annually, completing the focus group one time in approximately 1 hour.

Estimated Annualized Cost to Respondents

The cost to respondents was calculated using the Bureau of Labor Statistics (BLS) job codes with the most current data available from May 2022. For labor categories, there are two categories of respondents. First, the mean hourly wage for Social and Human Services Assistants [21-1093] is $20.00 per hour. To account for fringe benefits and overhead the rate was multiplied by two which is $40.00. Second, the mean hourly wage for General and Operations Managers [11-1021] is $59.07. To account for fringe benefits and overhead the rate was multiplied by two which is $118.14. The total annual cost to the respondents if all data collection instruments were employed in the same given year² is 381 hours and $19.615.84, respectively (please see Table 1). Labor categories and wage information was obtained from the following websites: https://www.bls.gov/oes/current/oes211093.htm and https://www.bls.gov/oes/current/oes111021.htm

Table 1. Estimated Annual Respondent Burden Hours and Annual Respondent Costs

Information Collection Title	Annual Number of Respondents	Total Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Annual Cost
Child Welfare Virtual Expo Exit Survey	300	1	0.0833	25	$40.00	$1,000.00
Child Welfare Virtual Expo Registration Form	3,000	1	0.05	150	$40.00	$6,000.00
Jurisdiction Lead Interview	40	1	1	40	$118.14	$4,725.60
Jurisdiction Lead Observation Debrief Protocol	25	1	0.25	6	$118.14	$708.84
Jurisdiction Lead Survey Related to Lived Experience	30	1	0.25	8	$118.14	945.12
Outcomes of and Satisfaction with Tailored Services Survey Appended Items (Section 4)	40	1	0.05	2	$118.14	$236.28
Peer Group Focus Group Protocol	150	1	1	150	$40.00	$6,000.00
Estimated Annual Burden Total:				381	Estimated Annual Cost Total:	$19,615.84

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There are no additional costs to respondents.

14. Annualized Cost to the Federal Government

The cost to the Federal government was calculated using the Bureau of Labor Statistics (BLS) job code with the most current data available through May 2022. For Social Science and Related Workers, All Other [19-3099] the mean hourly wage was $45.46 per hour (https://www.bls.gov/oes/current/oes193099.htm). To account for fringe benefits and overhead the rate was multiplied by two, which is $90.92. The estimate of annualized costs to the Federal government is $90.92 multiplied by 444 hours which is $40,368.48. The estimated annual costs to the Federal government for all activities associated with each instrument are outlined in the table in Table 2.

Table 2. Estimated Annual Costs to the Federal Government by Instrument

Instrument	Administration Activities	Staff Time (Hours)	Total Cost
Child Welfare Virtual Expo Exit Survey	Online Survey Development, Distribute Electronic Survey, & Follow-up	56	$5,091.52
Child Welfare Virtual Expo Registration Form	Online Survey Development, Distribute Electronic Survey, & Follow-up	30	$2,727.60
Jurisdiction Lead Interview	Schedule & Conduct Interviews	166	$15,092.72
Jurisdiction Lead Observation Debrief Protocol	Schedule & Conduct Debriefs	18	$1,636.56
Jurisdiction Lead Survey Related to Lived Experience	Online Survey Development Distribute Electronic Survey, & Follow-up	36	$3,273.12
Outcomes of and Satisfaction with Tailored Services Survey Appended Items (Section 4)	Online Survey Development, Distribute Electronic Survey, & Follow-up	5	$454.600
Peer Group Focus Group Protocol	Schedule & Conduct Focus Groups	133	$12,092.36
Total		444	$40,368.48

15. Explanation for Program Changes or Adjustments

Estimates of response burden are outlined in Table 1. We have revised the burden estimates from initial approval to account for lower than expected response rates for most instruments during previous administration of instruments as well as the elimination and addition of instruments to reflect programmatic changes. For most instruments that are continued in this package, the total number of respondents estimated is lower in this submission compared to the previous submission with two exceptions. The estimated total respondents increased from 225 to 300 for the Child Welfare Virtual Expo Exit Survey and the estimated total respondents increased from 1,000 to 3,000 for the Child Welfare Virtual Expo Registration Form due to unexpected increases in Virtual Expo attendance. We also updated labor rate to calculate costs to respondents, to reflect the most current (2022) data.

16. Plans for Tabulation and Publication and Project Time Schedule

Tabulation: Frequency distributions will be calculated to generate summaries of survey items, as well as to examine variability in the data. Descriptive information such as means and percentages will be assessed for each quantitative item. Cross-tabulations and significance tests along with other analyses to answer questions of interest will be conducted as appropriate, including analyses to understand subgroup variation. Content analysis will be conducted on open-ended survey items and interview transcripts, and will entail systematic coding, creation of a hierarchy of codes, and cross-case and cross-source thematic analysis.

Publication: Findings specific to Center for States service delivery will be reported on an ongoing basis to Center Staff and CB to inform planning and service delivery. Reports of research findings will include descriptive analyses and the implications of the findings. The Center will explore other opportunities for publication, such as conference presentations. There is currently no plan to make the data collected available on the agency’s website or data.gov or in a restricted-access environment as the data are being used to guide internal decision making and are not anticipated to be of benefit to the public.

Project Timetable: The Child Welfare Virtual Expo Registration Form will be used annually for participants to register for the Expo which occurs each year. The Child Welfare Virtual Expo Exit Survey will be administered at the end of each Expo.

Jurisdiction Lead Interviews will be conducted on an ongoing basis as service projects’ annual workplans close throughout the year. For service projects that are ending and not continuing into a new annual workplan, the interviews will occur approximately six months after the workplan closes. For service projects that are continuing into a new workplan, these interviews will occur approximately one month after a workplan closes.

Jurisdiction Lead Survey Related to Lived Experience: This survey will be administered once per fiscal year to a purposive sample of projects selected based on their topical focus.

Peer Group Focus Groups will be conducted on an ongoing basis, annually, until data is collected for all peer groups. The Center is currently overseeing approximately 22 peer groups.

The Jurisdiction Lead Observation Debrief Protocol will be used following the Center’s observation of service delivery for up to 25 service projects annually.

The Outcomes of and Satisfaction with Tailored Services Survey Appended Items (Section 4) will be administered as service projects end/close out throughout the year.

17. Reason(s) Display of OMB Expiration Date is Inappropriate

The OMB expiration date for the information collection will appear on the instruments.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are requested.

1 A hard copy of the surveys will be provided to those who cannot access the surveys online.

2 The annual respondent burden and annualized cost varies by year and depends upon the data collection strategies employed.