Evaluation of the Child Welfare Capacity Building Collaborative: Part One
Supporting Statement Part A
Department of Health and Human Services
Children’s Bureau
Washington, DC
Contact person:
Brian Deakins
Children’s Bureau
Administration on Children, Youth and Families
Mary E. Switzer Building
3rd Floor, Mailstop 3602
330 C Street, SW
Washington, DC 20201
202-205-8769
Brian.Deakins@acf.hhs.gov
The Evaluation of the Child Welfare Capacity Building Collaborative is sponsored by the Children’s Bureau (CB) in the Administration on Children, Youth and Families (ACYF), Administration for Youth and Families (ACF), U.S. Department of Health and Human Services (DHHS) which seeks approval for the data collection instruments and procedures described herein. The proposed information collection is necessary to track, monitor, and evaluate the activities of the Capacity Building Collaborative which includes three Federally funded centers (Center for States, Center for Tribes, and Center for Courts) that deliver national child welfare expertise and evidence-informed training and technical assistance services to State, Tribal, and Territorial public child welfare agencies and Court Improvement Programs (CIPs).
The CB funded Centers’ collective goal is to build the capacities of State, local, Tribal child welfare systems to successfully undertake practice, organizational, and systemic reforms necessary to implement federal policies, meet federal standards, and achieve better outcomes for the children, youth and families they serve.
This information collection request is for approval to extend use of previously approved data collection instruments for the first part of data collection for the evaluation of the Child Welfare Capacity Building Collaborative. The instruments under this information collection for which we seek approval were submitted and approved under OMB #0970-0484 in August 2016. Under this supporting statement for approval, four instruments were removed from the original approval (Tribal Organizational Assessment Interviews: Caseworker Interview; Community Provider Interview; Community Member/Elder Interview; Family Interview) because these instruments are not required collections by the federal government. No changes were made to the other instruments for which we seek approval to continue use.
Legislative Background and Purpose
Agencies that receive formula funding through the Child Abuse Prevention and Treatment Act, and titles IV-B and IV-E of the Social Security Act are eligible for technical assistance from the Children’s Bureau to support implementation of these programs, compliance with federal requirements, and improvement of outcomes. The proposed information collection is necessary to perform routine evaluation of quality and effectiveness and to inform future planning and decision making about the provision and improvement of technical assistance services authorized under multiple sections of the Child Abuse Prevention and Treatment Act (CAPTA) and title IV of the Social Security Act. This information collection also complies with the statutory requirement for training projects authorized by Section 5106 of CAPTA to be evaluated for their effectiveness. A copy of the relevant section of CAPTA as Amended by P.L. 111-320, the CAPTA Reauthorization Act of 2010 can be found in Appendix A.
In 2014, the CB restructured its technical assistance delivery structure and invested in new approaches to service delivery. The Centers’ services have been organized into three major categories: (1) product development and information dissemination, including the creation and release of website content, publications, and other resources; (2) training and peer networking, including the delivery of online courses or “learning experiences,” virtual presentations, and facilitated peer discussions; and (3) jurisdiction-specific consultation and coaching, including workshops and onsite visits to states and tribes to provide customized support. Each service category has been designed to achieve specific outcomes that require different levels of engagement and interaction between the Center and its targeted service recipients.
This OMB package represents the first extension request for this project. Through this first phase of data collection, information was collected and used by the Centers and the Children’s Bureau to improve the development and design of services in each category. Findings were regularly shared in webinars and during Collaborative meetings for formative purposes. This included findings from satisfaction surveys, interviews with jurisdiction leadership, and surveys of collaborative members. In final reporting, evaluation findings will be shared with other providers and service recipients to increase knowledge about technical assistance strategies and approaches. Evaluation findings will also inform future decision making about service delivery.
Consistent with this approach, in 2015 Children’s Bureau released findings from its prior evaluation of 15 training and technical assistance providers to the public (OMB #: 0970-0377). Findings from the final report were summarized in an executive summary and several briefs to make key information accessible to technical assistance providers, evaluators, and consumers of services (see https://www.acf.hhs.gov/cb/capacity/cross-center-evaluation). The prior evaluation findings were shared with other federal agencies and departments that fund training and technical assistance systems and results were presented at national conferences. The Children’s Bureau used data from the prior information collection and findings from its analysis as a basis for the changes to its current service delivery system. The current evaluation design was shared with other federal agencies and departments as well as researchers that study capacity building services. https://www.acf.hhs.gov/cb/resource/cross-center-evaluation-design
The Centers’ services are being evaluated by both Center-specific evaluations and a Cross-Center Evaluation. The Center-specific evaluations are designed to collect data on Center-specific processes and outcomes. The Cross-Center Evaluation is designed to respond to a set of cross-cutting evaluation questions posed by the Children’s Bureau. Data collected will address several critical evaluation questions. Research questions for the Cross-Center Evaluation and Center-specific evaluations are provided in Appendix B.
Proposed Cross-Center Evaluation data sources for phase one include:
(1) satisfaction surveys to assess recipients’ satisfaction with services, such as the Learning Experiences Satisfaction Survey;
(2) a leadership interview, administered to all State child welfare directors, Tribal child welfare directors, and CIP directors that are receiving services from the Centers; and
(3) a collaboration survey, an annual web-based survey administered to the directors and staff of the three Centers.
Center-specific data sources for phase one include:
(1) assessment tools such as the Center for Tribes Needs and Fit Exploration Tools; and
(2) service-specific feedback forms, such as the Center for States Intensive Projects instrument and the Center for Courts CQI Workshops instrument.
This OMB package represents the first request to extend the instruments used to gather data for this project.
Wherever possible and appropriate, information technology will be used to capture information and reduce burden relative to alternative methods of data collection. Administration of the majority of evaluation surveys will be web-based, utilizing email notification and Internet-based survey technologies creating efficiencies for survey administrators, allowing flexibility and convenience for participants, and ideally resulting in a user-friendly experience for respondents. Based on the services provided, survey respondents will receive an email notification inviting them to complete the appropriate survey instrument by accessing a web-link to an online version of the survey.
Nearly all of the targeted respondents are expected to be able to access the web-link to the surveys.1 The majority of questions in most surveys are closed-ended response items that can be completed quickly (within 10 – 15 minutes), allowing descriptive and comparative analyses. One in-person survey – the CQI Workshop survey, is administered by paper at the conclusion of the workshop. The survey takes fewer than 10 minutes to complete.
The proposed instruments are intended to uniformly collect data that will allow for the evaluation of center-specific processes and outcomes and to answer a set of cross-cutting evaluation questions posed by CB. CB has required its Cross-Center and Center-specific evaluators to ensure data collection is necessary and complementary. The information collection and the Center-specific evaluation activities have been coordinated to avoid potential duplication of data collection and reduce burden to respondents. Each of the three Centers have met with the Cross-Center evaluation team and reviewed each of the Cross-Center data collection instruments. The instruments have been revised to address potential overlap and the timing of data collection activities is being closely coordinated to minimize burden. While Center-specific data will yield important and relevant information it will not be sufficient to meet the Cross-Center purposes for the proposed information collection.
The full range of information will be requested of all respondents. The size of the organization will not affect the relevance of particular questions. Several efforts are in place to minimize respondent burden, regardless of organizational size, for each of the data collection strategies described herein. Skip patterns have been included in the survey instruments based on the types of services received, and the timing of data collection activities is being coordinated to minimize respondent burden. Information being requested has been held to the minimum necessary to respond to the intended evaluation questions.
In order to improve the Center’s services and collaborate effectively to provide coordinated support to State, Tribal, and Territorial public child welfare agencies and Court Improvement Programs, CB and its providers need timely data on the provision of services delivered by the Centers, the accessibility of services, the perceived effect and quality of the services received, and the interactions of service providers with one another. Less frequent data collection would inhibit the timely use of the information by CB and providers to improve service coordination and service quality and to potentially make decisions about service delivery.
There are no special circumstances associated with this data collection.
Following publication of the notice that appeared in the Federal Register, Volume 84, Number 79, Wednesday, April 24, 2019, page 17168-17170, Proposed Continued Information Collection Activity; Evaluation of the Child Welfare Capacity Building Collaborative (OMB Number: 0970-0484), no requests were received from the general public for copies of the proposed information collection instruments.
However, numerous opportunities were provided for direct stakeholders to review the proposed instruments and to contribute to their development throughout the original design phase of this study. The Cross-Center and Center evaluation teams were responsive to stakeholders’ comments whenever possible and have used their feedback in revising the data collection instruments. In preparation of the first OMB clearance package for phase one, instruments were pilot-tested with individuals who were knowledgeable of the topics addressed and who had served in positions similar to the potential respondents (i.e., State/Tribal Child Welfare Directors, CIP Directors, State technical assistance liaisons, current and former Center staff members and consultants). Following stakeholder review and pilot testing, revisions were made to instruments based on comments to improve clarity of instructions and items and, in some cases, to shorten the instruments. None of these revisions have added to the burden of completing the instruments and forms. The total burden has been reduced due to a revised estimate of the number of annual respondents.
No payments or gifts are provided to respondents for completing this information request.
No assurances of complete confidentiality are provided to respondents. However, all respondents are informed of the importance of maintaining their privacy and that reported data are aggregated; they are not attributed to individuals. All respondents are informed that participation is voluntary.
Personally Identifiable Information
The Capacity Building Centers will collect personally identifiable information (PII) on several instruments: The Web Pages and Products Satisfaction Survey; Learning Experiences Satisfaction Surveys; Webinars, Events and In-Person Meetings Satisfaction Survey; and other Center for States’ forms including Information and Referral Survey, Intensive Projects Survey, and Constituency Groups Survey; the Center for Tribes’ Contact Form, Demographic Survey, and Needs and Fit Exploration Tools. The Cross-Center evaluation team, led by JBA, will collect PII on the following instruments: Assessment and Capacity Building Work Plan Satisfaction Survey; Leadership Interviews; and Annual Collaboration Survey. Survey recipients are informed that participation in surveys is voluntary.
The Web Pages and Products Satisfaction Survey, Learning Experiences Satisfaction Surveys, Webinars, Events, and In-Person Meetings Satisfaction Survey will include data elements of jurisdiction and business role, as will the Center for States Information and Referral Survey and Constituency Groups Survey. The Center for States’ Intensive Projects Survey and Brief Tailored Services Survey will include the element of business role.
The Center for Tribes’ instruments will include the following data elements: name, email, tribal organization, role, address, and phone number. The Cross-Center evaluation’s Assessment and Capacity Building Work Plan Satisfaction Survey includes the following data elements: name, email, jurisdiction, and role. The Leadership Interviews include: name, email, jurisdiction, role, agency, length of time in current position, length of time with the agency/CIP/tribe. The Annual Collaboration Survey includes the following data elements: name, email, role, Center, length of time working with the Center, primary level of service planning, development, or delivery in which respondent works; and estimated FTE for work with the Center.
All PII collected by the three Centers and Cross-Center evaluation team are kept private and only select data such as jurisdiction and professional role will be shared with evaluators from the respective Center. Only the Cross-Center evaluation team will have access to identifiers such as contact name and email address for those surveys administered by the Cross-Center evaluation team and Center evaluators will store all PII contact data in separate files on their respective servers in password protected, secure data systems in order to ensure privacy. Data collected with Cross-Center evaluation instruments will be coded using identification numbers, and links between identification numbers and names will be stored in password protected secure files. Identifiers will not be used in any evaluation reporting.2
No questions of a sensitive nature are included in this evaluation.
Having applied hourly wage estimates to burden hours in each respondent category, the current annual cost to the respondents is as follows: (1) $34,361.60 for the Satisfaction Surveys; (2) $6,525.40 for the Center for Tribes Contact Forms, Demographic Survey, and Tribal Needs and Fit Interviews; (3) $11,687.90 for the Center for States Forms; (4) $2,285.82 for the CIP Survey and Center for Courts CQI Workshops; (5) $11,779.25 for the Leadership Interviews; and (6) $5,294.57 for the Collaboration Survey.3 The total annual cost to the respondents if all data collection instruments were employed in the same given year is $71,934.54. This cost information is based on the most current data available from May 2018. For labor categories, “Social Scientists and Related Workers” ($41.30) was used for all respondents completing the surveys; “Lawyers, Judges and Related Workers” ($67.23) was used for respondents completing the CIP Annual Meeting Survey and Center for Courts CQI Workshops Survey; “Management Positions, Chief Executives” ($96.22) was used for those participating in the Leadership Interviews; and the category “Operations Specialties Managers” ($63.79) was used for respondents completing the Collaboration Survey. Labor categories and wage information was obtained from the following website: https://www.bls.gov/oes/current/oes_nat.htm#00-0000,%2021-0000,%20Community%20and%20Social%20Service%20Occupations%20mean%20hourly%20=%2021.79
Exhibit A-1. Annual Burden Estimates
Instrument |
Total # Respondents |
Annual # of Respondents |
# Annual Responses Per Respondent |
Average Burden Hours Per Response |
Total Annual Burden Hours |
Average Hourly Wage |
Total Annual Cost |
1. Webpages and Products Satisfaction Survey |
4,680 |
1,560 |
1 |
.08 |
125 |
41.30 |
5,162.50 |
2. Learning Experiences Satisfaction Survey (single)4 |
1,500 |
500 |
1 |
.33 |
165 |
41.30 |
6,814.50 |
3. Learning Experiences Satisfaction Survey (intensive)5 |
2,700 |
900 |
1 |
.08 |
72 |
41.30 |
2,973.60 |
4. Webinars, Events, and In-Person Meetings Satisfaction Survey |
16,506 |
5,502 |
1 |
.08 |
440 |
41.30 |
18,172.00 |
5. Assessment & Capacity Building Plan Satisfaction Survey |
1,350 |
450 |
1 |
.066 |
30 |
41.30 |
1,239.00 |
6. Center for Tribes Contact Form |
150 |
50 |
1 |
.05 |
3 |
41.30 |
123.90 |
7. Center for Tribes Demographic Survey |
60 |
20 |
1 |
1.75 |
35 |
41.30 |
1,445.50 |
8. Center for Tribes Needs and Fit Exploration Tool Phase 1 |
180 |
30 |
1 |
1.5 |
45 |
41.30 |
1,858.50 |
9. Center for Tribes Needs and Fit Exploration Tool Phase 2 |
75 |
25 |
1 |
3.0 |
75 |
41.30 |
3,097.50 |
10. Center for States Information and Referral Survey |
36 |
12 |
1 |
.05 |
1 |
41.30 |
41.30 |
11. Center for States Intensive Projects Survey |
990 |
330 |
1 |
.33 |
109 |
41.30 |
4,501.70 |
12. Center for States Constituency Groups Surveys |
1,200 |
400 |
1 |
.33 |
132 |
41.30 |
5,451.60 |
13. Center for States Brief Tailored Services Survey |
375 |
125 |
1 |
.33 |
41 |
41.30 |
1,693.30 |
14. CIP Annual Meeting Survey |
600 |
200 |
1 |
.13 |
26 |
67.23 |
1,747.98 |
15. Center for Courts CQI Workshops |
144 |
48 |
1 |
.17 |
8 |
67.23 |
537.84 |
16. Leadership Interview – States, Territories |
56 |
19 |
2* |
1 |
38 |
96.22 |
3,656.36 |
17. Leadership Interview – CIPs |
52 |
17 |
2* |
1 |
34 |
96.22 |
3,271.48 |
18. Leadership Interview - Tribes |
39 |
13 |
2* |
1.25 |
33 |
96.22 |
3,175.26 |
19. Leadership Interview Part II – Tribes |
39 |
13 |
2* |
.67 |
17 |
96.22 |
1,676.15 |
18. Annual Collaboration Survey |
690 |
230 |
1 |
.36 |
83 |
63.79 |
5,294.57 |
Total |
|
|
|
|
1,512 |
|
$71,934.54 |
*Reflects the total number of responses per the extension period (three years) rather than the number of annual responses per respondent. For example, leadership interviews with states and territories will be conducted two times with 56 agencies over the three-year extension. Therefore, the annual number of respondents was reduced to reflect this administration (56 interviews divided by 3 years = 18.7 interviews per year). Each interview with the states or territory will require one hour, resulting in a total burden for the extension period of 112 hours (56 interviews x 1-hour x 2 administrations), or 38 hours per year.
Estimates of other total annual cost burden to respondents and record keepers
No additional cost burden will apply for respondents or record keepers.
The associated costs for administering the surveys are outlined in Exhibit A-2 below. The annual cost to the Federal government for administration is (1) $18,701.28 for the Satisfaction Surveys; (2) $4,155.40 for the Center for Tribes Contact Forms, Demographic Survey, and Tribal Need and Fit Exploration Tools; (3) $5,598.84 for the Center for States Forms; (4) $86.58 for the CIP Survey and Center for Courts CQI Workshops; (5) $7,734.48 for the Leadership Interviews; and (6) $76.96 for the Collaboration Survey.6 The total annual cost to the respondents if all data collection instruments were employed in the same given year is $36,353.98.
Exhibit A-2. Annualized Costs for Survey Administration
Instrument |
Administration Activities |
Staff Time |
Total Cost |
Webpages and Products Satisfaction Survey |
Distribute electronic survey & follow up |
132 |
$5079.36 |
Learning Experiences Satisfaction Survey7 |
Distribute electronic survey & follow up |
32 |
$1231.36 |
Learning Experiences Satisfaction Survey8 |
Distribute electronic survey & follow up |
42 |
$1616.16 |
Webinars, Events, and In-Person Meetings Satisfaction Survey |
Distribute electronic survey & follow up |
190 |
$7311.20 |
Assessment & Capacity Building Plan Satisfaction Survey |
Distribute electronic survey & follow up |
90 |
$3463.20 |
Center for Tribes Contact Form |
Distribute electronic survey & follow up |
1 |
$38.48 |
Center for Tribes Demographic Survey |
Distribute electronic survey & follow up |
2 |
$76.96 |
Center for Tribes Needs and Fit Exploration Tool Phase 1 |
Conduct interview |
30 |
$1,154.40 |
Center for Tribes Needs and Fit Exploration Tool Phase 2 |
Conduct interview |
75 |
$2,886.00 |
Center for States Information and Referral Survey |
Distribute electronic survey & follow up |
2 |
$76.96 |
Center for States Intensive Projects Survey |
Distribute electronic survey & follow up |
66 |
$2539.68 |
Center for States Constituency Group Survey |
Distribute electronic survey & follow up |
40 |
$1539.20 |
Center for States Brief Tailored Services Survey |
Distribute electronic survey & follow up |
37.5 |
$1443.00 |
CIP Annual Meeting Survey |
Distribute electronic survey & follow up |
2 |
$76.96 |
Center for Courts CQI Workshops |
Distribute electronic survey & follow up |
0.25 |
$9.62 |
Leadership Interviews – (States, Territories, CIPs, and Tribes) |
Conduct interview |
201 |
$7734.48 |
Annual Collaboration Survey |
Distribute electronic survey & follow up |
2 |
$76.96 |
Total |
|
|
$36,353.98 |
No significant changes were made to the Cross-Center or Center-specific instruments for which we seek an extension of approval. Four instruments were removed from the original approval (Tribal Organizational Assessment Interviews: Caseworker Interview; Community Provider Interview; Community Member/Elder Interview; Family Interview).
Tabulation: Frequency distributions will be calculated to generate summaries of survey items, as well as to examine variability in the data. Parameter estimates, such as variances and means, will be established for each quantitative item. Cross-tabulations and significance tests will be conducted as appropriate. Content analysis will be conducted on open-ended survey items and interview transcripts, and will entail systematic coding, creation of a hierarchy of codes, and cross-case and cross-source thematic analysis. Analyses will be conducted to determine subgroup variation.
Publication: The findings from the annual information collections will be summarized and tabulated in a series of annual briefings and reports to the CB beginning in the first year of data collection in FY 2020. For Centers (Center for States, Center for Tribes, and Center for Courts), findings specific to each Center’s service delivery will be reported on an ongoing basis to the respective Center staff to inform planning and service delivery. Reports of research findings will include descriptive analyses, and the implications of the findings. In the last stage of analysis beginning in FY 2022, data will be merged from multiple sources to enable final summative analyses to address major questions on the cumulative, overall results of the three centers. A final synthesis report of the project’s findings for all years will be submitted to the CB in FY 2022 for dissemination to federal, state and tribal stakeholders.
Project Timetable: Pending approval, The Webpages and Products, Learning Experiences, and Webinars, Events, and In-Person Meetings surveys will be administered by Centers on an ongoing basis as services are delivered to individuals and groups of child welfare professionals throughout the project period.
The Center for Tribes Contact Form and Center for Tribes Demographic Survey allows tribes to complete and send a short form describing a request for services. A tribe can access an electronic version through a website and submit it to the Center for Tribes or, meeting/conference attendees may pick up a paper version, complete, and return the form in-person to a Center representative. The Center for Tribes Needs and Fit Exploration Tool Phase 1 will be administered when a tribe makes a request for services; it will be administered by the Center for Tribes site lead through a phone discussion. The Center for Tribes Needs and Fit Exploration Tool Phase 2 will be administered by the site lead during the first on-site visit. Used together, information from both tools provides the Center with a picture of the tribes’ needs and goals which helps the Center begin to develop the tribe’s work plan for brief projects or plan for intensive projects.
The Center for States Information and Referral Survey will be administered to child welfare professionals after each information request that results in the provision of information to the requestor and referral by the Center to additional resources or services. The Center for States Intensive Projects Survey will be administered once annually for every intensive “tailored” services engagement between the Center and a state that lasts for more than 12 months. If the intensive project continues for subsequent years, this survey will be administered again using the same time frame. The Center for States Constituency Group Survey will be administered to every “constituency group” (a peer group of child welfare professionals – e.g., adoption manager, foster care managers) served by the Center once annually after constituency group activities and services are launched. The Center for States Brief Tailored Services Survey will be administered for every brief service (tailored consultation between the Center and an individual state lasting fewer than 12 months), once after services are closed.
The CIP Annual Meeting Survey will be administered annually following the Annual CIP Meeting. The Center for Courts CQI Workshops survey will be administered once following each CQI Workshop.
The Leadership Interviews will be administered in years FY 2020 and 2022. The Collaboration Survey will be administered once per year in FY 2020, 2021, and 2022. The Assessment and Capacity Building Work Plan Satisfaction Survey will be administered by the Cross-Center evaluation team on an ongoing monthly basis between FY 2019 through FY 2021 after Centers complete the annual assessments/workplans with the jurisdictions they serve.
The OMB expiration date for the information collection will appear on the instruments.
No exception is requested to the certification statement identified in Item 19, “Certification for Paperwork Reduction Act Submissions,” of OMB Form 83-I.
1 A hard copy of the surveys will be provided to those who cannot access the surveys online.
2 JBA’s IT staff will dispose of PII data using a commercial product that meets the U.S Department of Defense’s 5220.22-M (ECE) standards for sanitizing media. The timing of the disposal of the PII data will be in accordance with the guidelines outlined in the contract with CB.
3 The annual respondent burden and annualized cost varies by year and depends upon the data collection strategies employed.
4 For Learning Experiences that consist of a single event (e.g. on-line session or in-person training)
5 For more intensive Learning Experiences that require administration of multiple surveys over a series of events, modules, or units
6 The annual respondent burden and annualized cost varies by year and depends upon the data collection strategies employed.
7 For Learning Experiences that consist of a single event (e.g. on-line session or in-person training)
8 For more intensive Learning Experiences that require administration of multiple surveys over a series of events, modules, or units
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | James DeSantis |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |