Evaluation of the Child Welfare Capacity Building Collaborative: Part Two
Supporting Statement Part A
Department of Health and Human Services
Children’s Bureau
Washington, DC
Contact person:
Brian Deakins
Children’s Bureau
Administration on Children, Youth and Families
Mary E. Switzer Building
3rd Floor, Mailstop 3602
330 C Street, SW
Washington, DC 20201
202-205-8769
Brian.Deakins@acf.hhs.gov
Overview
The Evaluation of the Child Welfare Capacity Building Collaborative: Part Two is sponsored by the Children’s Bureau (CB) in the Administration on Children, Youth and Families (ACYF), Administration for Youth and Families (ACF), U.S. Department of Health and Human Services (DHHS) which seeks approval for the data collection instruments and procedures described herein. The proposed information collection is necessary to facilitate, track, monitor, and evaluate the activities of the Capacity Building Collaborative. The Collaborative includes three Federally funded centers (Center for States, Center for Tribes, and Center for Courts) that deliver national child welfare expertise and evidence-informed training and technical assistance services to State, Tribal, and Territorial public child welfare agencies and Court Improvement Programs (CIPs).
The CB-funded Centers’ collective goal is to build the capacities of State, local, Tribal child welfare systems, and CIPs to successfully undertake practice, organizational, and systemic reforms necessary to implement federal policies, meet federal standards, and achieve better outcomes for the children, youth and families they serve.
There are two parts to the evaluation of the Child Welfare Capacity Building Collaborative.
Part One: Part One includes information collection approved under OMB #0970-0484. A request for an extension with minor revisions was submitted for review on August 19, 2019. More information about Part One is provided in section A2.
Part Two: This information collection request (ICR) is specific to the second part of the data collection for the evaluation. This ICR is for three additional years of approval and includes the following:
Removal of three approved instruments: Capacity Survey, P.L. 113-183 Key Informant Survey, and P.L. 113-183 Key Informant Interview. No additional data collection is needed for the PL 113-183 instruments as the timeline for implementation of the law has passed. We plan to submit a revised Capacity Survey for approval at a later date following input from stakeholders of the Child Welfare Capacity Building Collaborative.
Addition of one new instrument (Center for States Innovation Survey).
Extension of approval for all other previously approved instruments.
Agencies that receive formula funding through the Child Abuse Prevention and Treatment Act (CAPTA) and titles IV-B and IV-E of the Social Security Act are eligible for technical assistance from the CB to support implementation of these programs, compliance with federal requirements, and improvement of outcomes. The proposed information collection is necessary to perform routine evaluation of quality and effectiveness and to inform future planning and decision making about the provision and improvement of technical assistance services authorized under multiple sections of CAPTA and title IV of the Social Security Act. This information collection also complies with the statutory requirement for training projects authorized by Section 5106 of CAPTA to be evaluated for their effectiveness. A copy of the relevant section of CAPTA as Amended by P.L. 111-320, the CAPTA Reauthorization Act of 2010 can be found in Appendix A.
Purpose of and services provided by the Collaborative. In 2014, the CB restructured its technical assistance delivery structure and invested in new approaches to service delivery. The Centers’ services have been organized into three major categories: (1) product development and information dissemination, including the creation and release of website content, publications, and other resources; (2) training and peer networking, including the delivery of online courses or “learning experiences,” virtual presentations, and facilitated peer discussions; and (3) jurisdiction-specific consultation and coaching, including workshops and onsite visits to States and tribes to provide customized support. Each service category has been designed to achieve specific outcomes that require different levels of engagement and interaction between the Center and its targeted service recipients.
Purpose and use of this information collection. This OMB package represents an extension request with revisions for Part Two of this project. Data collected from this proposed information collection will be used by the Centers and the CB to improve the development and design of services in each category. Evaluation findings will be used to learn about the outcomes associated with technical assistance strategies and approaches, the effectiveness of particular capacity building services, and to identify which service types are most used by jurisdictions to build their capacities across several areas. Evaluation findings will be used to increase knowledge about technical assistance strategies and approaches and to inform future decision making about service delivery.
Use of information collection from prior evaluations of CB’s Technical Assistance. The information garnered from the original OMB submissions for this evaluation (OMB # 0970-0484 and OMB #0970-0494) were used by Centers and the CB in a similar manner. Evaluation findings were regularly shared in webinars and during Collaborative meetings for formative purposes. This included findings from satisfaction surveys, interviews with jurisdiction leadership, and surveys of collaborative members.
Consistent with this approach, in 2015 the CB released findings from its prior evaluation of 15 training and technical assistance providers (OMB #: 0970-0377) to the public. Findings from the final report were summarized in an executive summary and several briefs to make key information accessible to technical assistance providers, evaluators, and consumers of services. (see https://www.acf.hhs.gov/cb/capacity/cross-center-evaluation). The prior evaluation findings were shared with other federal agencies and departments that fund training and technical assistance systems and results were presented at national conferences. The CB used data from the prior information collection and findings from its analysis as a basis for the changes to its current service delivery system and for the development of data collection instruments in both parts one and two.
Evaluation design and data sources for the proposed information collection. The Centers’ services are being evaluated by both Center-specific evaluations and a Cross-Center Evaluation. The Center-specific evaluations are designed to collect data on Center-specific processes and outcomes. The Cross-Center Evaluation is designed to respond to a set of cross-cutting evaluation questions posed by the CB. Data collected will address several critical evaluation questions. Research questions for the Cross-Center Evaluation and Center-specific evaluations are provided in Appendix B.
Proposed Cross-Center Evaluation data sources for part two include:
a tailored services satisfaction survey which captures satisfaction with tailored services;
a foundational assessment to capture contextual data regarding the organizational health and functioning of child welfare agencies and courts; and
a follow-up survey that will examine short-term and intermediate outcomes among CIPs that receive different levels of tailored services following continuous quality improvement (CQI) workshops
Cross-Center Evaluation data sources previously approved for this second part, but not included in this extension request include the Capacity Survey, the P.L. 113-183 Key Informant Survey, and the P.L. 113-183 Key Informant Interview. OMB approval for the Capacity Survey will be sought at a later time after the instrument has been reviewed and vetted by stakeholders of the Capacity Building Collaborative. The P.L. 113-183 Key Informant Survey and Interview are not included because no additional data collection is needed using these instruments as the timeline for implementation of the law has passed.
Center-specific data sources for part two include:
registration forms for webinar registration and for CapLEARN, a learning management system;
service-specific feedback forms and interviews, such as the Center for States Tailored Services interviews and the Center for Courts Universal and Constituency Services survey.
a new instrument, Center for States Innovation Survey, which will allow the Center to pilot new service strategies or innovative techniques and receive feedback on those strategies.
There are only minor changes to the webinar and online learning registration forms. These changes are intended to more accurately capture information on the characteristics of users.
This OMB package represents a request to continue data collection for the Evaluation of the Child Welfare Capacity Building Collaborative Part Two using the data sources described above.
Use of improved information technology and burden reduction
Wherever possible and appropriate, information technology will be used to capture information and reduce burden relative to alternative methods of data collection. Administration of the majority of evaluation surveys will be web-based, utilizing email notification and Internet-based survey technologies creating efficiencies for survey administrators, allowing flexibility and convenience for participants, and ideally resulting in a user-friendly experience for respondents. Based on the services provided, survey respondents will receive an email notification inviting them to complete the appropriate survey instrument by accessing a web-link to an online version of the survey.
Nearly all targeted respondents are expected to be able to access the web-link to the surveys.1 The majority of questions in most surveys are closed-ended response items that can be completed quickly (within 10 – 18 minutes), allowing descriptive and comparative analyses. The Center for Tribes may facilitate the completion of the Foundational Assessment Surveys by distributing hard copies and completing them in small groups together, subsequently entering the data online. The CapLEARN Registration form and the Center for States Webinar Registration forms are fully web accessible and can be completed in 5 minutes or less.
The proposed instruments are intended to uniformly collect data that will allow for the evaluation of Center-specific processes and outcomes and to answer a set of cross-cutting evaluation questions posed by the CB. CB has required its Cross-Center and Center-specific evaluators to ensure data collection is necessary and complementary. The information collection and the Center-specific evaluation activities have been coordinated to avoid potential duplication of data collection and reduce burden to respondents. Each of the three Centers have met with the Cross-Center Evaluation team and reviewed each of the Cross-Center data collection instruments. The instruments have been revised to address potential overlap and the timing of data collection activities is being closely coordinated to minimize burden. While Center-specific data will yield important and relevant information it will not be sufficient to meet the Cross-Center purposes for the proposed information collection.
Several efforts are in place to minimize respondent burden, regardless of organizational size, for each of the data collection strategies described herein. Skip patterns have been included in the survey instruments based on the types of services received, and the timing of data collection activities is being coordinated to minimize respondent burden. Information being requested has been held to the minimum necessary to respond to the intended evaluation questions.
In order to improve the Centers’ services and collaborate effectively to provide coordinated support to State, Tribal, and Territorial public child welfare agencies and CIPs, CB and its providers need timely data on the provision of services delivered by the Centers, the accessibility of services, the perceived effect and quality of the services received, and the interactions of service providers with one another. Less frequent data collection would inhibit the timely use of the information by CB and providers to improve service coordination and service quality and to potentially make decisions about service delivery.
There are no special circumstances associated with this data collection.
Following publication of the notice that appeared in the Federal Register, Volume 84, Number 191, October 2, 2019, page 52506, no requests were received from the general public for copies of the proposed information collection instruments.
However, numerous opportunities were provided for direct stakeholders to review the proposed instruments and to contribute to their development throughout the design phase of this study. The Cross-Center and Center evaluation teams were responsive to stakeholders’ comments whenever possible and have used their feedback in revising the data collection instruments. In preparation of the first OMB clearance package for part two, instruments were pilot-tested with 9 or fewer individuals who were knowledgeable of the topics addressed and who had served in positions similar to the potential respondents (i.e., former recipients of technical assistance services). Following stakeholder review and pilot testing, revisions were made to instruments based on comments to improve clarity of instructions and items and, in some cases, to shorten the instruments. Two instruments (Webinar registration and CapLEARN registration) have been revised for this extension request in order to refine the information obtained on the characteristics and role of users. None of these revisions have added to the burden of completing the instruments and forms.
No payments or gifts are provided to respondents for completing this information request.
No assurances of complete confidentiality are provided to respondents. However, all respondents are informed of the importance of maintaining their privacy and that reported data are aggregated; they are not attributed to individuals. All respondents are informed that participation is voluntary.
Personally Identifiable Information
The Capacity Building Collaborative Centers will collect personally identifiable information (PII) on several instruments: the Collaborative’s CapLEARN Registration form, and Webinar Registration form; the Center for States Innovation Survey. Assessment and Workplanning Survey, and Tailored Services Interviews; and the Center for Courts Universal and Constituency Services Survey. The Cross-Center Evaluation team, led by James Bell Associates, will collect PII on the following instruments: Tailored Services Satisfaction Survey, Foundational Assessment Survey, and the CQI Workshop Follow-Up Survey. All survey recipients are informed participation is voluntary.
The Collaborative’s CapLEARN Registration form and Webinar Registration form will include the following data elements: name of contact, job title, organization, email address. The CapLEARN Registration Form also will include age, gender, race/ethnicity, education and tribal affiliation. The CapLEARN information will be used to create an account within the system that allows users to track their own learning progress, provide information regarding webinar access, schedule meetings and coordinate service delivery opportunities, and allow for understanding of whether service delivery is reaching the intended audience in an aggregate. A Privacy Impact Assessment (PIA) was submitted for the items in the CapLEARN registration and questions from the Office of the Chief Information Officer were addressed.
The Webinar Registration data are held in third party software Adobe Connect. Adobe Connect data security protocols include all data traffic being SSL enabled (HTTPS by default; RTMPS is optional; both 128-bit and 256-bit are supported), isolated and restricted to respective users. The data are collected by the Adobe Connect server and remains in the data center where the account was provisioned until it is downloaded, moved, or deleted. Adobe Connect is TRUSTe and Joint Interoperability Test Command (JITC) certified. The Adobe Connect hosted deployment is also Service Organization Controls 2 (SOC 2) compliant. SOC 2 replaces SAS 70; portions of a SOC 2 Report can address some aspects of Federal Information Security Management Act (FISMA), ISO27001, and Health Insurance Portability and Accountability Act (HIPAA) certifications. Adobe Connect Managed Services (ACMS) is fully HIPAA compliant. Full information for Adobe Connect data security can be found here: https://wwwimages2.adobe.com/content/dam/Adobe/en/products/adobeconnect/pdfs/security/Adobe-Connect-hosted-security.pdf
The Center for States instruments (Innovation Survey, Assessment and Workplanning Survey, Tailored Services Survey) will include the following data elements: organizational type and role. The Center for Courts Universal and Constituency Survey will obtain data on the jurisdiction.
The Cross-Center Evaluation instruments which include the Tailored Services Satisfaction Survey, Foundational Assessment Survey, and the CQI Workshop Follow-Up Survey will include the following data elements: jurisdiction (name of State/Territory/Tribe), name of contact, job title, and email address. This information will be collected for survey administration purposes, to provide aggregate-level descriptions of survey respondents, to compare aggregate responses from jurisdictions over time, to link jurisdiction responses on the Foundational Assessment Survey and Tailored Services Satisfaction Survey to service delivery data saved in CapTRACK (the Capacity Building Collaborative’s service delivery database), and to allow for analyses of data by role and jurisdiction.
All PII collected by the three Centers and Cross-Center Evaluation team are kept private and only select data such as jurisdiction and professional role will be shared with evaluators from the respective Center. Only the Cross-Center Evaluation team will have access to identifiers such as contact name and email address for the surveys it administers, and those items will be removed and replaced with identification numbers. The Cross-Center Evaluation team and Center evaluators will store all PII contact data in separate files on their respective servers in password protected, secure data systems in order to ensure privacy. Data collected on Cross-Center Evaluation instruments will be coded using identification numbers, and links between identification numbers and names will be stored in password protected secure files. Identifiers will not be used in any evaluation reporting.2
No questions of a sensitive nature are included in this evaluation.
Having applied hourly wage estimates to burden hours in each respondent category, as shown in Exhibit A-1, the current annual cost to the respondents is as follows: (1) $1,156.40 for the Foundational Assessment Survey; (2) $806.76 for the CQI Workshop Follow-Up Survey; (3) $1,569.40 for the Tailored Services Satisfaction Survey; (4) $7,847.00 for CapLEARN and Webinar Registration forms 3; (5) $2,890.89 for the Center for Courts Universal and Constituency Services Survey; (6) $2,478.00 for the Center for States Tailored Services Interviews; (6) $1,569.40 for the Center for States Assessment and Work Planning Survey; and (7) $165.20 for the Center for States Innovation Survey. The total annual cost to the respondents if all data collection instruments were employed in the same given year is $18,483.05. This cost information is based on the most current data available from May 2018. For labor categories, “Social Scientists and Related Workers” ($41.30) was used for all respondents completing the Foundational Assessment Survey, Tailored Services Satisfaction Survey, Center for States’ surveys, and CapLEARN and Webinar Registration forms. Labor rates for “Lawyers, Judges and Related Workers” ($67.23) was used for respondents completing the CQI Workshop Follow-up Survey and the Center for Courts Universal and Constituency Services survey. Labor categories and wage information were obtained from the following website:https://www.bls.gov/oes/current/oes_nat.htm#00-0000,%2021-0000,%20Community%20and%20Social%20Service%20Occupations%20mean%20hourly%20=%2021.79
Exhibit A-1. Annual Burden Estimates
|
Instrument |
Total # of Respondents |
Annual # of Respondents |
# of Responses Per Respondent |
Average Burden Hours Per Response |
Total Annual Burden Hours |
Average Hourly Wage |
Total Annual Cost |
1 |
Foundational Assessment Survey |
831 |
277 |
1 |
.1 |
28 |
$41.30 |
$1,156.40 |
2 |
CQI Workshop Follow-Up Survey |
144 |
48 |
2 |
.12 |
12 |
$67.23 |
$806.76 |
3 |
Tailored Services Satisfaction Survey |
1,386 |
462 |
1 |
.083 |
38 |
$41.30 |
$1,569.40 |
4 |
CapLEARN Registration |
1,800 |
600 |
1 |
.083 |
50 |
$41.30 |
$2,065.00 |
5 |
Webinar Registration |
13,950 |
4,650 |
1 |
.03 |
140 |
$41.30 |
$5,782.00 |
6 |
Center for Courts: Universal and Constituency Services |
312 |
104 |
1 |
.41 |
43 |
$67.23 |
$2,890.89 |
7 |
Center for States: Tailored Services Interview |
180 |
60 |
1 |
1 |
60 |
$41.30 |
$2,478.00 |
8 |
Center for States: Assessment and Workplanning Survey |
450 |
150 |
1 |
.25 |
38 |
$41.30 |
$1,569.40 |
9 |
Center for States: Innovation Survey |
150 |
50 |
1 |
.083 |
4 |
$41.30 |
$165.20 |
|
Total |
|
|
|
|
413 |
|
$18,483.05 |
No additional cost burden will apply for respondents or record keepers.
The associated costs for administering and analyzing the surveys are outlined in Exhibit A-2 below. The annual cost to the Federal government for administration and analyses is (1) $2,424.24 for the Foundational Assessment Survey; (2) $2,886.00 for the CQI Workshop Follow-Up Survey; (3) $4,502.16 for the Tailored Services Satisfaction Survey; (4) $5,367.96 for Webinar Registration and CapLearn Registration;4 (5) $230.88 for the Center for Courts Universal and Constituency Services Survey; and (6) $17,361.00 for the Center for States Tailored Services Interview, Assessment and Workplanning Survey, and Innovation Survey. The total annual cost to the federal government for data collection and analyses is $32,772.24.
Exhibit A-2. Annualized Costs for Survey Administration
|
Instrument |
Administration Activities |
Staff Time |
Total Cost |
1 |
Foundational Assessment Survey |
Distribute electronic survey & follow up |
42 |
$2,424.24 |
2 |
CQI Workshop Follow-Up Survey |
Distribute electronic survey & follow up |
50 |
$2,886.00 |
3 |
Tailored Services Satisfaction Survey |
Distribute electronic survey & follow up |
78 |
|
4 |
CapLEARN Registration |
Online registration |
36 |
$2,077.92 |
5 |
Webinar Registration |
Online registration |
57 |
$3,290.04 |
6 |
Center for Courts: Universal and Constituency Services |
Distribute electronic survey & follow up |
4 |
$230.88 |
7 |
Center for States: Tailored Services Interview |
Conduct interview |
180 |
$10,389.60 |
8 |
Center for States: Assessment and Work Planning Survey |
Distribute electronic survey & follow up |
110 |
$6,394.20 |
9 |
Center for States: Innovation Survey |
Distribute electronic survey & follow up |
10 |
$577.20 |
|
Total |
|
|
$32,772.24 |
Minor changes, which did not impact respondent burden, were made to the Webinar and CapLEARN registration forms to more accurately capture information on the characteristics of users. The Center for States Innovation Survey has been added because the Center for States plans to use this instrument to pilot new service strategies or innovative techniques and receive feedback on those strategies. The Capacity Survey is not included in this extension request but will be submitted for OMB approval at a later date, after the instrument has been reviewed for continued applicability and vetted by stakeholders of the Child Welfare Capacity Building Collaborative. The previously approved P.L. 113-183 Survey and Interview are being removed because the timeline for implementation of the law has passed, and no further data are needed.
Tabulation: Frequency distributions will be calculated to generate summaries of survey items, as well as to examine variability in the data. Parameter estimates, such as variances and means, will be established for each quantitative item. Cross-tabulations and significance tests will be conducted as appropriate. Qualitative analysis will be conducted on open-ended survey items and interview transcripts, and will entail systematic coding, creation of a hierarchy of codes, and cross-case and cross-source thematic analysis. Analyses will be conducted to determine subgroup variation.
Publication: The findings from the annual information collections will be summarized and tabulated in a series of annual briefings and reports to the CB beginning in the first year of data collection in FY 2020. For Centers (Center for States, and Center for Courts), findings specific to each Center’s service delivery will be reported on an ongoing basis to the respective Center staff to inform planning and service delivery. Reports of research findings will include descriptive analyses, and the implications of the findings. In the last stage of analysis beginning in FY 2022, data will be merged from multiple sources to enable final summative analyses to address major questions on the cumulative, overall results of the Centers. A final synthesis report of the project’s findings for all years will be submitted to the CB in FY 2022 for dissemination to federal, state and tribal stakeholders.
Project Timetable: Pending approval, the Tailored Services Satisfaction Survey will be administered by the Cross-Center Evaluation team on an ongoing basis as services are delivered to child welfare agencies throughout the project period. The Foundational Assessment survey will be administered by both the Cross-Center Evaluation team and individual Centers in alignment with their assessment process, prior to the work plan, as often as a new assessment is conducted, but no more frequently than annually. The CQI Workshop Follow-Up Survey will be administered by the Cross-Center Evaluation team at 6 and 12 months after the CQI Workshop. The Center for States Webinar Registration will be utilized for registration prior to every webinar and the CapLEARN Registration form is required once to create an account to access the Collaborative resource library, including on-line trainings. The Center for Courts Universal and Constituency Services Survey will be administered annually to CIP Coordinators and Directors. The timing of the survey will occur approximately 6 months after CIP self-assessment reports are submitted. The Center for States Tailored Services Interviews will be administered once for a sample of “tailored” services engagements during the year that the project is underway. The Center for States Assessment and Work Planning Survey will be administered to half the States at the completion of the assessment process during each fiscal year. (The Cross-Center team will administer its Assessment and Capacity Building Plan Satisfaction Survey, which was submitted in the first OMB extension request for this project, to the other half of States at the completion of the assessment process during each fiscal year.) The Center for States Innovation Survey will be administered two times annually to a total of 50 unique respondents who participate in new service strategies or innovative techniques to obtain feedback on their experiences.
The OMB expiration date for the information collection will appear on the instruments.
No exception is requested.
1 A hard copy of the surveys will be provided to those who cannot access the surveys online.
2 JBA’s IT staff will dispose of PII data using a commercial product that meets the U.S. Department of Defense’s 5220.22-M (ECE) standards for sanitizing media. The timing of the disposal of the PII data will be in accordance with the guidelines outlined in the contract with CB.
3 The annual respondent burden and annualized cost varies by year and depends upon the data collection strategies employed.
4 The annual respondent burden and annualized cost varies by year and depends upon the data collection strategies employed.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | SUPPORTING STATEMENT |
| Author | hafford |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |