Supporting Statement PartB_NSCAW_revised January 2021_Clean

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

National Survey of Child and Adolescent Well-Being-Third Cohort (NSCAW III)

OMB Information Collection Request

0970 - 0202

Supporting Statement

Part B

Approved September 2020

Revised January 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Christine Fortunato

Mary Bruce Webb

Part B

B1. Objectives

Study Objectives

The following are the objectives of the Phase III information collection:

• To use NSCAW III as a platform to collect workforce data from child welfare agency directors/administrators, supervisors, and caseworkers.

• To gain a better understanding of child welfare agency operations and priorities, as well as the characteristics and work experiences of child welfare workers.

• To collect data that will inform ACF’s efforts to increase child welfare workforce satisfaction and stability and to inform programs and practice related to the public child welfare workforce.

• To make the workforce data available to the research community for analysis.

Sixty-one public county child welfare agencies currently participate in NSCAW III. Surveys will be conducted with respondents from these 61 agencies; primarily by telephone and/or web. An in-person mode may be offered to respondents in the future if COVID-19 infection rates indicate that it is safe to conduct in-person surveys in the respondent’s state and county.

Generalizability of Results

This proposed information collection is intended to produce nationally representative estimates of the workforce constructs measured in the agency director/administrator, supervisor, and caseworker surveys. The NSCAW III sampling frame consists of all counties in the U.S. except counties whose state law prohibits the release of identified child maltreatment records required for the survey and states with five or more sampled counties that were unable to participate. Therefore, NSCAW III public child welfare agencies represent approximately 77% of the child welfare population. Public child welfare workers employed in agencies that do not allow the release of identified child maltreatment records will not be represented in the estimates.

Appropriateness of Study Design and Methods for Planned Uses

This proposed information collection was designed to address questions of importance on workforce characteristics and competencies, training and professional development, and organizational factors. The surveys will also collect information about the potential impacts of the COVID-19 pandemic on child welfare agencies and practice. (See Section A.2 for a complete listing of research questions.) The study design and methods were selected to produce externally and internally valid results of high quality. Probability sampling procedures are being used to ensure that child welfare agencies that meet the target population definition are fully represented in the NSCAW III. The workforce surveys were designed in collaboration with subject matter experts. Nonresponse at all three levels (agency director, supervisor, and caseworker) will be minimized through repeated contact attempts by field interviewers who will follow-up via phone, email, and/or text. Field interviewers will be trained to obtain valid responses with minimal unit and item nonresponse. Data quality monitoring activities will be used to ensure high quality information collection. Finally, survey statisticians will weight the data to compensate for disproportionate sampling, nonresponse and potential frame noncoverage issues and will produce the estimates to ensure valid inferences. Limits to representativeness and any other contextual details relevant to interpreting the findings will be clearly documented in the text of all written materials associated with the study.

As noted in Section A.2, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

For each of the approximately 61 public child welfare agencies participating in NSCAW III, we will collect information from agency directors/administrators, supervisors, and caseworkers.

Sampling and Site Selection

The two-stage sample design that guided the selection of child welfare agencies and children in Phase I and II of NSCAW III (OMB# 0970-0202) is included for reference in Appendix C. The current request has three sampling frames corresponding to the three levels of employees working in public child welfare agencies. For agency directors/administrators, the sample consists of all directors/administrators associated with the 61 agencies. The supervisor sample will be randomly selected from rosters of supervisors employed within the 61 agencies (for details, see Q39 the attached Instrument 1_Agency Director Survey). The caseworker sample will be randomly selected from rosters of caseworkers who report to the supervisors participating in the study. Initially, one supervisor will be selected with certainty from each agency (for details, see Q19 in the attached Instrument 2_Agency Director Survey). For agencies having two or more supervisors, additional supervisors will be randomly selected for participation. Up to three caseworkers will be selected for each supervisor. If a supervisor has more than three caseworkers, three caseworkers will be selected at random. For supervisors having fewer than three caseworkers, the caseworkers reporting to that supervisor will be selected with certainty. An average of two supervisor and three caseworker surveys per agency is anticipated. If there are fewer than 390 completed caseworker surveys at the end of the initial sampling process, additional caseworkers will be randomly selected across all agencies. This nested sampling approach is shown in Exhibit 1.

Exhibit 1. Nested Design for Sampling Child Welfare Staff

As previously noted, agency directors/administrators in all participating NSCAW III agencies will be selected to participate in the workforce data collection. In determining the required sample sizes for the supervisors and caseworkers, we considered the power required to detect a 1/2 point difference in means for two groups of equal size for an item measured on a 5-point (1 to 5) ordinal (e.g., Likert) scale. This analysis suggested an effective sample size of at least 100 supervisors. However, because NSCAW III agencies were selected with unequal probabilities, we anticipate a design effect which we estimate to be approximately 1.3 considering both unequal weighting and moderately large clustering effects. This translates into a target sample size of 130 completed supervisor interviews.

The same power analysis was applied to caseworkers because most of the items in the caseworker survey are also measured on a 5-point, ordinal scale. We expect clustering effects to be somewhat larger for caseworkers because supervisory influences could affect responses for caseworkers working under the same supervisor. In addition, it may be desirable to compare more highly disaggregated caseworker groups (for example, a 20% grouping characteristic). Thus, we estimate that a sample of 390 completed caseworker interviews will provide sufficient power for the key comparisons and analysis objectives. Table 1 summarizes these sample sizes and the expected response rates.

Table 1. Anticipated Sample Sizes and Response Rates

Respondent Type	Sample Size (N)	Response Rate (%)	Completed Surveys (N)
Agency Director	61	92	56
Supervisor	173	75	130
Caseworker	520	75	390

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

A panel with expertise in the child welfare workforce were convened to provide input on preliminary research questions, survey respondents, sampling, and potential constructs and measures to be included in the data collection instruments. Section A8 provides additional detail on outside experts who contributed to this effort. A review of the workforce literature was also conducted to identify knowledge gaps and additional constructs and measures of interest. Standardized and validated measures were incorporated into the draft data collection instruments whenever possible. Draft data collection instruments were developed and underwent multiple iterations of review by the project team and selected experts. The table provided in Appendix D summarizes the research questions, the constructs associated with each of the three workforce respondents, and the source (author/publisher) of the items used to assess the construct. Instruments 1, 2, and 3 contain the full workforce surveys for agency directors, supervisors, and caseworkers, respectively.

B4. Collection of Data and Quality Control

The Contractor will collect all data from sampled agency respondents. Workforce surveys will be administered by field interviewers familiar to participating NSCAW III agencies due to their work on the baseline and 18-month follow-up efforts. Prior to conducting workforce surveys, interviewers will be trained on the data collection protocol and workforce survey administration.

Lead letters will be mailed to the sampled agency personnel prior to contacting them to schedule an appointment. Lead letters include information on the study’s purpose, how the respondent was selected, survey topics to be covered, and assurances of privacy. Lead letters for agency directors/administrators, supervisors, and caseworkers are provided in Appendices E, F, and G.

Interviewers will conduct workforce surveys primarily by phone and web. If in-person surveys are deemed safe in participating NSCAW III counties by the contractor’s Infectious Disease Response Team, interviewers will secure a private setting within the participating NSCAW III agency. Prior to conducting the survey, interviewers will read a consent form aloud and ask the sampled worker to provide their verbal or signed consent for participation. The interviewers will provide a copy of the consent form with the respondent. Consent forms for agency directors/administrators, supervisors, and caseworkers are provided in Appendices H, I, and J.

The primary mode of the agency director survey will be a paper-and-pencil instrument administered over the telephone by a field interviewer. Data from the paper-and-pencil instrument will then be entered into a computer-assisted data entry program after the survey. The primary mode of the supervisor and caseworker surveys will be a combination of telephone and/or web, with the most sensitive items (e.g., secondary trauma, burnout) being answered by caseworkers and supervisors via a web survey.

Computer-assisted programs improve survey data quality by eliminating routing errors, implementing logical range checks, and increasing response rates. During the data collection period, several data quality monitoring activities will be employed on the study, including telephone verification of completed surveys and data frequency reviews.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

As noted previously in Table 1, we expect a 92% response rate for the agency directors and a 75% response rate for both supervisors and caseworkers. The anticipated supervisor and caseworker response rates are based on the current NSCAW III caseworker survey response rate. While a 100% agency director response rate was achieved for the prior NSCAW (NSCAW II) launched in 2008, we have assumed a slightly lower response rate among agency directors in NSCAW III.

The anticipated response rates reflect the actual risk of nonresponse bias for inferences to the population covered by the 61 agencies in NSCAW III. It may be possible to apply coverage adjustments using frame and National Child Abuse and Neglect Data System (NCANDS)¹ data to extend interferences to nearly 100% of the population and that option will be explored in the analysis. Weighted and unweighted response rates will be calculated. The formula used to calculate response rates will comply with AAPOR Response Rate 1 (RR1) formula (see https://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf) which is the most conservative approach to calculating response rates. RR1 seems appropriate for this study given our knowledge of the eligibility of the sample members for the survey. Interviewers will make multiple attempts to contact each workforce respondent to schedule appointments. Interviewers will be experienced in NSCAW III, highly trained, and familiar with best practices for recruiting respondents. Standard nonresponse conversion techniques will be applied to gain survey participation. However, if a sample member refuses to participate and requests that the interviewer cease further attempts to conduct the interview, the case will be classified as nonresponse and no further attempts will be made to interviewer the respondent.

NonResponse

Separate sets of weights will be computed for the agency director, supervisor, and caseworker surveys. The three sets of weights are needed so that the estimates for each unit of analysis reflects its respective target population. For the agency director survey, design weights will be constructed that reflect the unequal probability sample of the agencies, which are the NSCAW III primary sampling units (PSUs). For the supervisor and caseworker surveys, the design weights will reflect both the agency and caseworker within agency selection probabilities.

To determine whether the design weights should be adjusted to compensate for nonresponse, the design weighted estimates will be compared to available sampling frame totals. It is unlikely that nonresponse weighting adjustments will be necessary or available for the agency director surveys because their response rate is expected to be high. For the supervisor and caseworker surveys, response rates are expected to be lower and nonresponse weighting adjustments may be warranted. If so, various levels of adjustments (e.g., agency, state, or region) will be compared for their impact on the mean squared errors of the estimates. Adjustments providing the best balance between bias reduction and variance inflation will be performed. Characteristics from the sampling frame, as well as from NCANDS data, will be considered in these adjustments.

Item nonresponse will also be evaluated and controlled to the extent possible. We expect that item missingness will be minimal, especially for surveys administered over the phone or in-person by an interviewer. The level of item nonresponse may be higher for surveys conducted via web. Imputation will be considered for cases where a response to a missing item can be inferred from the patterns of responses in the data.

B6. Production of Estimates and Projections

Estimates produced by this work will be prepared for internal use by ACF and external release by the agency in the form of reports, research briefs, and other publications or presentations. The estimation methods that will be employed in the analysis of the study data will follow statistical best practices. Estimates, including tabulations, descriptive analyses and model estimates, will be appropriately weighted for probabilities of selection and if deemed necessary, adjusted for nonresponse bias. Estimates and their standard errors will be produced using statistical software, such as SUDAAN, that is designed for weighted clustered samples from finite populations.

Deidentified workforce data will be archived at the National Data Archive on Child Abuse and Neglect (NDACAN). A data file user’s manual (DFUM) will accompany the dataset. The DFUM will include documentation of sampling methods, response rates, population of inference, construction of scoring and derived variables, and construction and appropriate use of survey weights. Also included will be annotated survey specifications and a codebook. The survey specifications will include the question text, variable names, response categories, and skip logic. The codebook will contain frequencies of all the variables in the dataset.

B7. Data Handling and Analysis

Data Handling

To enhance data quality, the agency director, supervisor, and caregiver surveys will be interviewer-administered. The supervisor and caseworker surveys will utilize a computer-assisted interviewing (CAI) program and the agency director instrument will use a paper instrument with a CAI data entry program. The questionnaire programs will ensure data quality by implementing skip logic, filling specific wordings based on answers previously provided by the respondent, and implementing consistency and range checks. The programs will also provide greater expediency with respect to data processing and analysis. Several backend processing steps, including editing, coding, and data entry, will be part of the data collection process.

Data Analysis

Analysis will address the identified research questions and most important knowledge gaps about current workforce in public child welfare agencies, including child welfare staff characteristics and competencies; workforce issues (e.g., recruitment, hiring, retention/turnover); training and professional development; and organizational/agency factors. Several analysis strategies are planned for the workforce surveys. First, we will describe the characteristics of caseworkers, supervisors, and agency directors currently working within the child welfare system. We will use univariate methods to derive subpopulation estimates (proportions and means in particular) of important variables related to child welfare workforce characteristics and competencies, training and professional development, organizational factors, and policy to practice issues with a high degree of precision. The analysis plan will make calculation and interpretation of these estimates a priority.

Second, we will explore and evaluate a number of predictors of workforce competencies, job satisfaction, retention, burnout, and turnover variables, primarily through the use of group comparisons and bivariate correlations. These include contingency table (crosstab) analysis with appropriate statistical tests (e.g., Pearson’s χ2) and simple regression and correlation procedures. This will enable us, for example, to examine outcomes across type of child welfare workforce (caseworkers, supervisors, and agency directors), education level, age groups and gender, on the relationship between competencies, training, organizational factors, and policy with job satisfaction, retention, and turnover.

Multivariate models will play an important part in analysis of NSCAW III-workforce survey. There are circumstances, for example, in which important questions about what variables predict an outcome must use multivariate methods to take into account confounding variables that limit interpretation. Methods with multiple predictors may be needed to address the likelihood that some consumers of NSCAW results will interpret bivariate relationships as causal. Multiple indicator methods like structural equation modeling may be needed to adapt to measurement error that is likely to arise for some NSCAW measures (Biemer et al., 2009). Questions about the relationship between agency variables and child welfare workforce outcomes necessitate multivariate analysis, since they require hierarchical linear models that can take into account sampling both at the agency and child level. The project team will be focused and selective about our use of multivariate methods; the project team will not conduct multivariate methods for every dependent variable and/or if simpler methods will be more appropriate.

The special requirements of NSCAW preclude using “off the shelf” methods in many circumstances, and require careful programming with sophisticated statistical software such as SUDAAN, MPlus, or R.

Table 2 summarizes the analysis strategies, and data sources likely to be used to address each research question.

Table 2. Research Questions, Analysis Strategies, and Instruments/Data Sources

NSCAW III- workforce survey Research Questions	Population estimates	Group comparisons/ bivariate correlations	Multivariate models	Constructs	Potential Respondents of Data Sources	Potential Data Collection instrument, measure or items
What are the demographic and educational characteristics of the child welfare workforce, including caseworkers, supervisors, and agency directors? What are the primary roles and responsibilities of the child welfare workforce? What personal characteristics and competencies (knowledge, skills, and abilities) do child welfare staff members possess and/or demonstrate?	✔	✔		Workforce Characteristics and Competencies	Caseworker Supervisor Agency director	Title and function within agency; primary roles and responsibilities; prior CWS training/ experience when hired; years of experience; competencies; education; demographics; caseload characteristics variables.
What training did child welfare staff possess when they were hired? Did their training include any courses in child welfare? What recruitment practices do agencies use to attract and hire child welfare candidates? What onboarding practices do agencies have for new child welfare workers? What types of training and professional development opportunities are offered to child welfare staff? What factors influence child welfare staffs’ abilities to engage in training and professional development opportunities? What training and professional development opportunities do child welfare staff engage in? How does this training and professional development influence job satisfaction, retention, turnover, increased knowledge, and practice?	✔	✔		Training and Professional Development	Caseworker Supervisor Agency director	Types of training received; types of professional opportunities; mentoring; caseworker safety; perception of casework preparedness (e.g. court appearances; conducting risk assessments; dealing with resistant clients); supervisory support; quality of supervision
What is the public child welfare agency landscape? This includes the number of child welfare staff, caseload size, turn-over rates, ratio of supervisors to workers, vacancy rates, salary levels and benefits, and union/non-union. To what extent are public–private child welfare partnerships taking place? What is the extent of privatization and contracting out? What is the organizational culture and climate within the public child welfare agency? How do organizational culture and climate influence workforce issues, including role conflict, burnout, turnover, supervision, caseload/workload, or education and training requirements? How do child welfare staffs’ perspectives about organizational culture and climate influence workforce issues? How has the COVID-19 pandemic potentially influenced the work of child welfare staff?	✔	✔	✔	Organizational Factors	Caseworker Supervisor Agency director	Supervisory support; quality of supervision; frequency and content of staff meetings; salary level and benefits; stress/burnout (secondary traumatic stress); organizational culture, organizational climate, work attitudes; mastery of work; agency structure and organization

Data Use

As mentioned previously, we will release a data file user’s manual (DFUM) to inform and assist researchers who might be interested in using the data for future analyses. The manual includes (1) background information about the study; (2) information about the sample design on the number of study participants, response rates, and weighting procedures; (3) an overview of the data collection procedures, data collection instruments, and measures; and (4) data preparation and the structure of data files, including data entry, frequency review, data edits, and creation of data files.

As noted in Section B6, dissemination of study findings may include reports, research briefs, 1-page data spotlights, webinars, and/or other publications or presentations. Dissemination materials will include a discussion of the limitations of the data and guidance on how to interpret the findings. Priority audiences for these products will include researchers, program staff, and technical assistance providers interested in the characteristics of and challenges faced by the child welfare workforce. Deidentified data files and documentation made available to the research community via NDACAN will contextualize and assist in the interpretation and use of the data. Data collected from agency directors will be merged with other agency-level contextual information available in NCANDS and AFCARS. Data from all workforce surveys will be analyzed to address the research questions provided in Section A.2.

We anticipate significant interest from the research community given NSCAW’s standing as a nationally representative data source and the employment of a nested design intended to represent the perspectives of multiple levels of child welfare agency personnel.

B8. Contact Person(s)

Christine Fortunato

Senior Social Science Research Analyst
Office of Planning, Research, and Evaluation

Administration for Children and Families

370 L’Enfant Promenade, S.W.

Washington, DC 20447

Christine.Fortunato@acf.hhs.gov | 202.401.5460

Paul Biemer

Distinguished Fellow, Statistics

RTI International

Post Office Box 12194

Research Triangle Park, NC 27709

ppb@rti.org | 919.541.6056

Attachments: Added January 2021

Appendix F-1: Adolescent Fact Sheet

Appendix G-1: Caseworker Telephone Mode Consent Form

Previously Approved Attachments: Modified January 2021

Appendix B: NSCAW III Child Interview

Appendix C: NSCAW III Caregiver Interview

Appendix D: NSCAW III Caseworker Interview

Appendix F: Lead Letters and Fact Sheets

Appendix G: Consent and Assent Forms

Appendix H: Data Linkage Forms

Appendix I: HIPAA Authorization Forms

Previously Approved Attachments

Appendix A: NSCAW III Summary of Interviews

Workforce ICR Appendix A: NSCAW III Phase II Research Questions

Workforce ICR Appendix B: 60-Day Federal Register Notice

Workforce ICR Appendix C: NSCAW III Phase I-II Site Selection and Sampling

Workforce ICR Appendix D: Research Questions and Constructs by Respondent Type

Instrument 1: Agency Director Survey

Instrument 2: Supervisor Survey

Instrument 3: Caseworker Survey

Workforce ICR Appendix E Agency Director Lead Letter

Workforce ICR Appendix F: Supervisor Lead Letter

Workforce ICR Appendix G: Caseworker Lead Letter

Workforce ICR Appendix H: Agency Director Consent Form

Workforce ICR Appendix I: Supervisor Consent Form

Workforce ICR Appendix J: Caseworker Consent Form

References

Bai, Y., Wells, R., & Hillemeier, M. M. (2009). Coordination between child welfare agencies

and mental health service providers, children's service use, and outcomes. Child abuse & neglect,

33(6), 372-381.

Biemer, P., Christ, S., & Weisen, C. (2009). A general approach for estimating scale score reliability for panel survey data. Psychological Methods, 14(4), 400–412.

Folsom, R. E., & Singh, A. C. (2000). The generalized exponential model for sampling weight calibration for extreme values, nonresponse, and poststratification. In Proceedings of the 2000 Joint Statistical Meetings, American Statistical Association, Survey Research Methods Section, Indianapolis, IN (pp. 598–603). Alexandria, VA: American Statistical Association.

Jacobson, N.S. & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 12–19.

1 The National Child Abuse and Neglect Data System (NCANDS) is a voluntary data collection system that gathers information from all 50 states, the District of Columbia, and Puerto Rico about reports of child abuse and neglect (OMB #0970-0424).

11