SSB - CfS Eval Ancillary IC_fnl 6.26.23

Center for States Evaluation Ancillary Data Collection

OMB Information Collection Request

0970 - 0501

Supporting Statement Part B –

Statistical Methods

July 2023

Submitted By:

Children’s Bureau

Administration for Children and Families

U.S. Department of Health and Human Services

1. Respondent Universe and Sampling Methods

Only data from State and government agencies will be collected. Each of the proposed will be administered annually. The following list includes instruments proposed for this revision request. For instruments that were previously approved under OMB #0970-0501 and we are proposing to continue to use, we provide response rates in Table 1, below.

• Child Welfare Virtual Expo Exit Survey: This survey will be administered to all participants of the Child Welfare Virtual Expo (CWVE) at the conclusion the Expo.

• Child Welfare Virtual Expo Registration Form: This collection will be administered to all registrants of the CWVE.

• Jurisdiction Lead Interview: Jurisdictional leads will be invited each year after a project workplan closes to participate in a virtual interview via email.

• Jurisdiction Lead Observation Debrief Protocol: The state or jurisdiction project lead identified for each project observed will be asked to participate in a debrief of the meeting either in-person or virtually.

• Jurisdiction Lead Survey Related to Lived Experience: Jurisdictional project leads will be purposively selected each year based on the topic of the project if it aligns with Center workstream or priority and asked to complete the survey.

• Outcomes of and Satisfaction with Tailored Services Survey Appended Items (Section 4): The ten (10) new items will be appended to the cross-center evaluator Capacity Survey under current OMB approval (#0970-0576) and will be administered as intensive service projects¹ end throughout the year. The appended survey items will be used for up to 40 projects per year and one respondent will be invited to participate.

• Peer Group Focus Groups: Peer group members will be invited to participate in the focus groups by email messaging and will self-select a date/time slot that is convenient for them from a populated list of potential dates/times available. Some individuals are members of multiple peer groups and will be asked to participate only in the peer group with which they are most active or that is closest to their main topic of work.

Table 1. Estimated Annual Respondent Counts and Actual Response Rate Achieved*

Data Collection Instruments Title	Estimated Total No. of Respondents	Collection Conducted Previously	Previous Annual Respondent Estimate	Actual Total No. of Respondents	Actual Response Rate Achieved
Child Welfare Virtual Expo Exit Survey	300	Yes	225	270	16%
Child Welfare Virtual Expo Registration Form	3,000	Yes	1000	3736	N/A
Jurisdiction Lead Interview Adapted from two instruments Assessment and Service Delivery State Lead Interviews – Supplemental Questions and the Jurisdiction Interview Protocol (OMB#0970-0576).	40	Yes	30	17	65%
Jurisdiction Lead Observation Debrief Protocol Adapted from two removed instruments, Assessment Observation – Group Debrief and Service Delivery and Tracking and Adjustment Observation – Group Debrief) *NEW*	25	Yes	130	11	61%
Jurisdiction Lead Survey Related to Lived Experience *NEW*	30	No	N/A	N/A	N/A
Outcomes of and Satisfaction with Tailored Services Survey Appended Items (Section 4) *NEW*	40	No	N/A	N/A	N/A
Peer Group Focus Group Protocol *NEW*	150	No	N/A	N/A	N/A

2. Procedures for the Collection of Information

No statistical methodology for stratification and sample selection will be used for any of the data collection instruments. Details about plans for outreach and collection of data are provided in section A4.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Maximizing response rates is critical to the administration of the information collection activities. The content and format of the instruments were developed in close consultation with key representatives and were informed by previously developed, OMB-approved instruments.

Though these data collection activities are not designed to produce statistically generalizable findings and participation in the data collection activities is wholly at the respondents’ discretion, response rates will be collected when applicable for quality improvement purposes.

Data collection strategies that emphasize flexibility, privacy, and a respect for the respondent’s time facilitate timely participation. The following strategies will be implemented to maximize participation in the data collection and achieve the desired response rates². Referenced recruitment messages are included in Appendix A.

1. Introduction and notification:

Strategies to introduce and notify respondents about data collection are used for several instruments.

For the Child Welfare Virtual Expo Registration Form: The Center engages in a series of dissemination, marketing, and outreach activities to build registration and attendance numbers, this includes direct marketing of the event through partner social media channels, webpages, peer groups, and direct emails. Messaging is monthly for the 4 months prior to the event, and biweekly during the event month through these methods.

For the Child Welfare Virtual Expo Exit Survey: Participants are told at the beginning of the Expo that their feedback is critical to helping us improve the program and are asked to respond to the survey at the Expo’s conclusion. A follow-up email reminder is also sent after one week to increase response rates.

For the Outcomes of and Satisfaction with Tailored Services Survey Appended Items (Section 4): The Center for States or cross-center evaluators will inform potential participants about the survey prior to its release. All participants will receive an email invitation asking them to please participate in the survey. They will be provided an email link to take the survey electronically. A follow-up email reminder will be sent from the Cross-Center Evaluators after one week to increase response rates. A second follow-up e-mail will be sent one day before the survey closes.

For the Jurisdiction Lead Interview: The Center for States will notify potential Jurisdiction Lead participants by email of the option to participate in these voluntary interviews to help improve service delivery and better understand the effectiveness of the Center for States’ services. Potential interviewees will opt-in if they are interested. A follow-up email with an invitation to participate in an interview will be sent after one week if the Jurisdiction Lead has not responded to increase response rates. If a Jurisdiction Lead agrees to participate in an interview, follow-up emails will be sent to share logistical information, and to address any questions about the information collection.

For the Jurisdiction Lead Observation Debrief Protocol: Following an in-person observed meeting the observer will request to meet informally with a key jurisdiction leader who had participated in the meeting. If time is not available following the meeting, the observer will schedule a time to conduct the interview via e-mail or web-based meeting at a time convenient for the key jurisdiction leader. For virtual observations, these debriefs will occur via email (or at the request of the lead could occur via a phone call or web-based meeting).

For the Jurisdiction Lead Survey Related to Lived Experience: The Center for States will inform potential participants about the survey prior to its release. All participants will receive an email invitation asking them to please participate in the survey. They will be provided an email link to take the survey electronically. A follow-up email reminder will be sent from the Center after one week to increase response rates. A second follow-up e-mail will be sent one day before the survey closes.

For the Peer Group Focus Group Protocol: The Center will work with the peer services team to recruit peer group members for focus groups. All members of peer groups will be invited to participate. Reminders will be sent weekly to support volunteers’ ability to participate in the focus group.

2. Timing of data collection:

Individualized discussions will be held with respondents to determine optimal periods for data collection to minimize respondent burden and to facilitate recall, and to coordinate timing of data collection with the Cross-Center evaluation team. The Child Welfare Virtual Expo Exit Survey will be administered throughout the virtual Expo period following the administration guidelines described in B1. The Outcomes of and Satisfaction with Tailored Services Survey Appended Items (Section 4) will be administered as individual service projects end/close out throughout each year. The Jurisdiction Lead Observation Debrief will occur after observations of service project delivery via email or virtually through at the conclusion of the identified meetings. Jurisdiction Lead Interview will occur when a service project work plan ends, either after the work plan ends if the service project is continuing into a new work plan or six months after a work plan ends if the service project is not continuing into a new work plan. For the Peer Group Focus Group Protocol, the Center will conduct one focus group with each peer group annually.

3. Pre-interview preparation:

A copy of the interview or debrief protocols or focus group guides will be sent to participants in advance of any interview, observation, or focus group. For interviews, focus groups, and debriefs, evaluators will be flexible and accommodating with respect to scheduling and any needed rescheduling as necessary to be responsive to participants’ availability.

4. Administration:

For the Child Welfare Virtual Expo Exit Survey, at the conclusion of the Expo, participants will receive a link to the survey in a thank you e-mail after the Expo. A follow up email reminder will be sent out after one week to increase response rates. For the Outcomes of and Satisfaction with Tailored Services Survey Appended Items (Section 4) and for the Jurisdiction Lead Survey Related to Lived Experience, all participants will receive an email asking them to please participate in the survey. They will be provided an email link to take the survey electronically. A follow-up email reminder will be sent out to increase response rates. Electronic participation will allow respondents the flexibility to complete the survey at the most convenient time with minimal burden. For all interviews and focus groups (Jurisdiction Lead Interview, Peer Group Focus Group) – administration will be conducted via phone or virtual meeting (e.g., Teams or Zoom). For the Jurisdiction Project Lead Observation Debrief, administration will be conducted in-person, by phone or via virtual meeting.

5. Alternate response methods:

Respondents will be given the option to use an alternate method for responding to all interviews, focus groups, debriefs, and surveys. If a respondent prefers to submit written responses to a virtual interview or debrief, we will provide them with an electronic version to complete via email or mail. Similarly, paper versions of any of the electronic surveys and/or forms will be sent to respondents upon request or will be administered through phone or online interview if requested to accommodate any special needs.

6. Non-response:

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. The Center team will, however, track refusal rates to gain an understanding of potential patterns in data collection participation and refusal.

4. Test of Procedures or Methods to be Undertaken

The Center for States’ evaluation instruments contained herein were subject to review and feedback by key stakeholders, including Center for States staff.

User access and responsiveness to the web-based methodology for completing the Child Welfare Virtual Expo Registration and Child Welfare Virtual Expo Exit Survey, were assessed and updated throughout the last approval period to ensure each instrument was easy to use and navigate. The appended questions for the Outcomes of and Satisfaction with Tailored Services Survey (Section 4), survey items on the Jurisdiction Lead Survey Related to Lived Experience, and Jurisdiction Lead Interview were informally tested by a small group of Center for States staff to ensure utility and clarity with question wording. Small modifications were made to ensure question clarity and utility.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Capacity Building Center for States

Ms. Christine Leicht, Lead Evaluator

Christine.Leicht@icf.com

ICF

1902 Reston Metro Plaza

Reston, VA 20190

(703) 225-2208

National Cross-Center Evaluation Contractor

Dr. James DeSantis, Project Director

DeSantis@jbassoc.com

James Bell Associates

3033 Wilson Boulevard Suite 650

Arlington, VA 22201

(703) 247-2628

Should you have any questions about the contents of this OMB submission package, please contact one of the following individuals:

• Beth Claxon, Children’s Bureau, 4th Floor, Mary E. Switzer Building

330 C Street, SW Washington, D.C. 20201, telephone: 202.205.3398, email: Beth.Claxon@acf.hhs.gov

• Christine Leicht, ICF, 1902 Reston Metro Plaza, Reston, VA 20190, telephone: 703.225.2208, email: Christine.Leicht@icf.com

1 Intensive projects provide tailored, expert coaching and consultation on projects that aim to improve child welfare outcomes and support agencies’ strategic and long-term goals and are intended to last at least nine months.

2 Strategies that pertain to two or more data collections are discussed together.