NonSub Change Memo - CWCC - Response Rates

To: Jordan Cohen

Office of Information and Regulatory Affairs (OIRA)

Office of Management and Budget (OMB)

From: Mary Mueggenborg and Laura Hoard

Office of Planning, Research and Evaluation (OPRE)

Administration for Children and Families (ACF)

Date: December 11, 2020

Subject: Non-Substantive Change Request – Building Capacity to Evaluate Child Welfare Community Collaborations to Strengthen and Preserve Families (CWCC) Cross-Site Process Evaluation (OMB #0970-0541)

This memo requests approval of non-substantive changes to the approved information collection, Building Capacity to Evaluate Child Welfare Community Collaborations to Strengthen and Preserve Families (CWCC) Cross-Site Process Evaluation (OMB #0970-0541).

Background

In February 2020, OMB approved a new information collection request (ICR) for the Building Capacity to Evaluate Child Welfare Community Collaborations to Strengthen and Preserve Families (CWCC) Cross-Site Process Evaluation. The cross-site process evaluation is designed to help OPRE and the Children’s Bureau (CB) understand how communities come together to develop and implement integrated approaches to preventing child maltreatment. The cross-site evaluation includes documenting project and organizational leadership approaches, service integration and alignment strategies, and recruitment and assessment methods to identify and serve families most in need of prevention services. The study will provide a rich, detailed description of how grantees work to achieve their projects’ goals and illuminate challenges of and strategies for doing this work.

The ICR that OMB approved in February 2020 included instruments for:

• Recruiting survey sample and fielding a survey, annually; and

• Conducting in-person semi-structured interviews, annually.

In response to the COVID-19 pandemic and associated disruptions and travel constraints, and in order to learn more about the COVID-19-related challenges grantees are facing and how they are responding to those challenges, we designed an alternative approach to conducting in-person site visits and modified some of the instruments for COVID-impacted data collection. OMB approved these through a non-substantive change request in May 2020.

We fielded the survey to all participating individuals at four grantees’ sites in late June through early August 2020. Although we had cooperation from all CWCC grantee project directors, who introduced the survey and sent out weekly reminders following our approved protocol, we nevertheless obtained lower than hoped for response rates, ranging from 42% to 65% complete responses across the four grantees. In addition, two background questions designed to allow for subgroup analysis did not generate accurate data. We propose several changes in order to increase response rates and reliability of survey results for the seven future rounds of the survey data collection.

In addition, we propose a change to the process of obtaining consent for the interviews to ensure respondents have time to process the information prior to the interview itself.

Overview of Requested Changes

This memo describes the following non-substantive changes:

1. A single, unique link to the online survey for each respondent rather than for each grantee’s project.

2. Language CWCC grantee project directors can use to raise awareness of an upcoming survey fielding period.

3. Shifting responsibility for sending reminder emails to survey invitees, from CWCC grantee project directors to the CWCC cross-site process study team.

4. Use of telephone and text message reminders to non-respondents.

5a. Providing frequent opportunities to save responses in the survey.

5b. Revised background question language to better match grant participants’ roles.

5c. Addition of a location background question to the surveys that collect data from individuals associated with grantees that serve distinct geographic locations.

6. Additional information on respondents’ leadership role in the grant.

1. Single, Unique Survey Link

In our approved protocol, we send the same survey link to all survey invitees associated with the same grant, and all responses feed into the same database, anonymously. We propose a revision to this survey methodology, in that we would shift to sending each survey invitee a unique link to the survey. Responses would still be private to the extent permitted by law. We documented this change in SSA and SSB. This change required minor revisions to the email template to be used by the Abt survey team when sending survey invitations (See updated Appendix D: Email from Project to Survey Invitees including the Survey Link, Revised December 2020 and Appendix D-2: COVID Email from Project to Survey Invitees including the Survey Link, Revised December 2020).

The justifications for this change are that it would enable the survey team to more effectively focus survey reminders on non-respondents rather than sending reminders to all potential respondents and address any technical difficulties that respondents might experience in completing the survey. In addition, it will facilitate the respondents’ completing the survey in multiple sittings if needed.

2. Language for Raising Awareness of Survey

Going forward, we would like to provide more support for publicizing the survey by providing CWCC grant project directors with a flyer promoting the survey several weeks in advance. We will customize flyers for each grantee and each wave of data collection.

• We propose a new flyer – Appendix H (Flyer to Promote Collaboration Survey) – that CWCC grant project directors could use to promote awareness of the survey, beginning one month prior to survey launch.

The justification for this is to provide CWCC grant project directors with a readymade tool for promoting the survey.

3. Shifting Responsibility for Survey Reminders

In our approved protocol, we asked CWCC grant project directors to use OMB-approved templates to introduce the survey to their grant project team and then to send periodic reminders encouraging survey participation. Following approval, the survey team will track which survey invitees have not yet completed the survey, and the survey team will take over sending email reminders and encouragement to non-respondents. We propose to revise Appendix E (Reminder Email(s) from Project Directors to Survey Invitees to Increase Response Rate, Revised December 2020) to be authored by cross-site project staff and include shortened email templates for reminders.

The justification for this revision is to reduce the burden on the CWCC grant project directors while increasing the efficiency of sending reminders only to non-respondents.

4. Use of Telephone and Text Message Reminders for Survey

In addition to email reminders, we propose to use phone and/or brief text message reminders, as needed.

• We propose a new Appendix E-2: Reminder Phone and Text Messages to Increase Response Rate, which is a series of phone and text messages to remind invitees to complete the survey.

The justification for this revision is to provide more options to communicate with survey invitees who may not check email regularly or might have missed email messages about the survey.

5a. Providing Frequent Opportunities to Save Responses in Survey

In the first round of survey administration, the survey was programmed to allow respondents to save their responses before survey completion by hitting the Next button at 3 points in the survey. Given that 36% of surveys were started but not completed, we surmise that respondents may not have used the Next button to save their work. We would like to make saving data easier for respondents and reduce the amount of missing data by offering the Next button after each short section and after each background question (adding 14 opportunities for respondents to save their responses). To help ensure that respondents are aware of the Next buttons, we will also make them a prominent color. In addition, if each person has a unique link to the survey, then they can return to the survey at any time using their original link and pick up where they left off. We propose to revise the instructions in the Navigating the Survey portion of Instrument 2 (Online Annual Collaboration Survey).

The justification for this change is to improve the quality of the survey data.

5b. Revised background question language

We asked several background questions to assess respondent characteristics. The first item, Background Question 1, was designed to capture respondents’ roles on their grants. We received a significant number of nonresponses and fill-in-the-blank other responses. To improve data quality, we propose to: 1) revise the way Background Question 1 is worded; and 2) revise the categories in Background Question 1 to better reflect the roles of non-client participants on the CWCC grants. In addition Background Question 2 asked about the type of organization with which the respondent was associated. The number of missing and other responses indicate that our response categories did not map well onto respondents’ understanding of their organizational type. We propose to drop Background Question 2 as we already collect the name of each survey invitee’s organization on the Survey Invitee Template.

• We made an exhaustive search of grantee program documents to realign our role categories and also included categories that respondents listed in the “Other” open response areas in the survey in Background Question 1. In addition, we added a sentence to re-emphasize that respondents should be selecting their role on the CWCC grant project (please see Instrument 2: Online Annual Collaboration Survey, Revised December 2020).

The justification for this change is to improve the quality of the survey data.

5c. Addition of a Location Background Question to Survey

After communication with the CWCC grant project directors, it became clear that some grants are implemented in distinct locations and may rate their collaboration differently by location. To make survey results more informative for grantees, we propose adding a location question to the background portion of the survey for the grantees implementing programming in distinct geographic locations. The response categories will be tailored for each of the grantees to reflect their location terminology. We propose adding an additional item, Question 7, to Instrument 2 (Online Annual Collaboration Survey).

The justification for this change is to generate increased support for the survey by making its results more relevant to the subset of grantees who serve geographically distinct locations.

6. Additional Information on Respondents’ Leadership of Grant

Upon reviewing our hypotheses, we realized we need a measure to capture whether survey respondents were part of their grant’s leadership team. With two CWCC grantee project directors, we conducted cognitive testing (with fewer than 10 respondents) of an approach to add an item to the respondent survey and determined that it would result in poor data quality. Instead, we propose to add an item to the Survey Invitee Template which CWCC grant project directors complete to generate a list of survey invitees. We propose additional language in the instructions in Instrument 1 (Survey Invitee Template, Revised December 2020) to request that CWCC grant project directors add an ‘x’ to the row corresponding to each survey invitee who is part of grant leadership.

The justification for this change is to improve the quality of the survey data.

4