Part B Support Statement - Full revised 102616

Phase II Evaluation Activities for Implementing a Next Generation Evaluation Agenda for the Chafee Foster Care Independence Program

Full OMB Information Collection Request

New Collection

Supporting Statement

Part B

July 2016

Updated November 2016

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C St, SW

Washington, DC 20201

B1. Respondent Universe and Sampling Methods

The respondent universe for the formative evaluations include youth transitioning to adulthood from foster care and practitioners from state and local agencies, private organizations, employers and other program partners knowledgeable about employment, postsecondary education, and other programs for youth transitioning to adulthood from foster care. The team, in consultation with ACF, will identify promising programs across the country in the domains of interest above and conduct formative evaluations of between 6 and 8 programs. ACF will work with the project team to narrow promising program areas but the programs will likely include a mix of employment and postsecondary (college success) programs (at least 3 of each), or other domains of interest.

Criteria for Selecting Programs

The project team is in the process of finalizing the identification of potential programs for formative evaluation. This process continues work the team began prior to the current Phase II activities. During Phase I, the team designed a conceptual framework outlining core developmental assets youth need for success in adulthood. They also created a typology of programs for youth transitioning to adulthood from foster care to help guide identification of programs. Using this typology to guide the search, the project team identified programs working in different areas through a review of literature, scan of websites, program materials and reports, and consultations with state and local agencies, private organizations, and other knowledgeable experts. From these scans and consultations, the team identified an initial list of potentially promising programs in different areas including but not limited to employment and postsecondary education. With OMB generic clearance (OMB 0970-0356) we are in the process of interviewing practitioners from state and local agencies, private organizations, employers, and providers knowledgeable about each identified program. Based on these interviews, we will be finalizing the list of programs we will invite to participate in formative evaluation. In finalizing the list of program, the project has prioritized programs with at least some preliminary outcome data and /or components that the research review conducted for Phase I had suggested would be promising features to explore. For example, among employment programs the team looked for promising components (e.g. ‘soft’ skills training, mentoring, and work experience). Program selection criteria for formative evaluation include:

• Programs that demonstrate a clear program model and that emphasize fidelity to that model

• Programs with program features identified as promising in the research literature

• Programs that demonstrate a clear understanding about the target population(s), have a clear process for identifying the target population(s), are deploying a model that is appropriate for achieving the program’s stated objectives and has a robust referral process

• Programs that operate in a location or context that could support a rigorous evaluation (e.g., a location that does not have several other programs providing similar services that would be available to a control group – and treatment group; a contextual factor that would jeopardize the likelihood of detecting program impacts).

• Programs with available data, or the potential for gathering data, for estimating sample size – including current program participation and potential program sizes.

• Programs with available data, or the capacity to gather data, that demonstrate the intensity of the treatment.

• Programs interested in taking part in formative evaluation and in the possibility of participating in future rigorous evaluation.

Universe of Respondents

The project team will select up to 8 programs (between 6-8) to participate in formative evaluations. The universe of respondents will include:

• Program leaders: approximately 48 program leaders (roughly 6 leaders across 8 programs)

• Program partners: approximately 80 program partners (up to 10 representatives from partner organizations or agencies in each of the 8 program study sites)

• Program front-line staff: approximately 128 program front-line staff (roughly 16 staff each across 8 programs). We are recruiting all front-line staff from each program to participate.

Program participants: approximately 200 program participants (up to 16 focus groups with 12 participants each.

B2. Procedures for Collection of Information

Recruiting program staff

The project team will contact program leaders, front-line staff, and program partners affiliated with programs the team selects for formative evaluations via email using contact information from program websites. The team will use an email outreach script (Appendix A), and attach a one-page project description (Additional Project Materials) to the email. If necessary, the team will follow-up the emails or phone calls to answer any questions respondents may have, and to schedule in-person interviews while the project team is on-site at the program.

Recruiting program participants

The project team will ask program leadership and staff at the selected programs to recruit participants for the focus groups and to provide the physical space for the discussions. As directed by the project team, program staff will recruit young adult participants age 18 and older who are currently participating or who have participated in the selected programs. The project team will ask program staff to recruit participants who can speak knowledgeably about the program based on their own experience, including both those who have experienced success in the program and those who have faced challenges. The research team will inform program staff of the purpose of the focus groups so staff are prepared to answer questions posed by prospective focus group participants and will provide program staff with an informational fact sheet to aid with recruitment that will describe the purpose of the study and address other logistical questions (see Additional Project Materials).

Interview data collection

The project team will interview program leaders, frontline staff, and program partners and other stakeholders during local visits to each program. The project team will use the informed consent scripts and discussion topic guides (see Appendices C, D, E, F).

• Site visit 1 is a one-day visit for project team leaders to interview the program leaders and program partners and other stakeholders about the program’s history and components, program oversight processes, program model, content, procedures, and staff roles and responsibilities. The project team will use the discussion topic guides for program leaders and program partners and stakeholders (see Appendices B and C).

• Site visit 2 will involve the project team interviewing the program’s front-line staff in addition to follow up interviews with program partners and other stakeholders. The interviews will focus on capturing the referral and service procedures, and intended outputs and outcomes from staff and other stakeholders and other details about the program (see Appendices D and E).

• Site visit 3 will occur after the project team has synthesized information gathered from the first two site visits (including documenting the program’s expected logic model and procedures). Site visit 3 will focus on interviewing front-line staff about their roles and responsibilities as well as, in the case of front-line staff with responsibility for collecting or maintaining program data, the program’s collection of administrative data. The project team will use the discussion topic guide for front line staff and focus on staff roles and responsibilities and program data elements and uses (see Appendix E).

• Site visit 4 will be the final site visit for collecting data for the formative evaluation. It will serve to capture changes to the program model or program procedures resulting from the research team’s review of the logic model and data collection activities from the second and third site visits. The visit will involve interviewing program leaders, front-line workers, program partners and other stakeholders about the program’s capacity to sustain or implement procedures that would support rigorous evaluation (Appendices C, D, E).

Focus group data collection

Data collection will include 2 focus groups with up to 12 participants at each program site. The focus groups will last approximately 2 hours and will be conducted by two members of the project team. The project team will hold these focus groups during site visit 2 to learn about participant’s experiences and impressions of the programs (see Appendix F).

Informed Consent

At the start of each interview, the project team will ask respondents for their verbal consent to participate. During the informed consent process, the team will cover the following with all participants: the study’s purpose and funder, the nature of the data that will be collected and the data collection procedures, the way in which the information will be used, the potential benefits and risks of participating, and assurance that participation in the study is voluntary. The team will also inform study participants that they may choose to skip any questions or stop participating in the interview at any time. A copy of the consent script that the team will read and give to participants to keep is included in Appendix F. The consent forms include contact information for research team leads.

Accessing Program Administrative Data

The project team will work with each program on procedures for securing transferring and storing data. Access will likely include securing data sharing and other agreements.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Expected Response Rates

ACF anticipates a high response rate among respondents. The project team is identifying promising programs and expects that program leaders and other staff will be interested in sharing their insights with ACF. ACF anticipates that programs agreeing to the formative evaluations will be willing to participate fully in the study. Additionally, the project team will conduct interviews on site and select dates that are most convenient for program staff. To make participating as easy as possible, the project team will work collaboratively with respondents to schedule the interviews at times that are most convenient.

Data Reliability

We anticipate collecting reliable information. The formative evaluation includes interviews and focus groups as well as analysis of program data. Since the project team will be interviewing all or most program staff and a sample of program participants, it will receive information from several different sources and perspectives and will triangulate the information to identify consistencies and inconsistencies in staff and participants views of the program.

• Program leaders, staff, partners, and other stakeholders: Because the formative evaluation is intended to help program staff to better track how well their programs and services are working, and offer tools for sites to assess their own progress (e.g., helping with refining logic models and making sure the information sites collect about participants and their participation is sufficient for analysis), we expect the respondents to answer questions as truthfully as they can. From the research team’s experience from other similar studies, program staff, partners and other stakeholders interested in taking part in a formative evaluation are generally interested in learning whether their programs are effective and ways they can better assess and improve participant outcomes.

• Program participants: Generally program participants who agree to take part voluntarily in a focus group are comfortable sharing their opinions. The project team will also ensure that program staff are not present during the focus groups or that any reports do not include participants names or information that could be used to identify participants (See Appendix F). We expect that these assurances will increase participants’ willingness to be candid although the team cannot promise complete privacy. The project team will request that all participants respect each other’s’ privacy and not share the information discussed with others outside of the focus groups.

• Administrative program data: We will work with program staff to examine whether current data the program tracks are reliable and complete, and sufficient for evaluation purposes. We anticipate that programs interested and willing to take part in a formative evaluation will want feedback on the quality of their data systems and processes.

Maximizing Response Rates

Program leaders and staff: To maximize response rates the project team will work closely with the identified programs to ensure program leaders and staff fully understand what to expect from the formative evaluation and that they are interested in participating. The project team has designed the first site visit to focus on meeting with program leaders and other staff and partners to discuss the formative evaluation and confirm that the program is able and willing to participate. The team will also explain the time commitment and the responsibilities of staff. All site visits will be scheduled with enough advance notice and with input from the programs to ensure the dates are convenient for most staff. From prior experience with similar evaluations the project team generally maximizes responses and participation by sufficiently informing staff about the process. The team also seeks buy in from program leadership so leaders also may encourage their staff and other partners and stakeholders to participate.

Program participants: To maximize focus group participation, the project team will ask program staff to help with recruitment. Participants may be more likely to take part in the groups if they are invited by staff members they know. Since the project team would like to hear from both very engaged program participants and program participants who may be less engaged with the program (and likely less engaged with program staff), the team will ask staff to make a special effort to recruit less active participants. To further minimize nonresponse bias in the focus groups, the project team will give each focus group participant $25 to offset the costs of taking part in the group and will work with program staff to schedule the focus groups during a time that is convenient for most participants. The incentive is intended to assist with transportation costs, child care, or other expenses that might prevent some in our target population from participating – i.e., those with the greatest financial challenges or other barriers, and whose absence could contribute to nonresponse bias. Based on prior experience this amount is high enough to support participation, but not so high that participants would feel the incentive is excessive or coercive. We have offered $25 in other studies approved by OMB involving similar populations of low-income young adults enrolled in programs (e.g., Same-sex relationships: Updates to Healthy Marriage and Relationship Education; OMB 0970-0479).

Dealing with Nonresponse

Program leaders and staff: In the event that staff are not available to participate in interviews, we will work closely with program leaders to either identify other staff with similar knowledge, or ways to schedule telephone interviews or follow-up conversations. Because the evaluation is voluntary, any member of the program may choose not to participate. Any substantial nonresponse from members of a program will be reported as a study limitation and may result in excluding the program from the analysis.

Program participants: In the event that the project team does not have sufficient participation in focus groups, the team may consider working with staff to reschedule. If participants choose not to participate or to contribute to the focus group discussion the project team will respect the participant’s decision. The team has experience engaging young adults and others in focus groups and has generally found that individuals tend to be willing and interested in speaking and sharing their opinions. Although ACF expects high response rates, the interviews are voluntary and respondents may choose not to participate.

Administrative program data: In the event that administrative program data have incomplete or missing information, the project team will note the data elements that are missing and work with program staff to understand why and develop procedures for collecting the information more consistently. Missing or incomplete data will be highlighted in the final report as a way to help assess programs’ readiness for rigorous evaluation.

B4. Tests of Procedures or Methods to be Undertaken

The project team does not plan to pilot the discussion guides. The team has used similar guides in prior exploratory interviews during Phase I of the project and have found them to be effective. Team members have also used similar questions in other studies.

B5. Individual Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The information for this study is being collected by the Urban Institute and Chapin Hall at the University of Chicago on behalf of ACF. Co-Principal Investigators Michael Pergamit and Mark Courtney led development of the study design plan and data collection protocols, and will oversee collection and analysis of data gathered through on-site interviews and telephone interviews.

The agency responsible for receiving and approving contract deliverables is:

The Office of Planning, Research, and Evaluation (OPRE),

Administration for Children and Families (ACF)

U.S. Department of Health and Human Services

The Federal project officer for this project is Maria Woolverton.

6