Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
State Temporary Assistance for Needy Families Case Studies
OMB Information Collection Request
0970 – New Collection
Supporting Statement
Part B
July 2019
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officer:
Girley Wright
Part B
B1. Objectives
Study Objectives
The objective of the State Temporary Assistance for Needy Families (TANF) Case Studies project is to document the implementation of innovative employment and training programs for low-income individuals, including TANF recipients. The study will examine how programs provide or link families to wraparound services, such as child care assistance.
Generalizability of Results
This study is intended to produce internally-valid descriptions of programs in chosen sites, not to document program effectiveness or promote generalization to other sites or service populations.
Appropriateness of Study Design and Methods for Planned Uses
The study’s purposive site selection and qualitative case study method are appropriate for the goal of collecting comparable information and understanding the program models of innovative employment programs. By beginning the case study process with publicly-available information, the project team will focus resources on increasing the consistency of high-level program descriptions and identifying uncommon program components or approaches.
The information collected under this request will improve federal, state, and local policymakers’, practitioners’, and other stakeholders’ knowledge and understanding of innovative employment and training models designed to help low-income individuals succeed in the workplace. The project will produce a single summary report consisting of summaries of site visits and case studies of each selected program. The results will be serve as a source of well-documented information about the strategies TANF agencies and workforce development agencies are implementing to help TANF recipients and similar low-income individuals become economically self-sufficient. The results will benefit state TANF administrators and also help ACF and other key stakeholders in setting program and evaluation priorities.
B2. Methods and Design
Target Population
The State TANF Case Studies project seeks to gather consistent information from programs that are implementing innovative employment and training approaches and/or linkages between employment and training services and wraparound supports, including child care, for low-income individuals and families, including TANF recipients. The project will include a purposive sample of up to 32 programs, representing a diverse range of service strategies, geographies and community contexts, target populations, and service environments. The 32 sites will include up to 12 comprehensive case studies and up to 20 shorter case studies.
Site visits for the 12 comprehensive case studies will include document review and interviews with program administrators, frontline program staff, and program participants. The 20 shorter case studies will include document review and interviews with program administrators.
Sampling and Site Selection
The data collection is intended to describe innovative program models, not to produce generalizable or representative information about typical employment services. To identify programs to be included in the purposive sample of programs, the project team will seek input from federal and external stakeholders; search key research clearinghouses and publications; and scan relevant recent conference proceedings.
After compiling a list of potential sites, we will purposively select promising programs for inclusion in the project. “Promising” will be defined using the following criteria: program strength (a well-articulated theory of change; use of evidence-based practices; or demonstrated ability to engage hard-to-serve participants); organizational capacity (including experienced program staff and established partnerships); and the availability of unusual wraparound supports for program participants.
To the extent possible, the sample will be designed to vary in target service population; geographic setting; whether the program is replicating another evidence-based approach; and program strategy, such as substance use disorder services and integrated care models, employer-based employment interventions, full-family approaches and provision of wraparound services, and collective impact and collaborative community initiatives.
We also intend to select programs that vary in complexity, such as the number of program partners in the model, number or types of services they provide, or number or types of staff involved in service provision. For the comprehensive case studies, we expect to include up to four complex programs and up to eight less-complex programs.
Proposed interview respondents, the instruments to be used, and the respondent selection processes are explained below.
Semi-structured interviews with program staff (Instrument 1). We expect to interview 200 program staff across up to 32 programs. For programs with telephone interviews only, we expect to interview two program staff per program (2 staff x 20 programs). In complex programs we visit on site, we expect to interview 8 program administrators and 12 frontline staff (20 staff x 4 programs). In less-complex programs we visit on site, we expect to interview 5 program administrators and 5 frontline staff (10 staff x 8 programs). Program administrators include staff who administer and supervise the case study program, TANF and employment and training programs, child care and other wraparound supports, and other workforce programs; partners, such as community colleges, adult basic education providers, and employers; and state decision makers, as appropriate. Frontline program staff include intake workers, case managers, job developers, and other direct service providers who work at TANF agencies and American Job Centers, employment and training providers such as community colleges, and providers of wraparound supports, such as child care subsidy frontline staff. Respondents will be selected purposively using organizational charts and information on each staff person’s role at the case study program and its partner organizations. Purposeful staff selection is appropriate because particular insights and information can only come from individuals with certain roles or knowledge.
Guided case reviews (Instrument 2). On site, we expect to conduct up to 48 guided case reviews (4 case reviews x 12 programs) with frontline staff. In each program, we will select two frontline staff members who participated in semi-structured staff interviews. We will ask these staff members to select one case in which, in his or her view, the client successfully overcame barriers or had positive program experiences and one case in which there were significant barriers that the client was unable to overcome or contributed to a less than positive program experience.
In-depth interviews with participants (Instrument 3). On site for the comprehensive case studies, we expect to conduct up to 24 in-depth participant interviews (with up to two purposively selected participants from each program visited). Before the site visit, we will ask program staff to identify two participants with different service needs and different program experiences.
B3. Design of Data Collection Instruments
Development of Data Collection Instrument(s)
The data collection instruments for the TANF Case Studies were developed by content experts at Mathematica and MEF, and informed by reviewing instruments used in similar data collection efforts, including from the Parents and Children Together study (OMB Control Number 0970-0403), the Evaluation of Supplemental Nutrition Assistance Program Employment and Training Pilots (OMB Control Number 0584-0604), Evaluation of Employment Coaching for TANF and Related Populations (OMB Control Number 0970-0506), Exploring Measurement of Performance Outcomes and Work Requirements in Programs Promoting Economic Independence study (OMB Control Number 0990-0421), and the Institutional Analysis of American Job Centers (OMB Control Number 1225-0090).
The instruments were developed to capture essential data for the study’s main research questions. The instruments were closely examined to confirm that they were streamlined and not collecting duplicative data across respondents.
Table 1 presents a crosswalk between the data collection instruments and the study’s objectives.
Instrument |
Document the implementation of innovative employment and training programs |
Examine the ways programs provide or link families to wraparound services |
1. Semi-structured program staff interview guide |
Questions A1-A5, B1, B2, C1-C9, D1-D6, E1-E3a, E4-11, F1-F7, G1, G2, H1-H15, H20-29, M1-M12, J1-J12, K1-5, L1-L5, M1-M2, M6, N1-N5 |
Questions E1-E3, E3b, E4-11, G1, G2, H1-H4, H6-H9, H16-H19, H20, H22a-c, H24-H29, K2-K3, M3, M4-M6, N4 |
2. Case review guide |
Questions 1-10, 12-17, 18b, 18c, 19-24, 32-38 |
Questions 11,18a, 25-31, 36-37 |
3. In-depth participant interview guide |
Questions 1-26, 40-47 |
Questions 27-39 |
The project team does not intend to conduct a pre-test of the data collection instruments. Each instrument allows questions to be tailored to the respondent based on his or her organization and role. Not all questions will be asked of all respondents, and interviewers will ensure they use the program names, acronyms, and terminology that is appropriate for each respondent to understand the questions that are asked. Prior to site visits, interviewers will learn from the program contact how to word questions to take into account program-specific titles or terminology.
B4. Collection of Data and Quality Control
For the comprehensive case studies, members of the project team will conduct semi-structured interviews with program staff, in-depth interviews with program participants, guided case reviews with program staff, and observations of program activities over the course of two or three day site visits. Site visits to complex programs will take place over three days; site visits to less-complex programs will take place over two days. Telephone interviews are planned for one hour. Document review will occur before the site visits and phone interviews. The remainder of this section describes the project team’s procedures for contacting the programs, collecting data, and quality control.
Comprehensive Case Studies
After up to 12 programs have been selected for inclusion in the project’s comprehensive case studies, the project team will send each program director an email inviting the program to participate (see Appendix 1: Program recruitment emails). This email will introduce the project and will be used to schedule a call to discuss the project and site visit in more detail. An attachment to the email will contain the project description (Appendix 3). During this telephone call, the project team will describe the purpose of the site visit, confirm the program’s willingness to participate, provide an overview of site visit activities, and begin working to schedule the visit and gather existing documents. The project team will review the following kinds of documents before the site visit: organizational or staffing charts; program logic models; and the results of any previous evaluations of the program. Through document review and in consultation with the program director or their designated representative, the project team will provide an overview of the data collection approach and identify the most appropriate respondents for the data collection.
For each program, the project team will conduct on-site data collection activities using the following procedures. Before conducting each site visit, the project team will identify sections or questions of each data collection guide that are relevant to the program and the particular role of the respondent in each program. This preparation will streamline the process and reduce respondent burden.
Semi-structured interviews with program staff (Instrument 1). Interviews with program administrators and frontline staff will be conducted through 60-minute individual interviews, depending on the staffing structure, roles, and number of staff in each role. The interviewers will offer privacy assurance as part of the introduction to the interview. With permission from respondents, interviewers will record interviews to support the notes taken during the interviews. The guide for the semi-structured interviews with program staff (Instrument 1) will not be administered in its entirety in each interview. Rather, for each interview, the project team will select and ask questions that are relevant to each respondent and program.
Guided case reviews (Instrument 2). Guided case reviews will be conducted individually with frontline staff and last approximately 45 minutes per case. The interviewers will offer privacy assurance as part of the introduction to the interview for the staff member and the case under review will be anonymized. With permission from the staff respondents, interviewers will record interviews to support the notes taken during the interviews. We will then walk through the case with the staff person using electronic files or other available records and asking questions to learn about the participant’s progress through the program.
In-depth interviews with participants (Instrument 3). In-depth interviews will be conducted individually with program participants for 90 minutes, in a location that is convenient for the participant. Prior to the site visit, the project team will work with program staff to identify participant respondents and contact participants to schedule interviews in advance (see Appendix 2: Participant recruitment email). Prior to the start of the interview, interviewers will ask participants for their consent to participate. During interviews, interviewers will inform participants that their responses will be kept private to the extent permissible by law and ask for permission to record the interview. For participants that do not provide this consent, notes will be taken in lieu of a recording. As each interview progresses, the interviewer will select and ask questions that are relevant to the participant, their current economic and family situation, and their experiences with the program.
Observations (Appendix 4). We will use a versatile observation guide that can be adapted for observations of different activities, such as intakes, referrals, a job club or other type of employment workshop, and a transitional job experience. Depending on the program design, the project team may observe a subset of the following elements: office layout, decor (including signage), intake activities, client-staff meetings (when both consent), delivery of classroom training, and job placement activity. Based on the design of each program, the interviewers will select the most appropriate section of the observation guide to document their observations. Each observation will last approximately 30 minutes.
Shorter Case Studies
After up to 20 programs have been selected and approved for inclusion in the project’s phone interviews, the project team will send each program director an email inviting the program to participate (see Appendix 1: Program recruitment emails). This email will introduce the project and request a phone interview.
The project team will conduct telephone interviews with up to 20 programs to gather information for a series of shorter case studies on programs using interesting strategies aimed at supporting employment and training among TANF recipients and other low-income populations. These shorter case studies will primarily be based on existing, available information. The telephone interviews will provide information for the case studies not otherwise available through existing sources. Existing sources the project team may review before the phone interview may include organizational or staffing charts, program logic models, and the results of any evaluations that may have been produced previously about the program.
The project team will use the guide for semi-structured interviews with program staff (Instrument 1) for this data collection activity, but it will not be administered in its entirety in each interview. Rather, for each interview, the project team will select and ask questions that are relevant to each respondent and program after considering what information is available through existing sources.
Quality control
The project team will participate in data collection training to ensure data are collected systematically and consistently. The training will cover the research questions, the research approach, the topics to be covered in the data collection, and techniques for data collection. Topics to be discussed include: protecting privacy, preparing post-visit summaries and case study reports, and procedures for ensuring data security. The training will also include the data collection instruments to ensure full understanding of their purpose and the collection of comparable, complete, and high quality data across the team. The project team will meet periodically during data collection to share experiences and receive any necessary refresher training.
After the data are collected, the project team will input the data into a standard template across all programs.
B5. Response Rates and Potential Nonresponse Bias
Response Rates
The study is not designed to produce representative or generalizable findings and site participation is wholly at the program’s discretion. Site-level response rates will not be calculated or reported.
NonResponse
The study is designed to produce accurate descriptions of participating sites and does not need to secure a set number of sites or participation rate to be of value to the field.
Within selected sites, the project team will aim to arrange staff and participant interviews that accommodate respondents’ scheduling needs. If there is an instance when a targeted respondent is unable to meet while the team is on site, a member of the project team will schedule a follow-up call at a more convenient time or request to meet with an alternate respondent in a similar position. Particularly in the case of program participants, the project team will be prepared to identify and recruit alternate respondents while we are on site.
To encourage programs’ participation, the project team will contact the selected program to describe the project, its goals, and the potential benefits of their participation in the study, including concise documentation of their program’s services and their contribution to the body of knowledge of innovative employment models. Well before the site visits, the project team will begin working with program staff to ensure the timing of the visit is convenient for the program and all staff involved.
The scheduling of on site and telephone interviews will be flexible to accommodate the particular needs of respondents and program operations. The project team will be especially flexible in scheduling interviews with program participants, including being available outside of business hours, scheduling interviews at locations that are convenient for participants, being flexible to accommodate changes to respondents’ schedules, and conducting follow–up telephone interviews if necessary. Program participants who participate in the in-depth interviews, which are estimated to take 90 minutes on average, will receive a $20 gift card as a token of appreciation.
Data Analysis
After
each visit or phone call, the project team will produce written
summaries of information gathered from each program and fully
transcribe the recordings of the in-depth interviews with
participants. The project team will: develop a coding scheme
according to the project research questions; code interview and other
data using qualitative analysis software; and identify common,
similar, and unique program approaches, services, activities, and
participant experiences.
The project team will use the
qualitative findings to develop a case summary of each program, that
describes the program and its components in detail, the innovative
employment and training strategies used by the program, how the
program uses wraparound supports, how employment and training
services and wraparound supports are integrated with each other, what
implementation challenges the program has faced, and implementation
lessons learned. The project team will give the programs the
opportunity to review their case summaries to confirm their accuracy
before publication.
Data Use
The case study reports published for this study will describe the data collection and analysis process, as well as information about the context in which each program operates, including target population and geography. The reports will also include appropriate caveats to explain that the data are not generalizable.
B8. Contact Persons
Mathematica developed the plans for this data collection. Leaders of the project team from the Office of Planning, Research, and Evaluation (OPRE), Mathematica, and MEF Associates who designed and will collect and analyze the data are as follows:
Girley Wright, Administration for Children and Families, OPRE, Federal Project Officer
Kimberly Clum, Administration for Children and Families, OPRE, Federal Project Officer
Linda Rosenberg, Mathematica, Project Director
Pamela Holcomb, Mathematica, Co-Principal Investigator
Julia Lyskawa, Mathematica, Task Lead
Mary Farrell, MEF Associates, Co-Principal Investigator
Asaph Glosser, MEF Associates, Principal Associate
Sam Elkin, MEF Associates, Principal Associate
Attachments
Instruments
Instrument 1: Semi-Structured Program Staff Interview Guide
Instrument 2: Case Review Guide
Instrument 3: In-Depth Participant Interview Guide
Appendices
Appendix 1: Program Recruitment Emails
Appendix 2: Participant Recruitment Emails
Appendix 3: Project Description
Appendix 4: Observation Guide
Appendix 5: 60 Day Federal Register Notice
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |