Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Next Steps for Rigorous Research on Two-Generation Programs
Formative Data Collections for ACF Research
0970 - 0356
Supporting Statement
Part B
FEBRUARY 2021
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officer: Kathleen Dwyer
Part B
B1. Objectives
Study Objectives
The Next Steps for Rigorous Research on Two-Generation Programs (NS2G) project is designed to build the capacity of the two-generation field for future ACF research. Two-generation programs aim to improve outcomes related to child development and family economic security by intentionally combining services for parents and children in the same family. NS2G has three main study objectives:
To conduct formative research and provide evaluation technical assistance (TA) to better understand program implementation, strengthen promising programs, build their capacity to engage in data-informed continuous quality improvement, and prepare them for evaluations of effectiveness
To promote a better understanding of relevant processes and outcomes of two-generation programs for children in low-income families and their parents
To build the capacity of programs and researchers to conduct rigorous and meaningful evaluations of integrated approaches to supporting child development and improving family economic security
The information collection that is the subject of this request is relevant to objectives 1 and 2.
Generalizability of Results
This study is intended to present internally-valid descriptions of four specific two-generation programs and help these programs build capacity for future summative and effectiveness evaluations. The study does not intend to promote statistical generalization to other programs or service populations.
Appropriateness of Study Design and Methods for Planned Uses
NS2G builds on an earlier project, Exploration of Integrated Approaches to Supporting Child Development and Family Economic Security, which demonstrated that to prepare for effectiveness evaluations, two-generation programs will first need to use formative evaluation to refine their models (Sama-Miller et al. 2017). The goal of the NS2G formative evaluations and TA is to strengthen a group of up to five two-generation programs as they refine their models and move them closer to readiness for evaluations of their effectiveness.
The proposed information collection will support the individual programs’ formative evaluations and TA. The instruments are designed to build understanding across the NS2G team and the program itself of how each two-generation program is designed and implemented, the challenges and opportunities to strengthen its program design, and collect feedback on the implementation of specific strategies. Information collected from these instruments will be used to inform program development and enhancement. The proposed formative evaluations and TA are a collaborative process, and findings from administering the instruments will be shared with program staff.
Because this study aims to conduct formative research to better understand program implementation with four example programs, data collection for this study is not intended to promote generalizability of findings. As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.
B2. Methods and Design
Target Population
NS2G defines two-generation programs as those that intentionally combine services intended to support both child development and family economic security. These programs intentionally serve parents and children (under age 12) from the same families. In 2016, the Exploration of Integrated Approaches to Supporting Child Development and Improving Family Economic Security (Integrated Approaches) project conducted a national scan to identify currently-operating two-generation programs. The scan identified 52 programs in operation. Of those programs, 21 programs were pilot programs or programs that had existed for less than three years, while only 2 programs had progressed to the point of participating in an outcomes evaluation. The relative newness of the field of two-generation programs suggested a need for formative evaluation and TA that would contribute to the continued development of the two-generation field and preparation of programs for evaluations of effectiveness.
Services provided by two-generation programs were influenced by the populations and areas that the programs served (Sama-Miller and Baumgartner 2017). All of the programs identified in the scan served low-income families; 19 served single mothers and their children only. Of the 52 programs, 44 served children ages birth to 3. The parent services generally focused on employment assistance and increasing skills and educational attainment through basic and postsecondary education, or individualized services such as case management (Sama-Miller and Baumgartner 2017). Services for children focused on school readiness and achievement, social-emotional and cognitive development, and physical and mental health. These programs frequently use ACF funding to provide early childhood services, such as TANF, Head Start, and Community Services Block Grants to support child services. Fourteen of the 52 programs provided Early Head Start or Head Start (Sama-Miller and Baumgartner 2017).
This will be a multisite study, involving four two-generation programs. We plan to involve a maximum of 176 program staff and parents across all the sites.
Sampling and Site Selection
The NS2G project used a purposive strategy to identify sites for NS2G. To narrow down the potential sites for selection, we considered programs that were (1) in the pool of programs identified in OPRE’s Integrated Approaches study, (2) nominated by ACF staff, or (3) recommended by one of the following two-generation program experts: Allison Holmes of the Annie E. Casey Foundation; Marjorie Sims of Ascend at the Aspen Institute; and Sharon McGroder, an independent consultant. From this narrowed list, the selected programs:
Were motivated and enthusiastic about participating in NS2G.
Were implementing two-generation services (not just be interested in providing services for low-income parents and their children).
Served populations of interest to ACF; at least one of the programs offered Head Start services
Had established partnerships to provide services, if partnerships with other organizations are necessary for them to provide two-generation services.
Had relevant data available and an appetite for using the data to inform program improvement. Most programs were able to determine whether parents and children in the same family are being served.
Had stable and committed leadership.
The project team is involving four programs in the NS2G formative evaluations and TA. These programs are:
Brighton Center, in Newport, Kentucky
Garrett County Community Action Committee, in Oakland, Maryland
San Antonio Dual Gen Initiative, in San Antonio, Texas
Valley Settlement, in Carbondale, Colorado
B3. Design of Data Collection Instruments
Development of Data Collection Instruments
Mathematica designed the Program Confirmation Call Protocol (Instrument 1), the Site Visit Topic Guide (Instrument 2), the Participant Focus Group Protocol (Instrument 3), the Program Staff Survey (Instrument 4) and the formative evaluation feedback survey (Instrument 5) specifically for the NS2G project.
Instruments 1 and 2 are informed by implementation science frameworks such as the Consolidated Framework for Implementation Research and the Active Implementation frameworks developed by the National Implementation Research Network (NIRN). These instruments contain semi-structured interview questions and topics that are designed to gather information about key components thought to contribute to strong implementation of evidence-based programs and practices, in order to document how two-generation programs are designed, implemented, and staffed.
Instrument 2 includes a series of interactive, design thinking exercises intended to engage program staff in identifying program challenges and brainstorming creative solutions. These activities are informed by human centered design approaches developed by the LUMA Institute.
Instrument 3 is intended to gather program participants’ experiences enrolling and participating in two-generation program services, and was informed by an instrument developed under and used in the Integrated Approaches study to collect information about the experiences of two-generation program participants during field work (Sama-Miller et al. 2017).
Instrument 4 is intended to collect rapid feedback about program staff members’ use of a strategy, process, or tool, and was informed by instruments used on a number of other ACF projects that have used the LI2 framework to conduct formative rapid-cycle evaluations (for example, McCay et al. 2017).
Instrument 5 was informed by research on how to measure the effectiveness of TA (Rachidi 2019).
The project team reviewed the instruments to ensure that they ask only questions necessary to achieve the objectives of the information collection. All instruments were created specifically for the project, do not include any scales or items that measure constructs, and do not require psychometric testing.
B4. Collection of Data and Quality Control
Two Mathematica NS2G project team members (“TA liaisons”) will be assigned to work with each program; these project team members will collect the data. Data quality and consistency will be ensured by assigning team members with extensive experience conducting interviews and surveys, facilitating focus groups, and providing additional training on the relevant instruments.
Each site’s formative evaluation will begin with a program confirmation call (Instrument 1). In this call, two NS2G project team members will ask guided questions to confirm our understanding of each program’s services and capacity to participate in the formative evaluation and TA, including their data systems, the data they collect, how they use data, their past experiences with research and evaluation, and staffing. Using this information, Mathematica and the programs will co-develop individualized formative evaluation and TA plans that specify the types of activities that programs will engage in during their formative evaluations.
Following the program confirmation call and development of the formative evaluation and TA plan, members of the NS2G project team will conduct aninitial virtual site visit. Part 1 of the site visit is part of the “Learn” phase, where we will work with the program and its partners to develop a comprehensive, shared understanding of their services and service challenges and deepen our understanding of the site’s TA needs and stage of development. TA liaisons will interview program staff partners to learn about the program’s community context, vision and goals, service delivery, staffing, and data management practices (Instrument 2). The TA liaisons will also conduct a focus group with program participants to collect feedback on the participants’ experiences in the program (Instrument 3).
In Part 2, the TA liaisons and program staff will co-design new strategies to address challenges in service delivery, such as improving the integration of services for parents and their children. First, to wrap up the “Learn” phase, TA liaisons will use human-centered design activities to help program staff develop a common understanding of their program’s intended outcomes, challenges, strengths, and opportunities, and articulate or refine a program logic model. Then, staff will begin developing creative solutions to the implementation challenges they identified earlier (“Innovate”). At the end of Part 2, TA liaisons will ask participating staff to rate the design-thinking session by responding to a few short questions through a web survey. TA liaisons will continue the “Innovate” phase during regular program TA calls using the lessons from the site visit to inform the rest of the formative evaluation and TA.
Once a program has developed a strategy to improve an aspect of its programming, TA liaisons will ask the program will complete a formative RCE to test its approach. The program staff survey (Instrument 4) will be used to pilot test a strategy to help the program “Improve.” In the formative RCE, the program will complete up to three sequential learning cycles. In each learning cycle, a small number of program staff (up to 10 staff per program) will pilot a strategy, tool, or process for a short period of time (up to eight weeks). For example, a program may develop and pilot a new process for sharing information about families between adult- and child-focused frontline staff. They will provide feedback through a 10-minute web-based survey administered about once per week. At the end of each learning cycle, TA liaisons will work with the programs to interpret data from the program staff survey to understand whether and how the approach was successful and what the program could do differently. Then, TA liaisons and the program will collaboratively refine the strategy, tool, or process and will begin another short learning cycle.
About two years after the start of their participation in the study, the sites will share what they learned from their formative evaluations and TA and develop action plans to continue their work through the end of the project.
Staff from the two-generation programs who are responsible for championing and leading the formative evaluation and TA work in their respective programs, will respond twice to a short feedback survey about the process of participating in the formative evaluations (Instrument 5). NS2G will administer the survey once at about the midpoint of the formative evaluations, in late 2021 or early 2022, and once at the end of the study, in early 2023. NS2G will review the feedback to understand whether and how program staff felt that their organization increased its capacity and functioning as a result of participating in the study.
Due to the COVID-19 pandemic, all activities will be completed virtually.
B5. Response Rates and Potential Nonresponse Bias
Response Rates
The data collection is not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. The data collection will document examples of programs that provide services to parents, children, and families. Because data collection is not designed to be representative, response rates will not be reported.
NonResponse
As participants will not be randomly sampled and findings are not intended to be representative, nonresponse bias will not be calculated.
B6. Production of Estimates and Projections
We will not produce estimates or weights for this project.
B7. Data Handling and Analysis
Data Handling
Qualitative data will be collected through note taking (including screen captures of images of virtual brainstorming) during the virtual site visit (Instruments 2 and 3). The survey data (Instruments 4 and 5) will be self-reported by program staff members. We will use audio recordings of interviews and focus groups (Instruments 2 and 3) to verify notes. As a part of the formative rapid-cycle evaluations, we will review summary information from Instrument 4 with program staff to confirm whether it aligns with their experiences and recollections of pilot-testing strategies. We will also review response rates for Instruments 4 and 5 to assess how much emphasis to put on the survey results for refining strategies to address implementation challenges.
Data Analysis
This project will not employ any complex analytical techniques. To analyze the data collected through interviews during the virtual site visit, we will use standard qualitative analysis techniques such as thematic identification; the primary purpose of the site visit is to inform future TA. For the formative rapid-cycle evaluation staff survey, we will conduct standard qualitative analysis of responses to open-ended items and calculate ranges and averages for the remaining questions.
Data Use
This study is intended to provide descriptions of two-generation programs’ capacity building and improvements to their readiness for summative evaluations of their effectiveness. When necessary, results will be labeled as examples. NS2G will develop a summative report and up to three briefs about the formative evaluations and TA. Although the information collected under this clearance will not be the primary subject of any published ACF reports, the information may be included in published products to share insights from the formative evaluations and TA. Topics could include: the process of capacity building, the development of tools used in strengthening two-generation program models, descriptions of program models and the activities that programs engaged in through formative evaluations and TA, or recommendations for next steps in strengthening two-generation programs for future evaluations of effectiveness. The products will not share quantitative findings nor hard data about the programs. In sharing findings, we will describe the study methods and limitations to generalizability and as a basis for policy.
B8. Contact Person(s)
Emily Sama-Miller
Senior Researcher
Mathematica
1100 First Street NE
Washington, DC 20002
esamamiller@mathematica-mpr.com
202-484-4512
References
McCay, Jonathan, Michelle Derr, and Ann Person. “Using a ‘Road Test’ to Improve Human Services Programs.” OPRE Report #2017-107. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Rachidi, Angela. “Is Your Training and Technical Assistance (T/TA) Effective? Considerations for Measuring Performance.” Washington, D.C.: Office of the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services, 2019. Available at: https://aspe.hhs.gov/system/files/aspe-files/261886/measuring-t-ta-considerations-toolkit.pdf
Sama-Miller, Emily, Christine Ross, Teresa Eckrich Sommer, Scott Baumgartner, Lily Roberts, and P. Lindsay Chase-Lansdale. “Exploration of Integrated Approaches to Supporting Child Development and Improving Family Economic Security.” OPRE Report # 2017-84. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Sama-Miller, Emily and Scott Baumgartner. Achieving Family Economic Security and Promoting Child Well-Being: Features of Contemporary Programs, OPRE Report #2017-49, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Aspen Institute. “National Network/2Gen Learning Community.” Available at https://ascend.aspeninstitute.org/two-generation/national-network2gen-learning-community. Accessed February 18, 2020.
Attachments
Instrument 2: Updated Site Visit Topic Guide (tracked)
Instrument 2: Updated Virtual Site Visit Topic Guide (clean)
Instrument 3: Updated Participant Focus Group Protocol (tracked)
Instrument 3: Updated Participant Focus Group Protocol (clean)
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Ann Person |
| File Modified | 0000-00-00 |
| File Created | 2023-10-17 |