Ssa_streams_fup2_2-13-19

Strengthening Relationship Education and Marriage Services (STREAMS) Evaluation

OMB Information Collection Request

0970-0481

Supporting Statement

Part A

February 2016

Updated June 2016

Updated March 2017

Updated January 2019

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Samantha Illangasekare

Overview

• Status of study:

  • As approved under OMB#0970-0481, the ACF Office of Planning, Research, and Evaluation (OPRE) is conducting the Strengthening Relationship Education and Marriage Services (STREAMS) evaluation with five Healthy Marriage and Relationship Education (HMRE) grantees.

    • Previous Approval: Data collection for the process and impact studies were previously approved and includes qualitative interview protocols, a baseline survey for youth and adults, and a follow-up survey for youth and adults administered about 12 months after completing the baseline. The universe of data collection efforts is detailed in section A2.

    • Current Request: The current data collection request is for (1) an extension of previously approved data collection instruments for the STREAMS impact study and (2) two additional data collection instruments for conducting a second follow-up with impact study participants in two sites.

• What is being evaluated (program and context) and measured.

  • The STREAMS evaluation includes two components: (1) an impact study to provide rigorous estimates of the effectiveness of HMRE program services and interventions to improve program implementation and (2) a process study to support the interpretation of impact findings and document program operations to support future replication.

• Type of study:

  • This is an impact evaluation and process study. The impact evaluation involves random assignment.

• Utility of the information collection:

  • To expand the research base on the full range of HMRE programs funded by ACF, the study is assessing the effectiveness of:

    • commonly used HMRE curricula for both youth and adults, as well as the integration of HMRE and economic stability services for adults

    • strategies for improving the quality of program delivery of HMRE programming in high schools and the effectiveness of curricula at different levels of dosage of HMRE programming

    • behavioral economics strategies to encourage participant engagement in services.

  • The information from this study is expected to be of use to policymakers, grantees, TA providers, and researchers. The information could inform policy making, program improvement, decision-making, program design, funding decisions, and the HMRE research field.

A1. Necessity for the Data Collection

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval for (1) an extension of previously approved data collection instruments for the STREAMS impact study and (2) two additional data collection instruments for conducting a second follow-up with impact study participants in two sites.

Study Background

Healthy Marriage and Relationship Education (HMRE) programs have been undergoing a transformation over the past decade. At first a new approach for serving vulnerable families, such programs have become an established presence in many communities, with connections to other agencies and a growing number of families served. In the early 2000s, ACF announced the Healthy Marriage Initiative, which provided funding to federal grantees through existing legislative authorities to add marriage education to their service offerings. This effort coincided with findings from the longitudinal Fragile Families and Child Well-being Study that suggested the period around a child’s birth could be an opportunity for intervening with unmarried couples, who typically were romantically involved and interested in marriage (McLanahan et al. 2001).

The Deficit Reduction Act of 2005 created the HMRE grant program, which authorized $150 million over five years to support program activities aimed at promoting and sustaining healthy marriages, providing relationship education services to youth, and fostering economic stability. The Claims Resolution Act of 2010 re-authorized this grant program, and three-year grants totaling $150 million were awarded in September 2011 (and subsequently extended through September 2015). In September 2015, ACF to awarded five-year grants to 46 HMRE grantees to serve a broad spectrum of couples, single adults, and youth.

Studies of the effectiveness of HMRE programs serving couples that were funded by the government have produced mixed results. Most federally sponsored HMRE research in recent years—including the Building Strong Families (BSF) evaluation, the Supporting Health Marriage (SHM) evaluation, and the ongoing Parents and Children Together (PACT) evaluation—has focused on programs serving low-income couples raising children together. BSF, which examined HMRE programs for unmarried parents, found no overall effects on relationship quality or status (Wood et al. 2012; 2014). Positive effects in one BSF site at 15 months generally did not persist at 36 months. SHM, which examined HMRE programs for low-income married parents, found small positive effects on relationship quality and declines in psychological abuse and distress at 12 and 30 months, but no effects on relationship status (Hsueh et al. 2012; Lundquist et al. 2014).

However, many HMRE programs serve youth; many others serve adults as individuals and do not serve couples. Little rigorous research has been conducted on these programs or the curricula they implement. One rigorous study of an HMRE youth curriculum, Relationship Smarts PLUS, conducted in 39 Alabama high schools found short-term positive effects on measures of healthy relationship beliefs and perceived conflict management ability (Kerpelman et al. 2009). Other prior studies used less rigorous research methods. A pre-post study of Love U2: Communication Smarts found improvements on measures of relationship attitudes and skills (Antle et al. 2011). A small pre-post study of the Within My Research curriculum for single adults reported improved relationship knowledge and skills (Antle et al. 2013). A small quasi-experimental study of another curriculum for single adults, PICK a Partner, reported increased participant knowledge and confidence in participants’ ability to develop healthy relationships (Van Epp et al. 2008).

To expand the research base on the full range of HMRE programs funded by ACF, in March 2015, the agency contracted with Mathematica Policy Research to conduct the STREAMS evaluation. The study entails designing and implementing a multi-site impact evaluation of HMRE programs. It aims to identify strategies for improving the effectiveness and delivery of HMRE programs by conducting multiple rigorous tests of program components and implementation factors. Sites were chosen from among the 45 ACF healthy marriage grantees that were awarded in October 2015. Sites were selected to answer a range of distinct research questions that address specific policy priorities for ACF and that will help to fill key gaps in the existing research on healthy marriage programming. In particular, the study is assessing the effectiveness of commonly used HMRE curricula for both youth and adults, as well as the integration of HMRE and economic stability services for adults. The study is also assessing the effectiveness of strategies for improving the quality of program delivery of HMRE programming in high schools, effectiveness of curricula at different levels of dosage of HMRE programming, and the effectiveness of behavioral economics strategies to encourage participant engagement in services. Sites were selected for their suitability for answering priority research questions, and not to be nationally representative of all healthy marriage grantees.

Legal or Administrative Requirements that Necessitate the Collection

This is a discretionary data collection authorized under Sec. 811 (b) Healthy Marriage Promotion and Promoting Responsible Fatherhood Grants of the Claims Resolution Act of 2010, Pub. L. No. 111-291, 124 Stat. 3064 (Dec. 8, 2010). A copy of the legislative authority is included as Attachment A.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

Previously Approved Request

The STREAMS evaluation includes two components, an impact study and a process study. The goal of the impact study is to provide rigorous estimates of the effectiveness of program services and interventions to improve program implementation. The impact study uses an experimental design. In each site, eligible program applicants are randomly assigned to either a program group that is offered program services or a control group that is not. STREAMS collects baseline information from eligible program applicants prior to random assignment and administers a follow-up survey to participants 12 months after random assignment. The goal of the process study is to support the interpretation of impact findings and document program operations to support future replication. STREAMS conducted semi-structured interviews with program staff and selected community stakeholders, conducted focus groups with program participants, administered a survey to program staff, and collected data on adherence to program curricula through an add on to an existing program MIS.

Both the impact study and the process study information collection requests were initially approved on July 5, 2016 (OMB #0970-0481). Data collection for the process study began in February 2017 and was completed in September 2018. Data collection for the impact study began in July 2016 and is ongoing.

Current Request

For this current data collection request, ACF seeks approval for 1) an extension of previously approved data collection instruments for the STREAMS impact study and 2) two additional data collection instruments for the impact study that would collect additional follow-up data at two of the five participating sites. This revision would extend data collection through spring 2022. The purpose of these two additional data collection instruments is to allow for an assessment of the longer-term impacts of HMRE programming by conducting an additional follow-up survey with impact study participants. The information from the entire STREAMS study is expected to be of use to policymakers, grantees, TA providers, and researchers. The information could inform policy making, program improvement, decision-making, program design, funding decisions, and the HMRE research field.

Research Questions

Questions specific to the ongoing data collection and new data collection included in this current request focus on longer-term participant impacts in two sites: More Than Conquerors, Inc. and University of Denver.

The goal of the STREAMS impact study is to provide rigorous estimates of the effectiveness of program services and interventions to improve program implementation in each of five sites. As shown in Table A.1, each site involves a different target population and addresses one or more distinct research questions. One site assesses the effectiveness of HMRE programming for youth in high schools. The four other sites focus on adult samples. The goal of the STREAMS process study is to support the interpretation of impact findings and document program operations to support future replication.

Table A.1. Overview of STREAMS impact study sites

Grantee	Target population	HMRE curriculum	Primary research questions	Information collection
More Than Conquerors, Inc.	1,900 students from two Atlanta, Georgia-area high schools	Relationship Smarts Plus	What is the effect of offering relationship skills education as part of the regular school curriculum? How does abbreviating the curriculum influence program effects?	Baseline survey Follow-up survey Second follow-up survey
University of Denver	1,000 low-income pregnant women or new mothers from the Denver Health hospital system in Colorado	Within My Reach	What is the effect on adult relationship quality and stability of offering relationship skills education and other support services to low-income pregnant women?	Baseline survey Follow-up survey Second follow-up survey
The Parenting Center	900 low-income romantically involved couples in Fort Worth, Texas	Family Wellness	How does an integrated approach to relationship skills and economic stability services affect the outcomes of participating couples?	Baseline survey Follow-up survey
Family and Workforce Centers of America	900 low-income job seekers from an employment center in St. Louis, Missouri	Within My Reach	What is the effect on adult relationship quality and stability of supplementing job readiness services with the relationship skills education?	Baseline survey Follow-up survey
University of Florida	1,500 couples participating in relationship education workshops across five Florida counties	ELEVATE, Smart Steps	Can text messages informed by behavioral insight theory improve attendance at relationship skills education group sessions for couples?	nFORM program participation data

Study Design

The STREAMS impact study uses an experimental design. In three of the impact study sites—the University of Denver, the Parenting Center, and FWCA—individuals are randomly assigned to either a group that is offered program services or a control group that is not. Grantee staff in each site conduct random assignment using a web-based system that STREAMS developed as an added component to ACF’s Information, Family Outcomes, Reporting, and Management (nFORM) information system (OMB #0970-0460). An extension of previously approved burden for this random assignment feature of nFORM is included in this data collection request.

In the site serving youth in high schools (More than Conquerors, Inc.), random assignment occurs at the classroom level and involves three research groups. Two research groups receive the curriculum, either the full 12-session version or a shortened 8-session version, and the third group is the control group. By randomizing groups of students by classroom, the evaluation will test the effectiveness of receiving either the full 12-session curriculum or the shortened 8-session curriculum relative to receiving no HMRE programming. There is no burden to grantee staff of random assignment in this site.

In all four of these evaluation sites, STREAMS collects baseline and follow-up survey data. Data from the baseline survey will be used for two purposes. First, the baseline data will be used to describe characteristics of the study sample. This step will enable ACF to understand the characteristics of the populations being served and to provide guidance on how the study sample and findings might generalize to a broader policy setting. Second, baseline data may also be used for exploratory subgroup analyses, to examine the demographic and personal characteristics that may moderate the impacts of healthy relationship education programming. Baseline surveys are administered at the time of study entry. In the site serving youth in high schools (More than Conquerors, Inc.), baseline data collection ended in February 2018. For the sites serving adults, an extension of previously approved burden for baseline data collection is included in this data collection request.

Follow-up surveys are administered 12 months after the baseline survey, to measure the effectiveness of HMRE services and program components on participant outcomes. The evaluation team will limit the primary analyses for each site to a small set of key outcomes. In selecting these primary outcomes for the impact analysis, the evaluation team will rely on the program logic model developed for each site. For sites serving adults, the team anticipates that most of these outcomes will be measures of the quality and stability of relationships. For the site serving youth in high schools, the primary outcomes are expected to be measures of relationship knowledge, skills, and attitudes. An extension of previously approved burden for the STREAMS follow-up survey is included in this data collection request.

STREAMS also intends to measure the longer-term effects of HMRE services by administering additional follow-up surveys for participants at two sites – one serving adults and one serving youth in high schools. Specifically, ACF is seeking approval for a second follow-up survey at 30 months for University of Denver participants and at 24-36 months for More Than Conquerors, Inc. participants.

A fifth site, the University of Florida, is not participating in baseline or follow-up survey data collection. In this site, participants are randomly assigned to different strategies to promote program participation, such as text messages using different techniques to motivate regular attendance. Baseline and follow-up survey data are not collected with the University of Florida site because the evaluation is testing the effectiveness of the text messages on program participation rather than the effectiveness of the program on participant outcomes.

Universe of Data Collection Efforts

To address these research questions, STREAMS uses several data collection sources. This subsection provides an overview of the new follow-up information collection request, the instruments that were previously approved and will continue to be fielded, and the instruments for which data collection is complete.

New Information Collection Request

As part of this current data collection request, ACF is seeking approval for two additional data collection instruments associated with the impact study:

• Second follow-up survey for youth (Instrument 12). For the STREAMS site serving youth in high schools (More Than Conquerors, Inc), ACF seeks to conduct a second follow-up with study participants during the 2019-2020 school year, approximately 24-36 months after the baseline survey. Data from the second follow-up survey will allow for an assessment of the longer-term impacts of the program two to three years after study enrollment. Study enrollment in the More Than Conquerors, Inc. site began at the start of 2016-2017 school year and ended in the spring semester of the 2017-2018 school year. The timing of the second follow-up survey will mean that for some students, the second follow-up will occur two years after they enrolled in the study, while for others it will be three years after they enrolled.

The survey will be administered via computer-assisted self-administered interviewing (ACASI) on a tablet device. In most cases, the survey will be conducted in the school through group administration. To achieve a high response rate to the follow-up survey, the evaluation team will contact youth who do not complete the follow-up in school and request that they complete the survey via computer-assisted telephone interviewing (CATI) or computer-assisted web interviewing (CAWI).

The survey will cover many of the same areas as the first follow-up survey (approximately 90 percent of the second follow survey is the same as the first follow-up), such as youth relationship attitudes and experiences. Because romantic relationships become more common as youth age, assessing these outcomes at a later point will provide a more definitive test of the program’s impacts on youth relationship outcomes. Attachment S provides a question-by-question justification of survey items and indicates the questions that are the same as those in the first follow-up and questions that have been added to the second follow-up.

• Second follow-up survey for adults (Instrument 13). ACF also seeks to conduct a second follow-up survey in one of the sites serving adults. In particular, the University of Denver site is assessing the effectiveness of an HMRE program for low-income pregnant women or new mothers from the Denver Health hospital system in Colorado. For this target population, a second follow-up survey will allow for an assessment of the program’s longer-term impacts on a range of outcomes of policy relevance to ACF, including relationship quality and status, father involvement, parenting, and child well-being.

As with the first follow-up survey, the second follow-up survey will be administered via CATI and CAWI. The second follow-up survey will be conducted approximately 30 months after study enrollment. To achieve a high response rate to the follow-up survey, the evaluation team will send the previously approved advance letters to study participants prior to attempting contact by telephone.

The second follow-up survey will include many of the same items as the baseline and first follow-up survey for adults, in addition to items on relationships, parenting, child well-being, and economic outcomes. See Attachment T for a question-by-question justification of these items and indicates the questions that are the same as those in the first follow-up and questions that have been added to the second follow-up.

Previously Approved; Data Collection Ongoing

ACF requests an extension to continue to collection information using the following data collection instruments for the STREAMS impact study. There are no changes requested to these instruments.

• Introductory script, grantee staff (Instrument 6). In STREAMS sites serving adults, grantee staff use this script to introduce applicants to the HMRE program and the STREAMS evaluation and to answer applicants’ questions about the study. Grantee staff read the introductory script to applicants prior to conducting random assignment.

• Introductory script, program applicants (Instrument 6). In STREAMS sites serving adults, program applicants are read this script as an introduction to the HMRE program and the STREAMS evaluation and to answer applicants’ questions about the study. This script is read to applicants prior to conducting random assignment.

• Add-on to nFORM to conduct random assignment (Instrument 7). In STREAMS sites serving adults, grantee staff use an added component to the nFORM system (OMB no. 0970-0460) to conduct random assignment. After reading the introductory script to applicants and determining eligibility, grantee staff enter applicant information to enroll sample members and perform random assignment.

• Follow-up survey for youth (Instrument 9). For the site serving youth in high schools, follow-up surveys are conducted via computer-assisted self-administered interviewing (ACASI) on a tablet device approximately 12 months after study enrollment. In most cases, the survey is conducted in the school through group administration. To increase the response rate to the follow-up survey, the evaluation team contacts youth who do not complete the follow-up in school and request that they complete the survey via computer-assisted telephone interviewing (CATI) or computer-assisted web interviewing (CAWI).

• Baseline survey for adults (Instrument 10). In sites serving adults, baseline surveys are conducted via computer assisted telephone interviewing (CATI). When an individual enrolls in the study, grantee staff call the Mathematica Survey Operations Center and connect the applicant to a trained telephone interviewer who then administers consent and conducts the baseline survey with the applicant. The baseline survey is used to collect information from study participants on their characteristics, pre-program measures of key outcomes, and contact information used to locate them for follow‑up survey completion.

• Follow-up survey for adults (Instrument 11). In sites serving adults, the follow-up survey is administered via CATI and CAWI. The survey is conducted approximately 12 months after study enrollment. To increase the response rate to the follow-up survey, the evaluation team sends previously approved advance letters to study participants prior to attempting contact by telephone.

Previously Approved; Data Collection Complete

Previously approved data collection instruments with which data collection is complete include:

• Topic guide for staff and stakeholder interviews (Instrument 1). The purpose of this information collection was to document manager, staff, and community stakeholder experiences and perspectives about the implementation and operation of HMRE programs. This guide was used during site visits to collect information from program managers, program staff, and community stakeholders on topics such as program plans and goals, staffing structure, recruitment and engagement, service delivery, enrollment and retention strategies, goal attainment, and community context.

Focus group guide for adults (Instrument 2). The purpose of the focus group guide for adults was to obtain information about adult program participants’ motivations for enrolling in the program and their perspectives on the availability, quality, and value of program services. The evaluation team also asked about participants’ level of satisfaction with the program and their self-assessment of knowledge and skill gains from program participation. The purpose of the focus groups was to document participants’ experiences and satisfaction with the program, as well as their perceptions of the knowledge and skills they gained through program participation. Focus group data will not be used to measure program effects. Program effects will be measured using data from follow-up surveys which cover a range of topics, including the quality and stability of romantic relationships and other outcomes.

• Focus group guide for youth in schools (Instrument 3). The purpose of the focus group guide for youth was to obtain information about youth program participants’ motivations for enrolling in the program and their perspectives on the availability, quality, and value of program services. The evaluation team also asked about participants’ level of satisfaction with the program and their self-assessment of knowledge and skill gains from program participation. The purpose of the focus groups was to document participants’ experiences and satisfaction with the program, as well as their perceptions of the knowledge and skills they gained through program participation. Focus group data will not be used to measure program effects. Program effects will be measured using data from follow-up surveys which cover a range of topics, including the quality and stability of romantic relationships and other outcomes.

• Staff survey (Instrument 4). The purpose of this survey was to obtain more systematic and potentially more candid information than can be gained from program staff through interviews during site visits. Site visit interviews of program staff were often conducted in group or other semi-public settings, which may have limited staff’s willingness to disclose certain work experiences. Staff completing the survey may have been more willing to candidly report on their experiences working with the program due to the anonymity afforded by the survey and the ability to complete the instrument in a private setting. The survey gathered information from program staff on their work activities, work experience, interactions with other staff members, opportunities to receive training, supervision, and perceptions of the supportiveness of the organization. The survey was self-administered as a web survey.

• Session adherence form (Instrument 5). The purpose of this form was to collect data on facilitators’ adherence to the HMRE curriculum during each group session. Facilitators reported on the materials, lessons, and activities they used during the sessions as well as any disruptions or difficulties with conducting the session as planned. The STREAMS evaluation team included an additional screen in the nFORM system (OMB no. 0970-0460) for use in STREAMS sites that facilitators could use to record this information.

• Baseline survey for youth in high schools (Instrument 8). The purpose of this survey was to collect information from study participants on their characteristics, pre‑program measures of key outcomes, and contact information used to locate them for follow-up survey completion. The survey was administered via audio computer-assisted self-administered interviewing (ACASI) on a tablet device during group administration in high school classrooms.

A3. Improved Information Technology to Reduce Burden

Previously Approved Request

The evaluation team is using technology to reduce burden for the baseline and first follow-up surveys. In the STREAMS site serving youth in high school, the evaluation team is conducting the surveys using ACASI. The respondent listens to a recording of the survey questions and then enters responses on a tablet device. ACASI allows programming of skip logic and response validation, creating a more streamlined experience for respondents than a paper-and-pencil survey and ensuring that respondents do not inadvertently complete sections of the survey that they should skip. The evaluation team will contact youth who do not complete the follow-up in school and request that they complete the survey by telephone using CATI.

In the STREAMS sites serving adults, the evaluation team is conducting the baseline survey using CATI. This technology is well suited to interviews with complex skip patterns, the need for interviewer probes, and the large number of respondents. CATI reduces respondent burden by automating skip logic and question wording adaptations and eliminating delays caused when interviewers have to determine for themselves the next question to ask. CATI is programmed to accept only valid responses based on pre-programmed checks for logical consistency across responses. Interviewers are able to correct errors during the interview, eliminating the need for burdensome call-backs to respondents.

The first follow-up survey for adults is administered using CATI. The survey is also being administered using web. Similar to CATI surveys, the web survey reduces respondent burden by automating skip logic and question wording adaptations and eliminating delays caused when interviewers have to determine for themselves the next question to ask. The web survey is programmed to accept only valid responses based on pre-programmed checks for logical consistency across responses.

All surveys administered using CATI are recorded for quality assurance. These recordings are used by phone center supervisors, as well as the project team, to monitor that surveys are being conducted correctly. Respondents can decline to be recorded and all recordings are stored on a secure server and destroyed at the end of the study.

New Information Collection Request

For the second follow-up, the evaluation team will use the same technology as the baseline and first follow-up surveys to reduce respondent burden.

A4. Efforts to Identify Duplication

The STREAMS evaluation will not require the collection of information that is available from alternative sources. To avoid potential duplication, the STREAMS evaluation team will rely on data being collected from grantees as part of performance measurement. This data collection is being overseen through the ACF FaMLE Cross-Site project with data being recorded in the nFORM system (OMB no. 0970-046).

At each stage of the evaluation, the evaluation team will ensure that they do not collect information that is available elsewhere. None of the instruments will ask for information that can be reliably obtained through other sources.

A5. Involvement of Small Organizations

No small businesses that are not HMRE grantees or their partners will participate in data collection. Some of the HMRE grantees and partners may be small entities, such as community‑based organizations and schools. We will only request information required for the intended use and minimize burden by restricting the length of surveys to the minimum required, conducting interviews on-site or on the telephone at times that are convenient to respondents, and convening focus groups in a convenient location.

A6. Consequences of Less Frequent Data Collection

All of the data collection instruments for the STREAMS impact study are one-time data collections.

A7. Special Circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on December 6, 2018, Volume 83, Number 234, page 62871, and provided a sixty-day period for public comment. A copy of this notice is attached as Attachment J. During the notice and comment period, no substantive comments were received.

Consultation with Experts Outside of the Study

In developing the data collections for the STREAMS impact study, the evaluation team consulted with external experts to complement the knowledge and experience of the evaluation team (Table A.2). Consultants included program and policy experts and researchers. Collectively, these consultants have specialized knowledge in HMRE programs for youth and adults, strategies for improving the quality of program implementation, and research design and data collection methods relevant to this work.

Table A.2. STREAMS expert group

Name	Affiliation
Paul Amato	Pennsylvania State University
Thomas Bradbury	University of California-Los Angeles, Relationship Institute
Carolyn Pape Cowan	University of California-Berkeley
Philip Cowan	University of California-Berkeley
William Doherty	University of Minnesota
Ariel Kalil	University of Chicago, Behavioral Insights and Parenting Lab
Jennifer Kerpelman	Auburn University
Wendy Manning	Bowling Green State University
Allison Metz	University of North Carolina, National Implementation Research Network
Shari Miller	RTI International
Kay Reed	The Dibble Institute
Mindy Scott	Child Trends
Renee Sieving	University of Minnesota
Scott Stanley	University of Denver, Center for Marital and Family Studies
Luis Torres	University of Houston
David Wolfe	University of Toronto

A9. Incentives for Respondents

The STREAMS second follow-up round extends the STREAMS longitudinal RCT study design. The critical data quality issues facing the study team at this point in the impact evaluation are (1) maintaining high response rates to ensure sufficient statistical power to detect meaningful longer-term effects of the interventions and (2) and minimizing differential non-response to ensure non-biased estimates of effects for the full baseline study population. The team proposes the use of incentives as part of the strategy for meeting these data quality goals. For a description of additional planned efforts, please see SSB, Section B3 (Methods to Maximize Response Rates and Deal with Nonresponse).

The following incentive structure was previously approved for the STREAMS impact study: For the site serving youth in high schools, respondents receive a $15 gift card for completing the follow-up survey in school. Those who cannot be reached in school are offered a $20 gift card for completing the follow-up survey by CATI or CAWI outside of school. For sites serving adults, respondents receive a $25 gift card for completing the follow-up survey by CATI or CAWI. As indicated in Table A.3, for the first round of data collection, STREAMS has achieved the goals of (1) maintaining high response rates and (2) minimizing differential non-response with this incentive structure. To date, the response rate for the first round of follow-up data collection is 85 percent for the site serving youth in high schools and 79 percent for sites serving adults, with a difference of less than five percentage points between the treatment and control groups.

Table A.3. Follow-up survey response rates in STREAMS impact study sites

Population	Treatment	Control	Total
Youth in high schools	84%	86%	85%
Adults	80%	77%	79%

Note: Response rates reflect follow-up surveys completed through January 11, 2019. For sites serving adults, response rates reflect cases released for at least six months.

Due to its demonstrated effectiveness in the first round of follow-up data collection, ACF proposes using the same incentive structure for the second follow-up surveys as was used in the initial round of follow-up surveys in these evaluation sites. As indicated in Section A2, the purpose of the second follow-up surveys is to assess the longer-term impacts of HMRE programming in each site. Such impact estimates can be subject to non-response bias if respondents differ substantially from non-respondents, particularly if those differences are correlated with assignment to the evaluation treatment or control groups. The risk of biased impact estimates increases with lower overall survey response rates or larger differences in survey response rates between the research groups (What Works Clearinghouse 2013). Thus, if low overall response rates or large differential response rates between the research groups are observed, differences between groups on key outcomes might be the result of differences in the characteristics of survey respondents and cannot be attributed solely to the effect of the HMRE intervention (What Works Clearinghouse 2013). The proposed incentive structure is critical to reducing non-response bias for several reasons.

First, the incentive can help increase overall response rates, thus reducing the proportion of non-respondents. This is particularly important for groups who might otherwise be underrepresented in surveys, such low-income and minority populations, which resemble the participants in the HMRE programs in the STREAMS evaluation. Using incentives was useful in achieving higher response rates from low-income and minority populations (Singer et al. 1999; James and Bolstein 1990) and people with lower educational levels (Berlin et al. 1992).

Second, the incentives can help reduce differential response rates between the treatment and control groups. Participants assigned to the control group may be less motivated to participate than those assigned to the treatment group because they are not receiving the intervention. They may also feel that the surveys are not relevant to them. Singer and Kulka (2002) showed that incentives reduced differential response rates and thus potential non-response bias. Incentives can also help compensate for lack of motivation to participate among control group members (Shettle and Mooney 1999; Groves et al. 2000).

Third, incentives can help stimulate cooperation. There is some evidence that monetary incentives can stimulate cooperation in situations where there is a dearth of other factors that might induce cooperation, such as an interest in the study topic. For example, Baumgartner and Rathbun (1997) found a significant impact of incentives on response rate in the group for which the survey topic had little salience, but virtually no impact in the high-salience groups. In another study, it was reported that a $5 incentive, enclosed with a mail questionnaire, was a success in motivating less-satisfied parents in continuing their participation in a school-sponsored panel survey (Martinez-Ebers 1997).

A10. Privacy of Respondents

As specified in ACF’s contract for the STREAMS evaluation, the contractor must protect respondent privacy to the extent permitted by law and comply with all Federal and departmental regulations for private information. In response, the contracting organization, Mathematica Policy Research, has developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ personally identifiable information (PII). All STREAMS staff – at sites or on the evaluation team – with access to private data receive study-specific training on (1) limitations on disclosure; (2) safeguarding the physical work environment; and (3) storing, transmitting, and destroying data securely. These programs are documented in training manuals. Refresher training will occur annually. In addition, all staff of the contracting organization sign the Mathematica Confidentiality Agreement, complete online security awareness training when they are hired, and receive annual refresher training thereafter. Training addresses security policies and procedures found in the Mathematica Corporate Security manual. Subcontractors and consultants do not handle PII.

A study identification number is used to identify each study member. A link file associates each study identification number with the name and other identifying information of each study participant. All analysis files will contain only the identification numbers and no identifying information. When private data is transmitted, the evaluation team will encrypt the data with SecureZIP, using the option to encrypt in Federal Information Processing Standard (FIPS) mode with Advanced Encryption Standard 256-bit encryption. All data from STREAMS will also be encrypted when being stored using FIPS 140-2 compliant cryptographic modules and in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable federal and departmental regulations. The evaluation team will establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process private information. Data will be securely destroyed at the earliest opportunity. The evaluation team will ensure all project staff report any suspected or actual computer incidents immediately to Mathematica’s security incident team and Mathematica will report immediately to OPDIV Senior Information Systems Security Officer, or other designated personnel. The evaluation team will submit a plan for minimizing to the extent possible the inclusion of private information on paper records and for the protection of any paper records, field notes, or other documents that contain PII that ensures secure storage and limits on access.

Through the informed consent process, respondents are informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. All surveys administered using CATI are recorded for quality assurance. Respondents are given an opportunity to decline to be recorded and all recordings are stored on a secure server and destroyed at the end of the study.

Due to the private and sensitive nature of some information that is being collected as part of this research (see Section A11), the evaluation obtained a Certificate of Confidentiality. The Certificate of Confidentiality helps to assure participants that their information will be kept private to the fullest extent permitted by law.

ACF granted an Authorization to Operate (ATO) to the Fatherhood and Marriage Local Evaluation (FaMLE) Cross-Site Project’s nFORM system, which is the system that STREAMS is using. The HHS Chief Information Security Officer (CISO) provided approval for the ATO following a careful review of the nFORM system’s security documentation package. The CISO determined that the boundary as presented was secure and that any identified risks and weaknesses were deemed to be reasonable and acceptable. The Office of Planning, Research and Evaluation also completed a Privacy Impact Assessment (PIA) for the nFORM system to ensure that information handling conforms with applicable legal, regulatory, and policy requirements regarding privacy; determine the risks of collecting and maintaining PII; assist in identifying protections and alternative processes for handling PII to mitigate potential privacy risks; and communicate an information system’s privacy practices to the public. The PIA is available online through the Department of Health and Human Services.

Upon receiving OMB clearance for previous information collection requests, Mathematica obtained approval for information collection instruments and data collection procedures from the New England Institutional Review Board (NEIRB) and the University of Denver Health System. NEIRB oversees the study for youth sites and each adult site except the University of Denver. The University of Denver Health System’s IRB oversees study activities with the University of Denver site. NEIRB and the University of Denver Health System has renewed STREAMS’ IRB approval annually. The study team will also seek IRB approval for the new data collection upon final OMB approval of this information collection request.

A11. Sensitive Questions

The baseline and follow-up surveys for the STREAMS impact study include sensitive questions on relationship topics, such as intimate partner violence and children’s behavior and parents’ discipline practices. Several of these questions are required for ACF’s FaMLE Cross-Site data collection and were thus previously approved by OMB under 0970-0460. For STREAMS, a limited number of additional sensitive questions were added beyond those required by the FaMLE Cross-Site data collection. These additional topics are described in Table A.4. When asked to complete surveys for the STREAMS impact study, all participants are informed that their identities will be kept private and they do not have to answer questions that make them uncomfortable.

Table A.4. Justification for sensitive questions

Question topic	Justification	Youth in high schools	Adults
Involvement with the criminal justice system	Recent research suggests that a history of incarceration and involvement with the criminal justice system may be fairly common among men in the STREAMS target population (Zaveri et al. 2014; Pearson et al. 2011). Incarceration has major negative effects on child and family well-being, including reducing the financial support and other types of support adults can provide to their partners, children, and families, thus, documenting the incidence is important. Further, because some relationship education programs encourage men to become more responsible, we want to explore whether the programs had any effect on criminal involvement. Similar questions have been included in other large national studies, such as the Fragile Families and Child Wellbeing Study, the National Job Corps Study, the Building Strong Families Study, and the Parents and Children Together evaluation. In the Building Strong Families survey, nonresponse was less than 1 percent for these items (Wood et al. 2010).		X
Relationship experiences and characteristics	Several questions on participants’ relationship experiences and characteristics are required for the FaMLE Cross-Site data collection and were previously approved by OMB under 0970-0460. For STREAMS, a limited number of additional questions on relationship experiences and characteristics were added to both the surveys for youth in high schools and the surveys for adults. For youth in high schools, the additional questions ask about the respondent’s current relationship status and, for those youth currently in a relationship, about their relationship satisfaction and experiences. For adults, the additional questions ask about the number of romantic partners the respondent has had in the past year and the characteristics of the respondent’s current romantic relationship. These questions are necessary both to understand the populations being served and as potential moderators and outcomes of healthy relationship education programming.	X	X
Sexual activity	For youth in high schools, healthy relationship education programming often includes information on decision making around sexual activity. To measure the potential impact of this program component, the STREAMS surveys for youth in high schools include four questions on youth sexual activity: (1) whether the respondent has ever had sex, (2) whether the respondent was sexually active in the past three months, (3) whether the respondent had sex without a condom in the past three months, and (4) whether the respondent had sex without any effective contraceptive method in the past three months. All of these questions derive from a recent federal evaluation of adolescent teen pregnancy prevention programs.	X
Sexual orientation	There is a growing emphasis in healthy relationship education on inclusivity with respect to sexual orientation. For the STREAMS impact study, we will ask respondents to self-identify their sexual orientation both to better understand the populations being served and as a potential moderator of relationship education impacts.	X	X
Social Security number	In evaluation sites serving adults, the respondent’s Social Security number is essential for two reasons. First, it will be used to collect important outcome data on sample members’ employment from the National Directory of New Hires. These data will be important for measuring potential employment effects in in sites that are providing a combination of relationship education and economic stability services. Second, Social Security numbers will also be used to collect information on the location of the study participant for the follow-up data collection.		X

A12. Estimation of Information Collection Burden

Table A.5 shows the previously approved burden for instruments that are no longer in use. Table A.6 summarizes the estimated reporting burden and costs for previously approved information collections that are ongoing in addition to two newly requested data collection instruments associated with the impact study. For information about previously approved burden, see the previously approved information collection requests: https://www.reginfo.gov/public/do/PRAViewICR?ref_nbr=201806-0970-006

Previously Approved Study Instruments with Burden Completed

Table A.5. Previously approved burden; data collection complete

Instrument	Total number of respondents	Annual number of respondents	Number of responses per respondent	Average burden hours per response	Estimated Annual burden hours	Average hourly wage	Total annual cost
Topic guide for staff and stakeholder interviews (Instrument 1)	150	50	1	1	50	$27.86	$1,393
Focus group guide for adults (Instrument 2)	120	40	1	1.5	60	$7.25	$435
Focus group guide for youth in schools (Instrument 3)	60	20	1	1.5	30	$7.25	$218
Staff survey (Instrument 4)	120	40	1	.5	20	$27.86	$557
Session adherence form (Instrument 5)	48	48	104	.08	399	$27.86	$11,116
Baseline survey for youth (Instrument 8	3,600	1,200	1	.5	600	$7.25	$4,350

Previously Approved Study Instruments with Burden that Remains

• Introductory script, grantee staff (Instrument 6). In sites serving adults, grantee staff introduce the evaluation to program applicants at study enrollment. Eight program staff will provide information about the HMRE program and the STREAMS evaluation to 600 more applicants. Each staff member involved with sample enrollment will conduct a total of about 75 of these meetings over the three year period, or 25 each year. During these meetings, program staff explain program services and the fact that the applicant will be randomly assigned to be eligible or not eligible for services; the meetings last 0.08 minutes. The total annualized burden for grantee site staff to work with remaining program applicants is 16 hours (8 staff members * 25 meetings * 0.08 hours).

• Introductory script, program applicants (Instrument 6). During the three year clearance period, we expect 600 more program applicants to participate in meetings with program staff to deliver the introductory script for sample enrollment. Each meeting lasts approximately 0.08 hours. Thus, the total burden for remaining program applicants is 16 hours (200 applicants per year * 0.08 hours) per year.

• Add-on to nFORM to conduct random assignment (Instrument 7). During the three year clearance period, we expect approximately 600 more adults to enroll in the study sample. This burden is based on the number of computer entries grantee site staff will make as they enroll remaining study participants. We estimate that eight program staff will conduct these entries for a total of 75 entries per staff member over three years, or 25 entries per staff member per year—each taking 0.08 hours to complete. Therefore, the total burden is 16 hours (8 staff member *25 entries per year * 0.08 hours) per year.

• Follow-up survey for youth (Instrument 9). During the three year clearance period, we expect 690 more youth to complete the follow-up survey. We expect each follow-up survey to last 0.5 hours. We estimate that the total burden for the remaining youth follow-up survey to be 115 hours (0.5 hours * 230 youth) per year.

• Baseline survey for adults (Instrument 10). During the three year clearance period, we anticipate that we will collect baseline survey data from 600 more adults. We expect each baseline survey to last 0.5 hours. We estimate that the total burden for the remaining adult baseline to be 100 hours (200 adults * 0.5 hours) per year.

• Follow-up survey for adults (Instrument 11). During the three year clearance period, we anticipate 2,300 more adults will complete the follow up survey. We estimate that the total burden for the remaining adult follow-up survey to be 575 hours (0.75 hours * 767 adults) per year.

New Instruments for Approval in Current Request

• Second follow-up survey for youth (Instrument 12). During the three year clearance period, we expect 80 percent of 1,900 youth to complete the second follow-up survey, for a total of 1,500 youth (1,900 youth * 80 percent response rate). We expect each follow-up survey to last 0.5 hours. We estimate that the total burden for the youth follow-up survey to be 250 hours (0.5 hours * 500 youth) per year.

• 7. Second follow-up survey for adults (Instrument 13). During the three year clearance period, we expect 80 percent of 1,000 adults to complete the second follow-up survey, for a total of 800 adults (1,000 adults * 80 percent response rate). We estimate that the total burden for the adult second follow-up survey to be 200 hours (0.75 hours * 267 adults) per year.

Table A.6. Total Burden – remaining (already approved) and current request

Instrument	Total number of respondents	Annual number of respondents	Number of responses per respondent	Average burden hours per response	Estimated Annual burden hours	Average hourly wage	Total annual cost
Burden that Remains for Previously Approved Information Collections
Introductory script, grantee staff (Instrument 6)	8	8	25	0.08	16	$27.86	$446
Introductory script, program applicants (Instrument 6)	600	200	1	0.08	16	$7.25	$116
Add-on to nFORM to conduct random assignment (Instrument 7)	8	8	25	0.08	16	$27.86	$446
Follow-up survey for youth (Instrument 9)	690	230	1	0.5	115	$7.25	$834
Baseline survey for adults (Instrument 10)	600	200	1	0.5	100	$7.25	$725
Follow-up survey for adults (Instrument 11)	2,300	767	1	0.75	575	$7.25	$4,169
Burden for New Information Collections
Second follow-up survey for youth (Instrument 12)	1,500	500	1	0.5	250	$7.25	$1,813
Second follow-up survey for adults (Instrument 13)	800	267	1	0.75	200	$7.25	$1,450.00
Total estimated annual burden (remaining plus new request)					1,288		$9999

Total Annual Cost

We estimate the average hourly wage for staff at the grantee organizations is the average hourly wage of “social and community service managers” taken from the U.S. Bureau of Labor Statistics, National Compensation Survey, 2010 ($27.86). We estimated the average hourly wage of program applicants based on the current federal minimum wage ($7.25).

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost for the data collection activities under this current request will be $13,696,095. This amount includes costs for (1) completing the data collection activities previously approved and (2) conducting the two additional follow-up surveys. Annual costs to the Federal government will be $4,565,365 for the proposed data collection.

A15. Change in Burden

This is an additional information collection request under OMB #0970-0481. This request is for (1) an extension of previously approved data collection and (2) new data collection.

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

For the STREAMS impact study, program impacts will be analyzed separately for each site. With a random assignment research design, unbiased impact estimates can be obtained by comparing mean outcomes for the treatment and control groups based on follow-up data alone. However, the precision of the impact estimates can be improved by estimating multi-variate regression models that control for baseline covariates, such as baseline measures of the outcome variables. Regression adjustment can also address any differences between the treatment and control groups in baseline characteristics that arise by chance or from survey nonresponse.

The empirical specification for the regression model will depend on the unit of random assignment. In sites that randomly assign individuals to the treatment or control groups, the regression model can be expressed as follows:

(1) y_i =β′x_i+λT_i+ε_i

where y_i is the outcome of interest for individual i; x_i is a vector of baseline characteristics; T_i is an indicator equal to one for individuals in the treatment group and zero for individuals in the control group; and ε_i is a random error term. The vector of baseline characteristics x_i will include demographic characteristics such as age, gender, race/ethnicity, and baseline measures of the outcomes. The parameter estimate for λ is the estimated impact of the program.

In the site serving youth in high schools, random assignment was conducted at the classroom level. In this site, the estimated regression model must account for the correlation of outcomes among individuals in the same cluster, as they will all be randomly assigned as a single unit, and each sample member cannot be considered statistically independent. To account for this dependence, the regression model used to estimate program impacts can be expressed as follows:

(2) y_is =β′x_is+λT_is+η_s +ε_is .

The general structure of the model is the same, but now y_is is the outcome measure for individual i in cluster s (and similarly for the treatment status indicator, T_is, vector of baseline characteristics, x_is and the error term ε_is). Most importantly, the error term in Equation (2) accounts for the clustering of youth within classrooms because of the inclusion of the cluster-level error term η_s—a cluster “random effect.” If this error term is excluded, the precision of the impact estimates could be overstated. As in Equation (1), the estimated impact of the program is λ.

Table A.7 displays the tentative timeline for data collection and reporting activities. Sample enrollment and baseline data collection for the impact study began in July 2016. Data collection for the first follow-up survey began in September 2017 and is expected to continue through May 2019 for youth and June 2020 for adults. The second follow-up data collection is expected to begin immediately after obtaining OMB approval, and to continue for three years. Site-specific impact reports will be released on a rolling basis as each round of follow-up data collection is completed.

Table A.7. Schedule for the STREAMS Impact Study

Activity	Timinga
Data collection
Sample enrollment and baseline surveys	July 2016 through December 2018
Follow-up survey for youth	September 2017 through May 2019
Follow-up survey for adults	September 2017 through June 2020
Second follow-up survey for youtha	September 2019 through May 2020
Second follow-up survey for adultsa	April 2019 through April 2022
Reporting
Impact study reports	October 2019 through September 2022.

^aSubject to timing of obtaining OMB approval.

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

13