Supporting Statement A

The Office of Planning, Research and Evaluation (OPRE) and the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS), are proposing a data collection activity as part of their efforts to develop recommendations for measures and measurement tools to assess healthy marriage and relationship education (HMRE) outcomes for high school age adolescents and couples in “complex” families.

For the purposes of this project, we are defining high school age adolescents as adolescents who are generally between the ages of 14 and 17. They are adolescents who are and are not in current romantic relationships. While many of the youth in our target population are in school, we also consider youth who are no longer in school or in alternative school settings. Complex families in our study refer to low-income parents with young children (under the age of five) who have no other children together (first-time parents together), but one or both partners have children from previous relationships. Parents in our priority population are in committed relationships (e.g., married, cohabiting or not living together, but in a committed romantic relationship).

The major goal of this project is to develop outcome measures that are relevant for HMRE programs that will be (1) sensitive across cultures associated with racial, ethnic, and socioeconomic characteristics, and (2) may be used in impact evaluations of future HMRE program models tailored to adolescents and complex families. As a step in developing these measures, OPRE requests permission to conduct three iterative rounds of cognitive interviews with nine individuals per target population per round for a total of up to 54 interviews. Cognitive interviews with high school age adolescents will include individuals ages 14-17 and interviews with complex families will include individuals ages 18-45. The purpose of the interviews is to help improve item wording and ensure that items are applicable to and well understood by specific target populations.

The information collected will be used for internal purposes only; it will not be released to the public. Methodological findings may be included in methodological appendices or footnotes or instrument user guides. When necessary, results will be labeled as exploratory in nature.

A.1. Necessity for the Information Collection (IC)

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval for the cognitive testing activities described in this request to support measurement development for the Health Marriage and Relationship Education: Models and Measures project. Having high-quality outcome measures will allow for a more complete and accurate assessment of the impact of HMRE programs. Such information is important in making decisions about healthy marriage policies and programs.

Study Background

Romantic relationships during adolescence are developmentally appropriate and important. Not only do they help shape other areas of subjective well-being and behavior (both positive and negative) during adolescence, they may also influence romantic relationships during adulthood (Collins, Welsh, & Furman, 2009). Unfortunately, some adolescents come into or encounter romantic relationships with negative, inappropriate, and/or sexualized views about romantic relationships and this may influence risky sexual behaviors as well as their attitudes about romantic relationships in adulthood. Others may get involved in unhealthy romantic relationships and become victims of dating violence, which puts them at risk for many negative psychosocial outcomes. In addition, some may experience unplanned parenthood before they are economically, socially, and emotionally ready to assume such a heavy responsibility. To improve adolescents’ knowledge, attitudes, expectations, and behaviors concerning healthy romantic relationships, it is important to develop and implement effective relationship quality education models for adolescents, building on the current programs that are in place for high school age students. A key component in program evaluation is using high quality measures that are appropriate for the target population and that tap into the domains that the program seeks to improve. There has been very little work on developing and testing HMRE outcome measures with diverse adolescents, however, and the information collected through the proposed cognitive interviews will provide recommendations for how to better measure these types of HMRE program outcomes for adolescents using items that have been specifically designed and tested with this target population.

The proposed IC is also necessary for improving our ability to measure and assess HMRE program outcomes for couples in complex families. An increasing proportion of children have seen their biological parents separate, have lived in a single-parent household, and have seen at least one parent form a new cohabiting or marital union that may include additional children (Brown, 2010). Together these changes mean that many children are living in complex families, which, in turn, can introduce economic and psychological stresses into a family, affect the quality and stability of couples’ relationships, and limit the availability of resources to children – including time with parents and money.

Children born to mothers with low levels of education and fewer economic resources are much more likely to experience relationship instability and family complexity than are other children. The risks and challenges faced by parents in complex families are magnified in the context of low economic resources (Adler-Baeder, Robertson, & Schramm, 2010). This combination of economic strain and complex family structure faced by many low-income parents motivates the need for more refined program models that are responsive to the challenges faced by lower-income parents with complex family and household arrangements. There is a lack of measures designed to capture dynamics that are unique to complex families, and it may be necessary to develop new or adapted outcome constructs and measures that can be expected to result from new program models for complex families.

The proposed IC is the final step in an ongoing process of identifying and developing HMRE outcome measures for high school age adolescents and complex families. We began with an extensive review of literature and evaluation evidence which informed the development of two logic models: one for each of our prioritized populations. Concurrent to this literature scan, we consulted with experts and program providers to inform the development of the logic models as well as the selection of measures. We also conducted an extensive review of existing outcome measures that may be relevant to our prioritized populations. Through the proposed IC, we will test how well the measures we have identified work among our prioritized populations (high school age adolescents and complex families). We will also test items that have been adapted or newly developed for domains in which existing measures did not exist or did not seem relevant without some revisions to the wording of the items.

We anticipate that findings will inform future work by providing a set of sound and reliable measures that will tap into multiple outcome domains that can be used for research and HMRE program evaluation. Having high-quality outcome measures is consistent with ACF’s goals because it will allow for a more complete and accurate assessment of the impact of HMRE programs. Such information is critical in making decisions about healthy marriage policies and programs.

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.

A.2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

The purpose of this information is to provide information to ACF about outcome measures for HMRE programs and their appropriateness for use with the two target populations: adolescents and parents in complex families. The study approach will include cognitive interviews with members of the target populations that tap into respondents’ understanding of items designed to measure different outcomes that HMRE programs typically seek to address, as well as their perspectives about what should or shouldn’t be included in such a measure.

The cognitive interviews will cover the general topics of adolescents’ and individuals in complex families’ understanding of and reaction to items that measure:

High School Age Adolescents	Complex Families
Relationship and Marriage Knowledge, Expectations and Attitudes	Knowledge of Healthy Complex Family Relationships
Concern for Partner Well-Being	Co-parental Relationship with Previous Partners
Multiple domains of a healthy adolescent relationship (e.g., support, shared interaction, communication, conflict)	Concern for Partner Well-Being
	Complex Family Relationship Status and Stability (including family bonding)
	Relationship and Marriage attitudes and expectations

Upon OMB approval of the proposed information collection, recruitment and data collection is expected to occur between September and December of 2014. In early 2015, the findings will be compiled in a report for internal ACF use.

Research Questions

This study will explore the following research questions:

Research Question 1: How do newly developed measures of knowledge, perceptions and expectations for healthy complex family relationships, concern for partner’s well-being, complex family relationship status and stability, and commitment to a stable lifestyle work for low-income parents in complex families?
Research Question 2: How well do established measures of co-parenting work for low-income parents in complex families with multiple co-parental relationships?
Research Question 3: How well do existing measures of healthy relationships developed for adults work for high school age adolescents?
Research Question 4: How do newly developed measures of relationship and marriage knowledge, attitudes, and expectations; concern for partner’s well-being; and dimensions of healthy relationships work for high school age youth?

Study Design

Plans for IC include a total of three iterative rounds of cognitive interviews. Cognitive interviews offer an ideal vehicle for measurement development because they can identify problems with item wording and questionnaire design and provide information about understanding respondents’ information retrieval and response formation (Willis, 2005). Cognitive interviews in this project will ensure that the tested items are clear, easily understood, and interpreted the way they are intended. They will determine ease of comprehension, recall of information, response formation, and other issues affecting the accuracy of the information collected in the surveys, such as formatting issues (e.g., skipping patterns are confusing), instructions, and flow of the scales. Additionally, cognitive interviews will provide information about the applicability to racially, ethnically and economically diverse populations.

The three iterative rounds will be conducted with up to 54 English speaking individuals, including a maximum of 27 high school aged (14-17) adolescents and a maximum of 27 mothers and fathers who fall under our definition of a “complex family.” Experienced interviewers will conduct all cognitive interviews. Respondents will be recruited in several ways including: (1) from websites (e.g., Craigslist) (Attachments I and J) and (2) from study fliers posted in partnership organizations in the DC metro area with whom Child Trends has existing relationships¹ (Attachments K and L). Internet ads will include information about the study including a phone number that participants can call to learn more about the study and participate in a brief screener interview to determine eligibility (Attachments O and P). Screener interviews will provide a brief overview of the study and will be used to ask the potential participant a series of questions to determine eligibility. In addition, there is also a parent screener for minors that need parent/guardian’s consent (Attachment P). This will occur after the teen is screened.

Following the screener interview, participants will have the option to participate immediately or schedule an interview for the near future. Participants recruited on-the-ground will receive a flyer (Attachments K and L) that contains similar information as the web-based advertisement. These participants will have the option to provide contact information on individual sign-up sheets so that a study team member can call them at a later time (Attachment N); call a study phone number provided on the flyer to participate in the screener interview at a later time (Attachment O and P); or participate in the screener interview on-site. Once a teen is determined to be eligible for the study, we will also request that the teen provide us their mailing address, and their parent/guardian’s phone number and mailing address (if different and we do not already have it) so that verbal parental consent can be obtained. Minors cannot participate in the interview until parental consent is given.

After verifying eligibility, participants will receive a letter introducing them to the study (Attachment U and V), a consent or assent form (Attachment F, G, and H), and the cognitive interview materials (Attachment A for parents in complex families, Attachment C for teens). They will receive these materials by email whenever possible and if needed, by post. Participants will receive a reminder call the day before their interview and a reminder text the morning of their interview (Attachment E). Parents of adolescents will also receive a letter introducing them to the study (Attachment V; this includes information about the time and date of their child’s interview), a consent form (Attachment G), and a copy of their child’s assent form. (Attachment H).

The survey instruments and the cognitive interview protocols for each target population are presented in Attachments A-D.

Universe of Data Collection Efforts

The research approach calls for the development of customized interview protocols for adolescents and parents in complex families. The instruments to be used for collecting data and the research questions (RQ) the data address are as follows:

HMRE Cognitive Interview Guide for Parents in Complex Families (RQ 1, RQ 2): To collect information from parents in complex families about their understanding of measures used to capture knowledge, perceptions, and expectations for healthy complex family relationships; co-parental relationships with previous partners; concern for their partner’s well-being; complex family relationship status and stability; and relationship and marriage attitudes and expectations (Attachment B).
HMRE Cognitive Interview Guide for High School Age Youth (RQ 3, RQ 4): To collect information from adolescents about their understanding of measures used to capture relationship and marriage knowledge, attitudes, and expectations; concern for their partner’s well-being; and dimensions of healthy relationships (Attachment D).

A.3. Improved Information Technology to Reduce Burden

Whenever possible, advanced technology will be used to collect and process data to reduce respondent burden and make data processing and reporting more timely and efficient. A digital audio recorder will be used in all cognitive interviews. Before using the audio recorder, participants’ verbal consent, or assent and parental consent when appropriate, will be obtained. To reduce participant travel burden and to get a geographically diverse sample, we will conduct most of the interviews by phone. In some instances when we are recruiting participants on-the-ground locally in Washington, DC we may conduct in-person interviews if it is easiest for the participant to be screened and interviewed on the spot. We will send participants a reminder via email and/or text with the date and time of the cognitive interview; unless they indicate a preference to receive this information via airmail (see Attachment E).

A.4. Efforts to Identify Duplication

Every effort has been made to determine whether similar measurement tools exist by searching various databases (e.g., national and scholarly), reviewing existing measures, and consulting with experts in the field. As we reviewed the extant literature, we found a number of measures used to assess the impacts of HMRE programs for teens and adults; however, many of these measures have not been directly used and tested for our populations and there were gaps in the measures used. For example, there are a number of validated measures that have been used to assess the effects of HMRE programs for adults, but these measures have not been tested for use with adolescents. Additionally, there are a number of measures that have been used for adults in more traditional families but not specifically for parents in complex families. Furthermore, we have identified additional outcome domains and developed new measures that have never been tested. We have also consulted with experts in the HMRE field and they concur that the field lacks appropriate and psychometrically sound measures for adolescents and parents in complex families that are applicable for use in program evaluation. We concluded that no existing data source can provide the data needed to answer the study’s research questions.

A.5. Involvement of Small Organizations

No information will be collected from small businesses or other small entities.

A.6. Consequences of Less Frequent Information Collection

This is a one-time data collection.

A.7. Special Circumstances

There are no special circumstances for this data collection.

A.8. Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13 and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995)), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on June 10, 2011, Volume 76, page 34078, and provided a 60-day period for public comment. During the notice and comment period, the government did not receive any comments in response to the Federal Register notice.

Consultation with Experts Outside of the Study

This project has benefited from consultation with an expert work group comprised of the experts listed below. They were asked to suggest existing high-quality outcome measures and recommendations for domains in which development of new measures is needed. Their feedback was incorporated into the cognitive testing plan.

Expert Workgroup Member	Affiliation
Francesca Adler-Baeder, Ph.D., CFLE	Director, Center for Children, Youth, and Families Professor and Extension Specialist, Human Development and Family Studies Auburn University
Alan Hawkins, Ph.D.	Professor, Department of Family Life College of Family, Home and Social Sciences Brigham Young University
Jennifer Kerpelman, Ph.D.	Jennifer Kerpelman Professor and Extension Specialist, Human Development and Family Studies Auburn University
Linda Malone-Colon, Ph.D.	Chair of Psychology Department Professor, School of Education and Human Development Hampton University Former Executive Director of the National Healthy Marriage Resource Center
Anupa Bir, Ph.D.	Researcher, RTI International, Division of Health Services and Social Policy Research

A.9. Incentives for Respondents

All participants who complete a cognitive interview will be given $50 as a token of appreciation for their participation (see Attachments F, G, and H). This amount expresses gratitude for the respondents’ participation, without being an excessive amount that could be coercive. In addition, Child Trends has used similar amounts in past studies involving cognitive interviews. For instance, parents who participated in cognitive interviews for the Redesign of the National Household Education Survey (NHES OMB Control No. 1850-0803) received a $60 gift. Most recently, parents and childcare providers who participated in cognitive interviews for a study of family-provider relationship quality (OMB Control No. 0970-0355) received $50 as a token of appreciation of their participation. Child Trends has found that the amount of $50 helps to reduce overall recruitment costs and effort as well as facilitates the recruitment of hard-to-reach populations (e.g., racial/ethnic minorities, low-income parents, etc.), which this study is particularly interested in reaching. The gift will be provided at the beginning of the study, and respondents who withdraw during an interview will still be offered the indicated amount.

A.10. Privacy of Respondents

Prior to the start of each interview, the researchers will assure the respondents that the information provided will be kept private to the extent permitted by law. Due to respondents’ ages, the consent process will be different for the two populations. Parents in complex families will be legal adults and prior to the interview will provide verbal consent on their willingness to participate in the study. We will not record consent as this adds additional documentation of personally identifiable information to the study, thus increasing participant risk. Parents of minors will provide verbal consent prior to conducting the minor’s interview. Minors will be asked to verbally provide their assent before the start of their interview. On the consent and assent forms, individuals will be made aware of the extent to which their (child’s) privacy will be protected as part of the study (see Attachments F, G, and H). Specifically, participants/parents will be assured, verbally and on consent forms, that their (child’s) names will not be documented on final reports, that their (child’s) responses will not be shared with others outside of the study team, and that their (child’s) personally identifiable information will not be linked to their responses during the cognitive interviews. Identifiable information will only be collected prior to the start of the cognitive interview and will not be linked to data collected during the cognitive interview. In order to protect participants’ privacy, a study-specific identification code will be assigned to each participant and will be used for all study materials.

As specified in the contract, Child Trends shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. All Child Trends staff involved in data collection for this project will sign a pledge of confidentiality specific to this project (see Attachment M). Child Trends will maintain the security of the data and the privacy of participants by storing electronic data (i.e., electronic computer files, audio electronic files) in a restricted access drive. Following the completion of each cognitive interview, Child Trends project staff will transfer the audio recording over to the secure drive and delete it from the portable recorder. Hard copies of completed recruitment materials or screener interviews will be stored in locked files in locked offices at Child Trends separate from cognitive interview data files (such as interview notes). Child Trends will also implement its institutional procedures to ensure the security of data transfer. Child Trends will immediately transfer the data onto the secure drive and delete it from the e-mail files. Child Trends will obtain IRB approval for all data collection activities from Child Trends’ IRB Board.

A.11. Sensitive Questions

We will be asking questions about relationship quality, stability, communication and other factors related to romantic relationships that may be sensitive. We have determined that these are core components of many of the curricula for HMRE programs serving our priority populations. We will take measures to ensure that participants are aware of their rights in the study and their ability to skip any questions they do not wish to answer.

A.12. Estimation of Information Collection Burden

Exhibit A-1 shows estimated burden of the information collection, which will take place within a one-year period.

HMRE Cognitive Interview Guide for Parents in Complex Families: Cognitive interviews with up to 27 parents in complex families over the telephone or in person, lasting an average of 90 minutes (Attachment B).

HMRE Cognitive Interview Guide for High School Age Youth: Cognitive interviews with up to 27 adolescents over the telephone or in person, lasting an average of 90 minutes (Attachment D).

The total annual burden includes 0.5 hours to conduct the screening interview to ensure eligibility, and 1.5 hours needed to review the consent form, review the instructions, and conduct the interview. The total annualized hours for this data collection activity is estimated to be 38 hours (annualized over the 3 year clearance period of generic clearance 0970-0355).

Exhibit A-1: Total Burden Requested Under this Information Collection

Instrument	TOTAL Number of Respondents	ANNUAL Number of Respondents²	No. of Responses per Respondent	Average Burden Hour per Response	Estimated Annual Burden Hours	Avg. Hourly Wage	Total Annual Cost
HMRE Cognitive Interview Screener for Parents in Complex Families	27	9	1	.5	5	$17.36	$86.80
HMRE Cognitive Interview Guide for Parents in Complex Families	27	9	1	1.5	14	$17.36	$243.04
HMRE Cognitive Interview Screener for High School Age Youth	27	9	1	.5	5	$7.25	$36.25
HMRE Cognitive Interview Guide for High School Age Youth	27	9	1	1.5	14	$7.25	$101.50
Totals:					38		$467.59

Total Annual Cost

There is an estimated annualized cost to respondents of $467.59. Adolescents’ average hourly wage was calculated at federal minimum wage, which is $7.25 (United States Department of Labor, 2014). An average hourly wage of $17.36 was assumed for adult respondents in complex families, based on the Bureau of Labor Statistic’s report on average earnings for high school graduates (Bureau of Labor Statistics, 2014).

There will be no direct cost to the respondents other than their time to participate in the study.

A.13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A.14. Estimate of Cost to the Federal Government

The total cost for the data collection activities under this current request will be $53,087. Annual costs to the Federal government will be $53,087 for the ICR under the OMB clearance number (0970-0355). This includes direct and indirect costs of information collection.

A.15. Change in Burden

This is an individual generic information collection request submitted under pre-testing generic clearance 0970-0355.

A.16. Plan and Time Schedule for Information Collection, Tabulation, and Publication

Analysis Plan

Given the nature of the data collected, the analysis of the cognitive interviews will not be conducted using descriptive statistics. Instead, a summary document will be prepared for the agency’s internal use. A qualitative assessment of participants’ interview responses will be conducted, and summary tables of the results will be included in a report. The summary tables will outline the key findings for each item, with corresponding suggested revisions. All information collected is for internal use and will be used to inform the development of new measures that may be used in evaluations of HMRE programs serving complex families and high school age adolescents.

There are no plans for tabulating and publishing the information gathered from this process. Hypothetically, information might be included as a methodological attachment or footnote in a report containing data from a larger study to design program approaches for healthy marriage-relationship education programs.

Time Schedule and Publication

Upon OMB approval of proposed instruments, we anticipate data collection to occur between September and December of 2014 (see Table A.2). Child Trends will recruit participants in the Washington, DC metropolitan area and across other large metropolitan areas and rural areas in the United States. Approximately two to three weeks will be allotted for recruitment and data collection in each round. Two weeks in between rounds will be used for conducting data collection debriefing meetings, data analysis, and edits to the instruments as needed based on the findings from the cognitive interviews. Given this timeline, recruitment efforts for Round 1 are scheduled to begin in September, pending OMB approval. We can expect to conduct our first cognitive interview for Round 1 within a few days of commencing recruitment. Data collection for Round 1 will be completed in five weeks; Round 2 data collection will begin five weeks after the start of Round 1 and will last for four weeks. Finally, Round 3 data collection will begin four weeks after the start of Round 2 (i.e. nine weeks after Round 1), and last for two weeks.

TABLE A.2

Activity	Timeline
COGNITIVE INTERVIEWS
Round 1	Commencement*	Duration	Completion*
Recruitment and Data Collection	9/15/14	(3 weeks)	10/3/14
Debriefing Meetings and Data Analysis	10/6/14	(2 weeks)	10/17/14
Round 2	Commencement*	Duration	Completion*
Recruitment and Data Collection	10/20/14	(2 weeks)	10/31/14
Debriefing Meetings and Data Analysis	11/3/14	(2 weeks)	11/14/14
Round 3	Commencement*	Duration	Completion*
Recruitment and Data Collection	11/17/14	(2weeks)	11/28/14
Debriefing Meetings, Data Analysis, and Reporting	December 2014 – February 2014
*Please note these dates are approximate and dependent on the date of OMB approval.

A.17. Reasons Not to Display of OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

References

Adler-Baeder, F., Robertson, A., & Schramm, D. G. (2010). Conceptual framework for marriage education programs for stepfamily couples with considerations for socioeconomic context Marriage & Family Review, 46(4), 300-322.

Brown, S. L. (2010). Marriage and child well-being: Research and policy perspectives. Journal of Marriage and Family, 72(5), 1059-1077.

Bureau of Labor Statistics. (2014). Earnings and unemployment rates by educational attainment (2013), from http://www.bls.gov/emp/ep_chart_001.htm

Collins, W. A., Welsh, D. P., & Furman, W. C. (2009). Adolescent romantic relationships. Annual Review of Psychology, 60, 631-652.

United States Department of Labor. (2014). Wage and Hour Division: Minimum Wage Laws in States-January 2014, from http://www.dol.gov/whd/minwage/america.htm#Texas

Willis, G. B. (2005). Cognitive interviewing: a tool for improving questionnaire design. Thousand Oaks, California: Sage Publications, Inc.

1 For example, the Latin American Youth Center (LAYC), Mary’s Center, Healthy Families Thriving Communities Collaborative Council.

2 Annualized over the full three year approval period of the overarching generic clearance (0970-0355).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	shickman
File Modified	0000-00-00
File Created	2021-01-26

Supporting Statement A

3M OMB PACKET_Part A_CLEAN.docx

Pre-testing of Evaluation Surveys