OMB supporting statement part A for SARHM (DraFT)
Self-Regulation Training Approaches and Resources to Improve Staff Capacity for Implementing Healthy Marriage Services for Youth (SARHM)
Generic Clearence for Pre-testing of Evaluation Data Collection Activities
0970 - 0355
Supporting Statement
Part A
June 2018
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
330 C Street, SW, 4th Floor
Washington, DC 20201
The Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services seeks approval to conduct pre-tests, interviews, and focus groups with staff and program participants at five Healthy Marriage and Relationship Education (HMRE) programs funded by the Office of Family Assistance (OFA) within ACF. This information collection is being carried out as part of the Self-Regulation Training Approaches and Resources to Improve Staff Capacity for Implementing Healthy Marriage Services for Youth (SARHM) project. The project aims to increase the ability of program educators to support adolescent and young adults’ self-regulation skills in the context of HMRE programs. This information collection (IC) will involve pre-testing data collection instruments with five HMRE programs serving youth to develop and refine them for use in a possible future evaluation of the training approaches and materials developed under SARHM. Approval for this data collection is requested under ACF’s generic clearance for pre-testing (0970-0355).
Self-regulation encompasses a critical set of life skills linked to individual success across the lifespan. Although adolescence is a critical time of self-regulation skill development, most evidence-based strategies for supporting self-regulation focus on young children. Healthy marriage and relationship education (HMRE) programs for youth—with their focus on goal setting, communication skills, and interpersonal relationships—provide a natural context for supporting adolescent and young adult self-regulation skill development.
The Deficit Reduction Act of 2005 created the HMRE grant program, which authorized $150 million over five years to support program activities aimed at promoting and sustaining healthy marriages, providing relationship education services to youth, and fostering economic stability. The Claims Resolution Act of 2010 re-authorized this grant program, and three-year grants totaling $150 million were awarded in September 2011 (and subsequently extended through September 2015). In October 2015, ACF awarded five-year grants to 46 HMRE grantees, including 31 grantees serving youth. Youth-serving grantees focus on teaching skills to promote healthy relationships, including conflict resolution, problem solving, goal-setting, and communication skills, and may also integrate job readiness and financial management skills such as budgeting, resume writing, and interviewing skills.
OPRE/ACF launched the SARHM project to develop training approaches and materials to enhance program educators’ ability to support adolescent and young adult self-regulation skill development in the context of HMRE programs. SARHM’s training for HMRE educators, covered during 8 to 10 hours of classroom training, includes two main components: knowledge development and strategies to promote youths’ self-regulation skill development. The knowledge development component will focus on understanding self-regulation, adolescent development, and the role of adults in supporting self-regulation skill development. The training will emphasize the importance of self-regulation as a building block for healthy relationships, successful employment, and academic achievement as well as overall lifetime health and well-being. The strategies component will focus on teaching educators to use evidence-informed strategies to support youth self-regulation in the context of HMRE programs. The training will cover strategies in three domains: (1) developing warm, responsive relationships with youth, (2) structuring the environment to support self-regulation, and (3) teaching and coaching youth on self-regulation skills.
SARHM will use an iterative process to develop the training approaches and materials and pre-test them with HMRE programs serving youth. This data collection effort will pre-test data collection instruments to develop and refine them for use in a possible future evaluation of the training approaches and materials developed under SARHM. ACF contracted with Public Strategies and Mathematica Policy Research to conduct the SARHM project.
This is a discretionary data collection authorized under Sec. 811 (b) Healthy Marriage Promotion and Promoting Responsible Fatherhood Grants of the Claims Resolution Act of 2010, Pub. L. No. 111-291, 124 Stat. 3064 (Dec. 8, 2010). A copy of the legislative authority is included as Attachment A.
The SARHM team will develop and pre-test a series of data collection instruments and procedures for use in a potential future evaluation of training approaches and materials developed as part of the project. The purpose of the pre-tests is to evaluate and improve the quality of the data collected using the instruments. ACF/OPRE will use the results internally to inform subsequent information collection requests. The results of these pre-tests may be used in reports on instrument development or instrument user guides. The data collected will not be presented as findings on implementation or effectiveness of the training approaches and materials.
Pre-tests will be conducted with purposive samples of ACF-funded HMRE program staff and youth participants ages 14-24. All data collection activities conducted under this generic clearance will be voluntary and low burden.
ACF will use three procedures to develop and pre-test a set of data collection instruments under this clearance (see below for more information about specific instruments included with this request):
Exploratory interviews: Semi-structured interviews will be conducted with HMRE program staff about their experiences with the training approaches and materials at three time points during the project. The first set of interviews will be conducted in-person, and two subsequent sets of interviews will be conducted by telephone. The results of these interviews will be used to refine semi-structured interview protocols and possibly to develop new survey instruments or observation tools for use in a subsequent evaluation.
Focus groups: Focus groups with youth will be conducted to explore their experiences in the HMRE programs and satisfaction with the program educators who received training. Youth will also provide feedback on a questionnaire completed prior to the focus group. The results will be used to refine the questionnaire and focus group protocol and possibly to develop a new survey instrument for use in a subsequent evaluation.
Pre-tests: Pre-tests of preliminary versions of an educator questionnaire, a youth questionnaire, a session assessment form, and a group session observation tool will be conducted. The results of these pre-tests will be used to refine questionnaire items and scales to prepare instruments for use in a possible subsequent evaluation.
Data collection instruments will undergo two rounds of pre-testing with five HMRE programs. In the first round, instruments will be pre-tested with staff and participants at two HMRE programs. In the second round, a subset of instruments will be pre-tested with staff at three different HMRE programs. If this iterative pre-testing results in changes to any of the instruments, ACF will upload revised materials to ROCIS as a non-substantive change between each round of testing and will provide a memo describing the rationale for the changes.
Clearance is being requested to pre-test the following instruments as part of SARHM:
Educator questionnaire (Instrument 1). The purpose of this questionnaire is to assess educator knowledge of self-regulation and strategies to support youth self-regulation. Educators will also report on their own use of these skills.
Semi-structured interview protocol (Instrument 2). The purpose of the semi-structured interviews is to document manager and educator experiences and perspectives about application of the training approaches during program activities.
Session assessment form (Instrument 3). The purpose of these forms, to be completed by educators after group workshops and individual client meetings with youth, is to collect information about educator self-reported use of behaviors taught in the training to support youth self-regulation, including frequency and ease of use. Educators will also be asked to identify areas in which they need additional training.
Group session observation form (Instrument 4). The purpose of this observation tool is to collect information from a trained observer about educators’ use of behaviors taught in the training to support youth self-regulation and the level of youth engagement during the group session. The observer will also document any disruptions that occur during the session, including the type and length of the disruption and the educators’ response.
Youth questionnaire (Instrument 5). The purpose of this questionnaire is to obtain information about youths’ knowledge and use of self-regulation skills and how educators’ behaviors and the program climate support youth self-regulation. This questionnaire will be administered to youth participating in focus groups before the focus groups begin.
Youth focus group protocol (Instrument 6). The purpose of this youth focus group protocol is to obtain information about youth perceptions of their own knowledge and skill gains during the program, including healthy relationship and self-regulation skills. The study team will also ask about their satisfaction with the relationship education services and their interactions with program educators. Questions are designed to elicit youth feedback about educators’ use of strategies to support youth self-regulation.
Semi-structured interviews and focus groups will be conducted by two study team members. One team member will lead the interviews and focus groups by asking questions and the second team member will take detailed notes on a laptop capturing verbatim key quotes and responses when possible. Site visit teams will audio record the interviews and focus groups with permission from respondents to later confirm direct quotes or other details from the interviews and focus groups. Semi-structured interviews will be conducted at a time convenient to staff and conducted by phone if needed to accommodate their schedules.
For ease of completion, in the first round of pre-testing, the session assessment form and group session observation form will be available to staff as online forms created using Survey Monkey. In the second round of pre-testing, the session assessment form, educator questionnaire, and group session observation form will all be available online using Survey Monkey. We will provide a link via email that HMRE program staff can use to access and complete the forms using a tablet or laptop. Survey Monkey is appropriate for this data collection effort because we are not collecting any personally identifiable information (PII) or sensitive information. All online forms will also be available as paper-and-pencil forms, in case data entry via a tablet or laptop is not feasible. The youth questionnaire will be completed via paper-and-pencil and collected by a project team member immediately upon completion.
The SARHM team is not collecting any information that is available elsewhere. None of the instruments ask for information that can be reliably obtained through other sources.
The HMRE grantees participating in the study are community-based organizations. The SARHM team will only request information required for the intended use. We will minimize burden by restricting the length of surveys to the minimum required, conducting interviews on-site or by telephone at times that are convenient to respondents, and convening focus groups at convenient locations and times.
The purpose of this data collection is to pre-test a set of data collection instruments for use in a possible future evaluation of HMRE program staff training approaches and materials. Iterative pre-testing with small groups of staff and youth is needed to test and refine each data collection instrument. Collecting the data less frequently would prevent the study team from being able to adequately test each instrument with small groups of staff from different HMRE programs serving youth in different program settings, and with youth with different characteristics.
There are no special circumstances for the proposed data collection.
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13 and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995)), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching pre-testing generic information collection activities. This notice was published on October 20, 2017; Volume 82, Number 202, page 48820, and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received.
The SARHM project team consulted with external experts to complement the knowledge and experience of the project team (Table A.1). Collectively, these experts have specialized knowledge in HMRE programming for youth, staff training approaches, self-regulation interventions for youth, measurement of self- and co-regulation, and study design and data collection methods relevant to this work.
Table A.1 SARHM expert group
Name |
Affiliation |
Joseph Allen |
University of Virginia |
Marc Brackett |
Yale University |
Joshua Brown |
Fordham University |
Ronald B. Cox |
Oklahoma State University |
Carolyn Rich Curtis |
Relationship Skills Center |
Abigail Gewirtz |
University of Minnesota |
Mark Greenberg |
Pennsylvania State University |
James Mazza |
University of Washington |
Velma McBride Murray |
Vanderbilt University |
David Osher |
American Institutes of Research |
Kay Reed |
The Dibble Institute |
Galena Rhodes |
University of Denver |
Emilie Smith |
University of Georgia |
The study team proposes to provide youth focus group participants with $25 gift cards as an incentive to their participation in the focus groups and completion of the questionnaire. The focus group is expected to take about 1 hour and 15 minutes and the questionnaire is expected to take about 15 minutes, for a total of 1.5 hours. These gift cards are important for this data collection because the youth participating in pre-testing are underrepresented and high-risk youth living in high poverty neighborhoods and young adults aging out of foster care, some of whom are teenage parents. These youth are overwhelmingly disconnected, highly mobile, and hard to engage. We make this proposal to cover incidental expenses, such as transportation and child care, and to increase the likelihood of participation. Although we will aim to recruit approximately 11 youth to obtain 8 focus group participants, accounting for no shows, incentives will still be needed to cover youths’ incidental expenses, especially for parenting teens who may need child care. Moreover, if we only recruit youth who are able and willing to participate without an incentive, we will likely recruit youth who face fewer barriers to participation and who have higher level of interest in the HMRE program content. For this pilot, we aim to recruit a cross-section of youth with a range of needs and levels of interest in the HMRE programming.
Research has shown that incentives are effective at increasing response rates for populations similar to the underrepresented and high-risk youth and young adult populations targeted for participation in this data collection. For example, Hinojosa et al. (2014) found that recruiting low-income, racially and ethnically diverse adolescents for focus groups is challenging due to barriers such as mistrust, constraints on time, and transportation. The study found that to recruit a sample high-risk youth evenly distributed by race and gender cost $264 per youth, including a $25 gift card incentive to reduce barriers such as transportation and encourage participation.
No incentives will be provided to any other groups of respondents included in this data collection.
Information collected will be kept private to the extent permitted by law. Only staff from Mathematica will handle data collected under this clearance. Public Strategies staff will not be involved in data collection or analysis. All Mathematica staff involved in the project will receive training on (1) limitations of disclosure; (2) safeguarding the physical work environment; and (3) storing, transmitting, and destroying data securely. All Mathematica staff sign the Mathematica Confidentiality Agreement (see Attachment B), complete online security awareness training when they are hired, and receive annual refresher training thereafter. Training addresses security policies and procedures found in the Mathematica Corporate Security Manual.
The instruments to be pre-tested will not collect any PII. To further protect the privacy of the participants, included at the beginning of each questionnaire is a reminder for respondents to not include any personal information such as names and contact information in their responses. A study identification number will be used to identify each HMRE program staff member participating in the study. Staff will provide this identification number when they complete the educator questionnaire, the session assessment form, and the group session observation form. All analysis files will contain only the study identification numbers and no identifying information.
There are no sensitive questions in this data collection.
Table A.2 summarizes the estimated reporting burden and costs for each of the instruments to be pre-tested under this information collection request. Figures are estimated as follows:
Pre-Test Round 1
Educator questionnaire. We expect to administer this questionnaire with 20 HMRE program educators (10 educators per site * 2 sites) for up to 20 minutes at two time points: before they receive the training and at a mid-point in the pre-testing period. Total burden for the educator questionnaire is 13 hours.
Semi-structured interview protocol. We expect to conduct semi-structured interviews with 28 HMRE program educators and managers (14 staff per site * 2 sites) for up to 20 minutes at three points in time during the pre-testing period. Total burden for the semi-structured interviews is 28 hours.
Session assessment form. We expect 20 HMRE program educators (10 educators per site * 2 sites) to each complete 36 session assessment forms. Each form will take 10 minutes to complete. Total burden for the session assessment form is 120 hours.
Group session observation form. We expect 1 HMRE program supervisor in each of the HMRE programs to conduct 3 observations of each of the 10 program educators (1 supervisor * 10 educators * 3 observations * 2 sites). Each observation will take 1 hour to complete. Total burden for the group session observation measure is 60 hours.
Youth questionnaire. We expect to administer this questionnaire with up to 32 youth program participants (2 focus groups * 8 participants * 2 HMRE programs). We expect each questionnaire to take 15 minutes to complete. Total burden for the youth questionnaire is 8 hours.
Youth focus group protocol. We expect to conduct focus groups with up to 32 youth program participants (2 focus groups * 8 participants * 2 HMRE programs). We expect each focus group to last up to 75 minutes. Total burden for the youth focus groups is 40 hours.
Pre-Test Round 2
Educator questionnaire. We expect to administer this questionnaire with 30 HMRE program educators (10 educators per site * 3 sites) for up to 20 minutes at one time point during the field testing period. Total burden for the educator questionnaire is 10 hours.
Session assessment form. We expect 30 HMRE program educators (10 educators per site * 3 sites) to each complete 12 session assessment forms. Each form will take 10 minutes to complete. Total burden for the session assessment form is 58 hours.
Group session observation form. We expect 1 HMRE program supervisor in each of the HMRE programs to conduct 3 observations of each of the 10 program educators (1 supervisors * 10 educators * 3 observations * 3 sites). Each observation will take 1 hour to complete. Total burden for the group session observation measure is 90 hours.
We estimate the average hourly wage for staff at the HMRE programs is the average hourly wage of “social and service managers” taken from the U.S. Bureau of Labor Statistics, National Compensation Survey, 2010 ($27.86). We estimated the average hourly wage of youth based on the current federal minimum wage ($7.25).
Table A.2 Total burden requested under this information collection
Instrument |
Total annual number of respondents |
Number of responses per respondent |
Average burden hours per response |
Annual burden hours |
Average hourly wage |
Total annual cost |
||
Pre-Test 1 |
||||||||
1. Educator questionnaire |
20 |
2 |
.33 |
13 |
$27.86 |
$362.18 |
||
2. Semi-structured interview protocol |
28 |
3 |
.33 |
28 |
$27.86 |
$780.08 |
||
3. Session assessment form |
20 |
36 |
.167 |
120 |
$27.86 |
$3,343.20 |
||
4. Group session observation form |
2 |
30 |
1 |
60 |
$27.86 |
$1,671.60 |
||
5. Youth questionnaire |
32 |
1 |
.25 |
8 |
$7.25 |
$58.00 |
||
6. Youth focus group protocol |
32 |
1 |
1.25 |
40 |
$7.25 |
$290.00 |
||
Pre-Test 2 |
||||||||
1. Educator questionnaire |
30 |
1 |
.33 |
10 |
$27.86 |
$278.60 |
||
2. Session assessment form |
30 |
12 |
.16 |
58 |
$27.86 |
$1,615.88 |
||
3. Group session observation form |
3 |
30 |
1 |
90 |
$27.86 |
$2,507.40 |
||
397 |
|
$10,906.94 |
||||||
There are no additional costs to respondents.
The cost for data collection under this current request will be $616,988.
This is an information collection request under generic clearance 0970-0355.
The purpose of this clearance request is for pre-testing a set of data collection instruments and procedures to evaluate and improve their quality for use in a potential future evaluation of training approaches and materials developed as part of the project. Results from the exploratory interviews and focus groups may be used to develop new survey instruments for use in a possible subsequent rigorous evaluation.
The project team will use standard qualitative procedures to analyze and summarize information from semi-structured interviews and focus groups. Analysis will involve coding, triangulation, and theme identification. The project team will summarize quantitative data from the education self-assessment questionnaire, the session assessment form, the group session observation measure, and the youth questionnaire using basic descriptive methods. The results will be used to assess the suitability of the data collection instruments and procedures for studying the implementation and effectiveness of the training materials.
The information collected under this clearance will not be the primary subject of any published ACF reports; however information may be made public through methodological appendices or footnotes, reports on instrument development, or instrument user guides. When necessary, results will be labeled as exploratory in nature.
The pre-testing will begin in July 2018, after obtaining OMB approval, and continue through May 2019. Pre-test results will be reported in internal memos to OPRE during the study period and in a report to OPRE in August 2019.
All instruments will display the expiration date for OMB approval.
No exceptions are necessary for this information collection.
Hinojosa, Melanie Sberna, Hajar Kadivar, Daniel Fernandez-Baca, TaJuana Chisholm, Lindsay A. Thompson, Jevetta Stanford, and Elizabeth Shenkman. “Recruiting Low Income and Racially/Ethnically Diverse Adolescents for Focus Groups.” Journal of Maternal and Child Health, Volume 18, 2014, pp. 1912-1918.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Diane Paulsell |
| File Modified | 0000-00-00 |
| File Created | 2021-01-21 |