SARHM: OMB supporting statement part b
Self-Regulation Training Approaches and Resources to Improve Staff Capacity for Implementing Healthy Marriage Services for Youth (SARHM)
Generic Clearence for Pre-testing of Evaluation Data Collection Activities
0970 - 0355
Supporting Statement
Part B
June 2018
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
330 C Street, SW, 4th Floor
Washington, DC 20201
The Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services seeks approval to conduct pre-tests, interviews, and focus groups with staff and program participants at five Healthy Marriage and Relationship Education (HMRE) programs funded by the Office of Family Assistance (OFA) within ACF. This information collection is being carried out as part of the Self-Regulation Training Approaches and Resources to Improve Staff Capacity for Implementing Healthy Marriage Services for Youth (SARHM) project. The project aims to increase the ability of program educators to support adolescent and young adults’ self-regulation skills in the context of HMRE programs. This information collection (IC) will involve pre-testing data collection instruments with five HMRE programs serving youth to develop and refine them for use in a possible future evaluation of the training approaches and materials developed under SARHM. Approval for this data collection is requested under ACF’s generic clearance for pre-testing (0970-0355).
In September 2015, ACF awarded five-year grants to 46 HMRE grantees; 31 of these grantees serve youth. The SARHM project will conduct pre-testing in five sites chosen from among these youth-serving grantees. Pre-tests will be conducted of the following instruments: (1) educator questionnaire, (2) semi-structured interview protocol, (3) session assessment form, (4) group session observation form, (5) youth questionnaire, and (6) youth focus group protocol.
The project team is using data available in grant applications and other program documents to identify a purposive sample of programs that serve youth, employ at least four program educators, and collectively provide services in schools and community settings. Participation in pre-testing will be voluntary.
Within the selected programs, the study team will select purposive samples of respondents to pre-test each instrument as follows:
Educator questionnaire and session assessment form. All staff who provide services directly to youth, either in group sessions or one-on-one meetings, will be asked to pre-test these instruments. We assume 50 program educators will participate in the pre-test (10 educators * 5 programs participating in the first and second rounds of pre-testing).
Group session observation form. One purposively selected supervisor of direct service staff at each program will be asked to pre-test the session observation form. We assume 5 supervisors (1 supervisor * 5 programs participating in the first and second rounds of pre-testing).
Semi-structured interview protocol. All program directors, supervisors of direct service staff, and direct service staff will be asked to participate in interviews in the first round of pre-testing. We assume 28 program staff (10 educators + 4 directors/supervisors * 2 programs participating in the first round of pre-testing).
Youth questionnaire and focus group protocol. The project team will conduct 2 focus groups with 8 purposively selected youth in 2 programs during the first round of pre-testing. The project team will randomly select youth in each site that have participated in at least 2 program activities in the month prior to the focus group. Twelve youth will be recruited for each group to account for no shows. We assume 32 focus group participants (8 youth * 2 groups * 2 programs participating in the first round of pre-testing).
The project team will conduct pre-testing during on-site program visits, telephone interviews, and online forms created using Survey Monkey. No personally identifiable information will be collected, and respondents will be reminded to exclude their name and any other identifiable information from all forms.
Pre-Test Round 1. During the first round of pre-testing, six instruments will be pre-tested in two programs. The project team will pre-test the educator questionnaire on site using a pencil-and-paper instrument. In addition, the project team will conduct in-person and telephone interviews to pre-test the semi-structured interview protocol, and the team will conduct in-person focus groups to pre-test the youth focus group protocol. Immediately prior to the start of the focus groups, the youth questionnaire will be pre-tested using a pencil-and-paper instrument. Pencil-and-paper instruments will be used at this stage for two reasons: (1) the instruments are designed to be brief and free of complicated skip patterns, and (2) the expected sample sizes do not justify the costs related to developing and maintaining an automated survey instrument. Pencil-and-paper forms will be collected from respondents immediately upon completion.
During the first round of pre-testing, two online forms will be pre-tested using Survey Monkey. Program educators will be asked to complete 36 session assessment forms at regular intervals during the pre-test period, such as daily or weekly depending on the frequency of service delivery at each program. In addition, supervisors will be asked to conduct in-person observations of 30 group sessions using the group session observation form. Supervisors will be asked to complete the observation using an online Survey Monkey form. If data entry via a tablet or laptop is not feasible, the study team will provide a paper-and-pencil version of the observation form.
Pre-Test Round 2. After completing the first round pre-testing and seeking approval from OMB for any non-substantive changes the team determines to be necessary, three online forms will be pre-tested in three additional programs: (1) the educator questionnaire, (2) the session assessment form, and (3) the group session observation form. Program educators will be asked to complete the educator questionnaire using an online Survey Monkey form in the first week of the pre-testing period. Program educators will also be asked to complete 12 session assessment forms at regular intervals during the pre-test period, such as daily or weekly depending on the frequency of service delivery. In addition, supervisors will be asked to conduct in-person observations of 30 group sessions using the group session observation form. Supervisors will be asked to complete the observation using an online Survey Monkey form. If data entry via a tablet or laptop is not feasible, the study team will provide a paper-and-pencil version of the observation form.
Expected Response Rates
The project team will work closely with ACF to purposively select HMRE programs in which staff are interested in assisting with the pre-test. We also expect the activities to be low burden. Therefore, we expect high participation in the pre-tests.
Expected response rates will vary for each instrument to be pre-tested. In the first round of pre-testing, we expect a nearly 100 percent response to the educator questionnaire because we will administer and collect a paper-and-pencil version of the form in person. In the second round of pre-testing, we expect a response rate of at least 90 percent. We also expect a nearly 100 percent response rate to the semi-structured interview protocol because we will conduct these brief interviews in person, or if that is not feasible, by telephone. We will recruit more youth than required to pre-test the youth questionnaire and focus group protocol to account for no-shows. We expect at least a 90 percent rate of completion of the expected number of group session observations and session assessment forms.
Maximizing Response Rates and Dealing with Nonresponse
The project team will be accommodating of respondents’ schedules. We will schedule on-site pre-testing activities of the educator questionnaire and semi-structured interview protocol at dates and times that are convenient for staff. The programs will be contacted by their assigned study team liaisons, which will include at a minimum one senior and one junior staff member from the project team. The senior member will have significant experience working closely with programs and their stakeholders on previous evaluations and a high level of success in engaging program staff in data collection activities.
To maximize response rates on the youth questionnaire and focus group pre-test, we propose provide youth and young adult participants with a $25 gift card. These gift cards are important for this data collection because the youth participating in pre-testing are underrepresented and high-risk youth living in high poverty neighborhoods and young adults aging out of foster care, some of whom are teenage parents. These youth are overwhelmingly disconnected, highly mobile, and hard to engage. We make this proposal to cover incidental expenses such as transportation and child care and to increase the likelihood of participation. We will also recruit approximately 11 youth to obtain 8 focus group participants, accounting for no shows.
The study team will purposively select two supervisors in each program who are interested in conducting the observations. Because many HMRE program supervisors routinely conduct group session observations, we anticipate that this activity will be viewed as low burden by program staff.
The study team will use Survey Monkey to monitor progress in completing the expected number observations and session assessment forms and will follow with program staff as needed to encourage completion of all pre-testing activities. In addition, the study team liaisons will be available to assist program staff with any difficulties they have completing the pre-test activities.
The purpose of this information collection request is to pre-test data collection instruments. ACF expects that all the tests conducted under this clearance will result in improved questionnaires and thus reduced respondent burden.
Mathematica Policy Research is conducting this pre-test under subcontract to Public Strategies for contract number HHSP23337002T. Mathematica received input and guidance on the plans for the statistical analyses for this study from ACF staff. The Mathematica project team is led by the following individuals:
Ms. Diane Paulsell |
Dr. Julia Alamillo |
In the future, further input on analytic approaches may be sought from additional Mathematica staff and from outside consultants.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Diane Paulsell |
| File Modified | 0000-00-00 |
| File Created | 2021-01-21 |