Alternative Supporting Statement for Information Collections Designed for Research, Public Health Surveillance, and Program Evaluation Purposes
Success Sequence Interviews – Pre-Test
Pre-testing of Evaluation Data Collection Activities
0970 - 0355
Supporting Statement
Part B
September 2021
Submitted by:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officer:
Caryn Blitz
The objective of this proposed pre-test is to ensure that data collection instruments for the Success Sequence Interview study are appropriate for a diverse audience and can elicit a diverse range of responses. The aim of the pre-test is to identify, through participant feedback, changes to the wording or ordering of questions in the interview topic guide that will enable future participants to more accurately comprehend the questions and respond in depth, improving the overall quality of the interview data. ACF will use the pre-test to refine the Success Sequence Interview as part of the finalization of the instruments to be submitted to the Office of Management and Budget (OMB) as a new full information collection request (ICR) for the Success Sequence Interviews study.
The pre-test is intended to produce descriptive findings about the reliability and validity of data collection items for the target population to inform instrument development, not to promote statistical generalization to other sites or service populations.
As noted in Part A of the Supporting Statement, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.
The pre-test is designed to collect information efficiently from a diverse sample of participants. To adequately test the success sequence milestones requires that pre-test participants have a range of life experiences, including variation in education, employment, income, marriage, and family. Participants will be selected to vary by race, ethnicity, gender, and US geographic location. Such variation is critical to the purpose of the pre-test. Economic analyses conducted as part of this study found variation in the sequence of the milestones completed.1 Variation in the most common milestones occurred based on gender, race, ethnicity, and parental education (see Part A of the Supporting Statement, Section A2). Therefore, it is important that pre-test participants reflect differences in these characteristics, as well as in their life milestones.
Pre-testing the interview protocol and related screener will allow us to capture the wide range of demographics and life experiences needed to test all parts of the instrument. The pre-test is also designed to identify any sensitivity issues for specific groups. Table B.1 shows the specific information on targeted milestones and demographic variation that this proposed pre-test seeks to capture. Working with a market vendor, the Schlesinger Group, who has access to a wide range of participants, allows for efficient recruitment of a diverse testing pool.
As discussed in Section B2, the pre-test design is iterative. The iterative nature will enable the research team to test modifications made to the instrument on the basis of initial feedback and to ensure that variation in participant milestones is achieved. This approach will result in an instrument tested across participants with a variety of experiences related to the milestones under study. The pre-test administration will mirror that of the future Success Sequence Interviews study; interviews will be conducted virtually using QualBoard, an online platform that provides anonymity and is convenient for participants. QualBoard allows for more flexible participation because participants can complete the pre-test at any time during a given data collection window.
The target population for the pre-test consists of English- and Spanish-speaking adults ages 30 to 35. The Success Sequence Economic Analysis report, discussed in Section A2 of the Supporting Statement, indicates that individuals fall into 64 milestone sequence pathways. Due to the complexity and variation in pathways, it is necessary to pre-test the instrument across a broad range of participants. Such a range will enable the research team to gauge comprehension of the questions and garner feedback on the flow of the various pathways in the instrument and administration of the interviews.
We propose a two-stage pre-test, described in Section A2 of the Supporting Statement (Study Design). Twenty participants will be recruited for the first round. The instrument will then be revised on the basis of feedback from the initial 20 participants, and then it will be administered to an additional 20 participants to test the revisions. All participants in the pre-test study will be recruited in accordance with specific targets noted in Table B.1.
The research team at Mathematica will work with a market research vendor, the Schlesinger group, to recruit and screen 120 participants by telephone, using Instrument 1, the Success Sequence Pre-test Screener. Questions in the screener will identify success sequence milestone groups, as shown in Table B.1, to ensure a diverse pool of participants. The team estimates that completion of the screener with 120 participants will result in 60 participants who meet the eligibility criteria for age range and primary language (English or Spanish). The Schlesinger Group will recruit an initial 30 for round 1 and then another 30 for round 20. Of the 60 total eligible participants, the team estimates that 40 participants will complete the pre-test interview. The research team at Mathematica will use non-probability sampling to identify pre-test participants who meet a range of criteria in such areas as educational attainment, income, marriage, children, demographics, and geographic location. A primary goal of the proposed pre-test is to ensure the interview questions are tested by individuals from a wide range of demographic characteristics and milestone variations so that revisions to the interview instrument and administration protocol reflect the diversity of participant experiences. The research team will closely monitor the pre-test sample to ensure balance is achieved across the demographics and milestone variables.
Table B.1. Pre-test participant targets, by demographic and success sequence milestones
Success sequence milestone group |
Pre-test
Round 1 target |
Pre-test
Round 2 target |
Income (low, medium/high) |
4 |
4 |
Educational attainment (no degree, alternative high school credentials and/or an associate degree or higher) |
4 |
4 |
Employment (unemployed/employed) |
4 |
4 |
Marriage (married, never married, divorced/widowed, separated, cohabitating) |
4 |
4 |
Children (does not have children and not planning to have children in the future, has children and not married) |
4 |
4 |
a In Round 1, we plan for up to four participants in each milestone group to test variation in income, education, marital status, and presence of children. Within each group, we will seek variation in participant geographic region, urbanicity, race, ethnicity, and gender.
b In Round 2, we plan for up to four participants in each milestone group, focusing on areas most in need of testing and variation in participant geographic region, urbanicity, race, ethnicity, and gender.
Mathematica staff, in collaboration with ACF, developed the Success Sequence Screener and Pre-test Interview and Debriefing Protocol (Instruments 1 and 2) on the basis of findings from the literature review and economic analyses described in Section A1 of the Supporting Statement.2,3
The screener includes questions on each of the success sequence milestones, income level, and demographics to ensure variation in participant background, geographic location, and economic self-sufficiency. Items in the screener on educational attainment and income are based on those used in the National Longitudinal Survey of Youth. The data collection instrument (Instrument 2) starts with a short set of closed-ended questions at the beginning of the interview, which allows for more detailed follow-up questions tailored for individual participants on the basis of relevant milestones in their lives. Both instruments (Instruments 1 and 2) have been translated into Spanish to ensure that Spanish speakers are recruited and included in the proposed pre-test study.
ACF has contracted with Mathematica for this pre-test data collection. The research team at Mathematica has extensive experience with pre-testing data collection instruments with young adult populations in earlier studies sponsored by the U.S. Department of Health and Human Services and other federal agencies.4 As mentioned in Section B2, the Schlesinger Group will recruit participants from their standing panel using Instrument 1. The market research vendor will begin recruitment of pre-test participants by sending a prescreen email in both English and Spanish to its panel members (Appendix A: Success Sequence Pre-test Recruitment Materials: English and Spanish). Interested participants will be screened by telephone, using Instrument 1. Mathematica will monitor and select the participants in each target group for each pre-test round, described in Table B.1, recruiting up to 20 total in the first round, of which five will be Spanish speakers. Following the first round, the research team will assess the completed interviews by the participant targets, after which round 2 of recruitment will begin. The research team will again work closely with the vendor, monitoring the recruited participants to ensure that targets are met, again recruiting up to a total of 20 participants, of which five will be Spanish speakers.
To conduct pre-testing, the research team will use QualBoard, an online bulletin board platform where participants access and respond to the interview questions. As the platform is not a real-time chat, participants can log on at any time convenient for them. Responses are stored as an interview transcript. The research team will email recruited pre-test participants their individual QualBoard login credentials to enable access to the data collection platform (Appendix C: QualBoard Invitation Email Sample Screen Shot).
When participants enter the QualBoard site, they will be asked to enter their first name, initials, or an alias; to protect their privacy, they will not be asked to provide their full name. The name they enter, which will be their display name in the chat board, can be seen only by the research team interviewers. An electronic consent form will be displayed upon login for completion by the participant before they begin their online interview (Appendix B: Pre-test Consent Form: English and Spanish). Participants will not be able to access any interview questions until they complete the consent form.
Interview questions will be programmed onto the chat board for pre-test participants to answer at their convenience during a given data collection window of up to three days. The interview questions are grouped into five sections: 1) education, 2) employment and work experience, 3) family life, 4) financial status, and 5) final thoughts. Participants will type their responses to the questions posed and, for clarification or to elicit more information, a trained Mathematica research team interviewer will probe as needed, while the participant is completing the interview. Participants can easily see when an interviewer has added a probe during the online interview, and they also receive an email notification. Following each section, participants will be asked for feedback regarding the section and any questions they found unclear or sensitive. At the conclusion of the interview, participants will be asked to enter how long it took them to complete the interview and to provide any additional feedback they have regarding the interview questions and their experience as a participant. All debriefing questions are included in Instrument 2.
On the basis of findings from round 1 of pre-testing, the research team will revise the instruments and conduct round 2 of the pre-test with a new group of purposively selected participants. The second round will ensure clarity in the refined items.
The pre-test interviews are not designed to produce statistically generalizable findings and participation is wholly at the participant’s discretion. Response rates will not be calculated or reported.
As pre-test participants will not be randomly sampled and findings are not intended to be representative, nonresponse bias will not be calculated. The research team will work with the vendor to purposively select a diverse group of pre-test participants to ensure all interview questions are tested across varying participant life milestones.
The research team will examine item nonresponse as an indicator of the sensitivity of interview questions. Mathematica will use this information in revisions of questions across the phases of the pre-test, with a goal of minimizing item nonresponse in the final instrument.
The purpose of this clearance request is for pre-testing data collection instruments and procedures to evaluate and improve the instruments’ quality for use in the Success Sequence Interviews study, forthcoming in a future ICR. Statistical estimation is not intended with the pre-test qualitative interview data.
Screener data will be shared by the vendor weekly during recruitment with the Mathematica research team via a secure file transfer site. The data will be shared through an advanced file exchange website that users access with individual usernames and passwords.
Interview data retrieved from QualBoard will be saved on a secure drive accessible only to Mathematica research team members. Direct export of the QualBoard data to the secure drive will result in minimal processing.
The Mathematica research team will review the qualitative data from the pre-test interviews for problematic questions and use participant feedback to inform changes to the instrument items.
The research team will use the pre-test results to revise and finalize the Success Sequence Interviews study instruments. These final instruments will be submitted for review and approval through a full ICR to OMB. Instrument revisions and an accompanying memo will be shared with ACF at the conclusion of the pre-test. If OMB would like to review revisions and receive updates between the two rounds of pre-test activities, please let us know and this information will be provided.
Table B.2 lists the federal and contract staff responsible for the study, along with each individual’s affiliation and email address.
Table B.2. Individuals Responsible for Study
Name |
Affiliation |
Email address |
Caryn Blitz |
Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services |
Caryn.Blitz@acf.hhs.gov |
Kathleen McCoy |
VPD Government Solutions Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services |
Kathleen.McCoy@acf.hhs.gov |
Susan Zief |
Mathematica |
SZief@mathematica-mpr.com |
Tiffany Waits |
Mathematica |
TWaits@mathematica-mpr.com |
Jennifer Walzer |
Mathematica |
JWalzer@mathematica-mpr.com |
Elizabeth Mugo |
Mathematica |
EMugo@mathematica-mpr.com |
Appendices
Appendix A: Success Sequence Pre-test Recruitment Materials (English and Spanish)
Appendix B: Pre-test Consent Form (English and Spanish)
Appendix C: QualBoard Invitation Email Sample Screen Shot (English and Spanish)
Instrument
Instrument 1: Success Sequence Pre-test Screener (English and Spanish)
Instrument 2: Success Sequence Pre-test Interview and Debriefing Protocol (English and Spanish)
1 Inanc, H., A. Spitzer, and B. Goesling. “Assessing the Benefits of the Success Sequence for Economic Self-Sufficiency and Family Stability.” OPRE Report #2021-XX. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
2 Goesling, B., H. Inanc, and A. Rachidi. “Success Sequence: A Synthesis of the Literature.” OPRE Report #2020-41. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2020.
3 Inanc, H., A. Spitzer, and B. Goesling. “Assessing the Benefits of the Success Sequence for Economic Self-Sufficiency and Family Stability.” OPRE Report #2021-XX. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
4 Pregnancy Assistance Fund Study (OMB Control Number 0990-0424); Federal Evaluation of Making Proud Choices (OMB Control Number 0990-0452), The Strengthening Relationship Education and Marriage Services Evaluation (OMB Control Number 0970-0481), Head Start Family and Child Experiences Survey 2019 (OMB Control Number 0970-0151), Middle Grades Longitudinal Study of 2017-18 (OMB Control Number 1850-0911), Evaluation of Demonstration Projects to End Childhood Hunger (OMB Control Number 0584-0603), School Nutrition and Meal Cost Study (OMB Control Number 0584-0596), and Regional Partnership Grants National Cross-Site Evaluation and Evaluation Technical Assistance (OMB Control Number 0970-0444).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | OPRE |
| File Modified | 0000-00-00 |
| File Created | 2024-07-25 |