Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Runaway and Homeless Youth
Learning Agenda Support
Formative Data Collections for Program Support
0970 - 0531
Supporting Statement
Part B
August 2025
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers:
Caryn Blitz
Calonie Gray
Part B
B1. Objectives
Study Objectives
The Administration for Children and Families (ACF) contracted with the Urban Institute (contractor) to complete a multisite process evaluation of the Family and Youth Services Bureau (FYSB) Runaway and Homeless Youth Prevention Demonstration Program (RHY PDP). The evaluation aims to understand the implementation of prevention services and interventions across the 10 grant sites participating in the RHY PDP. ACF outlined the following six research areas for the evaluation: (1) coordination among site partners; (2) leading in partnership with youth and young adults with experience of homelessness; (3) community capacity expansion and enhancement; (4) program reach; (5) providers’ experiences of the RHY PDP; and (6) youth experiences of the RHY PDP. See Appendix A for the research questions and objectives associated with each of these research areas and their corresponding data collection activities.
Generalizability of Results
This study is intended to present an internally-valid description of the implementation the RHY PDP in chosen sites, not to promote statistical generalization to other sites or service populations.
Appropriateness of Study Design and Methods for Planned Uses
The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.
The main goal of the multisite process evaluation is to document how the RHY PDP is implemented across a range of settings and community contexts. This process evaluation will explore coordination among partners, strategies empowering youth to co-lead prevention plans, collaboration within communities, service delivery, providers’ experiences, and youth experiences in the RHY PDP. The evaluation data collection procedures and protocols have been approved by the Urban Institute’s Institutional Review Board (IRB).
The evaluation is intended to describe each RHY PDP project with the goal of building knowledge and evidence on the prevention of youth homelessness and informing the development of strategies, programs, policies, and practices to prevent and respond to youth homelessness. The multisite process evaluation is not intended to replace any local evaluations, nor to measure individual-level impact of the RHY PDP projects on participants. Through the evaluation, RHY PDP sites will help identify facilitators and barriers in program design and implementation and in working with young people to prevent homelessness.
To achieve the primary objectives of the study and answer the research questions, the evaluation comprises four primary activities:
Virtual quarterly site meetings with RHY PDP grantee site staff
Virtual listening sessions with RHY PDP grantees, providers, and community partners
Site visits to conduct key respondent interviews with RHY PDP grantees, providers, and community partners and conduct focus groups with young people
Document review and analysis, to further collect, distill, and organize information on project activities and progress1
There is some overlap across data collection activities and respondents; this is intentional as the contractor seeks to obtain cross-constituent input and use multiple approaches to minimize method bias and obtain a range of input and feedback. As noted, the study and the data collected are only intended to present an internally valid description of the RHY PDP implementation in chosen sites, not to promote statistical generalization to other sites or service populations. Additionally, the process evaluation does not include an impact evaluation, and the data and findings should not be used to assess RHY PDP participant outcomes. These limitations will be clearly written in the final report produced at the end of the study.
B2. Methods and Design
Target Population
The contractor will collect information from each of the 10 RHY PDP grantee staff, community partners, and young people. In particular, information will be collected from young people participating in RHY PDP programming and with members of a local Youth Advisory Board (YAB) established by the RHY PDP grantee. The YABs consist of young people who participate in their local RHY PDP and collaborate with the RHY PDP grantee and partners on designing the program’s youth homelessness prevention approach. All RHY PDP grantees are required to form a YAB. The data collection activities for the process evaluation will proceed as follows:
Quarterly site meetings (virtual, 60 minutes). The contractor will facilitate quarterly site meetings with the primary RHY PDP grantee staff contact at each site via video conference (e.g., Zoom).
Listening sessions (virtual, 90 minutes). The contractor will host two virtual listening sessions with RHY PDP grantees and a broader set of community partners during site visits (see below). The number of participants will vary based on the number of active partners within each site, but the contractor anticipates up to 10 participants in each site for a total of 100 participants. Each participant will only participate in one of the two listening sessions.
Site visits (in-person). One visit at each RHY PDP site which will include:
Eight key respondent interviews (60 minutes) with RHY PDP staff, providers, and community partners focused on RHY PDP site implementation, coordination, and provider experiences.
Two focus groups (90 minutes) with up to 160 young people addressing youth experiences with the RHY PDP co-design process and experiences within RHY PDP service receipt.
Document review and analysis. The contractor will review key project materials produced by each RHY PDP grantee and relevant partners. Contractor staff will preserve materials from meetings with the sites, including meeting agendas, notes/minutes, reports, plans, local RHY PDP evaluation reports, and other documents. Document review will enable the contractor to identify key themes and unique site experiences related to project activities and components, providing valuable insights on implementation facilitators and barriers to inform future activities. The document review does not engage participants and is not formally part of information collection.
Sampling
All 10 grantees participating in the RHY PDP are included in the multisite process evaluation. Table 1 shows the characteristics of each site. As noted, participants will not be randomly sampled and findings are not intended to be representative.
Table 1. RHY PDP Grantee Organizations
Site # |
HHS Region |
Grantee |
Geographic area |
State |
1 |
1 |
Waypoint |
Manchester |
NH |
2 |
3 |
Sasha Bruce Youthwork Inc. |
Prince George’s County |
MD |
3 |
3 |
Valley Youth House |
Philadelphia |
PA |
4 |
4 |
Pendleton Place |
Upstate South Carolina |
SC |
5 |
4 |
Oasis Center Inc. |
Metro Nashville |
TN |
6 |
4 |
Florida Network of Youth and Family Services Inc. |
South Florida |
FL |
7 |
5 |
Lighthouse Youth Services Street Outreach Program |
Hamilton County (Cincinnati) |
OH |
8 |
6 |
Seasons of Change Incorporated |
Tarrant County (Arlington) |
TX |
9 |
9 |
Our Family Services Inc. |
Pima County (Tucson) |
AZ |
10 |
10 |
Covenant House Alaska |
Anchorage |
The potential sample pool for each data collection activity is limited to individuals with direct involvement in RHY PDP sites. Specifically, the sampling frame for each RHY PDP site will be the roster of staff that support RHY PDP programs; young people who engage in local youth advisory boards that support the RHY PDP programs or participate in the RHY PDP program themselves; and community providers and partners that are involved in the implementation of the RHY PDP projects. The contractor will work with the RHY PDP contact to purposively identify potential respondents who have information on the evaluation’s key research areas, and recruit them to participate in key respondent interviews, focus groups, and/or the listening sessions. Based on available documentation, we estimate the number of program staff with direct site involvement to be approximately 10 (total of 100). The estimated number of youth program participants and YAB members ranges from 40 to 60 (total of 400-600). Community providers and partners likely include 20 to 30 per site (total of 200-300). Data collection activities will engage 10 site staff in 8 quarterly calls, 240 young people in focus groups, 100 providers/partners in listening sessions, and 80 professionals in key respondent interviews.
The proportion of participation across these subsamples includes nearly all (quarterly site calls in which all program leads will participate), more than 50% (240 young people from among 200-300), and 25-40% (professionals in listening sessions). These numbers represent fairly substantial proportions of each group and should effectively enable us to reach saturation, e.g., a point where additional data collection yields little to no new information or insights. However, given the voluntary nature of the activities, there is no guarantee that participation will be representative of the array of perspectives in the larger population of RHY PDP staff or community providers. Young people who participate in the youth focus groups will not be representative of the population of young people that the programs serve. Instead, the contractor aims to describe the range of young people’s experiences with the RHY PDP co-design process and with RHY PDP prevention services within a given site.
B3. Design of Data Collection Instruments
Development of Data Collection Instruments
The contractor developed data collection instruments drawing from the contractor team’s substantive experience and expertise developing protocols and conducting interviews across a wide range of topic areas, including child welfare, homelessness, and housing instability and homelessness prevention. In particular, the interview guides and protocols draw on previous studies (conducted at the Urban Institute and by other researchers) that have a similar research design and objectives, including the Urban Institute’s evaluation of the US Department of Housing and Urban Development (HUD) and Justice (DOJ) Pay for Success (PFS) Permanent Supportive Housing (PSH) demonstration.2 The contractor team’s past experience includes many projects entailing the engagement of individuals with experience of homelessness and other adversities both as partners and participants.
Active engagement is an overarching component of the RHY PDP process evaluation design. The input, co-design, co-interpretation and related feedback of experts with experience of homelessness, community members, subject matter experts, and providers is critical to the validity of the evaluation and its ability to meaningfully reflect implementation. The contractor is now actively engaging these constituent groups via informal and unstructured discussions to ensure the feasibility of evaluation strategies and materials, including all data collection instruments3.
Table 2 outlines the data collection instruments developed for each of the planned data collection activities and the associated objectives.
Table 2. RHY LAS Data Collection Activities, Instruments, and Objectives
Data Collection Activity |
Instrument |
Objectives |
Virtual Quarterly Site Meetings |
|
|
Listening Sessions |
|
|
Site visits – Key Respondent Interviews |
|
|
Site visits - Youth Focus Groups |
|
|
B4. Collection of Data and Quality Control
Overall Approach
The contractor team consists of both senior and early-career professionals, each assuming a specific role in conducting the process evaluation. Senior members oversee all project activities while early-career staff are trained to carry out specific activities such as reviewing documents, notetaking, and assisting senior members with data collection, analysis, and reporting.
Site teams: To facilitate data collection, the contractor will assign two research staff to each RHY PDP site, forming designated ‘site teams’ that will serve as the contractor’s primary point of contact with the sites. The roles of the site teams will be shared between a site lead and site liaison:
Site leads: Senior professional staff, with extensive experience with youth homelessness research and practice, will manage relationships between the research team and site partners. They will conduct on-site and virtual data collection, ensuring accuracy and rigor in alignment with the research design and data analysis plan.
Site liaisons: These are early career professional staff with substantive subject matter expertise. Within a site team, these individuals will support data collection and maintain records of interactions and information exchange between the research team and sites.
Each site team will be assigned to three or four sites, allowing contractor staff to compare and contrast activities and processes across sites, connect lessons learned, and optimize staffing efficiencies. Site teams will leverage the contractor’s expertise in youth and family homelessness, homelessness prevention, and child welfare, with close support from senior research team members.
Regular contractor team meetings will ensure alignment of site team activities, facilitate planning for upcoming evaluation activities, and enable review of data collection, analysis, and partner engagement. The contractor team senior leaders will also monitor project quality, ensure research staff follow protocols, and maintain strong engagement with sites and other partners. These meetings will provide a space for preparation, updates, discussion of analysis and partner management, and other evaluation tasks.
Virtual Quarterly Site Meetings (Instrument 1)
Data Collection Leads. Contractor site teams will be responsible for leading quarterly meetings with assigned RHY PDP grantee sites.
Recruitment. FYSB federal project officers will provide contractor site teams with contact information for RHY PDP grantees. These staff will be asked to participate in virtual, quarterly site meetings as part of their grantee activities. The contractor will not need a separate recruiting procedure for these respondents, as site teams will already be in contact with them as part of project start-up activities before data collection planning begins. RHY PDP grantee staff will be participating in their professional capacity and are not considered a vulnerable population. The calls will be low risk, focusing on operational updates rather than sensitive topics.
Data Collection Mode. To enable ongoing communication, site teams will have quarterly 60-minute virtual meetings (e.g., Zoom) with RHY PDP grantee site staff. The contractor anticipates one RHY PDP grantee site staff member per site will participate, for a total of 10 respondents. Site teams will facilitate meetings using the Quarterly Site Meeting Guide (Instrument 1) to:
Build relationships with key grantee staff at each site and provide a primary point of contact for information and coordination on the broader evaluation.
Gather information on the site’s ongoing activities, including primary tasks, partners, milestones, and challenges.
Explore site reports on implementation progress, challenges experienced, and strategies for managing those challenges.
Data Quality. The contractor will ensure high-quality data collection of interviews. All site teams will participate in training before data collection begins. The training agenda will be focused on qualitative data collection and review of the discussion guides. The training will include the following topics:
Consent procedures, privacy, and other human subject considerations
Data security plan for interviews
Scheduling interviews
Best facilitation practices to administer open-ended interviews and
Interviewing and notetaking practices, including instructions on how to scribe handwritten notes if the respondent declines audio recording of the interview
Geographic/community context, cultural sensitivity and relationship building
Assessing for follow-up and re-interviews
Listening Sessions (Instrument 2)
Data Collection Leads. One member of the contractor team (site lead) will work with the Principal Investigator (PI) and project director (PD) to conduct the listening sessions.
Recruitment. Site teams will work with RHY PDP grantee contacts to identify a broad set community partners associated with the implementation of the RHY PDP to participate in a listening session. The contractor will engage with partners that represent RHY PDP housing or service providers as well as other partners in the housing and homelessness, incarceration and law enforcement, child welfare, school, and health (physical, mental, behavioral, etc.) systems. The number of respondents will vary based on the number of active partners within each site, but the contractor anticipates up to 10 participants in each site for a total of 100 participants. These individuals will be participating in their professional capacity. The listening sessions will be low risk, focusing on partnerships and collaboration, and will avoid sensitive topics.
Data Collection Mode. The listening sessions will be conducted virtually (via Zoom). Sessions will occur within a two week period to minimize response variation due to timing (e.g., ensure sites are at approximately the same point in implementation). To facilitate discussion and engagement, sessions will include two 30-minute breakout groups, each facilitated by at least one member of the contractor staff. At the beginning of each listening session, contractor staff will request permission to audio record the meeting as a backup to notetaking; participants will be free to privately decline the recording (via chat) if they prefer handwritten notes only. Audio recordings will be permanently destroyed as soon as the contractor has a complete set of notes and completes any verification or clarification that is needed. Handwritten notes will be destroyed at the end of the evaluation contract.
Data Quality. The contractor will ensure high-quality facilitation of the listening sessions. Before the listening sessions, contractor staff will be trained on the following topics:
Listening session procedures, data privacy and security, and other human subjects considerations
Facilitation best practices to encourage discussion
Notetaking practices, including instructions on how to scribe handwritten notes if any participants decline audio recording of the listening sessions
Geographic/community context, cultural sensitivity and relationship building
Assessing the need for follow-up and clarification
Site Visits (Instruments 3 and 4)
Data Collection Leads. Contractor site teams will conduct site visits.
Recruitment.
Key Respondent Interview Recruitment. Site teams will work with the RHY PDP grantee staff to prioritize partners that represent RHY PDP housing or service providers as well as other partners in the housing and homelessness, jail and law enforcement, child welfare, school, and health (physical, mental, behavioral, etc.) system for interviews during site visits. Partners will be prioritized to ensure that different elements of the sites RHY PDP program are represented, such as referral partners, direct service provision, and data and implementation planning or monitoring partners. The respondents for these interviews will be similar to the participants for the listening sessions, and some individuals will participate in both the interviews and listening sessions. RHY PDP grantee staff will provide site teams with partner contact information over encrypted email. These interviews will involve minimal risk. Partners will be serving in their professional capacity and encrypted email will be sent to their work addresses. Site teams will email partners an invitation and coordinate interview logistics.
Before each interview, site teams will explain the evaluation and how interview findings will be used and inform respondents that they will be free to skip any question they do not want to answer. Site teams will request permission to audio record each interview as a backup to interview notes; respondents will be free to decline the recording if they prefer handwritten notes only. Audio recordings will be permanently destroyed as soon as the contractor has a complete set of notes and completed any verification or clarification that is needed. Handwritten notes will be destroyed at the end of the evaluation contract.
Youth Focus Group Recruitment. The contractor plans to facilitate two focus groups with young people: one with young people participating in RHY PDP programming and one with a local Youth Advisory Board (YAB) established by the RHY PDP grantee.
Site teams will consult with the RHY PDP grantee site and any relevant community partners to plan the two focus groups. In cases where site teams are not able to meet with the YAB, they will conduct two focus groups with youth participating in the RHY PDP. Up to twelve youth will participate in each focus group. Site teams will provide materials for partners to assist with recruitment, including a recruitment flyer to distribute to eligible young people. Those partners who agree to assist with recruitment will be asked to sign a confidentiality pledge stating that they understand their role, including responsibilities related to participants’ privacy.
The RHY PDP grantee site staff will invite the partners assisting with focus group recruitment to attend a one-hour required training facilitated by the contractor. The training will prepare partner staff to conduct initial outreach to youth to participate in focus groups. It will also prepare staff to answer potential questions from prospective focus group participants and emphasize that whether they participate in the focus groups or not, they will still receive all services offered by the RHY PDP.
Since the contractor does not have a formal relationship with the RHY PDP partners, they will coordinate with the primary RHY PDP grantee to collect and maintain any contact information necessary for recruitment and coordinate with focus group participants. Site teams will remind partners that full names are considered personally identifiable information (PII) and should not be shared with the contractor. Instead, in order to confirm participants in the focus group, the RHY PDP grantee and partners will only provide a list of first names and last initials. This list will be shared with the contractor via encrypted email and destroyed after the focus group ends.
Contractor site teams will work with RHY PDP grantee staff and partners to arrange focus group logistics (e.g., time, location, transportation, language accessibility). The partners will be available to youth who become distressed but will not be present in the space where the focus group is taking place. Site teams will respond to signs of distress by offering breaks to participants and ensuring that they are connected to partner staff who will be on-call to offer support. Partners who assist with recruitment will not be compensated for their participation in recruitment activities.
Data Collection Mode. In-person listening sessions, interviews, and focus groups.
Data Quality. All contractor site team members will participate in training prior to site visits. The training will cover the same topics as outlined above for quarterly site meetings. In addition to training, one of the contractor’s co-PIs will lead the first two visits to oversee data collection. Site leads and liaisons will attend site visits together in all subsequent visits. The contractor’s senior professional staff will be well positioned to provide feedback to site teams on appropriate interviewing techniques and relationship building strategies. Contractor staff will also meet on a regular basis as site visits occur to share experiences and learnings.
B5. Response Rates and Potential Nonresponse Bias
Response Rates
The key respondent interviews, focus groups, and listening sessions are not designed to produce statistically generalizable findings and participation is at the respondent’s discretion. Response rates will not be calculated or reported for interviews and focus groups.
NonResponse
As participants will not be randomly sampled and findings are not intended to be representative, the contractor will not calculate non-response rates or bias. However, the contractor will qualitatively assess non-response for key informant interviews conducted during site visits. This is to assess gaps in knowledge. Any limitations of the data collected for key respondent interviews will be reported in written materials.
B6. Production of Estimates and Projections
The data collected will not be used to generate population estimates, either for internal use or dissemination.
The contractor does not plan to produce any quantitative estimates. As noted, the key respondent interviews, focus groups, and listening sessions are not designed to produce statistically generalizable findings.
The contractor does not plan to archive data. However, the contractor will outline a plan for final disposition of complete data, including PII and/or sensitive data, with appropriate levels of security.
B7. Data Handling and Analysis
Data Handling
The contractor has multiple checks in place to ensure proper handling of the collected data.
Training. All contractor staff who conduct quarterly site meetings, key respondent interviews, focus groups, and listening sessions will participate in training before data collection begins. The training agenda will be focused on qualitative data collection and review of the discussion guides and protocols, where relevant. As related to data handling, the training will include assessing the need for follow-up and re-interviews. Retraining will be conducted, as necessary.
Team Meetings. Regularly scheduled contractor team meetings will occur throughout the duration of the contract to allow project leadership to provide feedback and respond to questions from site teams on data collection and analysis topics. This will also allow for site teams to calibrate qualitative coding decisions and improve inter-rater reliability.
Transcription. The contractor plans on using a secure service to transcribe discussion, interview, and focus group audio files into text. Key respondent interview and focus group transcripts will be analyzed by the contractor for emerging themes and summarized for dissemination. Transcript analysis will be preceded by verification of any and all quotes that might be extracted (and included in the report) using audio recording. All quotes will remain anonymous.
Data Analysis
The data collected will be predominantly qualitative. One of the primary tasks for analysis will be compiling and documenting the data consistently and coherently so that data can be easily analyzed and transformed into research findings. The contractor will construct a master data collection protocol that tracks and compiles all data collection activities and documents such as interview summaries and meeting notes, enabling consistent organization, security, and retrieval as needed.
Once compiled, the contractor will analyze the qualitative data to answer the research questions to be discussed in annual (non-public) briefings to ACF. The contractor will begin analysis with a broad synthesis of document review findings as well as a synthesis of conversations and interviews with RHY PDP grantee staff, partners, and youth. For interviews, the contractor will create a uniform coding scheme aligned with the interview guide and research questions. The contractor will use NVivo, a qualitative data analysis software, to code these data and identify common themes that emerge from the interviews. Key themes may arise across RHY PDP grantee sites or within a site. For each key theme, the contractor will identify supporting evidence from the data.
The contractor will present the findings of their analysis through data tables, charts, quotes, and narrative, along with pull out boxes in annual reports and briefings. The inclusion of quotes from quarterly calls, interviews, and focus groups will only be used in a manner that maintains anonymity and privacy of respondents; respondents will not be named or attached to any identifying characteristics in quotes. The contractor will work iteratively to ensure that consistent themes and unusual or rare observations are captured across data sources. In the case of incongruent interpretations of data across constituents, the contractor will report the multiple perspectives offered.
Data Use
The contractor will report on and disseminate findings to the public in a final report. The report will present RHY PDP implementation evaluation findings thematically with supporting data and illustrative quotes that correspond to research questions. The report will highlight similarities and differences across RHY PDP sites, to point to distinct approaches and variation in local contexts. The report will be written in plain language using accessible terms.
In reporting findings, the contractor will not include any names of or identifying characteristics of participants. Any future results, analyses, and other information developed as a result of data obtained from the quarterly site calls, site visit collection, or listening sessions will be published in summary, aggregated, or using descriptive statistics so that the identity of individuals contained in the data is not revealed. Any report of findings will acknowledge limitations of data and data use and their potential implications.
Research products will include tangible recommendations, tools, and tips that will be useful for a practitioner audience. The contractor will use the active engagement framework to work with individuals and groups on co-analysis and co-interpretation of data as well as co-development of products. The contractor’s communications team will also provide editorial and dissemination support in close collaboration with contractor staff, experts with experience of homelessness, advisory board members, subject matter experts, and relevant FYSB, OPRE, and other ACF staff.
B8. Contact Persons
Anne Farrell, Principal Investigator, afarrell@urban.org; Samantha Batko, project director, sbatko@urban.org
Attachments
Appendix A: RHY LAS Research Questions
Appendix B: RHY LAS Process Study Data Collection IRB Full Approval
Instrument 1: Quarterly Site Meeting Guide
Instrument 2: Listening Session Topic Guide
Instrument 3: Site Visit Interview Protocol – Key Respondent Interviews
Instrument 4: Site Visit Focus Group Protocol – Youth Focus Groups
1 This effort does not impose burden on respondents and therefore is not included as an “information collection” under the Paperwork Reduction Act for this request.
2 Read more about Urban’s evaluation of the HUD-DOJ PFS PSH Demonstration at https://www.huduser.gov/portal/PFS-PSH-study.html.
3 This effort does not include requesting the same information from more than 9 respondents and is not subject to the Paperwork Reduction Act.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Moraras, Pear (Naramon) |
| File Modified | 0000-00-00 |
| File Created | 2025-09-20 |