Supporting Statement B - AHSSD Revision

Office of Community Services Affordable Housing and Supportive Services Demonstration Data Collection

OMB Information Collection Request

0970 - 0628

Supporting Statement Part B –

Statistical Methods

September 2024

Type of Request: Revision

Submitted By:

Office of Community Services

Administration for Children and Families

U.S. Department of Health and Human Services

1. Respondent Universe and Sampling Methods

The Office of Community Services (OCS) awarded 9 Affordable Housing and Supportive Services (AHSSD) grants in Fiscal Year (FY) 2023 and anticipates a similar number of awards in 2024. The respondent universe and sampling methods incorporate the second cohort of grant recipients, though their characteristics remain unknown at this time. This section provides information on the respondent universe and sampling methods for each instrument.

Arizona Self-Sufficiency Matrix and Service Receipt Questionnaire

For the Arizona Self-Sufficiency Matrix and the Service Receipt Questionnaire, the respondent universe is intensively case-managed adult residents living in affordable housing funded by the AHSSD. OCS hopes to collect Matrix and Questionnaire data from the majority of these residents. As this is a demonstration program, there is a wide variety of the anticipated number of case-managed residents by different grant recipients, with some sites planning to only offer intensive services to a handful of residents and other sites planning to case-manage close to a hundred residents. Due to this variety and the fact that most grant recipients are already collecting the Matrix, OCS is not planning to stratify or take a random sample of intensively case-managed residents.

OCS estimates that the first cohort will enroll up to 400 residents in intensive case management services, based on early enrollment numbers, projections from grant recipients, and a review of project progress. By focusing only on the universe of residents who are receiving the highest intensity of services and who are in regular contact with their caseworkers, OCS anticipates a high overall response rate and measurable change on the Matrix and Questionnaire. These Matrix and Questionnaire results will not be generalizable to all the residents living in affordable housing funded by the demonstration program.

OCS plans to conduct the Matrix and Questionnaire at multiple time points (baseline and follow up) to evaluate change over time. At baseline, OCS anticipates a response rate of 85% for the two instruments based on response rates observed by current grant recipients. For the second time point, OCS anticipates that some residents may move away from the affordable housing community or finish receiving services, and thus the response rate for the follow-up is anticipated to be 75%.

Resident Focus Groups

OCS will conduct focus groups with residents to understand participant experiences with the program and perceived benefits of services. Most of the grant recipients have also previously conducted focus groups with their residents and have stated that they have been able to convene groups of 5-7 residents to discuss their experiences of the program. Based on grant recipient experiences arranging these types of discussion groups, OCS expects a high participation rate, above 90 percent. OCS will administer focus groups at the housing communities, with the support of grant recipients in recruiting participants and arranging logistics.

To accommodate OCS staff constraints and to be able to conduct the majority of the focus groups in person, OCS plans to conduct focus groups at only half of the sites. OCS will purposively select sites to include a range of approaches to service delivery and a diverse population of participants. In particular, OCS is interested in understanding resident experiences for individuals residing in transitional housing while receiving services and those living in permanent supportive housing. Additionally, OCS is interested in understanding the experiences of individuals receiving differing intensities of case management. Some sites employ a lighter touch approach, with case management staff supporting a larger case load, while other sites support a smaller case load with greater intensity of services. OCS hopes to purposively sample residents from a variety of backgrounds and lived experiences in the focus groups (including people of different genders, racial and ethnic backgrounds, marital and family status, and age).

Interviews with Program Staff

For the interviews with program staff (program directors and caseworkers), OCS anticipates a response rate approaching 100%. All grant recipients have expressed a willingness for their staff to be interviewed. Current grant recipients employ one program director and one or more individuals who serve as case workers or service coordinators. By interviewing several different staff members from each grant recipient organization, OCS hopes to obtain a holistic understanding of perceived program outcomes and implementation. Interviews will be conducted in person for staff at sites that are hosting focus groups. Interviews will be conducted virtually for other staff.

Semi-Annual Quantitative Reports and Quarterly Narrative Performance Progress Reports

During the initial round of data collection, OCS achieved a response rate of 100% from grant recipients completing the Quarterly Performance Progress Reports (PPR) and Semi-Annual Quantitative Mandatory Reports. OCS anticipates that only around a third of the grant recipients will complete the Semi-Annual Quantitative Optional Reports based on the preliminary reports submitted in March 2024. The questions in the Optional Reports are based on information that some of the grant recipients indicated in their applications that they were planning to collect.

Instrument	Respondent Universe	Intended Sample	Response Rate
Self-Sufficiency Matrix	Intensively case-managed adult residents living in affordable housing funded by the Affordable Housing and Supportive Services Demonstration	Entire respondent universe (maximum: 800 residents)	85% for baseline; 75% for follow-up
Service Receipt Questionnaire	Intensively case-managed adult residents living in affordable housing funded by the Affordable Housing and Supportive Services Demonstration	Entire respondent universe (maximum: 800 residents)	85% for baseline; 75% for follow-up
Resident Focus Group	Case-managed residents living in affordable housing funded by the Affordable Housing and Supportive Services Demonstration	105 residents receiving case management services	90%
Quarterly PPR Collections	All grant recipients funded by the Affordable Housing and Supportive Services Demonstration	18 grant recipients	100%
Semi-Annual Quantitative Report MANDATORY	All grant recipients funded by the Affordable Housing and Supportive Services Demonstration	2023 Version: 3 FY 2023 grant recipients opting 2024 Version: 6 FY 2023 grant recipients opting and 9 FY 2024 grant recipients	100%
Semi-Annual Quantitative Report OPTIONAL	All grant recipients funded by the Affordable Housing and Supportive Services Demonstration	2023 Version: 1 FY 2023 grant recipient opting 2024 Version: 2 FY 2023 grant recipients opting and 3 FY 2024 grant recipients	33%
Interviews with Program Directors	All program directors overseeing programs associated with the Affordable Housing and Supportive Services Demonstration	All program directors overseeing programs associated with the Affordable Housing and Supportive Services Demonstration (18)	100%
Interviews with Caseworkers	All caseworkers working on the Affordable Housing and Supportive Services Demonstration	All caseworkers working on the Affordable Housing and Supportive Services Demonstration (36)	100%

2. Procedures for the Collection of Information

This section reviews procedures for data collection for the instruments included in the AHSSD study.

Self-Sufficiency Matrix and Service Receipt Questionnaire

For the Self-Sufficiency Matrix (Instrument 1) and Service Receipt Questionnaire (Instrument 2), grant recipients invite residents to participate in the study during a case management session. They provide residents a copy of the study information sheet (Attachment E) and review it with them before administering the instruments. OCS created surveys on a virtual platform (e.g., Qualtrics) to automatically transmit data from the matrix and questionnaire administration to OCS. All grant recipients use OCS’s version of the questionnaire, and most use OCS’s version of the Matrix. Two grant recipients had collected the Matrix data in their MIS prior to the grant period, and they have elected to provide de-identified Matrix data via upload of an encrypted file to OCS’s Box account. Residents receive a token of appreciation after participation to ensure a high response rate and help ensure quality data with utility for the government. The Matrix is typically conducted via a collaborative conversation between a caseworker and a resident (see Attachment D), while the Questionnaire may be read aloud by the caseworker in an interview format. Most caseworkers had experience using similar instruments in the past. The case-managed residents are each assigned a randomized unique individual identifier as well as a unique randomized household identifier for the Matrix and Questionnaire, so that OCS can evaluate change over time at the individual level, at the household level, at the grant recipient level, and for the demonstration program collectively.

OCS will examine changes in service receipt and self-sufficiency measures from baseline to each six-month follow-up report. We will begin with basic descriptive analyses (e.g., frequencies, means, response distributions) to understand baseline levels of service receipt and self-sufficiency upon study enrollment. We will then explore changes in service receipt and outcomes by examining baseline and follow-up differences for the items and also overall service receipt and self-sufficiency. Additionally, we will explore linkages between the receipt of wraparound services and self-sufficiency by linking the service-receipt and self-sufficiency questionnaire data. We will use regression analyses and t-tests to explore the significance of changes and associations between services and self-sufficiency.

Resident Focus Groups

For the resident focus groups, OCS plans to work with grant recipients to identify residents who received services through the program. We will work with grant recipients to recruit a diversity of experiences, including those who reside in transitional housing while receiving services, and those who reside in permanent supportive housing. We are also interested in understanding perspectives of residents who experience different approaches to case management. OCS will send Attachments B and C to the grant recipients and ask them to recruit residents from a variety of backgrounds and lived experiences (including people of different genders, racial and ethnic backgrounds, marital and family status, and age). OCS staff will then travel to the sites and conduct the focus groups in the affordable housing complexes where the residents live using Instrument 3 to guide the discussion. OCS will work with the grant recipients to ensure that participants have adequate childcare (and transportation if needed). Recipients will be given a token of appreciation at the close of the discussion. The focus groups will be transcribed, and OCS staff will then analyze the transcripts with qualitative analysis software to understand how residents experienced the program.

Staff Interviews

For sites selected for focus groups, OCS will perform staff interviews on site. Other interviews will be performed virtually via Teams. OCS will use Instrument 9 to guide the interviews. OCS will transcribe the virtual meetings and then use NVivo to identify themes in the interviews to assess implementation and perceived best practices.

Semi-Annual Quantitative Reports and Quarterly Narrative Performance Progress Reports

For the Mandatory Semi-Annual Quantitative Reports, the grant recipients will use their existing databases to track service delivery, referrals, and demographics of residents. The recipients will then provide total counts of these indicators in the Mandatory Report (Instrument 5 or 6) and will also list organizational partners and attributes of the housing community. The Optional Report (Instrument 7 or 8) is a place for grant recipients to detail other activities or metrics that they may be tracking on resident outcomes. The grant recipients track these metrics in their MIS for the individuals in the study and provide aggregate reports to OCS. The grant recipients will supplement these Reports with the narrative PPRs, which will include information about the progress of implementing the program.

3. Methods to Maximize Response Rates and Deal with Nonresponse

The primary procedures to deal with non-response on the Arizona Self-Sufficiency Matrix and Service Receipt Questionnaire are to limit the respondent universe to intensively case-managed residents and to offer a token of appreciation for each completion of these instruments. By offering a monetary token of appreciation at each time point, OCS anticipates that the response rate will remain high for all collections and that residents will answer all or most of the questions on the Matrix. OCS’ goal is for most of the submitted matrices to be fully complete (i.e., a numerical selection on the scale from 1 to 5 selected for all self-sufficiency domains). In addition, full completion of the Matrix is necessary to calculate the full self-sufficiency score for an individual (i.e., the sum across all domains). OCS also needs a high item response rate in order to have sufficient power to detect change over time from administration to administration. The Arizona Matrix was chosen in part because most of the grant recipients are already familiar with this instrument, and many of the residents may have encountered it previously, which should thus improve both response rates and data quality.

For the Resident Focus Groups, OCS plans to offer tokens of appreciation to ensure that both a diversity of people is included and that participants give rich, clear answers. By offering a monetary token of appreciation, respondents who might ordinarily refrain from participating (i.e., busy parents) are more likely to take part in the focus group. This will give OCS a more complete understanding of the experiences of residents. Furthermore, if respondents feel that their input is valuable, they will be more likely to offer full answers to each focus group question.

OCS will maximize response rates for the Program Director and Caseworker Interviews by working closely with grant recipients to schedule interviews at times that are convenient to staff members. By also interviewing two of each staff member (i.e., two caseworkers from each site), OCS will improve the information gained about each grant recipient.

For the Reports, OCS is requesting metrics that almost all of the grant recipients already collect or have previously indicated that they would like to collect, which OCS believes will lead to a high response rate.

4. Test of Procedures or Methods to be Undertaken

The Arizona Self-Sufficiency Matrix is a validated and reliable measurement tool. In a review of ten different instruments for assessing the self-sufficiency of homeless clients, Culhane et al. (2007) found that that the Arizona Matrix was the only tool that “met acceptable reliability and validity standards” (p. 12-21). The instrument has subsequently been successfully tested and is used by many cities, counties, and states to identify the needs of people receiving housing stability services and to predict outcomes for clients based on baseline Matrix characteristics. The majority of the grant recipients have already used the Arizona Matrix with their clients.

The Questionnaire, focus group guide, and interview guides used items derived from other evaluations of programs with housing and/or supportive services components. OCS anticipates conducting an internal pilot of the Interview protocol with ACF staff serving as test respondents. Moreover, the report forms are derived from other reporting forms already in use by grant recipients at ACF and refined based on the December and March reports from grant recipients.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Dr. Jessica Hale, Program Evaluator, Division of Community Discretionary and Demonstration Programs, Office of Community Services, Administration for Children and Families, jessica.hale@acf.hhs.gov, (202) 401-4835

Juliana Melara, Policy Advisor, Office of the Director, Office of Community Services, Administration for Children & Families, juliana.melara@acf.hhs.gov, 315-744-2490

Benjamin Wills, Program Specialist, Office of Community Services, Administration for Children and Families, Benjamin.Wills@acf.hhs.gov, 206-550-2471

Regine Adrien, Program Specialist, Office of Community Services, Administration for Children and Families, regine.adrien@acf.hhs.gov, 202-205-9741

Brontë Nevins, Community Development Project Support Specialist, Division of Community Discretionary and Demonstration Programs, Office of Community Services, Administration for Children & Families, bronte.nevins@acf.hhs.gov, 240-274-1442

Julia Rabin, Program Specialist, Office of Community Services, Administration for Children and Families, Julia.Rabin@acf.hhs.gov, 202-207-9485

Bridgette Sullivan, Program Specialist, Office of Community Services, Administration for Children and Families, Bridgette.Sullivan@acf.hhs.gov, 202-207-3707

References

Culhane, D.P., Parker, W. D., Poppe, B., Gross, K.D., and Sykes, E. (2007). Accountability, Cost-Effectiveness, and Program Performance: Progress Since 1998. 2007 National Symposium on Homelessness Research. Retrieved from: https://aspe.hhs.gov/sites/default/files/migrated_legacy_files//84166/report.pdf