Appendix A_ORR Response to Public Comments on ORR-5

Appendix A:

Refugee Data Submission System for Formula Funds Allocations and Service Analysis (ORR–5)

Response to Public Comments

Submitted by

Office of Refugee Resettlement

(October 2020, Updated March 2021)

1. Information Collection Necessity and Practical Utility

   1. Appropriateness of the ORR-5 to report client needs

The Refugee Data Submission System for Formula Funds Allocations and Service Analysis (ORR-5) satisfies the statutory requirements of the Immigration and Nationality Act (INA), specifically section 412 (a)(3) of the Act that requires the Director of the Office of Refugee Resettlement (ORR) to make a periodic assessment, based on the refugee population and other relevant factors, of the relative needs of refugees¹ for assistance and services and the resources available to meet those needs. In the most recent OMB-approved ORR-5 (November 6, 2017), ORR expanded and successfully collected client-level service data. ORR believes the new proposed data points will better meet INA requirements and ensure the relative needs of the populations ORR serves are assessed.

The proposed ORR-5 has four functions:

1) to satisfy the statutory requirements of the INA section 412(a)(3) and enable the Director of ORR to make a periodic assessment of the needs of refugees for assistance and services and the resources available to meet those needs;

2) to assess the number of clients served in each state to determine the corresponding Refugee Support Services (RSS) allocations;

3) to collect data to inform evidence-based policy making and program design; and

4) to monitor implementation of the requirements put forth in Policy Letter 19-07, which provides guidance on RSS Family Self-Sufficiency Plans (FSSPs).

The first three aforementioned functionalities are not new to the form or state and replacement designee (hereinafter referred to as “states”) reporting requirements. The revision to include additional client-level data elements related to the FSSP will allow the ORR Director to better understand client goals, services utilized, and the outcomes achieved by the populations ORR serves. This deeper understanding of a client’s progress in reaching self-sufficiency will better inform programmatic and policy decision making and ultimately improve programs and services for the populations ORR serves.

2. Client-level data and Personally Identifiable Information (PII)

   1. Client-level data informing ORR policy

ORR has collected client-level data since 2009. In 2017, ORR expanded the ORR-5 to include information about client enrollment in four service areas (Refugee Cash Assistance, Refugee Medical Assistance, Refugee Medical Screening, and Refugee Support Services). By collecting the proposed additional client-level data for the FSSP, including primary goals, referrals, and the outcomes of the referrals, ORR will be better able to analyze the impact of the services and identify where additional service needs may exist. ORR is not authorized to request outcome data for all services to which clients are referred, because many of those services are not ORR-funded. However, by analyzing the type of service referrals (ORR-funded versus non-ORR funded; or state administered versus discretionary programs) and how initial referrals differ across geographic locations among clients with the same primary goals and similar demographics, ORR will be able to identify potential service gaps and service needs in order to inform its evidence-based policy and programmatic decisions. This level of analysis will also allow us to better prepare for changes in service needs amongst various refugee populations. The data also provides important information on how ORR populations are faring twelve months after enrollment in RSS-funded employment services. This will allow ORR and states to analyze longer-term service needs and identify potential programmatic or policy solutions.

2. Appropriate use of client-level data in relationship to performance outcomes

Several respondents discussed client-level performance outcomes and the possibility of this information being used to limit refugee admissions based on an individual’s or a group’s perceived ability to achieve economic self-sufficiency. ORR’s mission continues to be to help new populations maximize their potential in the United States by linking them to critical resources that assist them in becoming integrated members of American society. Analysis of client-level data will strengthen ORR’s ability to analyze client outcomes and potential gaps in services, and in turn, to make evidenced-based programmatic and policy decisions to better address barriers to self-sufficiency and integration. Trends in positive outcomes in one geographic location or population demographic may help identify opportunities to replicate or expand services to other geographic areas or populations. Similarly, analyzing client-level data may help ORR identify opportunities or establish priorities to further support positive outcomes.

It is important to note that ORR does not manage the U.S. Refugee Admissions Program, which guides the development of the resettlement priorities that the U.S. Government sets each year within the Presidential Determination. ORR does not share ORR-5 data with federal agencies such as U.S. Department of State Bureau of Population, Refugee, and Migration or U.S. Department of Homeland Security. ORR uses ORR-5 data solely for internal ORR business operations. In addition, this data should be helpful to states in their oversight of the program, providing a more consistent level of analysis for clients enrolled in RSS and receiving FSSPs within their first year of eligibility.

3. Consistency with other federally funded programs

Several respondents stated that the proposed information collection is inconsistent with data collected by other federal programs. This is not a block grant program. As stated previously (Section I.A), the Refugee Act emphasizes the focus on programs to support self-sufficiency. Enhanced and more consistent data collection across states is critical to meeting this mandate. In addition, it is important to consider the context of other similar large resettlement programs such as Canada and Australia which have developed comprehensive data collection systems to assess services and progress post arrival. In regards to respondents’ comments on individual client-level data, states will be informed of the authority, purpose and routine use of the collected information through a Privacy Act Statement. States are required to ensure the Privacy Act Statement is communicated to the clients that they are reporting data on and that individual information collected will be kept private. . It is important to note that the ORR-5 is not the first information collection within ORR that requires individual client-level data and outcomes. In addition to client-level data in the current ORR-5, ORR also requests demographic information, service utilization, and outcomes in the ORR funded, state-administered Unaccompanied Refugee Minor (URM) program as well as the Survivors of Torture program. ORR has found that collecting this type of data enhances ORR’s program performance assessment and outcome evaluation. ORR is exploring options for collecting similar data in other programs.

4. Trust between the client and the service provider and stress avoidance

Taking public comments and the risk of re-traumatizing clients into account, and to further decrease the reporting burden of states, ORR has removed the “Ethnicity” field from the ORR-5 data collection.

3. Consistency with a holistic approach to refugee resettlement

In accordance with ORR Policy Letter 19-07, FSSPs are intended to be holistic and assess the needs of employable refugees and their family members. Creating an FSSP is a necessary step in developing a strategy to achieve economic self-sufficiency for a family. The FSSP is intended to be a flexible, living document and to address the overall needs of a household in a culturally responsive manner. In contrast, ORR designed the ORR-5 to capture a synopsis of the client’s experience, using the fewest data points possible to ensure consistency across states and that states are not unduly burdened with excessive reporting. This collection emphasizes the establishment of a consistent baseline across states so that this level of data can be analyzed for the first time in the program’s existence.

ORR acknowledges that multiple goals might be identified during the initial assessment and that a family’s needs evolve. ORR established a narrow definition for “Initial Primary Goal” for reporting purposes, with the expectation that states will address and record additional goals in the case file to holistically support a client and his or her family. The “Initial Primary Goal” reported on the ORR-5 is defined as the goal that is identified during the initial assessment and will address the most critical obstacle an individual needs to overcome by the end of 12 months in order to work towards longer-term self-sufficiency and integration. ORR will use alternative information sources (such as monitoring) to comprehensively assess the needs of clients, service provision, regulation and policy compliance, and program performance.

ORR will outline additional clarifications and expectations for how FSSPs are conducted and documented by local service providers, and subsequently reported by states in a future policy letter upon approval of the revised ORR-5.

4. Duplication and use of similar information

ORR has conducted a thorough review of its current internal information collections that are similar to the proposed ORR-5, namely the ORR-6 Performance Report and Annual Service Plan (ASP), Annual Outcome Goal Plan (AOGP), Annual Survey of Refugees (ASR), State Plan, and program monitoring tools. These information collections cover different ORR services and focus on distinct programmatic and performance evaluation areas. ORR believes that current available data can complement, but not replace the full caseload, cross-programmatic quantitative data received via the ORR-5. The following chart summarizes the data ORR currently collects and the difference between it and the proposed ORR-5.

Current ORR Information Collection	Data Type	Data Collected on Other ORR Data Collections and Not the Revised ORR-5
Data Area 1: Services Utilized
ORR-6: Schedule A	Qualitative and Narrative	Description of each RSS program in the state, activities, accomplishments/new initiatives, and challenges/emerging issues.
ORR-6: Schedule B, C, and D	Qualitative and Quantitative	Unduplicated semiannual # of RCA recipients; # of participants receiving four core RSS Employability Services at the state level; and # of participants in RSS set-aside programs at the state level.
ORR-6: ASP	Quantitative and Measureable	Unduplicated annual # of participants receiving each and all RSS Employability Services and Other Services at the state level.
Proposed revision of ORR-5	Quantitative and Measureable	Annual # of individuals/households receiving RSS funded services, RCA, RMA, and RMS simultaneously at the individual, household, local, and state levels, where no current information is available. Individual- and household-level demographics, where no current information is available.
Data Area 2: Goals and Outcomes
ORR-6: Schedule A	Qualitative and Narrative	Description of state and program performance and outcome measures
ORR-6: Schedule B, C, and D	Qualitative and Quantitative	Semi-annual RCA and RSS set-asides outcomes; employment outcomes for core RSS Employability Services at the state level.
AOGP	Quantitative and Measurable	States’ progress toward aggregate caseload goals; annual employment outcomes for any four core RSS Employability Services at the state level.
ASR	Quantitative and Measurable	Subset of the population covered by other reports; # of households receiving RCA; much smaller sample size (1,500 households).
Monitoring	Qualitative and Quantitative	Qualitative analysis for a sample of cases; could collect certain outcomes but does not provide data for the full caseload of those receiving RSS support through the FSSP that the ORR-5 provides.
Proposed revision of ORR-5	Quantitative and Measurable	Partial, but more detailed RSS individual and household goal areas, the progress toward achieving goals, and outcomes for both employable refugees and their families. No information is currently available on service recipients’ families.
Data Area 3: Program Implementation
Monitoring	Qualitative and Quantitative	Ensures program compliance and permits case file review for in-depth analysis; limitations in the scope of information reviewed on-site by federal monitors; # of case files reviewed is limited.
State Plan	Qualitative and Narrative	Description of how the state administers the refugee program, including who is served, the services provided from federal funding and various assurances.
Proposed revision of ORR-5	Quantitative and Measurable	Measurable RSS FSSP programmatic information; allows for inferential statistical analysis examining whether, how, and to what extent programmatic and individual level factors affect the likelihood of achieving goals and economic self-sufficiency outcomes at the individual, household, local, and state levels to inform program design and resources.

2. Burden Estimate Accuracy

1. Realistic time frame for compliance

Based on public comments and after careful consideration, ORR has revised the implementation timeline of the new ORR-5 to provide states with ample time to build or modify existing data collection systems. The following chart illustrates the revised implementation timeline for populations served in federal fiscal year (FY) 2022, FY 2023, and FY 2024. (See Attachment 5: revised ORR-5 Instructions for more details):

	FY 2022 Served Population	FY 2023 Served Population	FY 2024 Served Population
FY 2023 Submission (Dec 2022 to Jan 2023)	Section I & II	N/A	N/A
FY 2024 Submission (Dec 2023 to Jan 2024)	Section III	Sections I & II	N/A
FY 2025 Submission (Dec 2024 to Jan 2025)	N/A	Section III	Sections I & II

2. Accuracy of estimated burden hours

The burden to respondents varies widely due to multiple factors, including, but not limited to, size of eligible population, complexity of a respondent’s existing information collection and management methods, and a state’s flexibility modifying or developing information systems.

In August 2020, ORR consulted with a sample of nine states to inform the burden hour estimate. Eight states responded with an average estimate of burden hours ranging from 40 to 128 hours for the current ORR-5, and 65 to 240 hours for the revised ORR-5. ORR also examined public comments where burden information is explicitly defined and consistent with OMB’s guidelines For instance, states are the direct respondents to ORR-5 collection and thus burden hours estimates were based on state level information collection and reporting. As a result, ORR revised the average burden hours to 90 hours for states to complete the current ORR-5, and 140 hours for the proposed ORR-5. Per OMB guidance, estimates of other annual cost burdens to states such as total capital and start-up cost, total operation and maintenance, and purchase of services are provided in a separate section in Supporting Statement A (A14).

3. Information Quality and Clarity

1. Standardization of the information requested

Several respondents raised concerns regarding the standardization of the information collected such as “Initial Primary Goal,” “Education Level” and “English Ability.” (“Initial Primary Goal” is addressed in Section I. C. of this document.)

Self-reported education level and language proficiency is not without issues. However, after analyzing the options ORR is choosing the self-reporting approach since it has been widely used and serves as a satisfactory proxy, without imposing a greater amount of burden on both clients and service providers.

While there is always some measurement error in data collection, the self-reported English Ability measurement proposed on the new ORR-5 is consistent with the approach used by the American Community Survey (ACS), which is considered the gold standard and has been widely used in integration research. The ASR also uses the same measurement for English ability level. Using the ACS standard is beneficial in that it enables ORR to conduct benchmark comparisons to broader population numbers.

While both number of years and level of education attainment are plausible ways of measuring education level, ORR believes recalling the number of years of education imposes a greater cognitive burden on clients who might have had unstable school placements and in some cases are older adults who may have difficulty recalling those details. The ASR uses a similar measurement for education level. The URM program uses number of years in schooling. However, URM demographics and program focus are different from those of the RSS program and the ORR-5.

Both language proficiency and education attainment are essential indicators of integration and both affect employment outcomes and long-term self-sufficiency. In addition, § 413(b)(5)(A) of the INA (8 U.S.C. § 1523(b)(5)(A)) requires ORR to report to Congress an evaluation of the extent to which services assist “refugees in achieving economic self-sufficiency, achieving ability in English, and achieving employment commensurate with their skills and abilities.” The FSSP assessment and corresponding ORR-5 reporting is an important method for ORR and our network to conduct such evaluations.

By analyzing education and language levels across demographic groups and geographic locations, ORR is able to identify potential service needs and gaps, and evaluate whether those outcomes are commensurate with the population’s skills and abilities. This analysis will provide valuable data for informing ORR’s decisions about service provision (such as English language training), resource allocation, programmatic design, and program evaluation.

2. Centralized database

Several respondents stated that it would be more efficient for ORR to develop a centralized database to collect the type and scope of information required on the revised ORR-5. ORR has also heard from states expressing concern over the establishment of a centralized database, creating a parallel system to existing state systems. ORR is exploring options to avoid duplication and the possibility of developing such a system, but it will take time to develop, test and implement. In the interim, ORR has determined that the internet Refugee Arrivals Data System (RADS) is the most efficient and secure mechanism to collect, store and analyze the data.

3. Data fields clarification

1. RADS Case and Individual IDs

RADS Case and Individual IDs serve as a data quality control tool to ensure ORR can connect individuals’ 12-month follow-up (ORR-5 Section III submitted in the current reporting FY) with their initial assessments (ORR-5 Section II submitted in the previous reporting FY). To protect PII security and avoid passing PII back and forth with states, ORR will use RADS-generated Case and Individual IDs when working with states to update or correct records.

2. RSS Enrollment and Exit Dates

While states might need to keep multiple records of a client’s enrollment and exit dates for various RSS services, states could extract information per client for their earliest enrollment and latest exit date using either basic or advanced data analytical tools. “RSS Enrollment Date” refers to the first RSS service a client received in the reporting state (not a client’s service dates in other states, if applicable). The “RSS Exit Date” is the date that a client last received services. The “RSS Enrollment Date” and “RSS Exit Date” fields are existing data fields in the current ORR-5, approved by OMB in 2017.

3. Scope of Sections II and III reporting population

Taking public comments into account and to decrease the reporting burden for states, ORR revised the requirement to report data in Section II “RSS FSSP Initial Assessment” to include individuals receiving RSS-funded employment-related services and their family members within one year of their eligibility date. Section II reporting is not required for clients receiving other non-employment related services in the state, or those receiving employment-related services after their first year of eligibility. The scope of reporting requirements for FSSPs will differ from programmatic and policy requirements. ORR will issue clarifying FSSP policy upon OMB approval of the expanded ORR-5. Section III, “RSS FSSP 12-Month Follow-Up” submission is required for all clients and their family members reported in Section II in the previous reporting FY regardless of employment status at 12 months.

4. Scope of Section III “RSS FSSP 12-Month Follow-Up” submission

RSS FSSP policy requires follow-up to be conducted minimally at six and 12 months post enrollment. While ORR encourages ongoing follow-up beyond the initial 12 months, this reporting requirement ends with the 12-month follow-up. The follow-up is an essential element of FSSP implementation emphasizing the importance of contact with the client for one year. By collecting data on the information source for Section III “RSS FSSP 12-month Follow-up,” ORR is able to reinforce the implementation of FSSP follow-ups, strengthen program oversight, and monitor program compliance. This data field also indicates acceptable alternative information sources for Section III that would otherwise be missing in cases where the provider is unsuccessful in their attempts to contact the client(s) at 12 months.

4. Data fields changes

Based on further examination of data fields on which the public provided comments, ORR proposes the changes outlined in the following table:

Data Fields in Revised ORR-5 (60-day Federal Register Notice)	Data Fields in Revised ORR-5 (30-day Federal Register Notice)	Action	Justification
Ethnicity	N/A	Eliminated	Cultural and re-traumatization sensitivity; reduces burden.
Second Last Name	N/A	Eliminated	Reduces burden.
Sex	Gender	Modified	Revised back to Gender after 30-day FRN; improves representativeness and social inclusion.
RSS FSSP Best Employment Status	RSS FSSP Employment Status	Modified	Improves clarity.
RSS FSSP First Employment Date; RSS FSSP Best Hourly Wage	34c RSS FSSP First Employment Date; 34d RSS FSSP Best Hourly Wage	Consolidated	Improves response skip pattern and reduces burden. If client has never been employed during reporting FY, states should skip #34.
N/A	County	Added	Not a new field and is included in current ORR-5. Enhances information accuracy.
N/A	City	Added	Enhances information accuracy

1 The term “refugee” is used throughout this document to refer to all ORR-eligible populations as described in Policy Letter 16-01, 45 CFR 400.43, 45 CFR 400.111, and 45 CFR 400.208.