2023 ASR Feasibility Test-- Supporting Statement B

Feasibility Testing of Online Annual Survey of Refugees Mode

Formative Data Collections for Program Support

0970 - 0531

Supporting Statement

Part B

March 2023

Submitted By:

Office of Refugee Resettlement

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Sara Tompkins

Overview of Study Objectives

The Office of Refugee Resettlement (ORR) funds an Annual Survey of Refugees (ASR) to collect information on refugees’ progress toward self-sufficiency and integration during their first five years of living in the United States. The survey data are used to meet reporting requirements for ORR’s Annual Report to Congress and inform refugee resettlement policy, by providing a unique source of nationally representative information on refugees’ experiences of adjustment to life in the United States. The ASR collects information on a range of information about refugees, including language proficiency, training and education, employment and income, public assistance, social connections, and access to health care. The annual survey is conducted by telephone in up to twenty languages.

ORR is interested in collecting evidence to inform potential transition to a multi-mode version of the survey, in which respondents could participate either by telephone or online. This study is a feasibility test of an online option to determine whether a multi-mode approach is possible in future versions of the ASR. The overall design of this project will be conducted by the Urban Institute (UI), along with two subcontractors, Research Support Services (RSS) and SSRS. SSRS will be responsible for selecting the study sample from the administrative data and updating contact information, which they will transmit to RSS. RSS will be recruiting participants, conducting cognitive interviews, and analyzing the results of the feasibility test.

B1. Respondent Universe and Sampling Methods

Target Population

Study participants will be drawn from administrative data, specifically an ORR database that captures all refugee arrivals, the Refugee Arrivals Data Systems (RADS).

This feasibility test will utilize a purposive sampling approach, selecting the sample from the same sampling frame as the 2022 ASR. Purposive sampling is a non-random sampling technique aimed at collecting information from the correct mix of respondents.

The initial sample will be of 600 potential test respondents (150 each who speak Arabic, Dari, Kiswahili, and Ukrainian) drawing from the 2022 ASR sampling frame (FY17-21 arrivals). As with the ASR, only principal applicants are sampled.¹ The following groups will not be present in the feasibility study sampling frame:

• Non-principal applicants

• Refugees who were already sampled for the FY2022 ASR (collected in spring 2023)

• Refugees who do not speak one of the four languages being tested (Arabic, Dari, Kiswahili, and Ukrainian)

• Refuges who cannot read well in their native language (this information is included in the RADS administrative data)

• Refugees who completed a prior year ASR (i.e. the FY2021 cycle collected in spring 2022)²

Once the sampling frame is complete, five cognitive interviews will be conducted per language while meeting the following conditions (one interview can satisfy multiple conditions):

• One interview with a male refugee

• One interview with a female refugee

• Three interviews where the family size is greater than 1

• One interview with a refugee 55 years of age or older

Four languages were chosen with input from ORR (see Table 1) which capture a range of geographic regions of origin, and include some of the most common languages among recent refugee arrivals and expected arrivals.

Table 1. Language Groups for ASR Online Feasibility Test

Language	Share in FY2021 admissions	Number of 2021 ASR households surveyed	Number of interviews planned	Countries of origin
Arabic	16%	125	5	Iraq, Syria, Sudan, Eritrea, and others
Kiswahili	16%	86	5	Democratic Republic of Congo and Burundi
Dari	4%	48	5	Afghanistan, Iran, and Pakistan
Ukrainian	4%	160	5	Ukraine and Moldova

Appropriateness of Study Design and Methods for Planned Uses

The information gathered in the feasibility test is not intended to be used as a basis for public policy decisions and is not expected to reach the threshold of influential or highly influential scientific information. Rather, it is intended as a test of whether an online mode for the ASR is likely to yield valuable information that may not be able to be obtained in the traditional telephone mode or can be collected at a lower cost.

B2. Procedures for Collection of Information

Data Collection Processes

RSS will recruit potential respondents through sending a mailed letter of introduction and making an initial telephone call. They will use addresses and telephone numbers from the administrative data that have been updated through standard tracing databases proven effective for prior ASRs. If an individual is deemed eligible and consents to participate in the cognitive interview, RSS will schedule a time for an interview. During the interview, RSS interviewers will observe respondents as they complete an online version of the ASR. This observation will be done virtually on a zoom call, as the interviewer observes the audio and video of the respondent, who will also be sharing their screen as they complete the online survey. Interviewers will take hand-written notes on how participants respond to survey questions, and any observations they have on their reactions as they take the survey. They will then ask respondents a series of questions about their experience completing specific questions and overall on the online survey, including ease or difficulty of using the online platform and question comprehensibility. Twenty total interviews will be conducted across four language groups and participants will receive a $40 token of appreciation as an incentive to encourage cooperation and participation in the feasibility test.

Data Handling

If an individual agrees to participate during the telephone recruitment and is deemed eligible, the interviewer will send a copy of the informed consent to their email address so that the participant has a copy they can refer to; if someone only provides a texting phone number, a hard copy of the consent language will be sent to their mailing address.

The project team will use a secure data file transfer process using Secure File Transfer Protocol (SFTP). The data received from HHS/ORR (RADS administrative data) is transmitted to Urban through an HHS SFTP. Subsequent transfers, including the RADS and ASR survey data, between UI, HHS, and SSRS are transmitted through the UI Secure Data Site, which is an on-premise SFTP server behind the UI firewall. The UI SFTP will be used to transmit sample data between SSRS and RSS, but UI will not download these data.

RSS will not transmit any individual-level records on the cognitive interviews back to SSRS or to UI; they will only be sharing de-identified interview summaries that include no information about individual participants other than age, gender, family size, and language. RSS will securely store in locked boxes, and then destroy, hand-written notes that are taken during the cognitive interviews.

In addition, SSRS will not transmit online survey responses to any partners; these data will not be analyzed.

Data Use

UI will determine whether the data from the test indicate that an online version of the ASR is feasible from both a technical and value perspective and will share those findings with ORR and other relevant parties.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Response Rates

Not applicable, as this feasibility test is limited to only 20 respondents by design.

Maximizing Response Rates and Dealing with Nonresponse

Not applicable.

B4. Tests of Procedures or Methods to be Undertaken

The purpose of this information collection request is to test a different method for data collection instruments and survey procedures. HHS expects that all qualitative and quantitative data gathered under this clearance will result in improved survey procedures and thus reduce respondent burden and increase data quality.

B5. Individual Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

HHS has contracted with UI to design the feasibility test, organize the data collection, and analyze the data. The Principal Investigator of the study at UI is Hamutal Bernstein, Senior Fellow, who can be reached at hbernstein@urban.org or 202-261-5840.

Attachments

• ASR survey protocol with cognitive interview probes

• Letter of introduction

• Telephone recruitment script

1 For an entering refugee family, there is a Principal Applicant (PA) whose refugee case is the basis for admission. This person is often the head of the household.

2 SSRS will flag refugees who completed a prior year ASR using the EIN number, which is a unique identification number assigned to all refugees.

6