Supporting Statement B - ASR Pretest

Pretest of the Annual Survey of Refugees

Information Collection Request

0970 - 0355

Supporting Statement

Part B

August 2017

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

ACF Point of Contact

Molly Jones

B1. Respondent Universe and Sampling Methods

The ASR Pretest utilizes two respondent universes. The first respondent universe is the collection of refugee Principal Applicants (PA), which represent a specific subset of all refugees entering the US. “Refugees” can be of any age; when they enter the US, many enter with their family members. For an entering refugee family, the Principal Applicant (PA) is the individual for whose refugee case is the basis for admission. The ASR surveys adult PAs who either came with family members or came to the US alone. PAs are typically the head of the household. The ASR Pretest will conduct pretest activities with adult PAs who entered the US between FY 2011-2015.

A second respondent universe for the ASR pretest will be community stakeholders from the two metro areas where site visits are being conducted. Community stakeholders refer to individuals who work with, regularly interact with and/or service refugees in their community. This includes but is not limited to persons associated with refugee resettlement service providers, community and faith-based organizations, non-refugee-specific service providers, school district and other municipal and state level officials, ethnic media, and others such as major refugee employers. Some flexibility is necessary in terms of the respondent definition because community stakeholders may differ by site, depending on the experiences of refugees in different communities.

Sampling refugees. Because this is a pre-test, a random sample of refugees is neither needed nor warranted. In fact, random sampling would be a detriment to the pre-test because of the need to ensure maximum diversity of the sample within existing budget and time constraints. Insights from the pretest are maximized through a sampling plan that captures the diversity of refugees and their integration experiences. This will be accomplished using a full-factorial design matrix that incorporates four factors related to survey understanding and participation, self-sufficiency and integration: (a) Language Group – 6 cells (two dialects of Arabic, Nepali, Sgaw Karen, Somali, Kiswahili); (b) Recency of arrival – 2 cells (2-3 years after arrival; 4-6 years after arrival); (c) Sex – 2 cells (male, female); and (d) Family status – 2 cells (families at arrival, singles at arrival).

The justification of why these factors were used in the design matrix appears below:

Language Group: Language group variation is critical to inform survey administration (surveys are translated and administered in different languages), but it also correlates with other key factors such as country of origin, demographic characteristics, and cultural factors that are important for informing the research objectives above. We prioritized the largest language groups in most recent refugee inflows reported in the Worldwide Refugee Admissions Processing System (WRAPS) for FY2011-2015, and chose an additional language to maximize diversity of experience and represent a growing country of origin group, the Congolese. See Table 1 below for details on the languages chosen. Note that Arabic – due to its wide usage across countries and cultures – will be divided into two major dialects.

The questionnaire will be translated into the designated languages by a team of two linguists, a principal translator and a proof-reader, working in sequence to ensure we achieve the highest standard of quality, focusing on accuracy and the preservation of meaning. The linguists will have had several years of experience working as translators, and some previous experience translating in the subject matter of the project. Once delivered, the translation goes through additional quality assurance checks to guarantee nothing was missed.

Recency of Arrival: The amount of time that has passed since refugees' arrival in the U.S. is related to their self-sufficiency and integration outcomes, and is also related to operational issues like mobility and residential moves. Refugees who have been here for more time may be more

Table 1. Language Groups for ASR Pretest

Primary Language	Share among FY2011-2015 admissions	Number of 2016 ASR households surveyed	Number of Pretest Interviews Planned	Country of origin
Arabic	19%	458	32	Iraqi, Syrian and other refugees
Nepali	17%	254	24	Bhutanese refugees
Sgaw Karen	8%	89	24	Burmese refugees
Somali	10%	115	16	Somali refugees
Kiswahili	2%	36	16	Congolese refugees

familiar with U.S. culture and institutions and may respond to the ASR questionnaire differently. The 2016 ASR sample was stratified by cohort: FY2015 entrants (524 ASR respondents), FY2013-2014 entrants (475 ASR respondents), and FY2011-2012 entrants (501 ASR respondents).

Sex: Evidence suggests that self-sufficiency and integration should be understood in tandem with sex. 2016 ASR respondents included 427 women and 1073 men. Note the sex imbalance aligns with the distribution in the universe of Principal Applicants, from which the sample was drawn.

Family Status: Just over half of refugees arrive with no family members in their household (52% among FY2011-2015 admissions), while slightly under half come with family members and are resettled as a family unit. Processes of self-sufficiency and integration will vary across these experiences. Among the 2016 ASR respondents, 640 respondents arrived as single people; while 860 arrived with family members.

Refugee target sample sizes for the pretest. Table 2 presents the target number of completed survey interviews for each cell of the 6x2x2x2 = 48 cell design matrix. For each cell in Table 2 showing n=2 completed survey pretest interviews, one respondent would additionally receive a cognitive interview, and the other would additionally receive an in-depth interview. For each cell in Table 2 showing a target of n=3 completed pretest survey interviews, one of the respondents would additionally receive a cognitive interview, while the other two would receive an in-depth interview. Under this plan a total of 112 completed pretest survey interviews will be collected, 64 in-depth interviews will be completed, and 48 cognitive interviews. It will provide equal numbers of pretest interviews by sex, family status and recency of arrival.

Refugee sample selection. The sample of refugees will be drawn from the pool of respondents to the 2016 ASR - the most recent iteration of the survey which was conducted in the winter of 2017. The 2016 ASR employed a probability sample of PAs arriving in the U.S. between FY2010 and FY2015. The 2016 ASR respondents will be assembled into strata defined by the design matrix and sampled as needed to achieve the desired number of completed interviews for each cell. Response rate will not be an issue, as this is a pretest and substantive population estimates will not be made from these data.

This sampling plan reduces the costs of recruitment (given limited resources) thus allowing cognitive and in-depth interviews to be conducted and valuable insights to be generated for the ASR redesign. The advantages and disadvantages of this approach were discussed in Part A of this information collection request.

Table 2. ASR Pretest Survey Interviews Design

Recency of Arrival (Years After Arrival)	Language Group	Male		Female		Recency of Arrival Totals
		Families at arrival	Alone at arrival	Families at arrival	Alone at arrival
2-3 Years	Arabic (dialect 1)	2	2	2	2	56
	Arabic (dialect 2)	2	2	2	2
	Nepali	3	3	3	3
	Sgaw Karen	3	3	3	3
	Somali	2	2	2	2
	Kiswahili	2	2	2	2
4-6 Years	Arabic (dialect 1)	2	2	2	2	56
	Arabic (dialect 2)	2	2	2	2
	Nepali	3	3	3	3
	Sgaw Karen	3	3	3	3
	Somali	2	2	2	2
	Kiswahili	2	2	2	2
	Subtotals	56		56		112
	Grand total pretest surveys	112

Sampling stakeholders. The sample of community stakeholders will be subjective and may well vary by site. At the onset of the pretest project, outreach efforts will be made to the refugee resettlement agencies in each site. These exploratory communications will inform the types of stakeholders that would best provide insights into the refugee resettlement experience in each site. Staff from the refugee resettlement agencies will include those serving as directors/managers and those who interact directly with refugees. Conversations with refugee agencies will lead to additional outreach to other stakeholders including community and faith-based organizations, non-refugee-specific service providers, school district and other municipal and state level officials, ethnic media, and others such as major refugee employers.

In each site, we have planned for two focus groups of eight participants each, and ten in-depth interviews. This leads to four total focus groups and twenty in-depth interviews across the two sites. One focus group will be conducted with refugee resettlement staff, as well as two to three in-depth interviews. The remaining focus group and in-depth interviews will be devoted to community stakeholders who are not explicitly involved in resettlement agency service provision.

Data use. Finally, note that HHS will use the data collected to inform future ASR survey design and questionnaire development activities. The pretesting activities undertaken as part of this clearance involve purposive samples with respondents selected to cover a broad range of demographic subgroups and to include specific characteristics related to the topic of the survey.

B2. Procedures for Collection of Information

Pretest Survey Administration. After drawing respondents from 2016 ASR respondents, updated contact information will be collected following best practice learned by the Urban Institute in implementing the 2016 ASR. Tracing resources will include Transunion batch tracing and National Change of Address (NCOA) through the U.S. Postal Service. Survey interviewers will then commence calling potential pretest respondents, again following best practice based on 2016 ASR administration. Interviewers will be matched to respondents by sex, language, and (when feasible) national origin group. Survey administration will avoid religious holidays and other important calendar dates. To the extent possible, interviewers will be the same interviewers who had individually reached respondents in the 2016 ASR (conducted in 2017). This will build on personal trust and rapport established from the earlier survey experience. Respondents will be sent a $40 gift card as a thank you for participating after completion of the interview and either cognitive interview or in-depth interview.

Participation will require 2 hours per administration: 1 hour for the survey protocol, and an additional hour for either the cognitive interview or the in-depth interview. It is anticipated that an appointment and second call will be needed to conduct the in-depth interview or cognitive interview.

Interviewers will go through a process of securing oral informed consent from respondents, explaining the project, how their information will be used and kept private, and assuring respondents of the voluntary nature of their participation in the pretest (see consent language included in Attachment A of clearance package). When audio-recording is sought, explicit consent for that will be secured; if declined, data collection will rely on field staff notes.

All interviews, cognitive interviews, and in-depth interviews will be audio-recorded with permission of the respondent, for monitoring purposes. Moreover, in-depth interviews will be transcribed and translated into English. Notes from cognitive interviews will be translated into English and summarized.

The collected survey data will be entered into a data set to examine frequency distributions and missing data rates (don’t know and refusals). Estimates of survey administration timing will be generated overall and by design matrix factor categories. In combination with interviewer notes, the collected refugee qualitative and quantitative data will allow the identification of problematic questions, provide insight into respondent burden and suggest ways to promote survey participation. Transcribed and English-translated in-depth interview responses, English-translated cognitive interview notes, and interviewer feedback will be used to identify problems, solutions and recommendations for improving survey instruments and participation rates.

Site Visit Data Collection. Site visits to two metropolitan areas – Boise, ID and Washington, DC (greater metro area) – will collect the perspectives of service providers and stakeholders in communities where refugees live. Interviews and focus groups that we conduct will have two main goals: 1) collect perspectives on refugee self-sufficiency and integration from the ground, and 2) learn from service providers how to effectively reach and communicate with this population. A key goal is to understand how ASR survey administration might better leverage established relationships between refugees and service providers to reduce survey non-response and increase the cost-effectiveness and efficiency of administration.

Site visits will be used to learn what self-sufficiency and integration mean from the perspectives of service providers, stakeholders, and communities that host and interact with refugees day to day. Data collections will explore key factors for self-sufficiency and integration, including understanding of the greatest barriers and facilitators to self-sufficiency; the impact of pre-resettlement experiences; the role of the local community context; the role that children play for refugee parents and families; how long integration takes with different refugees; and other on-the-ground perspectives that may inform priorities for the ASR. We will collect the perspectives of service providers and stakeholders who interact with refugees day to day.

Site visit information will also be collected to help improve survey implementation. The site visit collection activities will reveal community actors who interact with refugees after initial resettlement services end; practices that can help to maintain updated contact information and effective outreach strategies; potential use of technology such as smart phones and the internet; sources of sensitivity for different groups; and refugee perceptions of government and government-sponsored surveys. The visits will also explore the possibility for service providers to provide information from administrative data and contact information for refugee clients and/or participate in ASR survey collection.

Site Visit Structure. Two site visits will be scheduled to locations with differing histories of refugee resettlement and refugee demographics. This approach will provide us sufficient variation in refugee experiences and stakeholder perspectives to fulfill the objectives of the pretest. One site will be the greater Washington, DC area, a large metropolitan region that has resettled successive groups of refugees over the past four decades, and is currently home to a large and diverse refugee community. Although the city of Washington, DC itself receives few refugees, the suburban areas are major refugee resettlement communities. Among recent arrivals, suburban Maryland and Virginia have received refugees from Bhutan, Congo, Iran, Iraq, Ethiopia, Syria, Afghanistan and Burma (Myanmar). Baltimore, MD also receives fairly large numbers of refugees; in the past, Southeast Asia, Bosnia, Ethiopia, Eritrea and other African nations have been main refugee source countries.

The second site will be Boise, Idaho, a medium-sized metropolitan area and a more recent destination for refugees. Compared to the DC area, Boise has had a different experience and history with refugee reception and institutional infrastructure. In the past five years, Boise has received the vast majority of Idaho’s refugees, with Congolese, Iraqi, Burmese and Somalis making up two-thirds of the recently resettled. According to a Brookings Institution study that analyzed new refugee arrivals between 1983 and 2004, Metropolitan Washington had received over 41,000 new refugees, while Boise resettled just over 5,000. But in the past five years, Boise has received 5,000 refugees for resettlement while the Washington region received approximately 7,000.

We will identify and set appointments with refugee resettlement service providers, community and faith-based organizations, non-refugee-specific service providers, school district and other municipal and state level officials, ethnic media, and other stakeholders such as major refugee employers at each site. Each site will be visited for up to three days by a pair of researchers who will conduct two focus groups and ten in-depth interviews with staff and stakeholders per site. Refugees will not be interviewed in these site visit interviews or focus groups, since they will be providing extensive information through one-on-one, non-English culturally appropriate interactions via the telephone interviews described above. Moreover, the research staff will not have access to interpreters, and the use of such third-party services is not conducive to the generation of high quality qualitative data.

Data collection instruments. Attachments A-E provide data collection instrumentation for the refugee pretest survey, refugee cognitive interview, refugee in-depth interview, stakeholder in-depth interview, and stakeholder focus group.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Expected Response Rates

Not applicable, as this is a pretest that will employ subjective sampling.

Maximizing Response Rates and Dealing with Nonresponse

Not applicable.

B4. Tests of Procedures or Methods to be Undertaken

The purpose of this information collection request is to pretest data collection instruments and survey procedures. HHS expects that all qualitative and quantitative data gathered under this clearance will result in improved questionnaires and procedures and thus reduce respondent burden and increase data quality.

B5. Individual Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

HHS has contracted with the Urban Institute to design the pretest survey and other data collection instruments, organize the data collection, and analyze the data. The Principal Investigators of the study at the Urban Institute are:

Robert Santos, Chief Methodologist

202-261-5904

rsantos@urban.org

Audrey Singer, Senior Fellow

202-261-5647

asinger@urban.org

7