Supporting Statement A - ASR Pretest

Pretest of the Annual Survey of Refugees

OMB Information Collection Request

0970 - 0355

Supporting Statement

Part A

August 2017

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

ACF Point of Contact:

Molly Jones

A1. Necessity for the Data Collection

This information collection request is to conduct a pretest of the Annual Survey of Refugees (ASR), to be conducted by the Urban Institute under contract with the Office of the Assistant Secretary for Planning and Evaluation (ASPE), U.S. Department of Health and Human Services (HHS), in partnership with the Office of Planning, Research and Evaluation (OPRE) and the Office of Refugee Resettlement (ORR) under the Administration for Children and Families (ACF), HHS. This request seeks approval to pretest a revised ASR with 112 refugees and refugee service providers in two sites in order to identify questionnaire and procedural problems, suggest solutions, and measure the relative effectiveness of alternative solutions.

The pretest is necessary to assess the effectiveness of the revised survey instrument, supporting survey materials, and field protocols for capturing high quality data from this special population.

Study Background

The Office of Refugee Resettlement (ORR) is charged with helping refugees become effectively resettled in the U.S., with the primary goal of rapid self-sufficiency and adjustment. Since 1975, when Southeast Asian refugees first began arriving in the United States, ORR (and its predecessors) has commissioned periodic surveys of the refugee population. A primary purpose of the Annual Survey of Refugees has been to help inform a statutory reporting requirement to Congress. The Refugee Act of 1980 (Section 413(a) of the Immigration and Nationality Act) requires the Secretary of Health and Human Services to submit an Annual Report to Congress that "shall contain an updated profile of the employment and labor force statistics for refugees who have entered the United States within the five fiscal-year period immediately preceding the fiscal year within which the report is to be made…as well as a description of the extent to which refugees received the forms of assistance or services under this chapter during that period."

Since the 1980s the ASR has captured data on the economic adjustment of refugees entering the United States. The focus of the survey has been on income, employment and labor force data, English language ability, education, and the utilization of government services. Aggregate findings of the survey data appear in the Office of Refugee Resettlement’s Annual Report to Congress and are available to the general public. The findings are used in combination with federal administrative data (which are not a part of the ASR) to facilitate an overall assessment of the refugee resettlement program’s success in effectively resettling refugees, and for purposes of program planning, policy-making, and future budgeting processes.

The ASR was last updated in 1993, when the survey population was expanded from just Southeast Asian refugees to include all refugees. A recent review by HHS revealed a number of concerns about the design, implementation, and performance of the ASR. Shortcomings included an outdated questionnaire, an ineffective sampling approach, weighting errors, ineffective tracing methods, insufficient documentation, and limited language coverage. These concerns led to an initiative to take a fresh look at the ASR, its goals and measures, and its design and methods. Because of this, ASPE – in partnership with OPRE and ORR – is spearheading a redesign of the ASR based substantially on this pretest of proposed new survey questions and methods that is the subject of this information collection request.

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. This pretest data collection is being taken at the discretion of the agency.

A2. Purpose of Survey Pretest and Data Collection Procedures

Overview of Purpose and Approach

The objectives of this pretest are to identify questionnaire and procedural problems and to identify solutions to those problems. This includes revising the questionnaire so that respondents can more accurately comprehend and respond to questions, and interviewers can more easily administer it, reducing burden, increasing response rate, and improving the overall quality and efficiency of the survey data collected. HHS will use the pretest data to refine the ASR questionnaire and field procedures as part of finalization of an instrument to be submitted to OMB for approval for use in the annual data collection. Assessment of the questionnaire and procedures is the main objective of the activities in this clearance.

HHS will use the pretest findings internally to inform the ASR redesign effort as well as subsequent ASR information collection requests. Results of the pretest may be made public through methodological appendices or footnotes, reports on instrument development, instrument user guides, descriptions of respondent behavior, and other publications describing findings of methodological interest. As appropriate, pretest results will be labeled as exploratory in nature. The pretest data collected will not be presented as findings on the outcomes of refugee resettlement or as generally representative of the refugee population.

The approach involves two collection components – one involving refugees, and the other involving individuals who interact with refugees in the receiving communities. The refugee component calls for collecting pretest survey interviews with refugees using a revised survey questionnaire and a specific factorial design. Additionally, these interviewed refugees would also undergo either cognitive interviewing or an in-depth interview (but not both). In the second component, two site visits would be conducted in cities with refugee populations. Resettlement agency staff and other community stakeholders would undergo in-depth interviews about their perspectives on refugee experiences to add their viewpoint and validate the constructs being captured in the survey instrument.

All survey data collected from the pretest with refugees will be generated from purposive samples of refugees admitted to the U.S. between FY2011 and FY2015. Qualitative data will be collected from both refugees (via in-depth interviews and cognitive interviews) and stakeholders such as staff from resettlement agencies in two sites. All information collection activities conducted under this generic clearance will be voluntary and low burden on the respondents.

Research Questions

The ASR pretest – the subject of this information collection request – will produce both conceptual and operational information to help inform decisions about a redesigned ASR. Table 1 details how the pretest components will cover the specific research objectives.

Study Design

The study design was developed to maximize insights within the constraints of the limited budget and time line. The pre-test plan includes two design components: (1) telephone interviews of refugee respondents testing the revised instrument and capturing the experience of being administered the revised survey instrument as well as capturing refugee self-sufficiency and integration experiences; and (2) site visits to conduct qualitative interviews with service providers and community stakeholders to understand priority topics for refugee self-sufficiency and integration, and effective strategies to engage refugee respondents and effectively implement the ASR.

Table 1. Pretest Research Objectives and Sources

Research objectives	Telephone Interviews of Refugees	Site Visits with Agency Staff & Stakeholders
Verify that the survey instrument captures accurate policy- and community-relevant information	X	X
Identify survey questions in need of revision due to structural or conceptual issues	X
Validate comprehension and assess respondent cognitive burden	X
Measure the length and cost of administration for various languages and refugee subgroup	X
Explore cultural relevance and sensitivity issues	X	X
Explore mobility, locating, and other issues related to longitudinal survey implementation	X	X
Explore ways of securing better participation and acceptance	X	X
Explore how government sponsorship might affect participation, trust, anxiety, veracity	X	X

Refugee Collection Design Component. Pretest interviews are to be completed with 112 refugees following a factorial design which establishes quotas for interviews by language, year of arrival, sex and family status. See Part B for the explicit design matrix and sample size allocation to cells. The rationale is to capture the variation in survey administrative experience across factors that influence length of interview, cultural sensitivities, and self-sufficiency and integration trajectories.

Under this design, equal samples of 56 will be allocated to: males vs females, to families vs single entrants, and to entrants who arrived 2-3 years ago vs 4-6 years ago. With regard to languages, three of the five languages will be allocated 16 interviews (Somali, Kiswahili, and 2 dialects of Arabic), while 24 interviews each will be allocated to Nepali and Sgaw Karen. No English interviews will be pretested since this language is seldom used in the ASR.

The refugee pretest sample will be drawn from a list of the most recent ASR participants, conducted in 2017. A random sample is neither necessary nor desired given the research objectives of this pretest. The costs of tracing and securing participation from a fresh sample would be prohibitive and the information collected from this more expensive approach would not be expected to provide better quality data such as timing of survey administration or comprehension of the questions relative to the proposed sampling approach.

Data will be collected through three methods: a questionnaire, cognitive interviews and in-depth interviews. (See Attachments A, B, and C.) Each are discussed below, in turn.

Questionnaire. The data we gather from the survey administration will be critical to the ASR redesign process. The pretest will allow a concise calculation of respondent burden and identify problematic questions (e.g., question wording, cultural relevance or sensitivity, skip patterns, reporting on other household members).

Cognitive interview. The cognitive interviews will provide an understanding of the specific questions or concepts that respondents find difficult to comprehend or handle. The cognitive interviews will provide feedback on ways to better capture information being requested by understanding how this population thinks about the measures being requested in the survey.

In-depth interview. The in-depth interviews will allow refugees to tell their own self-sufficiency and integration stories and thus inform us as to what may be missing in the revised questionnaire and suggest recommendations for a future ASR. They will also help validate whether or not the information we are collecting is accurate and meaningful to the respondent.

The in-depth interviews will explore refugee reactions to various outreach strategies intended to improve participation (e.g., advance letter, postcard, phone calls, incentives), including potential different modes of survey administration (e.g., internet/smart phone, telephone). Finally, the in-depth interviews will help explore the extent to which government survey sponsorship might have an effect on refugee likelihood to participate and shed light on their perspectives of such government-sponsored surveys. Insights will also be gained on who refugees regularly communicate with and trust. This will help to inform future ASR outreach strategy.

Site visit design component. Two site visits will be used to collect perspectives of service providers and stakeholders in communities where refugees live. Interviews and focus groups will be conducted to collect perspectives on refugee self-sufficiency and integration and to gather insights about how to more effectively reach and communicate with refugees. A key objective of the site visits is to understand how ASR survey administration might better leverage established relationships between refugees and service providers to reduce survey non-response and thereby increase the cost-effectiveness and efficiency of administration.

On the conceptual side, the qualitative data collection from the site visits will facilitate a better understanding of what self-sufficiency and integration mean from the perspectives of service providers, stakeholders, and communities who host and interact with refugees on a day-to-day basis. The site visit conversations (both focus groups and in-depth interviews; see Attachments D and E) will explore key factors for self-sufficiency and integration, including understanding of the greatest barriers and facilitators to self-sufficiency; the impact of pre-resettlement experiences; the role of the local community context; the role that children play for refugee parents and families; how long integration takes with different refugees; and other on-the-ground perspectives that may inform priorities for the ASR.

On the operational side, the site visit conversations are also expected to reveal insights about the points of contact community actors may have with refugees after their initial resettlement services end; practices that could help maintain updated contact information and effective outreach strategies; potential use of technology such as smart phones and the internet; sources of sensitivity for different groups; and refugee perceptions of government and other community stakeholders. The visits will also obtain information about the capacity of service providers to provide information from administrative data and contact information available for refugee clients and/or participate in ASR survey collection.

For this qualitative component, one site will be in the greater Washington, DC metropolitan area (including DC suburbs in MD and VA as well as Baltimore), which is home to a large and diverse refugee community. Among recent arrivals, suburban Maryland and Virginia have received refugees from Bhutan, Congo, Iran, Iraq, Ethiopia, Syria, Afghanistan and Burma (Myanmar). Baltimore, MD also received refugees from Southeast Asia, Bosnia, Ethiopia, Eritrea and other African nations. The second site will be Boise, Idaho, a medium-sized metropolitan area and a more recent destination for refugees. Boise has had a different experience and history regarding refugee reception and institutional infrastructure. In the past 5 years, Boise has received the vast majority of Idaho’s refugees, with Somalis, Congolese, Burmese and Iraqi making up two-thirds of the recently resettled. It is important to capture the perspectives of resettlement staff and stakeholders from different geographic locations, as regional/local U.S. communities and economies can affect refugees’ paths to self-sufficiency.

Site visits will provide contact with service-providers and community stakeholders who interact with refugees regularly. Focus groups and in-depth interviews will be vehicles for participants to share reflections on cultural sensitivities, experiences and issues related to refugee resettlement that can inform ASR operations, as well as questionnaire revisions. This includes their experiences maintaining contact with refugees, how their experience is similar or different from other populations, and their perceptions about best contact/communication strategies related to technology (e.g., texting on cell phones). Site visit qualitative data will be used to inform strategies for approaching refugees (e.g., incentives, mail or email communication). The visits will gauge providers’ interests and capacities to support ASR data collection (e.g. through onboarding of participants or disseminating information about the legitimacy and utility of the ASR) and identify trusted organizations that may serve as effective intermediaries to reach refugee respondents.

Qualitative data from the site visits will provide a useful window into refugee integration and self-sufficiency from the perspective of the service provider staff and the receiving community. It will be used to identify barriers, facilitators and mediating factors that may be different from those recognized by refugees. The qualitative data collections will also provide feedback on perceived ASR priorities, and how to better allow stakeholder input to and support of the ASR.

Balancing insight and limitations, subject to resources. This pretest is being conducted to support redesign work of the ASR. The pretest will field and evaluate a refined survey instrument, validate measures being captured, measure respondent burden, and improve strategies to maximize survey participation. Yet resources are limited. The proposed design leverages available resources to address the research objectives, although some limitations are unavoidable and noted here. On the one hand, advantages of this design are maximization of pretest sample size given the cost. This is achieved by using a subjective sample of refugees from the pool of respondents who participated in the 2017 ASR. These individuals have been found/located, contacted, and were willing to participate in the ASR about 8 months prior to the scheduled pretest administration. As such, the per unit costs of locating them and securing their participation (a key driver of survey costs) will be substantially reduced. Moreover, the most recent ASR survey interviewers would be used in this pretest. These interviewers have already gained the trust and acceptance of the pretest subjects, which is another barrier to survey participation. This should engender more truthful and thorough responses from pretest participants to the survey, the cognitive interviews and the in-depth interviews. The period of time between the ASR and the pretest administrations is sufficiently long that the timing of survey administration and the cognitive interviewing measures should not be adversely affected; that is, the respondents will not remember the previous ASR question wording well enough to measurably affect pretest administration in terms of how long it takes to complete the revised survey or comprehension of questions and terms.

On the other hand, a disadvantage with this sampling frame is the absence of more recent arrivals to the U.S., specifically those who arrived within the last two years. Use of the most recent ASR participants in the pretest limits refugees to those arriving to the U.S. between two and six years before survey administration. The risk is that refugees arriving one year ago may comprehend survey questions differently than refugees who arrived in the U.S. two or more years ago. This risk is acknowledged, but expected to be low because human comprehension does not change dramatically from one year to the next (absent of a medical/health condition). Also, note that the accuracy of retrospective response is of no consequence since the objective is assessing comprehension, not recall. Moreover, the site visits will capture the perspectives of service providers and stakeholders on the experiences of first-year and more recent arrivals.

A second disadvantage of re-interviewing the 2016 ASR respondents is that the ASR nonresponders may be systematically different from those who did participate. It may be that those respondents who either were not able to be traced for current contact information or who were located but declined to participate may be systematically different from ASR survey participants in key ways: they may have low literacy, complicated family or employment situations, or other features that make them harder to reach. A Chi-square Automatic Interaction Detector (CHAID) analysis of nonresponse to the most recent ASR suggested that nonresponse bias could be mitigated via weighting adjustments using information on the sampling frame. These factors were built into the pretest design matrix for sampling, so it is expected that the threat of the nonrespondent risk for the pretest will be mitigated. Moreover, the site visits will help fill some of those gaps, as we hear from service providers about strategies they use to contact and communicate with the hardest to reach refugees.

Universe of Data Collection Efforts

Data collection efforts under this information collection request will involve a combination of survey pretest interviews, cognitive interviews and in-depth interviews from refugees, as well as in-depth interviews of resettlement agency staff and community stakeholders. These are detailed below.

Pretest interviews of refugees. A key component of the pretest study design is the use of pretest survey interviews. A revised questionnaire has been developed to reduce cognitive and time burden on the survey respondents. This pretest will yield estimates of the timing of questionnaire administration as well as the identification of problems with specific questions or concepts. The questionnaire will be translated into five languages (including two dialects) that have high prevalence among the pool of refugees arriving in the U.S. to capture variation in timing and comprehension across a wide array of refugees.

The revised survey instrument to be pretested appears in Attachment A. A number of questions in the survey questionnaire are from the current Annual Survey of Refugees (see Attachment A, in which these questions appear in emboldened text), with minor revision to reduce cognitive burden and improve validity. New survey questions were drawn from a variety of validated reference surveys, with modifications and improvements made to language to customize for the target population of refugees. The reference surveys for these new questions include:

• American Community Survey (Census Bureau)

• Current Population Survey (Census Bureau)

• American Housing Survey (Census Bureau)

• National Health Interview Survey (Centers for Disease Control and Prevention)

• Behavioral Risk Factor Surveillance System (Centers for Disease Control and Prevention)

• National Household Education Survey (Department of Education)

• National Agricultural Workers Survey (Department of Labor)

• Settlement Outcomes of New Arrivals (Australia Department of Immigrant and Border Protection)

• Longitudinal Survey of Immigrants to Canada (Statistics Canada)

• DC Promise Neighborhood Initiative Survey (Urban Institute)

• New Immigrant Survey (Princeton University)

• Children of Immigrants Longitudinal Survey (Portes and Rumbaut 1991-2006)

• Survey of Refugee Integration (Northwestern University)

• Refugee Integration Survey and Evaluation (Colorado Department of Human Services)

• Survey of College-Educated Immigrants in the United States (IMPRINT and World Education Services)

• Refugee Integration Scale (Beversluis et al. 2017)

Cognitive interviews of refugees. After completing the survey, roughly 60% of the refugee pretest participants will undergo cognitive interviews. These will provide insights into cultural and cognitive issues associated with specific questions in the revised survey instrument and will be useful for identifying more easily understandable language or terminology to capture the information being requested in the survey.

The cognitive interview protocol is provided in Attachment B.

In-depth interviews of refugees. The remainder of the pretest participants (i.e., those not doing the cognitive interview) will undertake an in-depth semi-structured interview. In-depth interviews will be audio-recorded (with permission of the respondent), transcribed, and translated into English.

The in-depth refugee interview protocol is provided in Attachment C.

In-depth interviews and focus groups of refugee resettlement agency staff and community stakeholders. Focus groups and in-depth interviews will be conducted in two sites with refugee resettlement service providers, community and faith-based organizations, non-refugee-specific service providers, school district and other municipal and state level officials, ethnic media, and other stakeholders such as major refugee employers at each site. This qualitative data collection will collect information from service providers and capture community perspectives and experiences about refugee integration and self-sufficiency, cultural sensitivities, communication and engagement strategies, and perspectives on barriers and facilitators to resettlement.

The in-depth interview protocol and focus group protocol are designed to elicit similar information from these different respondents and are provided in Attachments D and E, respectively.

A3. Improved Information Technology to Reduce Burden

This pretest is intended to collect information about how refugees navigate orally administered pretest survey questions in terms of comprehension, burden and timing. It is therefore unnecessary and would be highly inefficient from both cost and labor perspectives to program the pretest survey instrument into a Computer Assisted Telephone Interview system. Nor is this feasible given the available resources and schedule. Interviews will be completed in hard copy using experienced interviewers who have participated in the most recent ASR.

The qualitative information being gathered will be collected via written notes onto laptops and will be audio-recorded and transcribed (and translated into English when collected in a non-English language).

A4. Efforts to Identify Duplication

No similar data are available from other sources. There are no available national data on refugee self-sufficiency and integration in the U.S over the five-year period after arrival. This work provides critical information to help ORR respond to its statutorily required Annual Report to Congress, and this pretest information collection request is part of HHS’ efforts to explore ways to collect accurate, relevant and statistically valid data in future ASR iterations.

A5. Involvement of Small Organizations

There is some involvement of small organizations under this request. In-depth interviews and/or focus groups will be conducted with refugee resettlement offices, local community based organizations that serve refugees, and other stakeholder organizations such as small businesses that hire refugees, representatives from local churches patronized by refugees, and so on. All data collection involving small organizations will be scheduled for the convenience of the participants.

A6. Consequences of Less Frequent Data Collection

As a request to conduct a pretest, this issue is not applicable. Pretests of this importance and magnitude are conducted occasionally. The last redesign of the ASR occurred in 1993.

A7. Special Circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978 August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on September 15, 2014, (vol. 79, no. 178, p. 54985) and provided a 60-day period for public comment. ACF did not receive any comments in response to this notice.

Consultation with Experts Outside of the Study

A two-day roundtable convening of forty outside experts in refugee research and survey design was conducted to review the existing ASR survey design, survey instrumentation and existing data resources. In addition to a number of federal government staff, the list of attendees included the following:

• Abigail Fisher Williamson, Trinity College

• Aryah Somers Landsberger, Grantmakers Concerned with Immigrants and Refugees (GCIR)

• David Kallick, Fiscal Policy Institute (FPI)

• Devin Keithley, Researcher on Central Ohio Refugee Report

• Dina Birman, University of Miami

• Duncan Lawrence, Stanford Immigration Policy Lab

• Erol Kekic, Church World Service

• Galya Ruffer, Northwestern University, Center for Forced Migration Studies

• Graeme Rodgers, International Rescue Committee

• Jennifer Cochran, Massachusetts Department of Public Health

• Jeremy Ferwerda, Dartmouth College and Stanford Immigration Policy Lab

• Kay Bellor, Lutheran Immigration and Refugee Services

• Larry Yungk, United Nations High Commissioner for Refugees (UNHCR)

• Lyn Morland, McGill University

• Margaret Gibbon, International Rescue Committee

• Melanie Nezer, Hebrew Immigrant Aid Society

• Mike Mitchell, Hebrew Immigrant Aid Society

• Ngoan Le, Illinois State Refugee Coordinator

• Paul Stein, Independent Consultant and Former Colorado State Refugee Coordinator

• Phillip Connor, Pew Research Center

• Rachel Peric, Welcoming America

• Rami Arafah, Harder+Company

• Randy Capps, Migration Policy Institute

• Rebecca Hailemeskel, Ethiopian Community Development Council

• Robin Koralek, Abt Associates

• Sam Elkin, MEF Associates

• Sandra Vines, International Rescue Committee

• Sanja Bebic, Center for Applied Linguistics

• Tsehaya Teferra, Ethiopian Community Development Council

Findings from the expert input were used to revise the survey instrument to be pretested. Discussions with the experts also impacted the design of the pretest, including the use of site visits to gather stakeholder feedback.

A9. Incentives for Respondents

Refugee pretest participants are being asked to participate in one of two sets of activities:

• A survey pretest interview, followed by a cognitive interview; or

• A survey pretest interview, followed by an in-depth interview.

Given the burden of the survey administration (1 hour) as well as the request for a second collection (a cognitive interview or an in-depth interview, each expected to entail 1 hour), an incentive is conventional for such research activity and will be provided. Note that the use of incentives of this size are conventional. For instance, a $45 incentive was used in a U.S. Department of Education study involving cognitive interviews.

(See: https://nces.ed.gov/surveys/gemena/pdf/FINAL_ATES_cog_int_report_Dec_18_2013.pdf )

More importantly, participant incentives are needed for both quality and cost reasons. The pretest design is complex and utilizes a full factorial design with limited numbers of potential respondents within cells. It is important to secure the participation of refugees from those design cells and incentives will facilitate securing cooperation. Failing to achieve those cell sizes will reduce the insight and utility of the pretest. The costs of recruitment will increase considerably if no incentive is offered given the burden being asked of participants. The reason is that many more attempts at our reach to additional subjects will be required to secure a cooperating respondent.

Refugee pretest participants will be sent a $40 gift card as a thank you for participating after completion of the interview and either cognitive interview or in-depth interview. The pretest survey and implementation plan will be reviewed and approved by the Urban Institute’s Institutional Review Board prior to launching the field data collection.

Incentives will not be provided to stakeholders (refugee resettlement agency staff, community representatives) under this information collection request for their participation in either focus groups or in-depth interviews.

A10. Privacy of Respondents

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. Explicit consent will be obtained for audio recording being requested for refugee surveys, cognitive interviews and in-depth interviews; if declined, the collection will proceed with only note taking by the field staff. Site visit focus groups and in-depth interviews will not be audio recorded.

As specified in the contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed and will abide by a Data Safety and Monitoring Plan that assesses all protections of respondents’ personally identifiable information. A Human Subjects Protection plan will be reviewed and approved by the Contractor’s Institutional Review Board (IRB) prior to the commencement of data collection. The Contractor will ensure that employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.

As specified in the evaluator’s contract, the Contractor shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard.  The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor will submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.  All audio recordings will be destroyed at the end of the pretest study.

A11. Sensitive Questions

There are no sensitive questions in this data collection in terms of divulging risky or illicit behaviors. However, some refugees may consider some questions to be sensitive, such as questions about experiences of flight from their home countries, income and barriers to employment, health, and the experiences of their children. To the extent that these are perceived as sensitive by this population, interviewers have been trained to reassure subjects of the privacy of their reported data under federal law.

A12. Estimation of Information Collection Burden

We request approval of 224 total burden hours for refugees taking the pretest survey and cognitive or in-depth interview. We expect the pretest survey plus a cognitive or in-depth interview will take 2 hours for each of the 112 refugee respondents. We also request approval of 52 total burden hours for the focus groups (32 hours) and in-depth interviews (20 hours) at the two sites (Washington, DC, and Boise, ID).

Total Annual Cost

The total participant cost of this pretest across refugees and stakeholders is estimated to be $3,832.

Refugee Cost: The cost to refugees was determined by multiplying the total annual burden hours of refugees (Rows 1 and 4 of Column D) (128 + 96 = 224) by the average hourly wage of refugees ($12) (Column E) based data on wages among refugees 18 or over who are employed from the 2017 Annual Survey of Refugees. Thus, the total cost associated with refugee burden is (224 x $12) = $2,688.

Stakeholder Cost: For Stakeholders, the total burden hours (Rows 7 and 8 of Column D) (32+20 = 52) was multiplied by the July 2017 US average hourly wage of $22 (Column E) to produce a total Stakeholder cost of (52 x $22) = $1,144.

Summing the Refugee and Stakeholder costs yields the total annual cost: $2,688 + $ 1,144 = $3,832.

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

Newly Requested Information Collections

Total Burden Requested Under this Information Collection

		A	B	C	D	E	F
Reference Document	Instrument	Total/Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage*	Total Annual Cost
	Pretest Survey & Cognitive Interview	64	2	1	128	$12	$1,536
Attachment A	Pretest Survey	64	1	1	64	$12
Attachment B	Cognitive Interview	64	1	1	64	$12
	Pretest Survey & In-Depth Interview	48	2	1	96	$12	$1,152
Attachment A	Pretest Survey	48	1	1	48	$12
Attachment C	In-Depth Interview	48	1	1	48	$12
Attachment D	Stakeholder Focus Group	32	1	1	32	$22	$704
Attachment E	Stakeholder In-Depth Interview	20	1	1	20	$22	$440
	Estimated Annual Burden Total				276		$3,832
	* For Refugees: Rounded 2017 ASR Mean Hourly Wages Earned at Current Job for Refugees 18 or older who are employed
	For Stakeholders: Rounded July 2017 Average Hourly Wages in U.S. workers -- Source: BLS

A14. Estimate of Cost to the Federal Government

The total cost for the pretest data collection activities (including respondent incentives) is funded by a government contract totaling $ 242,358.

There will be no annual costs to the Federal government because this information request represents an ad hoc pretest to facilitate the redesign of the ASR.

A15. Change in Burden

This request is for additional data collection under generic clearance 0970-0355.

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

Analysis Plan

Statistical estimation is not intended with the pretest survey data. The data will be subject to an examination of frequency distributions to assess variation in responses as well as item missing data (Don’t know & refusal responses). Statistics will be generated on the average time to administer the survey. The mean administration times will be calculated overall as well as by language and by family status.

Cognitive interview notes will be reviewed for problematic questions and in-depth interviews will be assessed to validate existing constructs as well as to explore new aspects of measuring economic self-sufficiency and societal integration. These data will also be used to explore how to improve locating respondents and encourage better participation in the ASR.

Similarly, qualitative analysis will be conducted on the text data from in-depth interviews and focus groups of Stakeholders. This may (or may not) lead to suggestions for additional survey questions for refugees. The data will also be used to develop improved strategies for locating and securing refugee survey participation.

Time Schedule & Publications

The current project Schedule for the pretest appears In Figure 1 below. An internal-only memorandum of findings and recommendations will be produced. Results of the pretest may be made public through methodological appendices or footnotes, reports on instrument development, instrument user guides, summary descriptions of respondent behavior, and other publications describing findings of methodological interest. As appropriate, pretest results will be labeled as exploratory in nature. The pretest data collected will not be presented as findings on the outcomes of refugee resettlement or as generally representative of the refugee population.

Figure 1: Project Schedule for the ASR Pretest*

*Schedule is dependent on the date of OMB approval.

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

15