SUPPORTING STATEMENT – PART B
Overseas Citizen Population Survey: 0704-TBD
Description of the Activity
The Federal Voting Assistance Program (FVAP) issued Request for Quote (RFQ – 881873) for research and analysis services to examine the demographic profile of registered overseas civilian voters, quantify the population of these voters by country, and calculate a voter registration and participation rate for the population using records from state and local election officials. The component of this research covered under this Information Collection submission pertains to a survey of the overseas U.S. civilian registered population, focused on voting-relevant characteristics such as educational attainment, income, and mobility. This data will allow FVAP to compare with greater certainty the voting behaviors of overseas registered U.S. civilians to those of the registered civilian voting age population (CVAP) and active duty military (ADM) population who share similar characteristics. The survey will be conducted with 45,000 registered voters who requested a ballot to an overseas address in the 2016 General Election. It is anticipated that the survey will yield a response rate of approximately 18%, for a total of 8,100 completed surveys, based on the 2015 OCPS and factoring in lower response rates due to the election type and contact methodology.
Procedures for the Collection of Information
The following activities will be conducted as part of this survey data collection effort:
Develop a survey of overseas citizens
Administer the survey
Manage received survey data
Analyze the survey data
FVAP will collect data from the overseas citizen population through a web only survey, with paper communication materials, of a sample of registered civilians who requested a ballot to an overseas address in the 2016 General Election. Conducting such a survey will allow FVAP to collect specific, accurate information on voting relevant demographic variables (such as age, gender, race/ethnicity, education, and mobility) to make the comparisons between the overseas, domestic, and ADM populations that are so important to FVAP’s mission. This collection will also provide important information on voting related behaviors that might help address complex and/or unexpected findings within and across countries. Specifically, this data will address overseas voter awareness of FVAP, use of and satisfaction with FVAP services and other resources that these voters use. The survey instrument is designed to parallel FVAP’s Post-Election Survey of ADM and the Census’ Current Population Survey (CPS), facilitating FVAP’s ability to compare the registration and voting behavior of the overseas U.S. civilian population, domestic CVAP, and ADM.
Though the length, expected burden, and topics have not substantially changed, the 2017 OCPS has substantively changed questions from the 2015 OCPS. A review of the 2015 OCPS analyzed the response distribution, time, and item non-response of each question. In general, questions in 2017 were evaluated to determine if they should be removed if they had distributions with little variability (ex: 90% or more of respondents select one response option). Others that appeared to take too long or had high item non-response were considered for revision. The overall changes primarily include removing computer and cell phone usage questions; removing questions about State websites; removing questions about sources of candidate/election issue information; and removing questions about previous years’ voting experience. Additionally, certain questions were added to address additional research questions and align survey questions with other surveys. These include adding social media usage questions; adding absentee voting process questions related to requesting, receiving, and returning a ballot; adding U.S. citizen social connection questions; and adding questions concerning elections for which the individual was outside the U.S./non-host country/non-U.S. address. Lastly, open-end questions at the end of the survey suggested that respondents had difficulty answering certain “double-barreled” questions, including the key absentee voting question. They noted that they knew how they had voted, but did not know the mode of voting, so they answer the question with “don’t know”. This, along with a sizable percentage on item non-response and do not know response on absentee voting questions, led to creating the treatment survey that varies based on the type of absentee ballot questions. Adding all of these questions will improve the accuracy and power of voting questions and provide necessary variables on citizenship, social media, and mobility that will be included in analyses of the overseas citizen population.
The survey will utilize a mixed-mode approach in which respondents will receive an initial mail contact directing them to a web site where they can complete an online survey. In 2015, respondents were mailed a paper survey if they had not responded online after the first several contacts. However, approximately 86 percent of respondents responded online, and there were no large and significant demographic, geographic, or measure survey response differences between online and paper responders. Given the significant cost involved in international mailing of survey packets, the 2017 survey will be web-only. Respondents who do not respond to the online survey will then be sent mail and email reminders. This mixed-mode design1 has significant benefits over soliciting potential respondents by email. Specifically, it ensures that all registered U.S. civilians living overseas have a known probability of being contacted and having the potential to participate, rather than just those with a listed email address; as a result, this increases the likelihood that the final sample of respondents will be more representative of this population with respect to Internet use/access. In addition, costs can be high for printing and mailing paper surveys, especially to an international population; pushing respondents to complete an online survey will lower costs while still capturing a representative sample of the overseas citizen population.2
The communication strategy will be a two-pronged approach involving six international mailings and two email messages per participant, for a total of 8 communications.
The physical materials sent to each voter’s overseas address will include:
One cover letter including the URL for the online survey.
Two reminder cover letters including the survey URL, inclusive of an 8.5x11 one-page graphic treatment communication.
Three standard postcards including the survey URL.
If these mailings do not result in a survey response or other contact, follow-up postcards may be sent, including sending a postcard to the individual’s domestic address of record. It is anticipated that this will increase the overall response rate by including responses from some individuals who were residing overseas during the 2016 General Election but have subsequently returned to the United States.
The email contacts will correspond to the physical mailings, using similar wording and design choices. It is expected that around 20% of the sample frame will have a valid email address. Those without a valid email address may receive a postcard instead of an email. Emails will include:
An initial email message including the URL for the online survey.
A second, similarly worded email message including the survey URL.
The physical and electronic mailings will be sent concurrently. The reminder contacts will be sent approximately once per week, with the day of the week varying for each contact and/or to reflect international holidays. Once a respondent has completed the survey or unsubscribed, no further communications will be sent.
Because the survey frame will not include email addresses for all overseas registered voters, the post survey analysis will appropriately weight the email and non-email samples to control for potential error introduced by this two-pronged approach. Using all means of contact available will help mitigate the effects of uncertain overseas mail delivery and other obstacles that may arise from trying to contact a population that has been historically difficult to survey.
The survey will be administered to a sample of 45,000 respondents. Each respondent will receive an invitation to participate in a web-based survey.
Estimation Procedures.
Sampling probabilities will be known at each stage of the sampling process and will be incorporated into the analytic weights used for estimation. Sampling design decisions, including the decision of how to allocate the sample across different domains, are key aspects of a study design and have meaningful implications for survey estimation, often involving competing goals and tradeoffs among options. The sampling plan for this study cannot be created yet because the survey frame consisting of absentee voters in the November 2016 general election has not yet been assembled. However, the main decisions and priorities are described below.
In this study, one pair of competing goals involves the desire to make estimates of sufficient precision at the country level, while also aiming to produce accurate estimates overall. Thus, although a proportional allocation of sample across countries could lead to more accurate estimates overall, it will be necessary to use disproportionate sampling in order to achieve necessary levels of precision for voters in smaller countries and world regions.
For this study, variables that will be used for stratification and/or in determining selection probabilities are geography (e.g., country of mailing address and State in which the voter is registered) and voter characteristics as measured on the voter file (e.g., demographics and vote history).
After the survey data has been collected, full-sample weights will be developed. At a minimum, these weights will account for differential selection and response rates across strata.
Sample weighting will accomplish the following objectives:
Compensate for differential probabilities of selection among sample members.
Reduce biases occurring because the characteristics of non-respondents may have been different from those of the respondents.
Improve the precision of the survey-based estimates.3
After data collection is complete, experienced project leaders will oversee and direct each analysis (including planning and execution) to ensure proper procedures are applied, utilize reasonable analytical assumptions, and apply the correct approach given the characteristics of the data and objectives of the analysis. All analysis will be repeated by a second analyst to ensure error-free results. Some of the approaches used will include:
Regression analysis (OLS, 2SLS, logit/probit, etc.).
Machine learning model building (CART).
Geographic analysis including thematic and LISA maps.
Degree of accuracy needed for the Purpose discussed in the justification. The goal of using the survey data is to support an analysis of voting rates similar to that applied to the comparison between the ADM and domestic civilian population using the Post-Election Voting Survey of Active Duty Military. For a comparison between overseas and domestic civilians to have a similar degree of reliability, the number of survey completes for this survey should be similar to the number of completes for the ADM Post-Election Survey. Consequently, the aim will be to obtain approximately 8,000 completed surveys. Assuming an 18% response rate this implies that approximately 45,000 total survey invitations should be sent during the survey fielding.
Unusual problems requiring specialized sampling procedures. Not applicable.
Use of periodic or cyclical data collections to reduce respondent burden. Not applicable.
Maximization of Response Rates, Non-response, and Reliability
Like all surveys, there is concern that non-response, both to the survey as a whole as well as to specific questions, will undermine the accuracy of estimated voting propensity of the overseas registered population, and thus the validity of any comparisons made with the CVAP. Low response rates could undermine the accuracy of estimates of participation propensity by increasing sampling variability and thus uncertainty in how well the sample reflects the greater population. This problem would be reflected in larger confidence intervals around the estimates. More problematic, if non-response is systematically related to unobserved demographic or geographic characteristics that are also relevant to voting, is that the estimated model of participation may be invalid even if the final sample is large. The final report detailing the findings of the 2017 OCPS will include a non-response bias analysis based on the sampling frame. Though methodological changes are not expected to have a significant effect on the overall response rate, the survey instrument was revised to maximize the total number of cases with valid responses on questions such as residency and mobility that are necessary for inclusion in key analyses of interest.
Tests of Procedures
Online survey logistics: The online survey will be thoroughly tested by employees of the contractor who will enter at least 20 test surveys through the web-based instrument. Should the data check reveal errors, necessary changes will be made to the web-based technology and the checks will be conducted again until 100% accuracy is achieved.
Communication strategy: The 2015 OCPS showed that an ideal survey fielding window is about three months, which will allow respondents with less Internet access or in hard to reach mailing geographies enough time to respond to the survey.
Statistical Consultation and Information Analysis
Provide names and telephone number of individual(s) consulted on statistical aspects of the design.
Dr. Brian Griepentrog
Director
of Research
Fors Marsh Group
Phone:
571.858.3798
Email: bg@forsmarshgroup.com
Provide name and organization of person(s) who will actually collect and analyze the collected information.
Dr. Brian Griepentrog
Director
of Research
Fors Marsh Group
Phone:
571.858.3798
Email: bg@forsmarshgroup.com
1Lonna Rae Atkeson, Alex N. Adams, and R. Michael Alvarez, “Nonresponse and Mode Effects in Self- and Interviewer-Administered Surveys”, Political Analysis, published online May 28, 2014, doi: 10.1093/pan/mpt049.
2 Approximately 84% of participants in the 2014 OCPS completed their survey online and there were minimal differences in responses between modes.
3Skinner, C. J., Holt, D. & Smith, T. M. (Eds.). (1989). Analysis of complex surveys. New York: Wiley.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Patricia Toppings |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |