Incentive Lit Review

Evaluation of the Fresh Empire Campaign on Tobacco (EFECT)

INCENTIVES

Attachment 4: Incentives

Incentive Plan for the EVALUATION OF THE Fresh Empire Campaign ON TOBACCO

This section presents the rationale for the incentives we propose for participation in Fresh Empire surveys. Incentives will be provided to encourage completion of the 5 minute screener survey ($2), the 30 minute pre-campaign launch survey ($25) and the 45 minute post-campaign launch surveys ($25). The study design is intended to produce cross-sectional samples for the pre- and post-campaign launch surveys.

The incentive plan for this study is based on the need to ensure cooperative from this hard-to-reach and specific population of multicultural youth who affiliate with a Hip Hop peer crowd (Beebe et al, 2005). Proposed incentives are intended to recognize the time burden placed on participants, encourage their cooperation, and convey appreciation for contributing to this important study.

Relevant research on use of Incentives

A large body of empirical research examines the role of incentives in survey outcomes. Experiments compare the effects of different incentive characteristics (amount, timing and type) on outcomes such as response rate, attrition, sample composition, item nonresponse and measurement error. Investigators have also documented interactions between incentive and survey characteristics. Findings from this body of literature are to a large degree specific to particular study designs, and are not all relevant in the case of the Fresh Empire evaluation. For example, there is a robust body of evidence related to longitudinal surveys which are not particularly relevant given our evaluation plan consists of repeated cross-sectional surveys. This document, therefore, focuses on incentive characteristics that have been demonstrated to influence response to cross-sectional surveys, with a particular focus on mail surveys.

Overarching Finding

Evidence Supports Use of Incentives to Increase Response Rates and Reduce Costs

Studies indicate that pre-paid and promised incentives, used alone and in combination, significantly improve response rates. This finding is particularly persuasive for pre-paid incentives in one-phase studies and screener questionnaires. Studies demonstrate that incentives can be used to improve sample composition by increasing participation among racial/ethnic minority and lower SES individuals. Studies also show that incentives reduce the cost per survey completed through their influence on the amount of effort that is required to achieve a completed interview. For these reasons and because of the large body of evidence supporting these findings, incentives have been supported in many OMB-approved information collection efforts.

Detailed Findings

Evidence for the Effectiveness of Pre-Paid Incentives to Complete Surveys by Mail

Empirical research has demonstrated that the inclusion of pre-paid incentives in mail surveys with both address-based and list frame samples can lead to higher response rates, and reduce nonresponse bias. For example, in 2001, when the National Household Travel Survey faced increasing response rate challenges, the study included incentives in a mail survey (Freedman, et al). A prenotification letter to potential respondents was accompanied by a $5 incentive for most sample members. Following this letter, a telephone call was placed to the household to administer a household survey. That call was followed by a travel survey diary, which included $2 for each member of the household. Despite a trend of declining response rates, the 2001 NTHS’s response rate was 4 percentage points higher than the 1995 survey. Experimental findings from a National Household Education Survey pilot test support these findings, although in this case the outcome of interest was the effect of two incentive amounts on response rate to a mailed survey screener (Han et al 2012). The $5 incentive resulted in a significant increase in response rates compared to the $2 incentive (70.9% v 66.5%). Similarly, a 2009 study of recent veterans (Coughlin et al 2011) found that a $5 pre-paid incentive increased the odds of returning a completed survey by 52% compared to the group who was not offered an incentive. A $5 promised incentive increased these odds by 34%.

Evidence for the Effectiveness of Promised Incentives to Complete Surveys

Promised incentives significantly improve response rates compared to no incentive (Cantor, 2003; Yu and Cooper, 1983; Olson et al. 2004; Curtin et al. 2005). For example, Cantor et al. (2003) found an almost 10 percent increase in response rate when promising $20 (vs. no incentive) in an RDD survey of caregivers to children 0-17. Other studies have reported gains in response rates with offering relatively large amounts of money ($25 or greater) at the end of the data collection period (e.g., Olson et al. 2004; Curtin et al. 2005).

The Relative Effectiveness of Pre-Paid and Promised Incentives

Several studies and meta-analyses have evaluated the relative effectiveness of pre-paid versus promised incentives, but findings are inconclusive. For example, a meta-analysis of the effects of incentives on response rates in mail surveys (Church, 1993) found that incentives that accompanied the request to complete a mail survey elicited higher response rates than those that were promised to the respondent upon completion of the mail survey. However, another meta-analysis (Singer et al., 1998) found no difference in the effect of promised and pre-paid incentives on response rates. In a more recent meta-analysis, Singer (2002) found that pre-paid incentives do not differ significantly from promised incentives, although differences are in the expected direction. This finding was echoed by Datta, Horrigan and Walker (2001).

Evidence for the Combined Use of Pre-Paid and Promised Incentives

Strong evidence supporting the use of incentives is from studies that included both pre-paid and promised incentives. For example, the National Survey of Early Care and Education (NSECE, Administration for Children and Families), explore the effects of a pre-paid incentive in two amounts ($1 and $2) in combination with a promised incentive in several amounts. Specifically, they tested $1 and $2 pre-paid incentives with their mail screener and conducted a refusal conversion incentive experiment aimed at increasing household completion rates among respondents who were screener eligible for the household or home-based provider surveys. For the latter, two experimental conditions were fielded and households were randomly assigned to receive either a $5 pre-paid incentive or a $5 pre-paid incentive plus a $10 promised incentive upon completion of the interview. Based on the field test results, the $2 pre-paid screener incentive outperformed the $1 incentive. In order to increase response rates and reduce the amount of effort required to contact and gain cooperation from households, ACF staff indicate that going forward the study will seek to offer a $2 pre-paid incentive with the screener mailing and an additional $20 promised incentive for household and home-provider survey respondents.

An experiment conducted on the Racial and Ethnic Approaches to Community Health across the U.S. project produced a similar finding (OMB Control No. 0920-0805, Report on Incentives). The project contacted households who were not reachable via telephone about completing a self-administered mail questionnaire. Half of the sample served as a control group, and did not received an incentive. Within the group that did receive an incentive, all received $5 with the mailing that included the self-administered questionnaire. However, half of that group was promised another $10 to be sent upon receipt of the completed questionnaire. Overall, the group that received an incentive had significantly higher response rates compared to the group that received no incentive. Nearly 30 percent of households offered an incentive responded to the mailing compared to 11 percent of households in the control condition. Moreover, the amount of incentive had a significant effect on response rates. Approximately 32 percent of the households promised an additional $10 responded to the mailing compared to 28 percent of those provided with $5 only.

Use of Incentives to Improve Racial/Ethnic Minority and Lower SES Sample Composition

Incentives have been shown to bring traditionally underrepresented groups into the sample, such as the less educated, nonwhites, and those with lower incomes (Singer & Kulka 2002). For this reason, incentives can be used to improve sample composition. Research shows that survey incentives largely have a differential, positive impact upon response rates and survey costs when surveying ethnic and racial minorities. Investigators at the Mayo Clinic sought to examine the impact of pre-paid incentives on participation in a mail survey among Medicaid recipients (Beebe et al 2005). The study called for a two stage sampling design. The first sample consisted of a simple random sample (SRS) of Medicaid recipients in Minnesota. The second consisted of a random sample of recipients who were African American, American Indian, Somali, Hmong or Latino. Data collection was performed in waves. Two mailings were sent to respondents. Following the mailings, nonrespondents received a telephone follow-up call. An experimental condition offered an incentive to half of each strata (SRS or minority). Groups were either offered no incentive, or a $2 bill. The inclusion of the $2 incentive resulted in an increased response rate across all groups. Among the SRS sample, the sample that received an incentive had a response rate 5 points higher compared to the group that did not receive an incentive, following the telephone call. Among the racial and ethnic minority group, the group that received an incentive had up to a 10 point higher response rate than the group that did not. An exception to this finding was Latinos, whose response rate was not significantly different between groups that received the incentive and those that did not. These response rates are cumulative across all contacts. Incentives also have an impact on response rates and study attrition for households in the poverty stratum and significantly reduce item nonresponse rates among respondents within this population (Creighton et al, 2007; Clark, S.M. and Mack, S.P, 2009).

Use of Incentives to Reduce Cost Per Completed Interview

Incentives, and the amount of the incentive, can reduce the cost per case completed through their influence on the amount of effort that is required to achieve a completed interview. Evidence in support of this conclusion is provided by the incentive experiments conducted for the National Survey on Drug Use and Health (NSDUH, Substance Abuse and Mental Health Services Administration). Cost per interview in the $20 group was 5 percent lower than the control (no incentive), and in the $40 group costs were 4 percent lower than the control. The cost savings were gained by interviewers spending less time trying to obtain cooperation from respondents (Kennet et al., 2005). These savings were realized through reduced interviewer labor as well as reduced travel costs (mileage, tolls, parking, etc.) A similar finding was produced by an incentive experiment conducted for the National Survey of Family Growth (NSFG, National Center for Health Statistics) Cycle 5 Pretest which examined $0, $20, and $40 incentive amounts. As in the NSDUH experiments, the additional incentive costs were more than offset by savings in interviewer labor and travel costs (Duffer et al, 1994).

Respondents to a telephone study in the DC area about experience with TANF recipiency received a prenotification letter containing $2 (Markesich & Kovac, 2003). The letter noted that upon completion of the telephone interview, the respondent would receive either another $18 or another $33. While the small sample size and uneven distribution of the experimental conditions prevented the response rates among the two incentive amounts from differing significantly, there was a difference in the level of effort needed to achieve desired response rate targets. Those individuals who were offered $35 for completing an interview achieved a 68.7 percent response rate about 2 ½ weeks faster, and with fewer contact attempts, than did those sample members who were offered $20.

The Mayo Clinic study of Medicaid recipients, previously cited, similarly found that the overall cost per complete of surveys in the incentive group were lower than in the group that did not receive an incentive (Beebe et al 2005). This group required fewer contacts and fewer expensive telephone follow ups. The Latino subgroup was an exception to this finding, with the cost for Latino completed interviews being more in the incentive group.

Interactions Between Incentive Characteristics and Survey Characteristics

Studies show that the effect of incentives differ depending on other design features of the survey. For example, incentives have a larger impact on response rates when interest in the topic is low or when the burden of the interview is high (Singer 2002; Groves, Presser and Dipko 2004).

Incentive Amount is Linked to Response Rates

The choice of an incentive amount depends largely on the survey burden, including the survey length and other tasks that may be required of the respondent, and the survey topic. The National Household Education Survey (NHES, U.S. Department of Education) tested $2 and $5 pre-paid incentives for their mail screener, which was expected to take 2-8 minutes to complete depending on the survey version used. The study also tested the effectiveness of $5 and $15 pre-paid incentives to respondents who screened eligible for the topical survey. The study found that the larger pre-paid incentive amounts ($5 vs. $2, $15 vs. $5) achieved higher response rates. The NHES also offered a $5 promised incentive to a subset of respondents to encourage them to participate in the topical survey by telephone. Although higher response rates (6- 8%) were achieved with the $5 promised incentive, none of the observed differences were statistically significant (Tubman and Williams, 2010). However, Strouse and Hall, 1997, recommend that in order to be successful, promised incentives have to be in the $15 to $35 range, and many federally-funded surveys today provide incentives ranging from $20 to $125.

Incentive Amounts Offered by Other Federally-Funded Surveys

Many federally-sponsored surveys offer incentives to gain cooperation, and these incentives range from $15 to $125, depending on respondent burden. The National Immunization Survey (NIS, National Center for Immunizations and Respiratory Diseases), for example, offers a combination of $5 pre-paid and $10 promised incentives to encourage eligible nonrespondents to participate. The National Survey of Adoptive Parents of Children with Special Health Care Needs (Department of Health and Human Services) offers parents $25 for participation in a 35-minute telephone survey. Interviewers in the NSDUH currently offer $30 for an interview that averages 60 minutes, and the National Survey of Family Growth (NSFG, Centers for Disease Control and Prevention) offer $40 for interviews that are about 60 minutes for males and 80 minutes for females. Over rounds 1 through 10 of the National Longitudinal Survey of Youth 1997 cohort (NLSY97, Bureau of Labor Statistics), incentives offered to respondents ranged from $10 to $50 in an attempt to minimize attrition across waves of data collection. In order to improve response rates, reduce the number of contacts required to gain cooperation, and address respondent concerns about interview burden, the National Survey of Child and Adolescent Well-Being (NSCAW, Administration for Children and Families) in 2002 doubled the incentive offered to respondents from $25 to $50. The Early Childhood Longitudinal Study-Birth Cohort (ECLS-B, U.S. Department of Education) offered parent participants $50 and a children’s book for the first wave and $30 and a children’s book for subsequent waves of data collection. Incentives on the National Health and Nutrition Examination Survey (NHANES, National Center for Health Statistics) range from $20 to $125 depending on the survey and physical exam components in which respondents agree to participate.

5