ACS Methods Panel Test

Attachment A: 2025 Questionnaire Timing Test 

American Community Survey Research and Evaluation Program

March 28, 2025

ACS Research & Evaluation Analysis Plan (REAP)

2025 Questionnaire Timing Test

REAP Revision Log

Version	Date	Description	Authors
0.1	11/4/2024	Initial Draft for Feedback	Lauren Contard, Lindsay Longsine, Samantha Spiers
0.2	12/5/2024	Second Draft for Approval	Lauren Contard, Lindsay Longsine, Samantha Spiers
1.0	12/13/2024	Final REAP	Lauren Contard, Lindsay Longsine, Samantha Spiers
2.0	3/28/2025	Final REAP, updated to reflect date change to September 2025	Lauren Contard, Lindsay Longsine

TABLE OF CONTENTS

1. INTRODUCTION 1

2. BACKGROUND 2

2.1 2025 ACS Mail Contact Strategy 2

2.2 Evolution of the ACS Mail Contact Strategy 3

3. LITERATURE REVIEW 7

3.1 2020 Census Internet First Contact Strategy 7

3.2 2021 ACS Strategic Framework Test 8

3.3 QR Codes in Mail Materials 9

4. RESEARCH QUESTIONS AND METHODOLOGY 12

4.1 Sample Design 12

4.2 Experimental Design 12

4.2.1 Control Treatment – ACS Production Timing without QR Codes on Materials 12

4.2.2 Treatment 1 – ACS Production Timing with QR Codes on Materials 13

4.2.3 Treatment 2 – 2020 Census Internet First Timing 13

4.2.4 Treatment 3 – Questionnaire and Postcard Last 14

4.2.5 Treatment 4 – Questionnaire Last 15

4.2.6 Mailout Timing 15

4.3 Research Questions 16

4.4 Analysis Metrics 17

4.4.1 Self-Response Return Rates 17

4.4.2 Final Response Rates 18

4.4.3 Form Completeness 19

4.4.4 Response Distributions of Respondent Demographics 20

4.4.5 Standard Error of the Estimates 20

4.4.6 Cost Analysis 21

4.4.7 Additional Analysis Metrics 21

5. ASSUMPTIONS AND LIMITATIONS 21

5.1 Assumptions 21

5.2 Limitations 22

6. TABLE SHELLS 22

7. POTENTIAL CHANGES TO ACS 28

8. REFERENCES 28

Appendix A. Images of the Control Treatment Mail Materials (ACS Production) 31

Appendix B. Images of Treatment 1 Mail Materials 40

Appendix C. Images of Treatment 2-4 Mail Materials 46

Appendix D. Timeline of Treatments 57

1. INTRODUCTION

The American Community Survey (ACS) has a two-month self-response period where sampled households are contacted by mail and encouraged to respond to the survey by internet, paper questionnaire, or phone. Since the internet instrument was introduced in 2013, internet self-response has increased over time whereas paper response has decreased (Mills et al., 2022). Responding online is both convenient for respondents and cost-effective compared to the other self-response modes. Additionally, the internet instrument has technological features that are designed to improve data quality, such as automated skip patterns and a review screen.

Over time, the ACS program has modified the ACS mail contact strategy to emphasize responding online over the other response modes (see Section 2). Currently, the paper questionnaire is the third mailing sent to households and is mailed three weeks after the first contact. Mailing the paper questionnaire later may push respondents to respond online sooner, reducing the number of paper questionnaires sent and decreasing costs of questionnaire printing, assembly, mailing, and processing.

We created the 2025 Questionnaire Timing Test (QTT) to compare alternate mail contact strategies that mail the paper questionnaire later. The purpose of the test is to determine which contact strategy decreases operational costs without decreasing overall self-response.

In addition to changing the timing of the paper questionnaire mailing, we are also interested in testing the addition of a Quick Response (QR) code to the ACS mailing materials. A QR code is a two-dimensional barcode that can be read by smartphones and tablets (either by a QR reader or a camera mobile application) and sends the user to a linked website. Most internet responses are from personal computers but responses from smart devices (i.e., smartphones and tablets) have steadily increased (Mills et al., 2022).

Currently, an ACS respondent must type the URL provided in the mailing materials to access the survey. However, with a QR code, respondents would not have to open a browser and type in the URL, and instead they could simply scan the code with their smart device. Providing a QR code might be helpful for respondents to access the survey quicker, reducing respondent burden.¹

2. BACKGROUND

   1. 2025 ACS Mail Contact Strategy

The ACS mail contact strategy focuses on encouraging residents in mailable sampled addresses to self-respond to the survey. Figure 1 shows the current (2025) ACS mail contact strategy.

Figure 1. 2025 ACS Mail Contact Strategy

To encourage self-response, the Census Bureau sends as many as six mailings to a mailable sampled address. The list of mailable sampled addresses is updated three times during this process to remove households that have responded. The sooner an ACS response is received from a household, the fewer mailings we send to that address.

The Census Bureau sends the first two mailings to all mailable addresses in the monthly sample. These addresses comprise the initial mailing universe (M1). The first mailing is an initial pressure seal mailer that includes an internet user ID and instructions for how to respond online. It also provides the Telephone Questionnaire Assistance (TQA) phone number which respondents can call if they have questions or need help. A week later, the same addresses are sent a second mailing – a reminder pressure seal mailer.

Addresses in the monthly sample for which a response is not received by a cutoff date after the second mailing comprise the second mailing universe (M2).² These nonresponding addresses are sent the third and fourth mailings. The third mailing is a package that includes a letter, a paper questionnaire, and a business reply envelope. Four days later, these addresses are sent a fourth mailing (reminder postcard) which encourages them to respond.

Addresses in the M2 universe for which a response is not received by a cutoff date after the fourth mailing comprise the third mailing universe (M3). These addresses are sent the fifth mailing about three weeks after the fourth mailing. The fifth mailing is the Due Date Letter, a pressure seal mailer which asks recipients to respond online or by mail prior to the printed due date.

Two to three weeks later, responding addresses are removed and the unmailable and undeliverable addresses (from the initial sample) are added to create the universe of addresses eligible for the Computer-Assisted Personal Interview (CAPI) nonresponse follow-up operation.³ Of this universe, a subsample is chosen to be included in the CAPI operation. Mailable addresses from this subsample comprise the fourth mailing universe (M4). These addresses are sent the sixth mailing (CAPI Internet Letter) at the start of the interviewing month, a pressure seal mailer that asks recipients to respond online to avoid an in-person interview.⁴ Census Bureau field representatives (FR) may first attempt to interview those selected for CAPI by phone, if possible. If the FR is unable to complete a phone interview, they visit the address to conduct an in-person interview.

Additional information can be found in the ACS and Puerto Rico Community Survey (PRCS) Design and Methodology Report (U.S. Census Bureau, 2024).

2. Evolution of the ACS Mail Contact Strategy

Since its full implementation in 2005, the ACS program has modified the mail contact strategy based on new research and changing data collection procedures. The following details major changes in the mail contact strategy related to timing, addition or removal of mail pieces, and mail type; additional design changes, such as changes in messaging, are not included.

ACS Mail Contact Strategy Pre-Internet

Figure 2 shows the ACS mail contact strategy before the internet response mode was introduced. The ACS program developed the original (2005) ACS mail contact strategy based on the 2000 Census mailing strategy, which included three mailings (prenotice letter, paper questionnaire package, and reminder postcard) followed by a nonresponse follow-up operation. The ACS program made some changes to this strategy, including tailoring language to reflect the survey and mandatory message, shortening the time between the first three mailings, and adding a second questionnaire mailing to nonrespondents.⁵ The contact strategy also included a Computer-Assisted Telephone Interview (CATI) operation that called nonrespondent households with telephone numbers available to elicit a response before the CAPI operation.

The mail contact strategy changed slightly in 2011 with the addition of a second reminder postcard. This postcard was sent during the start of the CATI operation to households where no phone number was available for the address (i.e., households not eligible for the CATI operation). In addition to the postcard, a multilingual brochure was also added to the prenotice letter to assist non‑English speakers with responding.

Figure 2. 2011 ACS Mail Contact Strategy

Internet Response Mode

In January 2013, the ACS program introduced a new internet response option for the ACS. Figure 3 shows the 2013 ACS mail contact strategy. The following details how it differs from the 2011 strategy:

1. The first paper questionnaire package was replaced with an internet invitation package with a letter, a card with instructions about how to respond online, and the FAQ brochure.

2. An internet instruction card was added to the paper questionnaire package.

3. The paper questionnaire package was mailed a week earlier than before.

4. A reminder postcard was added four days after the questionnaire mailing.

Figure 3. 2013 ACS Mail Contact Strategy

Prenotice Removal

The mail contact strategy changed again in August 2015 based on results from the 2015 ACS Mail Contact Strategy Modification Test (Clark et al., 2015). Figure 4 shows the 2015 ACS mail contact strategy, beginning with the August panel. The prenotice letter was replaced with an initial mail package – the previous internet invitation package but including a multilingual brochure. The initial mail package had the same timing as the prenotice, seven days before the first reminder. The first reminder was a letter with the internet user ID instead of a postcard.

Figure 4. 2015 ACS Mail Contact Strategy (beginning with the August Panel)

CATI Operation Eliminated

In October 2017, the ACS program eliminated the CATI operation.⁶ As a result, the fifth mailing, the second reminder postcard, changed to be sent to all remaining nonrespondents. Figure 5 shows the 2017 ACS mail contact strategy, beginning with the September panel.

Figure 5. 2017 ACS Mail Contact Strategy (beginning with September Panel)

CAPI Internet Letter and Pressure Seal Mailers

Several changes have been made over the years to the ACS mail contact strategy after the introduction of pressure seal mailers. Pressure seal mailers cost less than a letter with an envelope and can conceal personal information like the internet user ID. The following describes how pressure seal mailers have been used:

• In September 2018, the reminder letter and the second reminder postcard were replaced with pressure seal mailers based on results from the 2017 Pressure Seal Mailing Materials Test (Risley et al., 2018).

• In November 2020, the ACS program began sending the CAPI Internet Letter as a pressure seal mailer to all mailable addresses in the CAPI sample. Regional Offices had been sending this letter ad-hoc to addresses in the CAPI sample to encourage responding online as opposed to in-person. It was more cost-effective to send the letter as a pressure seal mailer from the National Processing Center (NPC) and to all mailable addresses in the CAPI sample as opposed to only some regions.

• The coronavirus (COVID-19) pandemic disrupted the ACS mail contact strategy from March 2020 to April 2021 due to an early shutdown of operations at NPC and supply and staffing shortages once operations resumed. Consequently, mailings overall were reduced, and pressure seal mailers were used in place of the initial mail package and questionnaire package for some panels (Longsine & Spiers, 2023). The panels that used a pressure seal mailer had unexpectedly higher returns than the ones that used the initial mail package, and thus the 2021 Initial Mailing Pressure Seal Test studied this effect during typical mail contact strategy conditions (Longsine & Spiers, 2023). Based on the results, the initial mail package was replaced with a pressure seal mailer in February 2022.

Figure 6 shows the 2022 ACS mail contact strategy, beginning with the February panel.⁷

Figure 6. 2022 ACS Mail Contact Strategy (beginning with the February Panel)

3. LITERATURE REVIEW

The following sections detail research that informed the mail material designs and mailout timing strategies of the treatments in the 2025 QTT. See Section 4.2 for a description of the treatments.

1. 2020 Census Internet First Contact Strategy

Before beginning in-person nonresponse followup (NRFU), the Census Bureau solicited a self‑response to the 2020 Census from mailable or deliverable addresses. The 2020 Census had two mail contact strategies: Internet First and Internet Choice. The Internet First contact strategy pushed for internet response and did not send a paper questionnaire initially; this strategy was used for most of the country. The Internet Choice contact strategy sent a paper questionnaire in the first and fourth mailings while also encouraging online response. This strategy was used for Census tracts with characteristics predicted to have low internet response. This section focuses on the Internet First contact strategy, which Treatment 2 of the 2025 QTT is based on.

An early version of the Internet First strategy was developed for the 2012 National Content Test, which tested several contact strategies to identify the best one for pushing respondents to complete the Census online. It was refined over multiple tests from 2014 to 2018, adjusting mail timings and types.

Table 1 shows the planned Internet First contact strategy for the 2020 Census with the expected in-home delivery date of each mailing. Housing units were divided into four cohorts with staggered delivery dates to distribute workloads and reduce burden on the USPS, 2020 Census call centers, and the 2020 Census internet instrument (Nichols et al., 2019).

The first mailing was an invitation letter encouraging online response and provided the Census ID and URL for the internet instrument. The second mailing, delivered four days after the first, was a reminder letter to all households while the third mailing, delivered ten days later, was a reminder postcard to nonrespondents only. If a response was not received after the third mailing, then a paper questionnaire was to be sent 13 days after the third mailing, but due to the COVID-19 pandemic, was sent 19 to 22 days afterwards. The fifth mailing was a reminder postcard, delivered about 12 days after the fourth and again sent to only nonrespondents. Due to the pandemic, two additional mailings were sent to nonrespondents: a pre-NRFU postcard and an additional questionnaire. These were sent almost three months after the fifth mailing to re-establish contact before the delayed NRFU operation began.

Table 1. 2020 Census Internet First Contact Strategy In-Home Dates by Cohort

Cohort	Mailing 1 Letter	Mailing 2 Letter	Mailing 3* Postcard	Mailing 4*+ Letter + Questionnaire	Mailing 5*+ “It’s Not Too Late” Postcard
1	March 12	March 16	March 26	April 8	April 20
2	March 13	March 17	March 27	April 9	April 20
3	March 19	March 23	April 2	April 15	April 27
4	March 20	March 24	April 3	April 16	April 27

^* Mailed only to nonrespondents.

⁺ Actual Mailing 4 production end dates were April 14-24 (in-home dates would be 1-2 days later). Actual Mailing 5 in-home dates were April 27 to May 6.

Consistent with tests, the self-response rate for the 2020 Census (70.0 percent) and the internet response rate (61.0 percent) were higher for the Internet First strategy than the Internet Choice strategy (57.0 and 26.5 percent, respectively) (Letourneau et al., 2024). Internet and total self-response rates had the largest daily increases in the two weeks leading into Census Day, April 1 (Letourneau et al., 2024).

For more information on the development and outcomes of the 2020 Census contact strategy, see the 2020 Census Self-Response and Return Rates Assessment report (Letourneau et al., 2024).

2. 2021 ACS Strategic Framework Test

The 2021 ACS Strategic Framework Mail Materials Test tested four new sets of ACS mail materials, which were developed based on research on best practices in survey messaging to gain respondent cooperation (Oliver et al., 2023). The treatments were evaluated based on response rates, effect on costs, and quality of responses. The Control treatment contained a fourth mailing reminder postcard that was based on materials tested in the 2018 ACS Mail Materials Test (Risley & Berkley, 2020), which had particularly strong messaging about the mandatory nature of the ACS and the importance of responding immediately to prevent a visit from a CAPI interviewer. It also put more emphasis on completing the paper questionnaire (which respondents receive a few days before the postcard) than on responding online.

Due to the timing of the development of the materials for this test, the experimental treatments’ fourth mailing postcards did not contain this strong messaging; they were based on an older, less aggressive version of the postcard, which did not emphasize responding by mail more than internet. Several of the experimental treatments gained higher self-response than the Control treatment earlier in the response period. However, after the fourth mailing postcard was mailed out, the Control treatment gained significantly higher self-response than all the experimental treatments. This suggests that sending a postcard with strong messaging about completing the paper questionnaire, timed to arrive a few days after the respondents receive the paper questionnaire, can significantly increase self-response.

3. QR Codes in Mail Materials

Researchers have been evaluating QR codes in surveys for some time. In the past, the research has mostly shown that the addition of a QR code to a mailing containing a URL has not improved response rates and that using only a QR code has hindered response (Marlar, 2018). However, a lot of this research took place before two major changes in QR code use:

• In 2017, smartphone manufacturers updated cameras mobile applications with the capability to read QR codes, eliminating the need for a separate QR code reading application for users that had the latest models.

• In 2020, with the emergence of the COVID-19 pandemic, many restaurants, medical offices, and other establishments began to use QR codes to lessen person-to-person contact.

Testing of QR codes in the ACS has been suggested in the past but was not a high priority at the time because they were not believed to be commonly used. There were also concerns over the quality of data collected from using a mobile phone compared to a computer. Since 2020, QR codes have become much more popular and widely used.

Pre-2020 Research

In 2015, Gallup conducted an experiment on the use of QR codes to access a survey (Marlar, 2018). In this experiment, 7,500 participants were asked to complete a survey via mail or internet. There were three treatments which offered different internet access options (all offered mail response). The three treatments were as follows:

• Group 1: A traditional URL link only

• Group 2: A QR code only

• Group 3: A QR code and the URL link

Gallup found no statistically significant differences in total response rates across the three groups, however some differences were found by mode, with Group 2 having lower internet response than the other two groups (Marlar, 2018). No significant difference in internet response was found between Groups 1 and 3, and only four percent of Group 3 used the QR code to access the survey.

They also found that respondents who used the QR code differed in age and education than those who used the URL to respond. The respondents who used the QR code were younger and more highly educated than respondents who used the URL (Marlar, 2018). However, Gallup noted that the number of respondents who used the QR code was too small to make a strong conclusion.

Other research supported the finding that older populations were less likely to be familiar with QR codes. Mendelson and Bergstrom (2013) found that older adults were only 13 percent as likely as younger adults to have used QR codes.

Post-2020 Research

In 2022 Gallup retested their earlier (2015) experiment using only two groups:

• Group 1: A traditional URL link only

• Group 2: A QR code and the URL link

This time they found that Group 2 showed statistically significantly higher response rates (19.4 percent) than Group 1 (18.0 percent) (Marlar & Schreiner, 2024). The QR code/URL combination also resulted in a higher number of internet responses.

They found that Group 2 had a significantly higher proportion of Hispanic respondents compared to Group 1 but found no other demographic differences between the two groups (Marlar & Schreiner, 2024).

Endres et al. (2023) conducted research on push-to-web surveys, which included testing a QR code. There were a variety of treatments in this test, but two treatments included a QR code experiment, with one treatment using a QR code in addition to a URL and another using only the URL.⁸ The survey was a statewide, public health survey that estimated the prevalence of various forms of violence, participant perceptions of safety in their community, and awareness of prevention services for adults between the ages of 18 and 65.

In this test, QR codes and the URL directed respondents to the survey consent page, however respondents were not required to enter a user ID or access code (each QR code/URL was personalized and could only be used once).⁹

The completion rate for the treatment that received both the QR code and URL was 2.54 percent compared to 0.65 percent for the URL-only treatment, a significant difference of 1.89 percentage points (Endres et al., 2023).

A study by RTI (Smith et al., 2023) tested the use of QR codes post-pandemic. In this study, mail materials were sent out with three response options.

• A URL in a call-out box towards the top of the letter.

• A toll-free phone number provided in a sentence following the call-out box.

• A QR code in the bottom right corner of the letter.

The results of this study showed that roughly half (48 percent) of the mobile response (response either by smartphone or tablet) was accessed through the QR code (Smith et al., 2023).

The respondents who were more likely to use the QR code fell into the following categories:

• Age 50 or younger

• Hispanic

• Other race

• Renters

• High-school degree or less

The respondents who were more likely to use the URL fell into the following categories:

• Older than 50

• White

• Black

• Owners

• Some college or associate degree

• Bachelor’s degree

These findings support other research that older populations are less likely to use QR codes, however, unlike Gallup’s results from 2015, these results show QR code use was correlated with lower levels of education.

QR Code Use at the Census Bureau

In 2022 and 2024, the Census Bureau conducted research studies on the effectiveness, efficiency, satisfaction, and technical difficulties while using a QR code to access a survey. Respondents were given instructions to scan a QR code to access a survey on their smartphone and were asked for feedback on the usability of the QR code and whether they would be likely to use a QR code to access a survey in the future. They were also asked about the wording of the instructions.

The first study (Rivas & Schulzetenberg, 2023) focused on older adults who may be less tech-savvy. There were 19 participants, with average age 63.7 years and most having previous QR code experience. They were given a paper invitation with step-by-step instructions on how to access the survey using the QR code. Most participants were able to use the QR codes and stated they would prefer using it over typing a URL. Some participants did have issues with tapping the link after scanning the QR code.

The second study (Rivas et al., 2024) was an observational study with twenty participants. All twenty participants were fluent in English and were iPhone users. The average age of the participants was 29.9 years, and all had previous QR code experience.

All respondents successfully scanned the QR code without any usability issues, taking an average time of 12.4 seconds to pull up the survey (Rivas et al., 2024). Most respondents (70%) reported being “extremely likely” to use a QR code to access a survey on a smartphone. However, two respondents (10% of the sample) reported being “somewhat unlikely” to access a survey through a smartphone due to a preference for larger screens. The respondents reported that the instructions were easy to follow but acknowledged that the instructions may not be as clear for those without previous QR code experience, those who are not technologically savvy, and those who are not fluent in English.

Research has also been conducted for the decennial census on the usability of QR codes. That research is wrapping up and results are forthcoming.

4. RESEARCH QUESTIONS AND METHODOLOGY

   1. Sample Design

The 2025 QTT will be conducted using the September 2025 ACS production sample. The monthly ACS production sample consists of approximately 295,000 housing unit addresses and is divided into 24 methods panel groups, where each group contains approximately 12,000 addresses. Each methods panel group is a representative subsample of the entire monthly sample, and each monthly sample is representative of the entire yearly sample and the country. The 2025 QTT will use all 24 methods panel groups. The control treatment will use four randomly assigned methods panel groups. Each experimental treatment will use five randomly assigned methods panel groups.

2. Experimental Design

This test will include a control treatment and four experimental treatments. All experimental treatments include QR codes on the mail materials.

1. Control Treatment – ACS Production Timing without QR Codes on Materials

The control treatment will use the ACS production timing and mail materials as described in Section 2.1. See Appendix A for images of the mail materials for the control treatment.

2. Treatment 1 – ACS Production Timing with QR Codes on Materials

In Treatment 1, the timing is the same as the control treatment. The mail materials are like the control treatment, but QR codes are included.¹⁰ The purpose of Treatment 1 is to study the effect of the QR code (which will also be included in Treatments 2-4) on response and data quality. See Appendix B for images of the mail materials for Treatment 1.

3. Treatment 2 – 2020 Census Internet First Timing

Treatment 2 is based on the 2020 Census Internet First mailout timing strategy (see Section 3.1). The questionnaire will be sent to all nonrespondents but moved from the third mailing to the fourth mailing. The motivation behind this treatment is that mailing the questionnaire package later may allow more respondents to respond by internet first, reducing costs if fewer questionnaire packages need to be mailed.

To reflect the new paper questionnaire timing, the content of the first through fourth mailings will be modified in the following ways:

• First mailing: The initial Pressure Seal Mailer (PSM) will not mention that a paper questionnaire will be mailed out in a few weeks. It will only mention the internet mode as a response option. This initial PSM will be the same as the one used in Treatments 3 and 4.

• Second mailing: The reminder PSM will not mention that a paper questionnaire will be mailed soon. It will focus on internet as a response option. This reminder PSM will be the same as the one used in Treatments 3 and 4.¹¹

• Third mailing: This will consist of a redesigned reminder postcard, reflecting how respondents will receive it before receiving the paper questionnaire. This reminder postcard will mention that a paper questionnaire will be mailed soon.

• Fourth mailing: The paper questionnaire package will include the paper questionnaire, letter, return envelope, and outgoing envelope. The questionnaire, return envelope, and outgoing envelope will match the control treatment. The letter will be redesigned to emphasize the urgency to respond soon, including removing some extraneous text, moving the sentence about random selection to the FAQs on the back, and adding a sentence from the control treatment postcard about how an interviewer may reach out if they do not respond.

In addition, a QR code will be added to all mail materials. See Appendix C for images of the Treatment 2 mail materials that will be different than Treatment 1.

4. Treatment 3 – Questionnaire and Postcard Last

1. In Treatment 3, the paper questionnaire will be sent in the fourth mailing, followed by the reminder postcard in the fifth mailing. This treatment keeps the ACS production order of questionnaire followed by postcard, which was shown to perform well in the Strategic Framework Test (see Section 3.2), while delaying the questionnaire to a later mailout. The content of all mailings will be modified to reflect the new questionnaire timing, as follows:

• First mailing: The initial PSM will not mention that a paper questionnaire will be mailed out in a few weeks. It will only mention the internet mode as a response option. This initial PSM will be the same as the one used in Treatments 2 and 4.

• Second mailing: The reminder PSM will not mention that a paper questionnaire will be mailed soon. It will focus on internet as a response option. This reminder PSM will be the same as the one used in Treatments 2 and 4.

• Third mailing: This mailing will be a newly designed PSM. It will incorporate messaging from the reminder PSM, modified to not be repetitive. It will mention that a paper questionnaire will be mailed in a few weeks.

• Fourth mailing: The paper questionnaire package will include the paper questionnaire, letter, return envelope, and outgoing envelope. The questionnaire, return envelope, and outgoing envelope will match the control treatment. The letter will be redesigned to emphasize the urgency to respond soon, including removing some extraneous text, moving the sentence about random selection to the FAQs on the back, and adding a sentence from the postcard about how an interviewer may reach out if they do not respond. This paper questionnaire letter will be the same as the one used in Treatment 2.

• Fifth mailing: The reminder postcard will be redesigned to incorporate messaging from the production Due Date Letter. It will include a due date on the front and back.

2. In addition, a QR code will be added to all mail materials. See Appendix C for images of the Treatment 3 mail materials that will be different than Treatment 1.

5. Treatment 4 – Questionnaire Last

In Treatment 4, the paper questionnaire will be sent in the fifth mailing to all remaining nonrespondents. The content of all mailings will be modified to reflect the new questionnaire timing, as follows:

• First mailing: The initial PSM will not mention that a paper questionnaire will be mailed out in a few weeks. It will only mention the internet mode as a response option. This initial PSM will be the same as the one used in Treatments 2 and 3.

• Second mailing: The reminder PSM will not mention that a paper questionnaire will be mailed soon. It will focus on internet as a response option. This reminder PSM will be the same as the one used in Treatments 2 and 3.

• Third mailing: This will consist of a redesigned reminder postcard, reflecting how respondents will receive it before receiving the paper questionnaire. Unlike the reminder postcard in Treatment 2, this postcard will not mention that a paper questionnaire will be mailed soon.

• Fourth mailing: This will consist of a newly designed PSM. It will incorporate messaging from the reminder PSM, modified to not be repetitive. It will mention that a paper questionnaire will be mailed in a few weeks. This reminder PSM will be the same as the one used in the third mailing in Treatment 3.

• Fifth mailing: The paper questionnaire package will include the paper questionnaire, letter, return envelope, and outgoing envelope. The questionnaire and return envelope will match the control treatment. The letter will include a due date and borrow messaging from the Due Date Letter in the control treatment. The back of the letter will match the letters in Treatments 2 and 3 except that the Spanish messaging will be moved to the front. The outgoing envelope will also include a due date.

In addition, a QR code will be added to all mail materials. See Appendix C for images of the Treatment 4 mail materials that will be different than Treatment 1.

6. Mailout Timing

In addition to changes in the materials described above, some treatments will also change the timing of when mailings are sent. The following summarizes the order of the mail pieces for each treatment:

• Control Treatment: This treatment will use the ACS production mailout timing. All mailable sample addresses in this treatment will be sent an initial PSM. Eight days later those same addresses will be sent a reminder PSM. Nonresponding addresses will be sent a paper questionnaire package 13 days after the reminder PSM, followed by a reminder postcard four days later. Nonresponding addresses will be sent a Due Date Letter 23 days after the reminder postcard.

• Treatment 1: This treatment will use the same mailout timing as the control treatment (i.e., ACS production mailout timing).

• Treatment 2: All mailable sample addresses in this treatment will be sent an initial PSM. Five days later those same addresses will be sent a reminder PSM. Nine days later those same addresses will be sent a reminder postcard. Nonresponding addresses will be sent a paper questionnaire package 14 days after the reminder postcard. Nonresponding addresses will be sent a Due Date Letter 20 days after the paper questionnaire.

• Treatment 3: All mailable sample addresses in this treatment will be sent an initial PSM. Eight days later those same addresses will be sent a reminder PSM. Nonresponding addresses will be sent a second reminder PSM 18 days after the first one. Remaining nonresponding addresses will be sent a paper questionnaire package 16 days after the second reminder PSM, followed by a reminder postcard with due date five days later.

• Treatment 4: All mailable sample addresses in this treatment will be sent an initial PSM. Eight days later those same addresses will be sent a reminder PSM. Nonresponding addresses will be sent a reminder postcard 13 days after the reminder PSM, followed by a second reminder PSM seven days later. Remaining nonresponding addresses will be sent a paper questionnaire package with due date 14 days after the second reminder PSM.

See Appendix D for a timeline showing the day each mail piece will be mailed for each treatment.

3. Research Questions

The 2025 QTT will answer the following questions related to the addition of a QR code to mail materials:

1. What is the impact of including QR codes on self-response return rates (overall and by mode)?

2. What is the impact of including QR codes on final response rates?

3. Are the overall form completeness rates different between Control and Treatment 1?

4. Are the demographic distributions of respondents different between Control and Treatment 1 before CAPI?

5. How would including QR codes affect the costs of data collection if implemented in ACS production?

6. What is the rate of respondents using the QR code to access the internet instrument on a computer? On a mobile device?

The 2025 QTT will answer the following questions related to changes in the mail contact strategy:

7. What is the impact of the different mail contact strategies on self-response return rates (overall and by mode)?

8. What is the impact of removing the paper questionnaire reference on self-response return rates?

9. What is the impact of each of the mail contact strategies on final response rates?

10. How do the treatments affect overall form completeness?

11. Are the demographic distributions of respondents different among treatments before the M3 cut? Before CAPI?

12. How would the experimental treatments affect the costs of data collection if implemented in ACS production?

4. Analysis Metrics

All self-response analyses, except for the cost analysis, will be weighted using the ACS base sampling weight (the inverse of the probability of selection). Cases in the CAPI subsample will have a CAPI subsampling factor that will be multiplied by the base weight unless they are self-responses. We will use a significance level of α=0.1 when determining significant differences between treatments. To control for the overall Type I error rate for a set of hypotheses tested simultaneously, we will perform multiple-comparison procedures using the Hochberg method (Hochberg, 1988). Section 4.4.1 provides the minimum detectable differences between self-response rates.

1. Self-Response Return Rates

To determine the effect of each treatment on self-response, we will calculate the self-response return rates at two points in time in the data collection cycle — before the M3 cut and before the start of CAPI (i.e., the start of the CAPI operation).¹² When appropriate, we will also calculate rates before the M2 cut to examine treatment differences in early response. Self-response return rates will be calculated for total self-response combined and separately for internet, mail, and TQA responses. If there are no significant differences in TQA rates between treatments, we may combine mail and TQA rates.

For comparing control to Treatment 1, the sample size will be able to detect differences of approximately 0.79 and 0.81 percentage points between the self-response return rates before the M3 cut and before the start of CAPI, respectively (with 80 percent power and α=0.1). For comparing experimental treatments, the sample size will be able to detect differences of approximately 0.74 and 0.77 percentage points between the self-response return rates before the M3 cut and before the start of CAPI, respectively (with 80 percent power and α=0.1).

The self-response return rates will be calculated using the following formula:

Self-Response Return Rate	=	Number of mailable and deliverable sample addresses that either provided a non-blank13 return by mail or TQA, or a complete or sufficient partial14 response by internet	* 100
		Total number of mailable and deliverable sample addresses15

If we receive more than one return from an address, then the return received first will be considered the response. In the rare case that we receive two returns on the same day, then we will choose the response in the following order: (1) mail, (2) TQA, and (3) internet.¹⁶

2. Final Response Rates

To determine the effect of the experimental treatments on overall response to the survey, we will calculate final overall response rates and how each response mode contributes to the overall final response rate. The final response rates will be calculated using the following formula:

Final Response Rate	=	Number of eligible sample addresses that either provided a non-blank return10 by mail or TQA, a complete or sufficient partial11 response by internet, or a complete CAPI interview	* 100
		Total number of sample addresses eligible to reply to the survey and not sampled out of CAPI

The denominator does not include UAAs (unless the address did respond or is in the CAPI sample) and does not include addresses that are found to be a business, demolished, under construction, etc.

If we receive more than one response from an address, then the response received first will be considered the response. In the rare case that we receive two responses on the same day, then we will choose the response in the following order: (1) mail, (2) TQA, (3) internet, and (4) CAPI.

3. Form Completeness

Form completeness provides a way to assess how much of the survey a respondent completed; it measures the number of questions on the form that were answered among those that should have been answered.¹⁷

The ACS has three question sections: a basic person-level demographic section, a housing section, and a detailed person-level section. We will calculate weighted completeness rates for each section of the questionnaire, as well as overall. Calculations will be made using the following formulas:

We will calculate the form completeness rates by mode because of known differences in form completeness between them. For example, the involvement of an interviewer can lead to higher form completeness than is seen in self-response modes.

4. Response Distributions of Respondent Demographics

We will compare demographic distributions of respondents to see if there are response distribution changes from the experimental treatments. Differences alone do not indicate treatment success or failure, but increased self-response with populations that are considered hard-to-count would be beneficial. We will also calculate distributions by mode to see if certain demographic groups switch modes in the experimental treatments.

We will calculate and compare the distributions of all non-blank self-responses for the following demographic and housing categories: age, educational attainment, Hispanic origin, race, sex, building type, and tenure.

Proportion estimates will be calculated using the following formula:

Valid in-scope responses will be included in the analysis. The demographic characteristics will be for the respondent, or Person 1, in the survey. We will use uncoded data for the race and Hispanic origin analysis.

This analysis will only use self-responses. In our calculations, we will calculate combined self-response and separate the distributions by mode: mail, internet, and TQA.¹⁸ We will use Rao-Scott chi-squared tests of independence to determine whether the response distributions are statistically different at the α=0.1 level (Rao & Scott, 1987).

5. Standard Error of the Estimates

We will estimate the variances of the point estimates and differences using the Successive Differences Replication (SDR) method with replicate weights – the standard method used in the ACS (see U.S. Census Bureau, 2024, Chapter 12). In calculating the different rates, we will use replicate subsampling adjusted weights, which account for the initial probability of selection (the base weight) and CAPI subsampling. We will calculate the variance for each rate and for the difference between rates using the formula below:

where:

the estimate calculated using the r^th replicate

the estimate calculated using the full sample

The standard error of the estimate (X₀) is the square root of the variance.

6. Cost Analysis

In evaluating the different experimental treatments, it is not sufficient to compare only the self-response return rates and final response rates. If one or more of the experimental treatments increases self-response, subsequent mailings and the CAPI workloads (which cost more per case to complete than self-response cases) would be smaller.

We will conduct a cost analysis to estimate the annual expected cost of implementing each experimental treatment into a full ACS production year. To assess impacts of the QR code on costs, we will compare the cost estimate for Treatment 1 with the control treatment. To assess impacts of the mail contact strategy on costs, we will compare the cost estimates for Treatment 2, Treatment 3, and Treatment 4 with Treatment 1. Since the cost model uses projected workload differences to project survey costs, this part of the analysis will not be weighted.

7. Additional Analysis Metrics

Prior to answering the research questions, we will investigate the underlying data to ensure there are no differences between treatments in metrics that could affect the research question results. We will examine the rate at which addresses are flagged by the USPS as being UAA, as return rates and response rates can be influenced by UAA rates.

We will investigate available metrics related to QR code usage, such as how many respondents accessed the internet instrument using a QR code or using a mobile device (i.e., smartphone or tablet).

5. ASSUMPTIONS AND LIMITATIONS

   1. Assumptions

1. A single ACS monthly sample is representative of an entire year (twelve panels) and the entire frame sample, with respect to both response rates and cost, as designed.

2. A single methods panel group (1/24 of the full monthly sample) is representative of the full monthly sample, as designed.

3. For mailings with the same mailout day, we assume that there is no difference between treatments in mail delivery timing or subsequent response time. The treatments will have the same sample size (with the control being smaller) and use the same postal sort and mailout procedures when the mailout timings are intended to be the same. Previous research indicated that postal procedures alone could cause a difference in response rates at a given point in time between experimental treatments of different sizes, with response for the smaller treatments lagging (Heimel, 2016).

1. Limitations

1. Group quarters and sample housing unit addresses from remote Alaska and Puerto Rico are not included in the sample for the test.

2. The relative cost analysis uses estimates to make cost projections. These estimates do not account for monthly variability in production costs, such as changes in staffing, production rates, or printing price adjustments.

3. Each treatment will have unique mail materials and timing. For most comparisons, for any significant differences detected, we will not be able to identify the specific elements in each treatment that caused those differences to occur.

4. Adding a QR code to mail materials, and removing references to the paper questionnaire in earlier mailings in some treatments, could increase calls to TQA if respondents have questions about the QR code or whether they will receive a paper questionnaire. Because TQA calls are not linked to the caller’s identity in the ACS sample, we will not be able to determine if any treatments cause more calls than others. There could be an impact to ACS operating costs that is not captured by this test.

6. TABLE SHELLS

Below are samples of tables that will be used in the final report to show results from this test.

Table 2. Sample Table for Self-Response Return Rates before [M2 Cut/M3 Cut/CAPI] (QR Code)

Mode	Treatment 1	Control	Difference	Adjusted P-Value
Total Self-Response	##.# (#.#)	##.# (#.#)	##.# (#.#)	#.##
Internet	##.# (#.#)	##.# (#.#)	##.# (#.#)	#.##
Mail	##.# (#.#)	##.# (#.#)	##.# (#.#)	#.##
TQA	##.# (#.#)	##.# (#.#)	##.# (#.#)	#.##

Source: U.S. Census Bureau, American Community Survey, 2025 Questionnaire Timing Test. DRB Approval Number: CBDRB-FY26-ACSO003-B00##

Note: Minor additive discrepancies are due to rounding. Standard errors are in parentheses. An asterisk (*) indicates a statistically significant result. Significance was tested based on a two tailed t-test at the α=0.1 level. P-values were adjusted for multiple comparisons using the Hochberg method.

Table 3. Sample Table for Final Response Rates (QR Code)

Mode	Treatment 1	Control	Difference	Adjusted P-Value
Overall Response	##.# (#.#)	##.# (#.#)	##.# (#.#)	#.#
Self-Response	##.# (#.#)	##.# (#.#)	##.# (#.#)	#.#
Internet	##.# (#.#)	##.# (#.#)	##.# (#.#)	#.#
Mail	##.# (#.#)	##.# (#.#)	##.# (#.#)	#.#
TQA	##.# (#.#)	##.# (#.#)	##.# (#.#)	#.#
CAPI	##.# (#.#)	##.# (#.#)	##.# (#.#)	#.#

Source: U.S. Census Bureau, American Community Survey, 2025 Questionnaire Timing Test. DRB Approval Number: CBDRB-FY26-ACSO003-B00##

Note: Minor additive discrepancies are due to rounding. Standard errors are in parentheses. An asterisk (*) indicates a statistically significant result. Significance was tested based on a two tailed t-test at the α=0.1 level. P-values were adjusted for multiple comparisons using the Hochberg method.

Table 4. Sample Table for [Response Mode] Form Completeness Rates (QR Code)

Section	Treatment 1	Control	Difference	Adjusted P-Value
Overall Form	##.# (#.#)	##.# (#.#)	##.# (#.#)	#.#
Basic Person	##.# (#.#)	##.# (#.#)	##.# (#.#)	#.#
Housing	##.# (#.#)	##.# (#.#)	##.# (#.#)	#.#
Detail Person	##.# (#.#)	##.# (#.#)	##.# (#.#)	#.#

Source: U.S. Census Bureau, American Community Survey, 2025 Questionnaire Timing Test. DRB Approval Number: CBDRB-FY26-ACSO003-B00##

Note: Minor additive discrepancies are due to rounding. Standard errors are in parentheses. An asterisk (*) indicates a statistically significant result. Significance was tested based on a two tailed t-test at the α=0.1 level. P-values were adjusted for multiple comparisons using the Hochberg method.

Table 5. Sample Table for [Overall/Response Mode] Demographic Distributions before CAPI (QR Code)

Section	Treatment 1	Control
AGE (p-value)	#.##	#.##
Under 18 years old	##.# (#.#)	##.# (#.#)
18 to 29 years old	##.# (#.#)	##.# (#.#)
30 to 49 years old	##.# (#.#)	##.# (#.#)
50 to 64 years old	##.# (#.#)	##.# (#.#)
65 years old or older	##.# (#.#)	##.# (#.#)
SEX (p-value)	#.##	#.##
Male	##.# (#.#)	##.# (#.#)
Female	##.# (#.#)	##.# (#.#)
HISPANIC ORIGIN (p-value)	#.##	#.##
Hispanic or Latino	##.# (#.#)	##.# (#.#)
Not Hispanic or Latino	##.# (#.#)	##.# (#.#)
RACE (p-value)	#.##	#.##
White alone	##.# (#.#)	##.# (#.#)
Black or African American alone	##.# (#.#)	##.# (#.#)
Other race alone	##.# (#.#)	##.# (#.#)
Two or more races	##.# (#.#)	##.# (#.#)
EDUCATIONAL ATTAINMENT (p-value)	#.##	#.##
High school, GED, or less	##.# (#.#)	##.# (#.#)
Some college or associate degree	##.# (#.#)	##.# (#.#)
Bachelor’s degree or more	##.# (#.#)	##.# (#.#)
BUILDING TYPE (p-value)	#.##	#.##
One-family home	##.# (#.#)	##.# (#.#)
Apartment	##.# (#.#)	##.# (#.#)
Other (boat, van, etc.)	##.# (#.#)	##.# (#.#)
TENURE (p-value)	#.##	#.##
Owned with a mortgage	##.# (#.#)	##.# (#.#)
Owned free and clear	##.# (#.#)	##.# (#.#)
Rented	##.# (#.#)	##.# (#.#)
Occupied without payment of rent	##.# (#.#)	##.# (#.#)

Source: U.S. Census Bureau, American Community Survey, 2025 Questionnaire Timing Test. DRB Approval Number: CBDRB-FY26-ACSO003-B00##

Note: Minor additive discrepancies are due to rounding. Standard errors are in parentheses. An asterisk (*) indicates a statistically significant result. Significance was tested based on a Rao-Scott chi-square test at the α=0.1 level. P-values were adjusted for multiple comparisons using the Hochberg method.

Table 6. Sample Table for Estimated Annual Cost Savings for Treatment 1 versus Control

Treatment	Printing, Postage, Assembly, and Data Capture Savings	CAPI Savings	Total Cost Savings
Control	--	--	--
Treatment 1	$###,###	$###,###	$###,###

Source: U.S. Census Bureau, American Community Survey, 2025 Questionnaire Timing Test. DRB Approval Number: CBDRB-FY26-ACSO003-B00##

Table 7. Sample Table for Self-Response Return Rates before [M2 Cut/M3 Cut/CAPI]

Mode	Treatment 1	Treatment 2	Treatment 3	Treatment 4
Total Self-Response	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Internet	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Mail	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
TQA	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)

Source: U.S. Census Bureau, American Community Survey, 2025 Questionnaire Timing Test. DRB Approval Number: CBDRB-FY26-ACSO003-B00##

Note: Minor additive discrepancies are due to rounding. Standard errors are in parentheses. An asterisk (*) indicates a statistically significant result. Significance was tested based on a two tailed t-test at the α=0.1 level. P-values were adjusted for multiple comparisons using the Hochberg method.

Table 8. Sample Table for Final Response Rates

Mode	Treatment 1	Treatment 2	Treatment 3	Treatment 4
Overall Response	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Self-Response	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Internet	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Mail	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
TQA	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
CAPI	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)

Source: U.S. Census Bureau, American Community Survey, 2025 Questionnaire Timing Test. DRB Approval Number: CBDRB-FY26-ACSO003-B00##

Note: Minor additive discrepancies are due to rounding. Standard errors are in parentheses. An asterisk (*) indicates a statistically significant result. Significance was tested based on a two tailed t-test at the α=0.1 level. P-values were adjusted for multiple comparisons using the Hochberg method.

Table 9. Sample Table for [Response Mode] Form Completeness Rates

Section	Treatment 1	Treatment 2	Treatment 3	Treatment 4
Overall Form	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Basic Person	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Housing	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Detail Person	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)

Source: U.S. Census Bureau, American Community Survey, 2025 Questionnaire Timing Test. DRB Approval Number: CBDRB-FY26-ACSO003-B00##

Note: Minor additive discrepancies are due to rounding. Standard errors are in parentheses. An asterisk (*) indicates a statistically significant result. Significance was tested based on a two tailed t-test at the α=0.1 level. P-values were adjusted for multiple comparisons using the Hochberg method.

Table 10. Sample Table for [Overall/Response Mode] Demographic Distributions before the M3 Cut

Section	Treatment 1	Treatment 2	Treatment 3	Treatment 4
AGE (p-value)		#.##	#.##	#.##
Under 18 years old	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
18 to 29 years old	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
30 to 49 years old	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
50 to 64 years old	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
65 years old or older	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
SEX (p-value)		#.##	#.##	#.##
Male	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Female	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
HISPANIC ORIGIN (p-value)		#.##	#.##	#.##
Hispanic or Latino	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Not Hispanic or Latino	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
RACE (p-value)		#.##	#.##	#.##
White alone	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Black or African American alone	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Other race alone	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Two or more races	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
EDUCATIONAL ATTAINMENT (p-value)		#.##	#.##	#.##
High school, GED, or less	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Some college or associate degree	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Bachelor’s degree or more	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
BUILDING TYPE (p-value)		#.##	#.##	#.##
One-family home	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Apartment	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Other (boat, van, etc.)	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
TENURE (p-value)		#.##	#.##	#.##
Owned with a mortgage	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Owned free and clear	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Rented	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Occupied without payment of rent	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)

Source: U.S. Census Bureau, American Community Survey, 2025 Questionnaire Timing Test. DRB Approval Number: CBDRB-FY26-ACSO003-B00##

Note: Minor additive discrepancies are due to rounding. Standard errors are in parentheses. An asterisk (*) indicates a statistically significant result. Significance was tested based on a Rao-Scott chi-square test at the α=0.1 level. P-values were adjusted for multiple comparisons using the Hochberg method.

Table 11. Sample Table for [Overall/Response Mode] Demographic Distributions before CAPI

Section	Treatment 1	Treatment 2	Treatment 3	Treatment 4
AGE (p-value)		#.##	#.##	#.##
Under 18 years old	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
18 to 29 years old	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
30 to 49 years old	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
50 to 64 years old	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
65 years old or older	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
SEX (p-value)		#.##	#.##	#.##
Male	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Female	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
HISPANIC ORIGIN (p-value)		#.##	#.##	#.##
Hispanic or Latino	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Not Hispanic or Latino	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
RACE (p-value)		#.##	#.##	#.##
White alone	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Black or African American alone	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Other race alone	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Two or more races	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
EDUCATIONAL ATTAINMENT (p-value)		#.##	#.##	#.##
High school, GED, or less	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Some college or associate degree	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Bachelor’s degree or more	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
BUILDING TYPE (p-value)		#.##	#.##	#.##
One-family home	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Apartment	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Other (boat, van, etc.)	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
TENURE (p-value)		#.##	#.##	#.##
Owned with a mortgage	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Owned free and clear	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Rented	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)
Occupied without payment of rent	##.# (#.#)	##.# (#.#)	##.# (#.#)	##.# (#.#)

Source: U.S. Census Bureau, American Community Survey, 2025 Questionnaire Timing Test. DRB Approval Number: CBDRB-FY26-ACSO003-B00##

Note: Minor additive discrepancies are due to rounding. Standard errors are in parentheses. An asterisk (*) indicates a statistically significant result. Significance was tested based on a Rao-Scott chi-square test at the α=0.1 level. P-values were adjusted for multiple comparisons using the Hochberg method.

Table 12. Sample Table for Estimated Annual Cost Savings for Treatments 2-4 versus Treatment 1

Treatment	Printing, Postage, Assembly, and Data Capture Savings	CAPI Savings	Total Cost Savings
Treatment 1	--	--	--
Treatment 2	$###,###	$###,###	$###,###
Treatment 3	$###,###	$###,###	$###,###
Treatment 4	$###,###	$###,###	$###,###

Source: U.S. Census Bureau, American Community Survey, 2025 Questionnaire Timing Test. DRB Approval Number: CBDRB-FY26-ACSO003-B00##

7. POTENTIAL CHANGES TO ACS

Based on the results of this research, the Census Bureau may revise the ACS production mail contact strategy, including the mail materials and timing.

8. REFERENCES

Clark, S., DiFiglia, L., Tancreto, J., & Raglin, D. (2015). 2015 Mail Contact Strategy Modification Test. U.S. Census Bureau. https://www.census.gov/library/working-papers/2015/acs/2015_Clark_03.html

Endres, K., Heiden, E., Park, K., Losch, M., Harland, K., & Abbott, A. (2023). Experimenting with QR Codes and envelope size in push-to-web surveys. Journal of Survey Statistics and Methodology, 12(4), 893-905. https://doi.org/10.1093/jssam/smad008

Heimel, S. (2016). Postal tracking research on the May 2015 ACS panel. U.S. Census Bureau. 2016 American Community Survey Research and Evaluation Report Memorandum Series #ACS16-RER-01.

Hochberg, Y. (1988). A Sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75(4), 800-802. https://doi.org/10.2307/2336325

Letourneau, E., Shia, L., & Moran, B. (2024). 2020 Census self-response and return rates assessment. U.S. Census Bureau. https://www.census.gov/programs-surveys/decennial-census/decade/2020/planning-management/evaluate/eae/2020-operational-assessment-self-response-return-rates.html

Longsine, L., & Spiers, S. (2023). 2021 Initial Mailing Pressure Seal Test. U.S. Census Bureau. https://www.census.gov/library/working-papers/2023/acs/2023_Longsine_01.html

Marlar, J. (2018, August 30). Do quick response codes enhance or hinder surveys? Gallup. https://news.gallup.com/opinion/methodology/241808/quick-response-codes-enhance-hinder-surveys.aspx

Marlar, J., & Schreiner, J. (2024). The use of QR codes to encourage participation in mail push-to-web surveys: An evaluation of experiments from 2015 and 2022. Journal of Survey Statistics and Methodology, 00, 1–17. https://doi.org/10.1093/jssam/smae024

Mendelson, J., & Bergstrom, J. (2013). Age differences in the knowledge and usage of QR codes. Proceedings of the 7th International Conference, Universal Access in Human-Computer Interaction 2013, Las Vegas, NV, USA (pp. 156-161).

Mills, G., Longsine, L., & Sawyer, R. (2021). Respondent device analysis for the internet instrument on the American Community Survey. U.S. Census Bureau. https://www.census.gov/library/working-papers/2021/acs/2022_Mills_04.html

Nichols, E., Konya, S., Horwitz, R., & Raim, A. (2019). 2020 Census research and testing report: The effect of the mail delivery date on survey login rates and helpline call rates. Research and Methodology Directorate, Center for Behavioral Science Methods Research Report Series (Survey Methodology #2019-01). https://www.census.gov/content/dam/Census/library/working-papers/2019/adrm/rsm2019-01.pdf

Oliver, B., Contard, L., & Barth, D. (2023). 2021 ACS Strategic Framework Mail Materials Test report. U.S. Census Bureau. https://www.census.gov/library/working-papers/2023/acs/2023_Oliver_01.html

Rao, J., & Scott, A. (1987). On simple adjustments to chi-square tests with sample survey data. The Annals of Statistics, 15(1), 385–397. https://doi.org/10.1214/aos/1176350273

Risley, M., Barth, D., Cheza, K., & Rabe, M. (2018). 2017 Pressure Seal Mailing Materials Test. U.S. Census Bureau. https://www.census.gov/library/working-papers/2018/acs/2018_Risley_01.html

Risley, M., & Berkley, J. (2020). 2018 ACS Mail Materials Test report. U.S. Census Bureau. https://www.census.gov/library/working-papers/2020/acs/2021_Risley_01.html

Rivas, A., Peterson, A., Schulzetenberg, A., & Wang, L. (2024). Usability of Quick Response (QR) codes as a method to access a survey using a smartphone. U.S. Census Bureau Working Paper rsm2024-05. https://www.census.gov/library/working-papers/2024/adrm/rsm2024-05.html

Rivas, A., & Schulzetenberg, A. (2023). QR codes as a method for older adults to access a mobile survey. In Q. Gao, & J. Zhou (Eds.), Lecture Notes in Computer Science: Vol. 14042. Human Aspects of IT for the Aged Population. 128-141. https://doi.org/10.1007/978-3-031-34866-2_10

Smith, A., Hopkins, B., Dever, J., Fobia, A., Gomori, S., Snee, E., & Williams, D. (2023, May 10-12). Just scan it: The effectiveness of including QR codes in panel recruitment materials [Presentation]. 2023 American Association for Public Opinion Research Conference, Philadelphia, PA.

U.S. Census Bureau. (2024). American Community Survey and Puerto Rico Community Survey Design and Methodology, Version 4.0. https://www.census.gov/programs-surveys/acs/methodology/design-and-methodology.html

1. Images of the Control Treatment Mail Materials (ACS Production)

Figure 7. Inside of Initial Pressure Seal Mailer (Control Treatment)

Figure 8. Inside of Reminder Pressure Seal Mailer (Control Treatment)

Figure 9. Outside of Initial and Reminder Pressure Seal Mailer (Control Treatment)

Note: Form number in lower right corner varies for each mailing.

Figure 10. Front of Letter in Paper Questionnaire Package (Control Treatment)

Figure 11. Back of Letter in Paper Questionnaire Package (Control Treatment)

Figure 12. Outgoing Envelope for Paper Questionnaire Package (Control Treatment)

Figure 13. Front and Back of Reminder Postcard (Control Treatment)

Figure 14. Inside of the Due Date Letter (Control Treatment)

Note: the printed letter will have the due date printed after “Due:” in the box.

Figure 15. Outside of the Due Date Letter (Control Treatment)

Note: the printed letter will have the due date printed under “Response Due:” in the box.

2. Images of Treatment 1 Mail Materials

The following table lists the Treatment 1 materials that will and will not be different from the Control Treatment. The following images are mock-ups of changed materials. The internet user ID and QR code on the materials are fictitious and there for placement.

Treatment 1 Mail Materials Compared to Control	List of Mail Materials
Same design between treatments	Outside of PSMs Back of Paper Questionnaire Letter Back of Reminder Postcard Questionnaire Package envelopes (outgoing and return) Questionnaire
Different design between treatments	Inside of Initial PSM Inside of Reminder PSM Front of Paper Questionnaire Letter Front of Reminder Postcard Inside of Due Date Letter
Removed from Treatment 1 (i.e., in Control Treatment)	None
Added to Treatment 1 (i.e., not in Control Treatment)	None

Figure 16. Inside of Initial Pressure Seal Mailer (Treatment 1)

Figure 17. Inside of Reminder Pressure Seal Mailer (Treatment 1)

Figure 18. Front of Letter in Paper Questionnaire Package (Treatment 1)

Figure 19. Front of Reminder Postcard (Treatment 1)

Figure 20. Inside of Due Date Letter (Treatment 1)

Note: the printed letter will have the due date printed after “Due:” in the box and after “Respond by” in the bolded sentence.

3. Images of Treatment 2-4 Mail Materials

The following table lists the materials in Treatments 2-4 that will and will not be different from Treatment 1. The following images are mock-ups of changed or added materials.

Treatment 2-4 Mail Materials Compared to Treatment 1	List of Treatment 2 Mail Materials	List of Treatment 3 Mail Materials	List of Treatment 4 Mail Materials
Same design between treatments	Outside of PSMs Back of Reminder Postcard Questionnaire Package envelopes (outgoing and return) Questionnaire Due Date Letter	Outside of PSMs Questionnaire Package envelopes (outgoing and return) Questionnaire	Outside of PSMs Back of Reminder Postcard Questionnaire Package return envelope Questionnaire
Different design between treatments	Inside of Initial PSM Inside of Reminder PSM Paper Questionnaire Letter Front of Reminder Postcard	Inside of Initial PSM Inside of Reminder PSM Paper Questionnaire Letter Front and back of Reminder Postcard	Inside of Initial PSM Inside of Reminder PSM Paper Questionnaire Letter Questionnaire Package outgoing envelope Front of Reminder Postcard
Removed from treatment (i.e., in Treatment 1)	None	Due Date Letter	Due Date Letter
Added to treatment (i.e., not in Treatment 1)	None	New Reminder PSM (3rd mailing)	New Reminder PSM (4th mailing)

Figure 21. Inside of Initial Pressure Seal Mailer (Treatments 2-4)

Figure 22. Inside of Reminder Pressure Seal Mailer (Treatments 2-4)

Figure 23. Front of Reminder Postcard (Treatment 2)

Figure 24. Front of Reminder Postcard (Treatment 4)

Figure 25. Inside of New Reminder Pressure Seal Mailer (Treatments 3 and 4)

Figure 26. Front of Letter in Paper Questionnaire Package (Treatments 2 and 3)

Figure 27. Back of Letter in Paper Questionnaire Package (Treatments 2 and 3)

Figure 28. Front and Back of Reminder Postcard (Treatment 3)

Note: the printed postcard will have the due date printed below “Response Due:” on the front and in the box on the back.

Figure 29. Front of Letter in Paper Questionnaire Package (Treatment 4)

Note: the printed letter will have the due date printed after “Due:” in the box and after “Respond by” in the bolded sentence.

Figure 30. Back of Letter in Paper Questionnaire Package (Treatment 4)

Figure 31. Outgoing Envelope of Paper Questionnaire Package (Treatment 4)

Note: the printed envelope will have the due date printed after “Response Due:” below the box.

4. Timeline of Treatments

Figure 32 shows a timeline of each treatment. It shows the day each mail piece will be mailed as well as the days new mail universes will be created (i.e., the mail universe cut date).

Figure 32. Self-Response Data Collection Contact Strategy for Each Treatment

Note:

Dates in black are weekends, and dates in red are federal holidays.

PSM = Pressure Seal Mailer

Qx = Questionnaire

Specified Due Date = Due date listed on mail material

M2 = Mailing Two Universe, M3 = Mailing Three Universe, M4 = Mailing Four Universe

1 The user could still type in the URL that is on the ACS mail materials to reach the internet instrument website. No matter which method they use to get to the internet instrument (typing URL or scanning QR code), they would still need to input the user ID that is on the ACS mail materials to respond.

2 Addresses deemed “undeliverable as addressed” (UAA) by the United States Postal Service are not included in the M2 and subsequent mailing universes.

3 CAPI interviews start at the beginning of the month following the fifth mailing.

4 This mailing, including the M4 universe, will not be altered for this test.

5 Sending a second questionnaire to nonrespondents was considered for the 2000 Census but was not operationalized due to the volume of mailings and other factors.

6 For more information on the decision to eliminate the CATI operation, see the following ACS program notification: https://www.census.gov/programs-surveys/acs/news/updates/2017.html

7 In addition to the pressure seal mailer changes, the 2022 contact strategy (Figure 7) differs from the 2017 contact strategy (Figure 6) in two ways: 1) the questionnaire package FAQ brochure and instruction card were removed in January 2020 and 2) a due date was added to the final reminder letter in October 2020.

8 The experiment had additional treatments that tested incentives and larger envelope sizes along with the QR code and URL.

9 It is not permitted for the ACS to provide personalized QR codes. The QR code for this test will link to the ACS landing page. Users will still need to log in to the survey by manually entering their user ID.

10 We had to make slight design changes to the ACS production materials to fit the QR code. Changes were minimal and avoided when possible. The QR code will follow the latest Census Bureau QR code guidance.

11 We are not including a treatment that only removes the reference from the first mailing. Removing the reference from the first two mailings in Treatments 2-4 aligns with the various questionnaire timings since the questionnaire will no longer be received within a few weeks. Additionally, the 2021 IMPS Test showed that removing the paper questionnaire reference from the first mailing increased self-response in early data collection compared to the de-emphasized message in control and Treatment 1 (Longsine & Spiers, 2023). Thus, since it has already been proven effective, we are not retesting the message removal in the first mailing alone.

12 See Appendix D for when the mailing cuts are for each treatment.

13 A blank form is a form in which there are no persons with sufficient response data and there is no telephone number listed on the form.

14 In general, a sufficient partial internet response is one that has at least minimal information, which indicates an attempt to respond. The specific definition of a sufficient partial internet response is sensitive and for Census Bureau internal use only.

15 We will remove addresses deemed to be UAA by the USPS if no response is received.

16 Priority is given to a mail return in this situation because it would have been completed and mailed earlier than a TQA or internet return received the same day.

17 The number of questions that should have been answered is determined based on questionnaire skip patterns and respondent answers. If it is not clear if a question should have been answered (because a prior question was left blank), it is excluded from the calculation.

18 We may combine some categories within a demographic if needed due to small cell sizes. Additionally, we may combine mail and TQA responses if there is no significant difference in TQA responses.