OMB Memo

Generic Information Collection Request:
For Cognitive Interviewing of the Annual Social and Economic Supplement (ASEC)

of the Current Population Survey

Request: The Census Bureau, through a contract with RTI International and Research Support Services (RSS), plans to conduct two rounds of cognitive interviewing to test the newly designed web-based questionnaire for the Annual Social and Economic Supplement (ASEC) of the Current Population Survey. This testing will be conducted under the generic clearance for questionnaire pretesting research (OMB number 0607-0725).

Purpose: Efforts are underway to add an internet self-response option to ASEC, which is currently conducted only in-person and by telephone. Creating a self-administered web version of the survey involves rewriting most of the questions to work for a visual format, rather than aural. In addition, without the interviewer to provide help and definitions, we have decisions to make about when and where to provide help instructions and definitions. Lastly, Census is trying to reduce the burden on respondents by condensing some question series and harnessing the technology of the web to employ visual formats to help respondents see the connections between the information they provide. We need to be confident that the redesigned questions are understood and we do not introduce any measurement error. Therefore, the Census Bureau intends to conduct cognitive testing of the ASEC web instrument to evaluate how participants understand the questions and what is being asked of them.

Overview of Testing Protocol: The RTI-RSS team will conduct two rounds of cognitive interview testing. The first round will begin in August 2024 and the second round of testing will begin in June 2025. Each round will separately test the health insurance and income & poverty sections with two different groups of respondents. The smaller modules (out of pocket expenses, migration, and employment) will be included in one of the two tests. Each cognitive interview will take approximately 90 minutes. Participants will complete a section of the survey (usually one screen/question) while thinking aloud and then an interviewer will probe on things like: how they arrived at their answer, could they have chosen a different response, how they understand terms in the question and response option, if they read and understood instructions, if they understood what was being asked of them, etc.

RTI oversees the project and will lead the participant recruitment and scheduling. Both RTI and RSS will conduct the interviews and collaborate on the reporting.

Population of Interest: ASEC surveys a nationally representative sample of households in the United States. We plan to recruit from this general population as described further below.

Timeline: The first round of testing will begin at the end of August 2024 and last for approximately 12 weeks, concluding in November. The second round of testing will begin in June 2025 and last for approximately 13 weeks, concluding in September.

Language: Testing will be conducted in English.

Sample: Each round will include 60 participants for cognitive interviews: 30 participants for health insurance and 30 participants for income & poverty. Therefore, across the two rounds, there will be 120 participants.

Recruitment: RTI will use a combination of online advertisements and offline recruitment methods for recruitment, including online advertising, personal networks, and word-of-mouth referrals (i.e., snowball).

RTI will place advertisements on www.craigslist.com. Advertisements will be posted in the jobs/et cetera section in different metropolitan areas across the country to recruit geographically diverse respondents to participate in the virtual interviews. They will also identify websites frequented by groups that reflect the targeted participant characteristics. These may include listservs, forums, blogs, and online social networks such as Facebook groups, Reddit, and Nextdoor.

The team will use personal and professional networks to recruit participants via word-of-mouth and snowball referrals. To maximize the utility of the word-of-mouth snowballing strategy, we will also include a “referral” question in the recruitment screener to allow prospective recruits to provide contact information of additional individuals who may be interested in participating in the study.

Interview participants will be recruited and screened by a web-based screener. The link to complete the screener may be accessed directly from recruitment advertisements. Interested participants may also call the project’s toll-free phone number to be screened over the phone by a recruiter, who will enter the participant’s data into the web screener. The toll-free phone number will be included on online advertisements and the messaging used for dissemination to personal networks and word-of-mouth recruitment efforts. Including the toll-free phone number and option to be screened via the phone will capture potential participations that have low/no internet usage.

Recruitment characteristics are divided into quotas (i.e., primary criteria) and soft targets (i.e., secondary criteria). Primary and secondary quotas are not mutually exclusive and participants may represent more than one of the primary or secondary characteristics. Separate recruitment characteristics are targeted for each topic (see Attachment 1). For the health insurance testing, primary quotas involve people who are/have: eligible for Medicare, not the policyholder, periodically uninsured, experienced unemployment, experienced significant life events, worked nonstandard jobs, and enrolled in Medicaid. For the income testing, primary quotas involve people who are/have: retired, unemployed, self-employed, second jobs, interests and dividends, and enrolled in public assistance. For both topics, at least 80% of respondents must live in a household with at least one other person.

Method: Staff from RTI and RSS will conduct cognitive interview sessions remotely using Microsoft Teams. Interviews will be recorded.

Protocol: Each cognitive interview will take approximately 90 minutes. Participants will complete a section of the survey (usually one screen/question) while thinking aloud and then an interviewer will probe on things like: how they arrived at their answer, could they have chosen a different response, how they understand terms in the question and response options, if they read and understood instructions, if they understood what was being asked of them, etc. Then, the participant will continue to the next screen/question until they have reached the end of the instrument or 90 minutes has passed.

Consent: We will inform respondents that their response is voluntary and that the information they provide is confidential and will be accessed only by employees involved in the research project. The consent form will also indicate that the respondent agrees that the interview can be audio and videotaped to facilitate analysis of the results. Verbal consent will be captured on recordings. Respondents who do not consent to be video and/or audio-taped will still be allowed to participate.

Incentive: All participants, across both rounds, will receive $40 for their participation.

Length of Interview: Each cognitive interview will take about 90 minutes.

MicroTesting: In addition to the interviews contracted through RTI International, this project will use MicroTesting, as needed, to test short sections of the web instrument with a small sample. These tests may be necessary because: a section could not be thoroughly tested in the contracted rounds of cognitive testing, our recruitment did not attract people with the unique characteristics needed to trigger a particular pathway in the instrument, we want to A/B test two versions of a question to decide which works better, or we want to pilot revised questions before placing them on the round 2 instrument.

MicroTesting will conduct cognitive interviews with approximately 5-10 participants for each test. The section of the instrument tested will include 3 - 10 questions. We expect the interviews to take no more than 20 minutes but allow an additional 10 minutes for camera set-up, consent, and instructions. The protocol is similar to the large-scale cognitive testing but scaled down. The participants will be recruited by Census using word of mouth, online targeting (e.g., Facebook groups), and asking for volunteers from the Affinity panel (small batch email invitations). The criteria for participation will be specific to the unique characteristics necessary to thoroughly test the desired section of the questionnaire. Census will conduct the interviews, which will not be recorded. No incentive is planned.

MicroTesting will be conducted as a series of small tests. The protocol will be repeated for each section that needs additional testing. MicroTesting will occur as needed, primarily between the main rounds of cognitive testing.

Burden hours: Screening questionnaires will take approximately 15 minutes per person. We estimate that we will screen 6 people for each successful recruit for the 120 contracted interviews. Therefore, we estimate that 720 people will be screened, involving 180 burden hours. We use the same calculations for MircoTesting (15 minutes for the screener; 6 people per recruit). To recruit up to 80 people, this we will screen up to 480 people. This results in up to 120 hours. The maximum total expected burden for the screening is 300 hours.

Each contracted cognitive interview will take a maximum of 90 minutes. For 120 participants, this is 180 burden hours. For the MicroTesting, we expect a maximum of 80 people with a time commitment of up to a half an hour, resulting in another 40 hours of burden. The maximum total expected burden for the interviews is 220 hours.

The total burden hours are: 520.

Below is a list of materials to be used in the current study:

Attachment 1: Quotas for recruitment, rounds 1 and 2

Attachment 2. Screening questionnaire

Attachment 3. Recruitment ads

Attachment 4. Consent Form

Attachment 5. Questionnaire for round 1, Health Insurance instrument

Attachment 6. Interview protocol (probes) for round 1, Health Insurance instrument

Attachment 7: Questionnaire for round 1, Income instrument

Attachment 8: Interview protocol (probes) for round 1, Income instrument

Contact: The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Jennifer Sinibaldi

Center for Behavioral Science Methods

U.S. Census Bureau

Washington, D.C. 20233

301.763.6777

Jennifer.Sinibaldi@census.gov