National Survey on Drug Use and Health:
Questionnaire Change Flexibilities
Online Panel/Platform Test
A. JUSTIFICATION
The Substance Abuse and Mental Health Services Administration (SAMHSA) is requesting OMB approval to survey a maximum of 420 participants recruited via two different online panels from the general population and six subpopulations to compare the effectiveness of the panels in recruiting participants for quick turnaround testing of questions for the National Survey on Drug Use and Health (NSDUH). This questionnaire change flexibilities package is submitted under the NSDUH Methodological Field Tests generic OMB clearance (OMB No. 0930-0290).
NSDUH is sponsored by SAMHSA’s Center for Behavioral Health Statistics and Quality (CBHSQ) and approved under OMB No. 0930-0110. The data collection is a national survey of the U.S. civilian, non-institutionalized population aged 12 and older. This survey is paramount in meeting a critical objective of SAMHSA’s mission—to maintain current data on the incidence and prevalence of substance use and mental health problems in the United States. NSDUH has been conducted on a periodic basis from 1971 to 1988, and annually since 1990.
NSDUH is authorized by Section 505 of the Public Health Service Act (42 USC 290aa4 – Data Collection). Section 505 specifically authorizes annual data collection for monitoring the incidence and prevalence of illicit substance use and mental health problems, as well as the abuse of licit substances in the U.S. population.
NSDUH is a crucial source of data on substance use in the United States for SAMHSA, and SAMHSA expects that the survey will keep abreast of changing use patterns. At the same time, maintenance of the trend data collected in NSDUH is crucial, and SAMHSA is appropriately conservative in making changes to the questionnaire, knowing that small changes to question wording, context, or order can disrupt trend lines. Finding the appropriate balance between a flexible questionnaire that can detect and respond to changes in drug use patterns and a stable questionnaire that consistently uses the same (or comparable) terms across years is challenging. For NSDUH to continue collecting high-quality data, periodic evaluation and redesign efforts are undertaken to reflect changes in substance use and mental health issues as well as innovations in data collection methodologies.
Recent changes in substance use trends and other health-related issues covered in NSDUH have given rise to the need to 1) define procedures to determine when and how to add questions to NSDUH and 2) develop approaches for rapidly testing new questions. To this end, two documents were developed - a standard operating procedures memo to help stakeholders understand the issues to be considered in adding questions to NSDUH and a document detailing different online panels and platforms that could be used to conduct quick turnaround testing of new questions proposed for NSDUH. Although developing the second document afforded the opportunity to narrow down the field of available panels/platforms to help with quick turnaround testing, this work also revealed the need for testing each panel, to more extensively assess the specific features of each in relation to their likely effectiveness for NSDUH.
Effectiveness for NSDUH includes assessing the ability to rapidly recruit subpopulations of interest for NSDUH, how well the response distributions of those recruited correspond to those who have responded to NSDUH in the past, level of cost and ease of logistics. This study initially approached five panels to participate, and three panels declined participation as they felt that they would have difficulty providing specific drug users and would have issues providing youth participants.
To determine the effectiveness and efficiency of utilizing online panels/platforms to recruit respondents from the desired subpopulations and reduce the timeline for future question testing, the following remaining two panels1 will be compared:
Ipsos’ KnowledgePanel
Amazon’s Mechanical Turk
A survey will be completed with recruited participants via the web with individuals from the following groups: general population; youth (aged 12 to 17); Spanish speakers; those who have received substance use treatment in the past 12 months; those who have used methamphetamine, fentanyl, or kratom in the past 12 months; those with specific mental health issues; and those who have received mental health treatment in the past 12 months.
The findings from the online panel testing will be delivered by March 31, 2022, to assist SAMHSA in determining efficient recruitment procedures for quick turnaround testing of future NSDUH questionnaire items.
The purpose of recruiting participants through the online panels is to provide a testing opportunity to assess the specific features of each panel in relation to their effectiveness for NSDUH.
A maximum of 420 participants (30 participants from each subpopulation, for a possible total of 210 participants from each of the two panels) will be surveyed in this study. Some participants may belong to multiple subpopulations, resulting in less than 420 unique participants overall. Additionally, one of the goals of this study is to determine how many screenings will have to be administered to reach 30 participants in each subpopulation, but screening estimates for each panel are approximately 10,000 each (Tables 4 and 5 in Section B1 provide more detail on sample goals).
Previously-fielded questions from the 2019 and 2020 NSDUH surveys will be used in order to enable a comparison for each subpopulation listed below between existing data and data collected from panel recruits. In addition to the general population, participants will be surveyed from the six subpopulations of interest: youth (aged 12 to 17); Spanish speakers; those who have received substance use treatment in the past 12 months; those who have used methamphetamine, fentanyl, or kratom in the past 12 months; those with specific mental health issues; and those who have received mental health treatment in the past 12 months. The populations and subpopulations will overlap in some areas (for example, youth with substance use treatment in the past 12 months may also screen in for mental health treatment in the past 12 months), but the expectation is to complete a maximum of 30 responses for each subpopulation for each panel.
As explained in Section A.1, the findings from the survey data will be used to determine which online panel is the most effective in recruiting participants in the desired subpopulations for quick turnaround testing of new questionnaire items for future NSDUH surveys.
Information technology will be used to administer both the screener within the panel platform and the web survey for participants who are determined to be eligible after screening by the panels. For those that pass the screener requirements, a link will be provided to complete the main survey online. Potential adolescent participants will be screened through their parents via the panels, after which parental consent will be obtained through the panel (KnowledgePanel) or collected over the telephone (MTurk).
Adult participants will be recruited through the online panels and will complete a short web screener within the panel environment that will be used to determine if they are eligible to complete the main survey. The Contractor will provide screening criteria (Attachment A) to the panels for programming. The screeners will be programmed by the panels and administered through their platforms so that no PII is handled by the Contractor. During the screening, the panels will collect information on the age, race/ethnicity, education, past year substance use, endorsement of past year substance use treatment and past year mental health treatment.
After completing the screener, and if eligible to complete the main survey, panels will provide the participant with a survey link. The main survey will be conducted online and consist of a subset of questions previously fielded from the 2019 and 2020 NSDUH surveys (Attachment E). This includes modules for basic demographics, past year drug use for alcohol, marijuana, cocaine, heroin, methamphetamines and kratom, as well as medical and nonmedical use of prescription drugs, substance use disorder questions (DSM IV), drug treatment, adult/adolescent depression, mental health, and adult/youth mental health service utilization. No new questions were added to these modules so that responses can be compared with existing NSDUH data.
CBHSQ is in contact with all major Federal health survey managers and is aware of no other efforts to assess the effectiveness of online panels in recruiting potential respondents for rapid turnaround testing of new questions proposed for the NSDUH questionnaire. To date, no duplication of effort has been identified.
This survey does not involve small businesses or other such entities.
The findings of this data collection will inform future evaluation and redesign efforts of the NSDUH so that new questions can be tested and added to the survey under tight deadlines. This project is a one-time collection and will not be repeated.
This information collection fully complies with 5 CFR 1320.5(d)(2).
No consultation outside the agency was conducted for this task.
Both panels have made a recommendation for incentives based on the level of effort and burden on the respondent. Since this task involves only a web survey and recruitment is conducted through online panels with no live interaction, the incentive amount has been adjusted for each panel to align with what their participants typically receive for similar participation. The incentive amount is based on the level of effort, time burden, and sensitivity of the questions. The survey will take, on average, 30 minutes to complete. This incentive amount will be sufficient for the participants’ time based on what is typically offered by panels for surveys of a similar length and content. The incentive amounts are recommended for adolescents as well as adults based on experience recruiting participants from a specific population – in this case, adolescent drug users, those with specific mental health issues, or who have received substance use or mental health treatment in the past 12 months. Incentives offered by both panels are listed in Table 1 below.
Table 1. Survey Incentives Offered to Participants by the Two Online Panels
Panel |
Screener incentive |
Survey incentive |
Ipsos’ KnowledgePanel |
None |
$10 |
Amazon’s Mechanical Turk |
$0.50 |
$8 |
The survey incentive is mentioned in the following materials: Parental Permission/Youth Assent (Attachment B), Participant Informed Consent Screens Adult (Attachment C) and Participant Informed Consent Screens Adolescent (Attachment D).
Concern for the confidentiality and protection of respondents’ rights has always played a central part in the implementation of NSDUH and will continue to be given the utmost emphasis.
The Contractor’s Institutional Review Board (IRB) was granted a Federalwide Assurance (Attachment F) by the Office for Human Research Protections (OHRP) and HHS in compliance with the requirements for the protection of human subjects (45 CFR 46).
The surveys will incorporate several procedures to ensure that respondents’ rights will be protected. Recruitment efforts through the panels will advertise to the participants that “All responses will be kept confidential under federal law”. In the parental permission script (Attachment B), and the participant introduction/informed consent screen for the survey (Attachments C and D), participants are informed that their responses are voluntary and are assured there will be no penalties if they decide not to respond, either to the information collection as a whole or to any question.
Since the recruitment and screening process will be conducted by the panels, no personal information will be collected by or transferred to the Contractor. Those who are eligible to participate in the survey after screening will be directed to the link by KnowledgePanel. Since MTurk does not collect any personal or contact information of users, those who are recruited and determined eligible by the screener will be invited to participate in the survey through a separate posting on the platform. In all cases, a panel ID will be used to identify the participant so that when the survey is completed, the panel can issue the incentive. The Contractor will have only a panel ID and no access to names or other personal information associated with the participants.
For surveys conducted with adolescents, parents will provide consent via the panel (KnowledgePanel) or an inbound phone call prior to the survey (MTurk). If parental consent is provided over the phone, youth will give their verbal assent over the phone at that time (Attachment B). Once they begin the survey, the adolescent will assent at the beginning of the the survey questions (Attachment D).
Adult survey participants will also acknowledge an informed consent screen within the survey before proceeding with the survey items (Attachment C).
Only those respondents who consent/assent to the survey will be allowed via the web program to complete the online survey.
During the survey, age and gender will be collected from all participants but only to inform the web survey program which questions to display and to tailor wording. No links to individual participants will be preserved, and personal identifying information will not be included in the data or final report delivered to CBHSQ.
Many of the questions asked concern topics that are likely to be of a sensitive nature, including alcohol use by persons under the age of 21, use of illegal drugs by participants of all ages, questions about substance use dependence and withdrawal, questions about depression and other mental health issues. Consequently, some of these questions could be distressing for some respondents. The questions that will be included in the survey are provided in Attachment E, Survey Questions.
Measures will be taken to reduce risks to the respondents. Primarily, respondents will complete the interview online and have no interaction with an interviewer. During the introduction/informed consent screens of the main survey, respondents will be told to make sure they are completing the survey questions in a private setting and will be asked to acknowledge that within the informed consent screens. Additionally, SAMHSA's 24-hour toll-free Treatment Referral Helpline (1-800-662-HELP) will be included within the web survey.
As noted in Section 10, potential participants and the actual participants will be assured during recruitment and in the survey that the information they provide is voluntary and will be handled in a confidential manner. These efforts will be made to help participants feel more comfortable with participating in the survey.
Raw data from the screening questionnaires will remain within the panels’ environment and will not be retained once a participant is 1) deemed ineligible, or 2) completes the survey and is provided with an incentive. Panels will not update participant information with any information collected during the screening process. Raw data from the web survey will be password protected during the interview process. None of this information will be retained once the data have been extracted and aggregated; nor will the information become part of a system of records containing permanent identifiers that can be used for retrieval. Eligible participants will only be identified within the panel platform with a specific Panel ID, which will be passed to the Contractor. The Contractor will let the panels know which participants have completed the main survey via Panel ID only.
Administration of the screener during the recruitment process will take an average of 5 minutes per participant. It is estimated that the average amount of time required to complete the survey will be approximately 30 minutes.
The recruitment, data collection, and data review phases for this study will span approximately 3 months, from January 2022 through March 2022.
The respondent burden for this study is shown in Table 2 below. The hourly wage of $19.13 was calculated based on weighted data from the 2019 NSDUH and respondents’ reported personal annual income.
Table 2. Estimated Burden for Surveys for Questionnaire Change Flexibilities
Activity |
Number of Respondents |
Responses per Respondent |
Average Burden per Response (Hours) |
Total Burden (Hours) |
Hourly Wage Rate |
Total Hour Cost |
Screening |
20,000 |
1 |
0.083 |
1,660.00 |
$19.13 |
$31,755.80 |
Interviews |
420 |
1 |
0.500 |
210.00 |
$19.13 |
$ 4,017.30 |
TOTAL |
20,420 |
– |
0.091576885407 |
1,870.00 |
– |
$35,773.10 |
There are no capital, startup, operational, or maintenance costs to respondents.
Total costs associated with the interviews are estimated to be $155,000 over a 10-month period. Of the total costs, $140,000 are for contract costs (e.g. recruiting/screening, collecting interview data, analyzing, and developing a report for CBHSQ), and approximately $15,000 represents CBHSQ costs to manage the task.
The total burden hours approved under the Methodological Field Tests was 2,741 hours. For the Questionnaire Change Flexibilities Online Panel/Platform Test, SAMHSA is requesting 1,840 burden hours, leaving a balance of 901 burden hours.
The recruitment and data collection process, combined with the survey data, will be used to evaluate the efficiency and effectiveness of the two online panels for recruiting respondents from the desired subgroups.
Along with logistics and cost information for both panels, unweighted response distributions from survey items in this study will be compared to unweighted response distributions received from the original 2019/2020 items, per subpopulation to see how well they correspond.
The results will be summarized in an internal report used to make recommendations for future testing of NSDUH questionnaire items.
The schedule for data collection for the NSDUH Questionnaire Change Flexibilities Online Panel/Platform Test is included in Table 3 below.
Table 3. Schedule for Data Collection for Online Panel/Platform Testing
Subtask |
Date |
Recruiting for survey participants & data collection begins |
1/10/2022 |
All surveys completed |
2/16/2022 |
Final report completed |
3/31/2022 |
The OMB expiration date will be displayed at the beginning of the survey.
The certifications are included in this submission and fully comply with 5 CFR 1320.9.
Attachment A Panel Recruitment Screening Criteria
Attachment B Parental Permission/Youth Assent Scripts
Attachment C Participant Introduction/Informed Consent Screens (Adult)
Attachment D Participant Introduction/Informed Consent Screens (Adolescent)
Attachment E Survey Questions
Attachment F Federalwide Assurance
1 For ease of discussion, the term panel throughout the rest of the document is used even though Amazon’s Mechanical Turk is a platform.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | 2003 National Survey on Drug Use and Health |
| Author | lky |
| File Modified | 0000-00-00 |
| File Created | 2022-05-08 |