HINTS memo Change Request Cycle 3

Date: May 28, 2013

To: Office of Management and Budget (OMB)

Through: Keith Tucker, Report Clearance Officer, HHS

Seleda Perryman, Program Clearance Officer, NIH

Vivian Horovitch-Kelley, PRA OMB Clearance Liaison, NCI

From: Bradford Hesse, Ph.D., HINTS Project Officer and Chief,

Health Communication and Informatics Research Branch

National Cancer Institute (NCI)/NIH

Subject: Non-Substantive Change Request and Cycle 3 Instrument for,

“Health Information National Trends Survey 4 (HINTS 4)”

(OMB NO. 0925-0538, Expiry Date 10/31/2014)

In accordance with the teleconferences between OMB and the HINTS program staff on November 29, 2010 and May 30, 2012, this memo is a non-substantive change request which summarizes the Health Information National Trends Survey (HINTS) decisions about the survey design and implementation for Cycle 3 data collection.

To date, five sub-studies have been approved conducted under OMB No. 0925-0589 for HINTS to finalize materials and test procedures. They are detailed in Appendix A.

This memo reviews the results of experiments conducted as part of the Cycle 2 data collection (OMB No. 0925-0538, exp. 10/31/2014) and the resulting decisions that have been made for Cycle 3. Specifically, the memo covers:

• changes to the mailing strategy to improve response from the Spanish speaking population; and

• final questionnaire content decisions.

As outlined in the Supporting Statement of the OMB package submitted for HINTS 4, the target population is all adults age 18 or older in the civilian non-institutionalized population of the United States. HINTS 4 uses an address-based sampling frame, selecting the sample from all residential addresses in the U.S., and uses mail data collection procedures and paper questionnaires.

Results from the HINTS Cycle 2 mailing experiment

The mailing procedures for Cycle 2 were altered from Cycle 1 in an attempt to improve response from Spanish speaking households. The mailing procedures in Cycle 2 continued the procedure of mailing both an English and Spanish questionnaire to households either:

1. in a linguistically isolated area; or

2) with the address associated with a Hispanic surname.

For these households, a Spanish questionnaire was mailed with the English questionnaire in the first mailing.¹ In addition, an experiment was conducted that used the above procedure for all households, not just those identified in one of the two above groups. Table 1 provides the sample sizes for the experiment. Note that this lists 10,000 as the sample size for the ‘control’ group, but only those in the two above groups actually got the Spanish questionnaire. Everyone in the treatment group got the Spanish questionnaire at all mailings.

Table 1. Cycle 2 Experimental Design

Condition	Sample Size
Condition 1: Spanish questionnaire mailed only to addresses with Hispanic surname or in linguistically isolated area	10,000
Condition 2: Spanish questionnaire mailed to all addresses in sample	2,000
Total	12,000

We examined several outcomes from the experiment. The response rate was slightly higher in condition 1 where the Spanish questionnaire was only mailed to the Hispanic surname and linguistically isolated strata. As shown in Table 2, this confirms that it is possible to mail both questionnaires to all types of households without absorbing a significant drop in the overall response rate.

Table 2: Weighted response rate by experimental condition

Final Outcome	Experimental Condition
	Condition 1: Spanish questionnaire mailed only to addresses with Hispanic surname or in linguistically isolated area	Condition 2: Spanish questionnaire mailed to all addresses in sample
Complete/Partial	41.0%	38.3%
Nonresponse	58.2%	60.3%
Refusal	0.9%	1.4%
Total	100%	100%
N	8,834	1,752

Table 3 shows that there was very little difference in the response rates between the experimental conditions in the high minority strata. The response rate was slightly lower for condition 2 compared to condition 1 in the low minority stratum.

Table 3: Unweighted response rate by strata and experimental condition

Final Outcome	Low Minority		High Minority
	Condition 1	Condition 2	Condition 1	Condition 2
Complete/partial	45.0%	41.6%	29.3%	28.6%
Nonresponse	54.1%	56.9%	70.4%	70.5%
Refusal	0.9%	1.6%	0.7%	.09%
Total	100%	100%	100%	100%
N	3,235	640	5,447	1,080

We also examined the percentage of Spanish mail returns within each condition. Table 4 shows a slightly higher percentage of Spanish language returns in condition 2 where the Spanish questionnaire was mailed to all households. However, this difference is very small and not consistent with the research that motivated the revised procedure.²

Table 4: Percentage of Spanish language returns by experimental condition

Language of questionnaire returned	Experimental Condition
	Condition 1: Spanish questionnaire mailed only to addresses with Hispanic surname or in linguistically isolated area	Condition 2: Spanish questionnaire mailed to all addresses in sample
Spanish	3.2%	4.1%
English	96.8%	95.9%
Total	100%	100%
N	3,099	587

As shown in Table 5, there was an overall increase in the percentage of Spanish returns in Cycle 2 compared to Cycle 1. A total of 122 Spanish questionnaires were returned in Cycle 2 compared to only 79 in Cycle 1. However we believe this is still considerably lower than what it should be. For example, Brick et al (2012) had 9% of returns in Spanish.

Table 5: Percentage of Spanish returns by Cycle.

Language of questionnaire returned	Cycle 1	Cycle 2
Spanish	2.0%	3.3%
English	98.0%	96.7%
Total	100%	100%
N	3,959	3,686

We also examined the percentage of the sample that reports Hispanic ethnicity. In Cycle 2, 8.1% of the sample reported Hispanic ethnicity versus 6.4% in Cycle 1. This is a slight improvement, but is still lower than the estimate of approximately 13% from the American Community Survey.

We believe that some of the low response to the mailing was the visibility of the Spanish questionnaire. The cover of the Spanish questionnaire was not obviously in Spanish. The English and Spanish questionnaires were the same color and the words “Health Information National Trends Survey” were shown in English on both versions. The package was also sorted with the English on top, with the Spanish instrument second. A second look at the package, as well as review by other survey researchers at Westat, raised the possibility that many Spanish speakers may not even have realized the Spanish-language questionnaire was in the package.

Mailing Decisions for Cycle 3

HINTS is planning to conduct an experiment with similar conditions to the Cycle 2 experiment. The experiment will continue Spanish mailings to targeted groups in condition 1 (control) and all households in condition 2 (treatment). We will vary the order in which the materials in the envelope appear. Condition 1 will continue with the traditional order of the mailing where the English cover letter and questionnaire appear on the top of the mailing package. In condition 2, the Spanish cover letter and questionnaire will appear on the top. We will conduct similar analyses to those shown in this memo to determine whether the experimental manipulation was successful at improving Spanish and Hispanic response. In addition, this will allow an analysis of order by comparing the targeted groups in condition 1 and 2 since these only differ by order. Comparing the targeted groups in condition 1 for cycles 2 and 3 would provide an estimate of the effect of the change in appearance for the targeted groups.

Table 5. Cycle 3 Experimental Design

Condition		Sample Size
Condition 1: Spanish questionnaire mailed only to addresses with Hispanic surname or in linguistically isolated area	English cover letter and English questionnaire on top	10,000
Condition 2: Spanish questionnaire mailed to all addresses in sample	Spanish cover letter and Spanish questionnaire on top	2,000
Total		12,000

To distinguish the Spanish instrument from the English instrument, we made changes to the appearance of the Spanish questionnaire for Cycle 3, which can be seen as part of Appendix D. We put the title of the study in Spanish, changed the color of the cover, put in photographs that were more Hispanic in nature, and added the words “en español” in large letters. This will be the cover for all Spanish questionnaires, regardless of the treatment group.

Results of the Format of the Spanish questionnaire

Findings from the cognitive interviews indicate that Spanish speakers have significant problems with the length and content of HINTS. This effect is partly a function of education, as Spanish speakers tend to be disproportionately in lower education groups. These individuals do not attend to health care issues in the same way as the general population (e.g., they tend not have a regular doctor). In Cycle 1, we found higher rates of missing data, particularly on grid questions, for Spanish speaking respondents and lower educated respondents. For Cycle 2, the format of the Spanish questionnaire was put into a single-column format in an effort to ease the difficulty that Spanish speakers seemed to be having with the instrument.

Tables 6a and 6b show the missing data for the same question across Cycles 1 & 2 for the question “Over the past 2 weeks, how often have you been bothered by any of the following problems?” Overall, Spanish missing data rates look similar to Cycle 1 or slightly lower in Cycle 2. There is still an increase in missing data for low education for both proficient English speakers and Spanish speakers. This would seem to indicate that the single column format had little effect on missing data.

Table 6a: Cycle 1 Percent of missing data by form, language ability and Education

Item	Proficient English		Less than proficient English3		Spanish
	More than H.S. (n=2,551)	H.S. or less (n=1,033)	More than H.S. (n=121)	H.S. or less (n=91)	More than H.S. (n=23)	H.S. or less (n=51)
Little interest	1.1%	4.6%	5.0%	2.2%	4.4%	11.8%
Hopeless	1.1%	4.9%	3.3%	1.1%	4.4%	5.9%
Nervous	1.1%	4.3%	4.1%	2.2%	4.4%	11.8%
Worrying	1.3%	4.4%	2.5%	1.1%	4.4%	13.7%

Table 6b: Cycle 2 Percent of missing data by form, language ability and Education

Item	Proficient English		Less than proficient English		Spanish
	More than H.S. (n=2,320)	H.S. or less (n=970)	More than H.S. (n=77)	H.S. or less (n=58)	More than H.S. (n=40)	H.S. or less (n=76)
Little interest	3.5%	6.0%	2.6%	10.3%	0%	13.2%
Hopeless	3.5%	7.4%	3.9%	12.1%	0%	9.2%
Nervous	3.4%	7.2%	5.2%	12.1%	0%	11.8%
Worrying	3.4%	6.7%	3.9%	13.8%	0%	14.5%

We also performed the analysis on single items between Cycle 1 and Cycle 2. We conducted this analysis to understand if the rate of missing data increased generally between the two cycles. The results show similar and generally low rates of missing data rates on single item measures.

Instrument Format Decisions

Given the results from the above analysis of missing data rates from Cycle 1 and Cycle 2, we plan to continue with the single column format for Cycle 3.

Instrument Content Decisions

Like Cycles 1 and 2, content for Cycle 3 data collection comes from the all-inclusive items list submitted as part of the original HINTS 4 OMB submission (Appendix B). As planned, approximately half of the Cycle 3 instrument is unchanged from Cycle 2. As noted above, the new items on the Cycle 3 instrument were cognitively tested through OMB No. 0925-0589 (approved 12/31/12). Changes as a result of cognitive testing included dropping some proposed questions or sub-questions for inclusion in the Cycle 3 instrument and altering the wording of some questions to improve data quality. The content of the full-length Cycle 3 instrument is shown in Appendix C.

The version of the Spanish instrument attached here is just an initial translation. It has not yet been through the full adjudication process. We anticipate there will be minor changes to the translation through adjudication. The Spanish language version is shown in Appendix D.

Tobacco Content

In response to questions raised about tobacco content in Federal questionnaires, we examined the historic level of tobacco questions in past HINTS instruments in addition to the Cycle 3 instrument being submitted with this memo. As shown in Table 7 below, the percentage of tobacco questions in HINTS has ranged from a high of 14.4% in 2005 to a low of 1.5% in 2011. The questions proposed for inclusion in the Cycle 3 instrument are 7.7% tobacco-related. The survey’s goals, research questions, and included constructs are outlined in the information sheet prepared for OMB and attached as Appendix E.

Table 7: Number of tobacco questions included in HINTS

Year of Administration	Total number of items	Number of tobacco items	Percent of items about tobacco
2003	223	16	7.2%
2005	195	28	14.4%
2007	185	19	10.3%
2009 (Puerto Rico only)	185	19	10.3%
2011 (Cycle 1)	205	3	1.5%
2012 (Cycle 2)	203	11	5.4%
2013 (Proposed cycle 3)	209	16	7.7%

LIST OF APPENDICES:

1. HINTS 4 sub-study approvals

2. List Of Potential Items

3. Instrument – English

4. Instrument – Spanish

5. HINTS Information Sheet

1 In Cycle 1, the Spanish questionnaire was mailed at the second mailing.

2 Brick, J. Michael, Jill M. Montaquila, Daifeng Han, and Douglas Williams. 2012. Improving Response Rates for Spanish Speakers in Two-Phase Mail Surveys. Public Opinion Quarterly. 76: 721-732.

3 In response to new Federal guidelines, the language ability question was changed between Cycle 1 and Cycle 2. In the tables, respondents with less than proficient English skills include those who reported being less than “completely comfortable” speaking English in Cycle 1 and those who reported that they speak English less than “very well” in Cycle 2.

7