Pilot Testing Report

[1205-0453: The National Agricultural Workers Survey, Part B]

National Agricultural Workers Survey (1205-0453)

Pilot Testing Report

1. Introduction

JBS International, Inc. (JBS), the contractor for the Employment and Training Administration’s (ETA) National Agricultural Workers Survey (NAWS), pilot-tested three versions of the health care, mental health, and digital information questions, and two versions of the education and training questions, that ETA is seeking Office of Management and Budget (OMB) approval to begin administering in the October 2015 NAWS interview cycle. ¹ Proposed changes to the NAWS at this time include modifications to existing questions on health conditions and health care utilization, and new questions about preventive health care, dependents’ utilization of health care, digital literacy and use of digital devices for health care information, mental health, and vocational education and training.

2. Summary

Three tests were conducted using an iterative questionnaire design process. All three locations in which the pilot tests were conducted are major agricultural areas and in each location the local farm labor force has a different demographic composition. Pilot testing was conducted in Spanish in all three locations. The first pilot was conducted in West Palm Beach, Florida on March 20, 2013. The questionnaire piloted in Florida included a section of several new questions on preventive health for the farmworker respondent and his or her family members. After piloting in Florida, some of the preventive health questions were modified to help improve response time. For example, the question “When did you use the health service?” tended to induce lengthy response times due to respondents’ need to recollect multiple exact dates. To simplify the recall process, the question was changed to “When was the last time you used the health service?” The revised questionnaire was piloted with respondents in Hidalgo County, Texas on March 25, 2013. Following this second pilot test, the injury and musculoskeletal sections were reordered in an attempt to improve respondents’ recall of injuries. This third version of the questionnaire was pilot tested in San Diego, California on July 06, 2013. Two NAWS interviewers helped administer the survey in the third pilot, to test the interviewer instructions.

Most of the difficulties with the new questions were related to disruptions in the flow of the interview that were primarily caused by redundancy of questions, formatting, or confusing or missing question instructions. Several questionnaire wordings and formats were tested over the succession of the pilots. The proposed questionnaire reflects the most successful version of the piloted questions, as well as some additional changes. To reduce redundancy of questions and facilitate flow of the interview, questionnaire testers and developers determined that the preventive health testing section should be divided into two separate sets of questions – one for the respondent and another for family members – and then incorporated into already-existing sections of the survey. Questions for the respondent were incorporated into the NH - Personal Health - Life History section and questions for family members were added to the Household Grid.

Interviewers provided specific feedback on how to improve the supplements for ease of administration. These suggestions included better highlighting, reducing the length of instructions, and the addition of arrows to direct the interviewers to continuation or skip patterns.

The pilots also underscored the need to train the NAWS interviewers about the various health conditions asked about in these sections. Several pilot test respondents had not heard of the health conditions they were asked about and wanted more information from the questionnaire administrator about what the health conditions were or meant. To handle the likelihood of such questions occurring during surveying, JBS will provide interviewers with training and scripted explanations for each of the health conditions listed in the questionnaire.

The remainder of this document describes the methodology used for the pilots and the results for each series of new questions, by location and overall.

3. Methods

As there are linguistic variations based on the farmworker’s country of origin and place of residence in the United States, a pilot was conducted in each of the main farmworker migrant streams (Florida in the Eastern stream, Texas in the Midwest stream, and California in the Western stream), and questions were tested for variations in comprehension and interpretation. Each of the pilots helped to test optional wording for comprehension, sequence of questions, response time, and overall understanding of the questions. The pilots also helped gather information for interviewer training and instructions for the final version of the questionnaire; particularly to minimize the length of the interview and obtain the best quality data possible from the piloted questions.

Questionnaire: Following is a list of the new questions that were piloted and the sections of the questionnaire to which they were added after pilot testing was completed.

1. Household Grid: New questions were added which ask farmworkers about their family members’ use of preventive health services in the last 12 months for illness or injury, routine or preventive care, and dental care, where and when the family member last visited a provider for one of these reasons, and for family members who did not utilize health care services, the reasons why the service was not used.

2. NH – Personal Health - Life History section:

   1. The following new health conditions and related questions were added to the list of health conditions that farmworkers are already asked about in the NAWS:

      1. “Cancer” - New questions asking respondents whether they have ever been diagnosed with it and the type or kind of cancer they were diagnosed with.

      2. “Diabetes” – A new question for female respondents who report being diagnosed with diabetes, asking whether their diabetes occurred during pregnancy.

   2. A new set of questions related to preventive health testing was added, which ask respondents whether they have ever been tested for high blood pressure, high cholesterol, HIV, urinary tract infection, and tuberculosis (and for female respondents only, whether they have ever had a Pap test), and if so, the results of the last test and when and where it was taken.

3. HA – Quality Of and Access To Health Care section: Some of the current NAWS questions related to the worker’s access to, usage of, and payment for health care services were combined with new questions about whether workers have sought health care services specifically for illness or injury, routine or preventive care, or dental care and if so, where they went the last time they received care, when the last visit was, and their level of satisfaction with the care they received.

4. GAD-7 mental health questions: New questions which ask respondents about the frequency with which they have felt nervousness or anxiety, and an inability to control worry.

5. DI. Digital Information section: A series of new questions that ask farmworkers about the types of digital information devices they and their family members have access to and their usage of these devices to obtain information related to health, employment, and education.

6. Education and Training section: A new section consisting of a series of questions about the types of work-related education programs, training, or classes respondents have participated in, where and when they attended, whether they completed the course and any certificate or license earned, and what they paid for it. Additional questions attempt to collect information about farmworkers’ knowledge of vocational training opportunities, their motivation to seek training, and the barriers to participation that they face.

For the purpose of the pilot, the new questions were embedded in a shortened version of the NAWS questionnaire that included the NAWS cover page, the Household Grid, the NAWS origin and language, and the Work Grid. These sections were included to obtain relevant background information for the new questions, and to test the flow of the interview.

The new questions were piloted first in Florida. Based on the findings from the first pilot, the wording and sequence of the new questions were modified for the pilot in Texas. After piloting in Texas, the new questions were modified further and piloted a third time in California.

Interviewers:

Two NAWS staff members conducted the interviews in Texas and Florida. In California, two NAWS interviewers joined the staff members and helped conduct the interviews.

Respondents:

Nine farmworker respondents participated in each of the three locations. Additional information on respondents is available in section E. Characteristics of Respondents of this document.

1. West Palm Beach, Florida:

   1. Females and origin: one from Mexico, one from El Salvador, 5 from Guatemala

   2. Males and origin: two from Guatemala

   3. All interviews were conducted in Spanish

   4. All participants from Guatemala were bilingual speakers (Spanish and K’iche’)

2. Hidalgo County, Texas:

   1. Females and origin: three from Mexico, one from the U.S.

   2. Males and origin: four from Mexico, one from the U.S.

   3. All interviews were conducted in Spanish.

   4. Only one of the U.S.-born participants was fully bilingual (English and Spanish)

3. San Diego, California:

   1. Females and origin: five from Mexico

   2. Males and origin: 3 from Mexico, one from Guatemala

   3. All interviews were conducted in Spanish

   4. One participant was bilingual (English and Spanish)

4. Summary of Findings by Questionnaire Section

1. NH – PERSONAL HEALTH - LIFE HISTORY

1. Health conditions questions

Florida:

• Two respondents did not know anything about “Asthma.” They asked for an explanation of these conditions. One of the two respondents also asked for an explanation of “Diabetes.”

• Although participants knew how a “High Blood Pressure” test is conducted, some asked for an explanation of “High Blood Pressure.”

• One respondent answered that a doctor told her to visit an “oncologist” because she appeared to have cancer. She mentioned that she never went to see the oncologist because she could not afford paying for the visit. Besides, she added, “I do not have ‘sangre mala’, so I do not have cancer.” She did not know if the doctor had actually performed a cancer test on her.

• Another participant answered that she had never been tested for any of the conditions, but wondered if she should be tested because sometimes she has to breathe very hard. She asked how one is tested for “Asthma”.

These kinds of responses by participants suggest that NAWS interviewers should be trained to explain the health conditions in a concise and homogenous way to participants who request explanation of the conditions they are asked about. Findings here also suggest that conditions such as “Cancer” and “Asthma” should be moved into a different category since there are numerous ways of testing for these conditions but not enough time to cover all of them in this section.

Texas:

• As in Florida, there were some participants who had heard about the various health conditions, but others who did not know “Diabetes”, “High Blood Pressure”, and “Tuberculosis” and asked for explanations of those conditions. Again, the health conditions must be operationally defined for the interviewers so that when asked to explain them, all interviewers are able to provide a standard brief and concise explanation of the condition in question.

• When asked if they have been tested for each one of the conditions, some participants responded “I do not know.” This response option was missing in the questionnaire, and it should be added in the final version.

• Some participants thought that it was too repetitive to ask for each condition if they had used a health service provider in the USA and in another country. This was particularly redundant for those who admitted being undocumented and had mentioned in the Work Grid that they had not gone back to their countries in the last 12 months.

California:

• For this pilot location, the formatting of the questions was changed such that respondents were asked whether they had ever been diagnosed with each of the health conditions “Heart Disease”, “Asthma”, and “Cancer”, but the questions about ever having been tested for them and the date of testing were deleted. Additionally, the questions about “Diabetes” and “High Blood Pressure” that were found to be repetitive when piloted in Florida were merged into the section containing these other health conditions, and the follow-up questions about ever having been tested for them and the date of testing remained.

• All participants understood the questions and were able to explain each one of the health conditions. Two participants felt that the last question in this section, “In the last 12 months, here in the USA, have you seen a doctor or nurse for this condition?” (column “g”) was redundant. They believed that if a respondent admits having a condition (column “a”), was tested for it (column “b”), and provides the location of the test (column “e”), it is redundant to ask whether they have seen a health care provider about the condition in the last 12 months. In order to prevent this type of misunderstanding, interviewers must be prepared to address this question by adding, when needed, “since the time you were tested (if more than 12 months), have you seen a doctor or nurse for this condition?”

2. Preventive health testing questions

Florida:

• Two of the health conditions asked about in this section, “Cholesterol” and “Sugar Levels”, were redundant as respondents had already been asked about these conditions in the NH – Personal Health – Life History section. The only difference was that the term “Diabetes” was used in the NH – Personal Health – Life History section and the term “Sugar Levels” was used in the preventive health testing section.

• Women were able to recall dates for “date tested (last time)” question much more quickly than men.

• All women understood the question about the Pap (Papanicolau) test.

Texas:

• Three participants answered “Yes” to the question about being tested for cholesterol, but mentioned that the doctors did not explain “Cholesterol” to them. They asked their interviewer for an explanation of “Cholesterol”.

• Instead of being asked the “date tested (last time)” questions, respondents who responded that “Yes” they had been tested for a particular health condition were asked first for the approximate year they were tested, then the month. This approach appeared to work better since it took less time for respondents to remember the date. As one participant said, “It was a long time in my country when I had the test”. When asked if it had been more than 1 year, 2 years, or 5 years, the respondent quickly answered, “More than 5 years ago”. These results suggest that since the questions do not reference a specific time period, many respondents will have a hard time remembering dates. It could be better to provide time period ranges as response options (e.g., last 12 months, between 2 to 3 years, etc.).

• All women participants answered “Yes” to the question about ever having been given a Pap test. Most of them said that it is a standard procedure for women before or after pregnancy.

California:

• The questions about “Cholesterol” and “Sugar Levels” were not asked to all due to the redundancy issue that emerged during the earlier two pilot administrations. Respondents were simply presented with the question about “Sugar Levels” and asked if they knew the meaning of it. One participant who knew about “Diabetes” did not know the meaning of “Sugar Levels” and asked for clarification.

2. HA – QUALITY OF AND ACCESS TO HEALTH CARE

Florida, Texas, and California:

• Participants understood the questions.

• To the question, “Who paid the majority of the cost (for this health service)?”, respondents who had used health services answered by stating the amount they paid for the service. Respondents said that they did not know who paid the “majority of the cost”.

• Similar to what was found with the preventive health testing questions, respondents tended to take too long in recalling exact dates that they received the health services.

• Asking respondents about their spouse and children’s use of health services in the same section was confusing, because the formatting of the questions did not allow for distinguishing which family member used the health service.

• The questions about family members’ health care utilization did not capture the reasons why family members who did not use the health services did not try to access them, even for preventive care.

3. DI. DIGITAL INFORMATION

Florida, Texas, and California:

• While all the participants understood the questions, none of the Florida respondents acknowledged ever having used any of the digital devices they were asked about.

• Some of the respondents in Texas owned smartphones and had access to the internet through family members. Three said that they had used the internet with the help of their children. This type of situation is not captured by the questioning and must be added to the final version of this section. One participant said that he used “face” (Facebook).

• In California, more participants had smartphones and access to computers. Although only two said that they used the Internet on their own, four other participants used the internet with the help of their children.

4. GAD-7

Florida, Texas, and California:

• When piloted in Florida, this section consisted of seven mental health questions. All respondents in Florida understood all of the questions, but some respondents thought that some of the questions were repetitive.

• To reduce question redundancy, only the first two questions (pertaining to feelings of nervousness or anxiety and inability to control worry) were piloted in Texas and California. All participants understood the two questions and were able to answer them with no problem.

5. EDUCATION AND TRAINING

Florida and Texas:

• Two different versions of the proposed Education and Training supplement were first pilot tested in Texas through two focus groups and nine cognitive interviews. A revised version of the Education and Training supplement was used in the one Florida focus group. The revisions to the supplement were based on the feedback from the Texas focus group participants and interview respondents.

• The results from the Texas and Florida focus groups and Texas interviews highlighted the need to revise the questions to minimize the time interviewers might spend explaining questions and probing for responses. Three primary issues were reported by participants and respondents in the pilot study. The first issue was the wording of questions in Spanish. The participants and respondents requested explanations for some of the words used, and felt some words were too formal. They suggested using more colloquial words. The second issue was the sequence of questions and perceived redundancy. The last issue was that all participants had difficulty recollecting exact dates of their participation in specified programs. Florida participants suggested that questions only include year of training and number of hours.

• In Florida, for Question 1a. – Worker Safety Training, participants answered with an example as opposed to a “yes” or “no” response and referred to the pesticide question of the NAWS survey.

5. Characteristics of Respondents

	PILOT INTERVIEWS IN WEST PALM BEACH, FLORIDA
	Location	Sex	Age	Origin	Marital Status	Years of Education	Year of entry to the USA	English Speaking	English Reading	Medical Last 2 years
1	FL	F	43	Mexico	M	8	1999	A little	Not at all	Yes
2	FL	F	33	El Salvador	M	0	2005	Not at all	Not at all	Yes
3	FL	F	30	Guatemala	M	5	2002	Not at all	Not at all	Yes
4	FL	F	40	Guatemala	O	0	1990	A little	Not at all	No
5	FL	F	36	Guatemala	O	2	2007	Not at all	Not at all	Yes
6	FL	M	36	Guatemala	S	5	2002	A little	Not at all	No
7	FL	F	40	Guatemala	S	0	2005	Not at all	Not at all	No
8	FL	F	34	Guatemala	M	6	1998	Not at all	Not at all	Yes
9	FL	M	31	Guatemala	S	6	2005	Not at all	A little	No

	PILOT INTERVIEWS IN HIDALGO COUNTY, TEXAS
	Location	Sex	Age	Origin	Marital Status	Years of Education	Year of entry to the USA	English Speaking	English Reading	Medical Last 2 years
1	TX	M	17	USA	S	11	N/A	Well	Well	Yes
2	TX	F	31	Mexico	M	6	2002	Not at all	Not at all	Yes
3	TX	F	27	USA	M	12	N/A	Well	Well	Yes
4	TX	M	35	Mexico	M	6	1999	Not at all	Not at all	No
5	TX	M	31	Mexico	M	1	1997	Not at all	Not at all	Yes
6	TX	F	28	Mexico	M	6	2004	Not at all	Not at all	Yes
7	TX	F	29	Mexico	M	6	2000	A little	Not at all	No
8	TX	M	30	Mexico	M	9	2004	A little	A little	No
9	TX	M	31	Mexico	M	9	2000	A little	A little	Yes

	PILOT INTERVIEWS IN SAN DIEGO, CALIFORNIA
	Location	Sex	Age	Origin	Marital Status	Years of Education	Year of entry to the USA	English Speaking	English Reading	Medical Last 2 years
1	CA	M	43	Mexico	M	4	1989	A little	Not at all	No
2	CA	F	41	Mexico	S	11	1995	A little	Somewhat	No
3	CA	M	44	Mexico	M	5	1991	A little	A little	Yes
4	CA	M	50	Mexico	M	9	1984	A little	Somewhat	Yes
5	CA	F	45	Mexico	O	6	1987	A little	Not at all	Yes
6	CA	M	52	Guatemala	O	0	2004	Not at all	Not at all	Yes
7	CA	F	34	Mexico	0	6	1998	A little	A little	Yes
8	CA	F	48	Mexico	M	9	1999	Not at all	Not at all	Yes
9	CA	F	24	Mexico	S	9	2002	A little	A little	No

1 Modifications to the National Institute for Occupational Safety and Health (NIOSH)-sponsored occupational injury and musculoskeletal (MSK) questions were also tested. OMB approved the changes to these questions, which were administered in the NAWS in fiscal years 2014 and 2015. The final version of the questionnaire does not include the injury and MSK sections, however, as NIOSH, ETA, and JBS agreed to exclude them from the survey for fiscal year 2016.

8