Appendix A

Appendix A: Summary of NAWS Non-Response Bias Studies

The NAWS has a low response rate for agricultural employers primarily because of employer list quality issues. As a result, NAWS interviewers must make several attempts to contact agricultural employers who turn out to be ineligible or who are unable to be contacted. Since there is no complete list of the universe of employers in crop agriculture, the NAWS sampling universe list is constructed from several lists of agricultural employers that vary in quality. The Bureau of Labor Statistics (BLS) provides a list of agricultural employers that file unemployment insurance quarterly returns with their state. This list includes all employers filing in states where the NAWS samples and BLS is able to obtain the state’s permission to supply the list. Additional lists of employers are needed for: a) States that do not give permission for BLS to supply the list; and b) Small employers in all but the four states that have universal unemployment insurance. To fill these gaps, information on agricultural employers are obtained from professional organizations, commercial marketing lists, internet searches, and knowledgeable individuals including local officials, service providers, and others.

The likelihood that an employer on the sampling list is eligible varies considerably. Many issues are responsible for the variations in list quality. First, there is a lag in receiving information from BLS, so some information is out-of-date. Other sources of employer information vary in terms of completeness and the degree to which the list is vetted. At the same time, agricultural operations are not static and changes to crops, technology, or labor practices may affect the employment and timing of agricultural workers and thus survey eligibility. Also, there are businesses that are bought and sold, as well as those that start up, liquidate or cease production. Even employers that have agricultural workers may not be eligible at all times of the year since agriculture is a seasonal industry and workers may only be needed for tasks at certain times of the year.

In constructing the employer lists, the NAWS faces a tradeoff between increasing response rates and the completeness of the sampling universe list. If the universe list is restrictive the likelihood that employers on the list are eligible increases thereby increasing the response rate. At the same time, restrictions regarding the vetting of the list may cause non-sampling error by excluding qualified employers. The NAWS policy is to be inclusive in terms of who ends up on the agricultural employer list used for sampling. This minimizes non-sampling error, but results in lower response rates, as more employers are unable to be contacted to determine their eligibility. The question remains whether this policy results in non-response bias.

To examine whether there might be possible non-response bias, data from fiscal years 2009 through 2010 were reviewed to assess the quality of the grower list and the response rate patterns by sampling region and industry (4 digit NAICS codes). During this two-year period, the grower list consisted of 4,445 growers. Of those growers, 30 percent were determined to be eligible for the NAWS survey and 29 percent were determined to be ineligible. In 31 percent of the cases, the grower’s eligibility was unresolved; those cases are classified as “Do Not Know/Unresolved”.¹ The remaining 10 percent of the growers on the list were classified as “incomplete” because the grower was unavailable or the main office was at a different address. In less than one percent of the cases, no attempt was made to contact the grower.

The population of identified growers consists of growers who are eligible, ineligible and the cases classified as “Do Not Know/Unresolved.” The proportion of eligible growers did not differ substantially across regions. However, in California and the Mountain I and II regions, there were higher proportions of eligible growers compared to the other regions and the national average. In the Corn Belt/Northern Plains and the Southern Plains regions, less than 30 percent of identified growers were eligible for the survey. The highest proportion of ineligible growers was more likely to be found in the Appalachia, Corn Belt/Northern Plains, Lake, and Northeast I regions.

The distribution of identified growers was examined by industry type. There is some variation across industry type. More than half of growers in the “Greenhouse, Nursery, and Floriculture Production” industry were determined to be eligible for the survey. About 42 percent of growers in the “Vegetable and Melon Farming” industry, and 40 percent of those in the “Fruit and Tree Nut Farming” industry were determined to be eligible. The proportion of eligible growers in the industries eligible for the survey ranged from a low of 20 percent to a high of 38 percent. This variation is consistent with labor practices in these industries.

Grower response rates were fairly consistent across regions and industries. Almost 70 percent of eligible growers agreed to participate in the survey, and 31 percent of eligible growers refused to participate. Interviews were completed at 60 percent of cooperating growers. Interviews could not be completed at the remaining nine percent of cooperating growers for several reasons. In some cases, the grower placed restrictions that impeded timely access to the workers. In other cases, the grower asked the interviewer to return on a specific date or at the end of the season or accommodate them in some other way.

Response rates among eligible growers did not vary substantially across the 12 sampling regions. The lowest response rate was in California, where 56 percent of eligible growers agreed to participate and interviews with workers were completed. The highest response rate was recorded in the Appalachia and the Mountain III regions, where the response rate was 66 percent.

The response rate ranged from 54 to 66 percent across the various industries eligible for the survey. The exception was the “Live Stock Farming or Production” industry, where the response rate was 29 percent. This is a relatively rare category, as livestock are excluded from the survey, however, a few livestock growers do produce crops using hired labor.

For the fall cycle in FY2009, some analysis was conducted to examine whether further effort at grower contact would increase grower response. Additional growers were contacted by phone after interviewers had left the area in order to try to confirm whether the growers were eligible. Attempts continued into the spring in case the growers had been out-of-season in the fall and would be available at a later date. The data were analyzed for all growers selected; the analysis also included a comparison of growers obtained from the BLS Unemployment Insurance (UI) list and growers coming from other sources.

Of the 1,132 growers in fall cycle for FY 2009 (cycle 62), 833 of them (74%) were from the UI list and 299 (26%) were from other sources. A total of 511 growers classified as “Do Not Know/Unresolved” and “Incomplete” in cycle 62 were contacted by phone. An additional 83 growers had their status changed: 30 growers were identified as eligible and 53 as ineligible. The overall CASRO response rate remained at 35 percent, the response rate for UI growers remained at 39 percent and the CASRO response rate for non-UI growers declined slightly from 31 to 29 percent. The results indicated that additional effort at grower contact was not cost-effective and would not improve response rates.

An earlier study of response rates used information obtained from grower contacts. This study relied on the fact that the NAWS may attempt to contact the same growers in different cycles and have different outcomes, that is interviewers may be unable to find a grower in one cycle and at another time be successful and able to determine the grower’s eligibility. The information used for this analysis was extracted from the grower contacts from 2001 to 2004, which contained the responses from 8,596 agricultural employers. Some growers were contacted in as many as seven different cycles, for a total of 11,533 contacts. A Markov analysis was conducted identifying transition probabilities from contact state to contact state (e.g., unknown to ineligible). The results of the analysis showed a five percentage point gain in the CASRO response rate from 35 to 40 percent.

Efforts to improve response rates are ongoing as new techniques emerge to improve response rates among eligible growers. The response rates are monitored closely and regular reviews of possible causes of non-response bias are conducted. One limit to conducting non-response bias studies is the limited comparison data available outside the NAWS frame. The Agricultural Resource Management Survey (ARMS) done by the USDA is as a possible source of information that can inform future studies. It would be possible to compare response rates by NAICS codes with Census of Agriculture data, excluding labor done by farm families. The ARMS is the only source identified so far that has this data, and would enhance a non-response bias analysis.

1 The main reasons establishments are determined to be ineligible are that they are not farm employers, the grower is no longer in business, the grower uses only family labor, or the establishment is not a farm. The “Do Not Know/Unresolved” cases occur when the interviewer is unable to initiate contact because there was an answering machine and was unable to speak with a person, there was no one at the address when the interviewer visited, or the interviewer was unable to find the address on the list.