NAWS improvement plan

Department of Labor, Employment and Training Administration

Second Quarterly Report on Action Steps for Improving the Statistical Methodology of the National Agricultural Workers Survey, OMB No. 1205-0453

Per the Office of Management and Budget’s (OMB) March 13, 2009 clearance terms for the National Agricultural Workers Survey (NAWS), the Employment and Training Administration (ETA) commissioned an independent review of the survey to investigate the extent of potential biases in the survey’s estimates related to potential errors in the survey’s weights. The review was completed in December 2009. On January 20, 2010, Division of Research and Evaluation staff in ETA’s Office of Policy Development and Research, which has responsibility for the NAWS, briefed OMB and the Bureau of Labor Statistics (BLS) on the findings and, as required by the clearance terms, provided and discussed ETA’s proposed action steps for responding to the findings (see document: 1205-0453 ETA Plans for Responding to Evaluation Findings_12.30.09.doc).

One action step is the quarterly submission of reports on ETA’s progress and/or technical difficulties completing the deliverables that are associated with improving the survey’s statistical methods. This is the second quarterly report. It provides an update, as of October 6, 2010 on ETA’s action steps for improving the survey’s statistical methods.

Action Steps and Deliverables

• Development of single-stage non-response adjustments and Assessment of weights for and accuracy of regional-level estimates

Status: As discussed during the January 20^th briefing, the contract with the evaluator expired in December 2009 and a new contract will be required. The NAWS COTR has developed the statement of work (SoW) for this requirement and submitted it to BLS on October 1, 2010 for review. It is anticipated that the contract will be awarded in November 2010. The SoW will also include an assessment of worker sampling, analysis of national-level survey findings, and an evaluation of the survey’s design.

• Development of forms/directives and a quality control program for randomly selecting workers for an interview

Status: The NAWS contractor convened a two-day interviewer training meeting in February 2010 to review current field sampling procedures. In consultation with the statistical team, which participated in the training, the contractor developed clearer field sampling instructions, a revised sampling tracking sheet, and a new sampling method. Instead of sampling every nth worker, interviewers are now using a lottery system. The new method and the accompanying instructions and tracking sheet are discussed in the revised supporting statement (Appendix B, pages 23-25). The new method promises to be more transparent and easier to document.

Per BLS and OMB recommendations, ETA will commission an independent evaluation of worker sampling methods.  The requirement will be included in the contract for the survey design evaluation. It is anticipated that further revisions will be made to worker sampling procedures following the completion of that study. Due to procurement delays, the target completion date of this activity is now October 1, 2011.

• Update and modification of the NAWS public access data set to add the 2007-2009 data and the new post-sampling weight variable

Status: The NAWS contractor revised the sampling and post sampling weight variables (following the independent evaluator’s programs and methods) and replaced the post-sampling weight variable in the public access data set with the corrected weight. The NAWS contractor also updated the data set with new variables and data for fiscal years 1989-2009. The NAWS COTR sent the revised data set to BLS, which approved it for confidentiality purposes on August 11, 2010. The target date for posting the updated data set to the NAWS Web page is October 30, 2010.

• Development and posting of errata announcements on the NAWS Web page –Changes to the statistical methods documentation, e.g. the formulation of new sampling and post-sampling weight variables, and any necessary corrections to previously released estimates or instructions to users of the public data for producing robust estimates will be posted.

Status: The revised statistical methods document (Part B of the supporting statement) was submitted to the OMB in March 2010 and has been uploaded on the Information Collection Review page for the NAWS. It is available at:

http://www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=201003-1205-005 . On October 6, 2010, ETA posted an errata announcement on the NAWS Web page regarding errors in the calculation of the survey’s weights and concerns that the errors could bias the survey’s estimates. ETA deleted the 1989-2006 data from the NAWS Web site and also announced that a corrected public data set would be available on the site by the end of October 2010.

• Interagency meeting with Federal partners to discuss the need to redesign the NAWS methodology – ETA will consult with all the Federal agencies that use NAWS data so that the redesign accomplishes the goal of reducing survey design effects while continuing to meet, as best as possible, each agency’s information needs. This meeting will take place during the last week of January 2010.

Status: The interagency meeting was held on January 27, 2010. The following Federal agencies were represented: 1) Environmental Protection Agency, 2) Health Resources and Services Administration (HRSA), 3) Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health (CDC/NIOSH), Division of Surveillance, Hazard Evaluations and Field Studies, 4) CDC/NIOSH, Division of Safety Research, 5) Occupational Safety and Health Administration, and 6) U.S. Department of Agriculture, Economic Research Service. On January 28^th, the NAWS COTR and the NAWS contractor met with the Department of Education, Office of Migrant Education, and the Department of Health and Human Services, Administration for Children and Families.

NAWS Federal partners were briefed on the independent evaluation findings and ETA’s plans for redesigning the survey. In addition, Federal partners’ input on the ideal regional configuration for sampling and reporting purposes was elicited. Except for HRSA, which would prefer that NAWS sampling regions match its regions, the configuration of regions is not an issue because NAWS Federal partners will only report data nationally. The Federal partners meet at least annually and the NAWS COTR will continue to update them on any changes to the statistical methods that arise from the ongoing evaluation work.

• Survey redesign to address each of the concerns discussed in the evaluator’s report

Status: The NAWS COTR sent the draft SoW for the redesign requirement to BLS, for preliminary review, on October 1, 2010. As noted above, it is anticipated that the redesign evaluation will commence in November 2010. In the meantime, a number of improvements have been made to the survey’s statistical methods. These include:

New sampling and post-sampling weights created

The NAWS contractor used the programs and procedures that Mathematica Policy Research, Inc. (MPR) developed during the evaluation to incorporate the first-stage selection probabilities into the sampling weights and then, using these, constructed new post-sampling weights. The new post-sampling weights are now used in the analysis of data.

Two-stage selection of Farm Labor Areas reduced to one stage

Rather than selecting, by probabilities proportional to size (PPS), the final farm labor areas (FLAs) from a standing roster of 90, which was pulled, PPS, from the universe of 498 FLAs, the final FLAs are now directly drawn, by PPS, from the universe.  This change, which will be implemented in the first interview cycle of fiscal year 2011, will eliminate the double PPS sampling that MPR identified as a likely source of large survey design effects.

The algorithm and formula for the first-stage selection probabilities appear on pages 7-8 in the revised statistical methods document (Part B of the revised PRA supporting statement), which was transmitted to OMB in March 2010 and is available at:

http://www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=201003-1205-005

Sampling frame enhancements

The employer frame has been augmented to increase the representation of establishments that use contracted labor. This was accomplished by revising the memorandum of understanding between ETA and BLS regarding the agricultural sectors that are included in the Unemployment Insurance 202 data file. Specifically, the file was modified to include data from agricultural firms that fall under North American Industrial Classification System (NAICS) 11511: Support Activities for Crop Agriculture.

• Submission of Paperwork Reduction Act (PRA) clearance package

Status: The NAWS currently has authorization until October 31, 2010. As part of the PRA process to continue the survey, a 60-day public comment notice was published in the Federal Register on November 30, 2009. ETA summarized the comments it received and submitted the PRA package to OMB in March 2010. The new package incorporates the revised and approved formulas for the survey’s weights.

As discussed during a telephone briefing with OMB on October 1, 2010, ETA will be submitting a new PRA package for the survey in June 2011. The new package will include a request to pilot new questions on pesticide exposure for the EPA.

• Quarterly reports – ETA proposes to submit quarterly reports to OMB to document progress and/or any technical difficulties regarding the completion of the aforementioned deliverables.

Status: This is the second quarterly report. The third report, which will cover October through December 2010, will be submitted to the OMB in January 2011.