NAWS Improvement Plan #3

Department of Labor, Employment and Training Administration

Update on Action Steps for Improving the Statistical Methodology of the National Agricultural Workers Survey, OMB No. 1205-0453

Per the Office of Management and Budget’s (OMB) March 13, 2009 clearance terms for the National Agricultural Workers Survey (NAWS), the Employment and Training Administration (ETA) commissioned an independent review of the survey to investigate the extent of potential biases in the survey’s estimates related to potential errors in the survey’s weights. The review was completed in December 2009. On January 20, 2010, Division of Research and Evaluation staff in ETA’s Office of Policy Development and Research, which has responsibility for the NAWS, briefed OMB and the Bureau of Labor Statistics (BLS) on the findings and, as required by the clearance terms, provided and discussed ETA’s proposed action steps for responding to the findings (see document: 1205-0453 ETA Plans for Responding to Evaluation Findings_12.30.09.doc).

One action step is the quarterly submission of reports on ETA’s progress and/or technical difficulties completing the deliverables that are associated with improving the survey’s statistical methods. This is the third report. It provides an update, as of June 10, 2011, on ETA’s action steps for improving the survey’s statistical methods.

Action Steps and Deliverables

• Development of single-stage non-response adjustments and Assessment of weights for and accuracy of regional-level estimates

Status: The NAWS COTR developed the statement of work (SOW) for these requirements and submitted it to BLS, for review, on October 1, 2010. BLS reviewed it and suggested that: 1) additional technical detail be added to tasks 2, 3, and 4 to help ensure that the prospective contractor will have a clear understanding of the work that will be required; 2) additional details be added regarding the time commitments and technical qualifications of key personnel; and 3) the SOW emphasize the requirement for the work to be conducted independent of the current NAWS contractor. The NAWS COTR incorporated all the suggestions and obtained BLS’ final approval of the SOW on November 19, 2011.

As discussed in the previous progress report, the SOW also included requirements for an assessment of worker sampling procedures, analysis of national-level survey findings, and an evaluation of the survey’s design.

Due to the unavailability of funds, the task order contract for these requirements was not executed until June 10, 2011. The contractor has been required to complete its assessment of the accuracy of previously released national-level point estimates by October 1, 2011. The contractor will have until June 30, 2012 to complete the other requirements.

• Development of forms/directives and a quality control program for randomly selecting workers for an interview

Status: Per BLS and OMB recommendations, ETA will commission an independent evaluation of worker sampling methods.  The requirement for this has been included in the contract for the survey design evaluation. Due to procurement delays, the target completion date of this activity is now June 30, 2012.

• Update and modification of the NAWS public access data set to add the 2007-2009 data and the new post-sampling weight variable

Status: The NAWS contractor revised the sampling and post sampling weight variables (following the independent evaluator’s programs and methods) and replaced the post-sampling weight variable in the public access data set with the corrected weight. The NAWS contractor also updated the data set with new variables and data for fiscal years 1989-2009. The NAWS COTR sent the revised data set to BLS, which approved it for confidentiality purposes on August 11, 2010.

The updated data set was posted to the NAWS Web page on January 5, 2011. The availability of the updated data set was announced in a Training and Employment Notice. The announcement included a description of the data, including its limitations: http://wdr.doleta.gov/directives/attach/TEN/ten2010/TEN_21-10.pdf

• Development and posting of errata announcements on the NAWS Web page –Changes to the statistical methods documentation, e.g. the formulation of new sampling and post-sampling weight variables, and any necessary corrections to previously released estimates or instructions to users of the public data for producing robust estimates will be posted.

Status: The revised statistical methods document (Part B of the supporting statement) was submitted to OMB in March 2010 and has been uploaded on the Information Collection Review page for the NAWS. It is available at:

http://www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=201003-1205-005 .

On October 6, 2010, ETA deleted the 1989-2006 public data set from the NAWS Web page and posted an errata announcement on the same page regarding errors in the calculation of the survey’s weights and concerns that the errors could bias the survey’s estimates.

• Interagency meeting with Federal partners to discuss the need to redesign the NAWS methodology – ETA will consult with all the Federal agencies that use NAWS data so that the redesign accomplishes the goal of reducing survey design effects while continuing to meet, as best as possible, each agency’s information needs. This meeting will take place during the last week of January 2010.

Status: The interagency meeting was held on January 27, 2010. The following Federal agencies were represented: 1) Environmental Protection Agency (EPA), 2) Health Resources and Services Administration (HRSA), 3) Centers for Disease Control and Prevention (CDC), National Institute for Occupational Safety and Health (NIOSH), Division of Surveillance, Hazard Evaluations and Field Studies (DSHEFS), 4) CDC/NIOSH, Division of Safety Research, 5) Occupational Safety and Health Administration (OSHA), and 6) U.S. Department of Agriculture, Economic Research Service. On January 28^th, the NAWS COTR and the NAWS contractor met with the Department of Education, Office of Migrant Education, and the Department of Health and Human Services (HHS), Administration for Children and Families (ACF).

NAWS Federal partners were briefed on the independent evaluation findings and ETA’s plans for redesigning the survey. In addition, Federal partners’ input on the ideal regional configuration for sampling and reporting purposes was elicited. Except for HRSA, which would prefer that NAWS sampling regions match its regions, the configuration of regions is not an issue because NAWS Federal partners will only report data nationally. The Federal partners meet at least annually and the NAWS COTR will continue to update them on any changes to the statistical methods that arise from the ongoing evaluation work.

NAWS Federal partners met again on April 19, 2011. The primary purpose of the meeting was to review health data that has been collected in the survey since 1999, discuss farm worker health information gaps, and identify potential health question domains that could be added to the survey in the future. NAWS Federal partners were also briefed on the status of the NAWS survey design evaluation. The following agencies were represented: 1) HHS, Food and Drug Administration, Center for Food Safety and Applied Nutrition, 2) CDC/NIOSH/DSHEFS, 3) CDC, National Center for Zoonotic and Emerging Infections, 4) Department of Homeland Security, U.S. Citizen and Immigrant Services, Office of Policy and Strategy, 5) EPA, Office of Pesticide Programs, 6) HHS/ACF, 7) HHS/HRSA, and 8) DOL/OSHA.

The next NAWS Federal partners meeting will be held during the first week of August 2011.

• Survey redesign to address each of the concerns discussed in the evaluator’s report

Status: The requirement to evaluate the survey’s design has been awarded as a task order to an existing Blanket Purchase Agreement. The evaluation will commence in June 2011 and the contractor will have one year to complete all requirements. In the meantime, several improvements have been made to the statistical methods. Please consult the prior reports for a description of those improvements.

• Submission of Paperwork Reduction Act (PRA) clearance package

Status: As discussed during a telephone briefing with OMB on October 1, 2010, ETA will be submitting a new PRA package for the survey to the Department of Labor, Office of Chief Information Officer in June 2011. The new package will include a request to pilot new questions on pesticide exposure for the EPA. The 60-day Federal Register Notice for the proposed EPA questions was published on April 5, 2011 (Vol. 76, No. 65, page 18798). ETA received no comments on the proposed questions.

• Quarterly reports – ETA proposes to submit quarterly reports to OMB to document progress and/or any technical difficulties regarding the completion of the aforementioned deliverables.

Status: This is the third report. The fourth report, which will cover July through September 2011, will be submitted to the OMB in October 2011.