NAWS Improvement Plan

Department of Labor, Employment and Training Administration

First Quarterly Report on Action Steps for Improving the Statistical Methodology of the National Agricultural Workers Survey, OMB No. 1205-0453

Per the Office of Management and Budget’s (OMB) March 13, 2009 clearance terms for the National Agricultural Workers Survey (NAWS), the Employment and Training Administration (ETA) commissioned an independent review of the survey to investigate the extent of potential biases in the survey’s estimates related to potential errors in the survey’s weights. The review was completed in December 2009. On January 20, 2010, Division of Research and Evaluation staff in ETA’s Office of Policy Development and Research, which has responsibility for the NAWS, briefed OMB and the Bureau of Labor Statistics (BLS) on the findings and, as required by the clearance terms, provided and discussed ETA’s proposed action steps for responding to the findings (see document: 1205-0453 ETA Plans for Responding to Evaluation Findings_12.30.09.doc).

One action step is the quarterly submission of reports on ETA’s progress and/or technical difficulties completing the deliverables that are associated with improving the survey’s statistical methods. This is the first quarterly report.

The status, as of March 25, 2010, of each action step/deliverable is provided below.

Action Steps and Deliverables

• Development of single-stage non-response adjustments and Assessment of weights for and accuracy of regional-level estimates – This requirement will be assigned to the current evaluator and will be completed by March 1, 2010.

Status: As discussed during the January 20^th briefing, the contract with the evaluator expired in December 2009 and a new contract will be required. The NAWS COTR will develop the statement of work (SOW), which it will vet with BLS. The estimated completion date for this requirement is August 1, 2010.

• Development of forms/directives and a quality control program for randomly selecting workers for an interview – This task will be assigned to the NAWS contractor. An independent evaluator, however, will also be tasked with assessing current worker sampling procedures and helping the NAWS contractor develop appropriate forms/directives. This work will commence immediately and will be competed by October 1, 2010: the first interview cycle of fiscal year 2011.

Status: The contractor convened a two-day interviewer training meeting in February 2010 to review current field sampling procedures. In consultation with the statistical team, which participated in the training, the contractor developed: 1) clear field sampling instructions, 2) marked and unmarked sampling tags, and 3) a revised sampling tracking sheet. Instead of sampling every nth worker, interviewers are now using a lottery system. The new method and the accompanying instructions and tracking sheet are discussed in the revised supporting statement (Appendix B, pages 23-25). The new method promises to be more transparent and easier to document.

Per BLS and OMB recommendations, ETA will commission an independent evaluation of worker sampling methods.  We anticipate making further revisions following the completion of that study. The target completion date of this activity remains October 1, 2010.

• Update and modification of the NAWS public access data set to add the 2007-2009 data and the new post-sampling weight variable – As in the past, OPDR will send the revised data set to BLS for approval prior to making it available to the public via the NAWS Web page. The NAWS COTR will coordinate efforts with the contractor to complete this work by April 1, 2010.

Status: The NAWS contractor has nearly completed the task of revising the sampling and post sampling weight variables (following the independent evaluator’s programs and methods). Upon completing this task, the contractor and OPDR’s COTR for the NAWS will update the list of variables to be included in the new public access data set. The contractor will then construct the data set and the COTR will submit it to BLS for review. The new target completion date for this task is June 1, 2010.

• Development and posting of errata announcements on the NAWS Web page –Changes to the statistical methods documentation, e.g. the formulation of new sampling and post-sampling weight variables, and any necessary corrections to previously released estimates or instructions to users of the public data for producing robust estimates will be posted. OPDR staff will coordinate with the NAWS contractor to complete this task by April 1, 2010.

Status: The revised statistical methods (Part B of the supporting statement) have been transmitted, via email, to BLS. Upon approval of the revised methods and the revised public access data set, the NAWS COTR and the NAWS contractor will develop the necessary announcements regarding the updated materials for posting to the NAWS Web page. The new target completion date for this task is June 1, 2010.

• Interagency meeting with Federal partners to discuss the need to redesign the NAWS methodology – ETA will consult with all the Federal agencies that use NAWS data so that the redesign accomplishes the goal of reducing survey design effects while continuing to meet, as best as possible, each agency’s information needs. This meeting will take place during the last week of January 2010.

Status: The interagency meeting was held on January 27, 2010. The following Federal agencies were represented: 1) Environmental Protection Agency, 2) Health Resources and Services Administration (HRSA), 3) Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health (CDC/NIOSH), Division of Surveillance, Hazard Evaluations and Field Studies, 4) CDC/NIOSH, Division of Safety Research, 5) Occupational Safety and Health Administration, and 6) U.S. Department of Agriculture, Economic Research Service. On January 28^th, the NAWS COTR and the NAWS contractor met with the Department of Education, Office of Migrant Education, and the Department of Health and Human Services, Administration for Children and Families.

NAWS Federal partners were briefed on the independent evaluation findings and ETA’s plans for redesigning the survey. In addition, Federal partners’ input on the ideal regional configuration for sampling and reporting purposes was elicited. Except for HRSA, which would prefer that NAWS sampling regions match its regions, the configuration of regions is not an issue because NAWS Federal partners will only report data nationally.

• Survey redesign to address each of the concerns discussed in the evaluator’s report – While much of this work can be completed by the NAWS contractor, some portions of it may require an independent researcher with highly specialized statistical sampling expertise. This task will require additional funding and portions of the requirement may need to be competed. While some minor changes will be introduced immediately, and others beginning in October 2010, it may not be possible to implement all aspects of the redesign until October 2011: the first interview cycle of FY 2012. As there are three data collection cycles per year, with counties and growers being pulled and the interviews being allocated across the cycles before the first cycle begins, any major methodological changes would need to be implemented by October 1^st of each year, the start-date of the first cycle. The envisioned timeline for incorporating the necessary methodological changes is as follows:

Immediate Changes:

• Incorporating first-stage selection probabilities into the weight variable

• Application of the new weight variable to all future data analysis

• Addition of the new weight variable to the public data set

Changes that could be introduced by October 1, 2010 (the first interview cycle of FY 2011):

• Using simple random sampling at the second-stage of selection

• Using probability proportional to size sampling of growers

• Using newly developed worker sampling procedures and quality control program

• Incorporating single-stage non-response adjustments in the weight variable

Changes that could be introduced by October 1, 2011 (the first interview cycle of FY 2012):

• Any other changes that are deemed necessary as a result of the survey re- design effort

Status: The NAWS COTR is working on the SOW for the redesign requirement and, upon completing it, will vet it with BLS. The estimated completion date of the SOW is May 1, 2010. The redesign itself, and any changes stemming from it, should be completed/implemented by September 30, 2011.

In addition to making measurable progress on revised field sampling methods (discussed above), the NAWS contractor is on target for revising other sampling methods and procedures for the FY 2011 survey, the first cycle of which, pending OMB approval, would begin in October 2010.

Specifically, beginning in October 2010, Farm Labor Areas (FLAs) will be selected in a single stage.  This will eliminate the double probability-proportional-to-size (PPS) sampling that the independent evaluator identified.  Rather than selecting the final FLAs from a standing roster of 90, which was pulled from the universe of 498, the final FLAs will be directly drawn, using PPS, from the universe of 498.  The algorithm and formula for the first-stage selection probabilities appear in the revised statistical methods document (see pages 7-8).

In addition, to begin addressing survey design effects, starting in October 2010 the number of workers to be interviewed at each randomly selected establishment will be determined PPS.  Specifically, the allocation will be the square root of the number of workers employed at the establishment the day on which interviews are conducted, up to 12 workers (see page 5 of the revised methods document).

• Submission of Paperwork Reduction Act (PRA) clearance package – The NAWS currently has authorization until March 31, 2010. As part of the PRA process to continue the survey, a 60-day public comment notice was published in the Federal Register on November 30, 2009. Upon summarizing any comments it receives, ETA will submit the PRA package to OMB. The new package will incorporate the recommendations from the independent evaluation, including the revised and approved formulas for the survey’s weights, as well as any feedback ETA receives from NAWS Federal partners. The new PRA package, which will be submitted to OMB by February 15, 2010, will request a continuation of the survey, with the methodological changes discussed above, for 18 months, from April 1, 2010 until September 30, 2011, during which time all necessary survey design changes will be developed.

Status: The PRA package has been revised as discussed above and will be submitted to OMB by March 31, 2010.

• Quarterly reports – ETA proposes to submit quarterly reports to OMB to document progress and/or any technical difficulties regarding the completion of the aforementioned deliverables.

Status: The first quarterly report will be submitted to OMB in April 2010.