1205-0426 Supporting Statement Part B_FINAL 7.7.16

Placement Verification and Follow-up of Job Corps Participants

OMB Control No. 1205-0426

June 2016

B. Collections of Information Employing Statistical Methods

This information collection does employ statistical methods.

B1. Description of the Population and Sampling to Be Used

Goals of the Data Collection

No statistical methods leading to inferences of larger populations are used. In all reports and other publications and statements resulting from this work, no attempt is made to draw inferences to any population other than the set of participants responding to the data collection effort.

There are three primary goals of the data collected through the instruments and methods described in this request:

• to fulfill performance measurement and reporting requirements for the Job Corps program as specified in the WIOA regarding reporting on the employment and education outcomes of program participants to determine the performance of each center and the larger program;

• to independently verify, as specified by the Office of the Inspector General, the initial placement and wage reports made for individual participants reportedly placed by contractors in order to ensure the integrity and quality of placement services provided by Job Corps centers and career transition service (CTS) agencies; and

• to enhance the quality of the Job Corps program through collection of information on customer satisfaction of former participants, employers and educational institutions.

Populations to be surveyed

In addition to the employers and educational institutions taking the employment verification survey, two Job Corps populations are contacted in this data collection effort: graduates and former enrollees. Former enrollees are defined in Section 142 of the WIOA as an individual who has voluntarily applied for, been selected for, and enrolled in the Job Corps program, but left the program prior to becoming a graduate. A graduate is defined as an individual who has voluntarily applied for, been selected for, and enrolled in the Job Corps program and who, as a result of participation in the Job Corps program, has received a secondary school diploma or recognized equivalent, or completed the requirements of a career and technical education and training program that prepares individuals for employment leading to economic self-sufficiency or entrance into postsecondary education or training. This collection projects to survey 44,200 graduates and former enrollees during the second and fourth quarter surveys for a total of 88,400 surveys (see Supporting Statement Table 1)

The historical response rate for the National Job Corps survey is based on interviewing placed individuals (i.e., those participants who were placed in a job, education, or military after leaving the program). Under WIOA, the survey will attempt to interview all participants who exit Job Corp (placed or not placed) at two points after exit date: (1) to capture post-program information for quarter 2 (Q2) after exit date, and (2) to capture post-program information for quarter 4 (Q4) after exit date. As in previous surveys, the contractor will attempt to survey participants during an 8-week window.

Inasmuch as the respondents for the new WIOA survey (i.e., exiters) may have different characteristics as the current respondents (i.e., placed individuals), we estimate future response rates based on recent experience. In 2014, the contractor analyzed the response rate for the survey conducted six months after placement. The overall response rate as well as the response rate by a variety of demographic characteristics for the 6-month survey is approximately 64%. The response rate varies only slightly by demographic.

When Job Corps incorporates the use of administrative data to measure student outcomes, employment and earnings data will also be collected on uncommitted students who do not complete their career preparation period and become participants a subsequent information collection will be issued at that time.

Uses of the Collected Data

The first two goals for this data collection strongly suggest that survey results need to be precise at the level of the individual Job Corps center and CTS agency (the contractors responsible for participant placement). Job Corps' performance-based contracting methodology uses these survey-reported outcomes to determine incentive and award fees for the private contractors that operate the individual centers and the CTS (placement) agencies. Further, Job Corps' ranking of center and CTS contractor performance uses data collected from the surveys. The relative ranking of centers and CTS agencies provides one of the major criteria used by Job Corps to determine whether a contract option is renewed and in evaluating contractor past performance for new contracts, and the ranking reports are very sensitive to small changes in measured performance.

Following the employment modules, the focus of the questionnaire shift to post-program education. Specifically, for each education or training program, the questionnaire asks about the type of program, start date of the program, duration of the program, the number of credit hours that will be earned for completion of the program.

Using the data collected in the Q2 and Q4 surveys, the contractor will compare post-program outcomes for Job Corps participants who completed the program and other participants who left the program prior to completion. The analysis will also compare the employment rate and earnings of “completers” and “non-completers.” Other analysis will examine the employment rate of male and female Job Corps participants, older and younger participants, and participants in different occupational training programs.

While a great deal of effort will be made to maximize the response rate, there will inevitably be a small percent of survey non-respondents. The non-respondents to the survey create a potential for non-response bias. That is, the respondent sample may not be representative of the population.

As in past years, we will conduct a thorough non-response bias analysis by comparing the characteristics of Job Corps participants as they enter the program with the characteristics of survey respondents who complete the Q2 and Q4 surveys. Fortunately, as students enter Job Corps there is a great deal of information on their baseline characteristics: age, gender, location, prior education level, and other important characteristics. We will compare these characteristics for respondents and non-respondents to determine if the respondent sample is systematically different than the non-respondent sample.

In past non-response bias analysis we found that both male and female response rates exhibit the same pattern over time. Female graduates on average exhibited a slightly higher response rate than males. Similarly, response rates were fairly stable across age groups, with a minor uptick at the older end of the spectrum. Overall, in past surveys, we found little potential for non-response bias. Nonetheless, we will continue to conduct regular non-response analysis to ensure that the response rate in the new survey is similarly not biased.

Sampling weight calculations

The contractor is highly experienced in a variety of sample weighting processes. For this project, the contractor will explore a variety of weighting approaches including: (1) non-response adjustment to adjust for failure to obtain a completed interview from some respondents and (2) non-response adjustment for respondents in demographic groups with differential response rates (e.g., male, female). This weighting process, if necessary, will adjusts survey results to reflect the demographic characteristics of the target population. These adjustments will help to make the sample representative of the target population by mitigating non-sampling errors and bias.

Information collected in the survey is self-reported. We inform respondents that all data collected will remain private and that no individual data will be released. As a result, we do not believe that there are significant accuracy and reliability issues. Moreover, at the conclusion of collecting information on each job, we review the information provided and confirm that the information recorded is accurate.

For verification jobs, information on wages is provided by JCDC and, thus, the purpose of the survey questions is to verify the employment and the earnings on that job. Thus, for these jobs we have two sources of information.

The WIOA legislation requires that Job Corps report "information on the performance of each Job Corps center and the Job Corps program…." To meet this WIOA requirement, it is necessary for the Post Enrollment Data Collection (PEDC) to collect information on a census of placed graduates and former enrollees. With this census information, the Job Corps can satisfy the WIOA requirement that the program provide information on "each Job Corps center." Random sampling will not provide sufficiently accurate information to assess the performance information of each Job Corps center.

B2. Statistical Methodology for Stratification and Sample Selection

As indicated above, all groups are a census of the population. No sample selection or stratification is applicable.

The key variables to be collected in the Q2 and Q4 questionnaires are presented in the following diagram. The questionnaire will begin with questions designed to re-verify placement information obtained from JCDC (Job Corps Data Center). For those who are not placed, the questionnaire will skip the re-verification questions and begin with questions about their current job. Once information on the current job is captured, the questionnaire is designed to capture information on additional jobs held during the quarter. For each job, we will ask information on employer name, start date and end date, weekly hours worked (including overtime), hourly wage rate, any additional payments (such as tips, bonuses, or commissions) and other information needed to calculate total earnings over the quarter.

Using the data collected in the Q2 and Q4 surveys, the contractor will compare post-program outcomes for Job Corps participants who completed the program and other participants who left the program prior to completion. The analysis will also compare the employment rate and earnings of “completers” and “non-completers.” Other analysis will examine the employment rate of male and female Job Corps participants, older and younger participants, and participants in different occupational training programs.

B3. Methods to Maximize Response Rates and Address Non response

The following strategies are used to maximize response rates:

• The data collection instrument consists primarily of well-tested items.

• The contractors work closely with Job Corps placement staff and JCDC to obtain adequate tracking and locating information about all Job Corps graduates and former enrollees.

• The online survey instrument maximizes the opportunity for collection by utilizing electronic contact information like e-mail, text and smart phone applications. Maintaining contact during the first three months after exit will improve response rates during the data collection periods.

• Regular contact between students and career transition specialists following termination, and the enhancements to the Career Information System significantly improve the quality of available student locating information.

• Job Corps employs sample-locating and refusal-avoidance techniques that have been proven to maximize locating and enlisting the cooperation of youth populations. Graduates receive monetary incentives from Job Corps during the 12-month service eligibility period that will likely enhance cooperation with the data collection effort.

Research suggests that reminders can be the single most important technique for producing high response rates. Providing periodic reminders via electronic means helps track graduates and former enrollees who have moved since their separation from Job Corps or for whom traditional contact information such as phone numbers and mailing addresses is inadequate. Job Corps employs additional searching techniques that have been proven to maximize the location of respondents and the elicitation of their cooperation, including:

• contacting parents, relatives, and neighbors;

• sending address-correction letters;

• searching on-line nationwide databases (for example, Accurint, White Pages,

Directory Assistance, Lexis/Nexis, reverse lookups, among others);

• requesting, where possible, information from public agencies _{(for example, motor vehicle}

departments and corrections departments); and,

• providing a toll-free line for respondents to call.

Job Corps uses online survey techniques and telephone procedures through a support contract that maximize response rates after respondents have been located. Telephone interviewers are trained to carefully follow these procedures. The contractors carefully monitor and retrain interviewers to correct any weaknesses in their contact and interviewing techniques. Two to three weeks after the first contact with a respondent who initially refused to participate a senior interviewer will contact the respondent and address the respondent's concerns about completing the interview. The data collection contractors will maintain databases to track survey data, will generate regular reports to identify non responders, and will support follow-up efforts. These procedures have been used successfully in Job Corps’ data collection efforts to achieve response rates consistent with those projected for these follow-up surveys.

Because this revised survey will collect data from different groups of post enrollment Job Corps students than before, there is presently no response data and therefore no statistical basis for a non-response analysis.

Throughout the survey we make a concerted effort to avoid missing data. For example, when the questionnaire includes a question on earnings, the respondent may answer “Don’t Know” or “Refused.” The interviewers are trained to probe further and obtain useful information by providing the respondent opportunities to respond in categories rather than in exact earnings amount. This approach is usually effective in obtaining a response and avoiding missing data. Similarly, if the respondent refuses to provide information, interviewers are directed to a probe that reminds the respondent of the private nature of the survey and that information provided by the respondent will only be used for analysis of the Job Corps program and individual information will not be shared.

Where we encounter missing data, we will consider imputing missing values using a “hot-deck procedure.” This approach enables imputation of values given a set of constraints. Specifically, the hot-deck procedure randomly selects a “donor” with the same values on a set of classing variables for each respondent with a missing value (the “recipient”). The donor’s observed value on the variable of interest is then used to replace the missing value for the recipient. A sequential (with-replacement) nearest-neighbor hot-deck procedure can be used, implemented using a SAS macro. The number of recipients per donor is generally limited to two.

B4. Test Procedures

As mentioned in B3 above, this collection will continue to utilize the currently approved telephone survey collection methodology and there will not be a need for further testing.

B5. Contact Information & Privacy

The contact information for the contractors who were consulted on the design of the PEDC is found below:

IMPAQ International, LLC

10420 Little Patuxent Parkway

Columbia, MD 21044

(443) 259-5500

Battelle Memorial Institute

505 King Avenue
Columbus, OH 43201

Phone: 800-201-2011 or 614-424-6424

Decision Information Resources, Inc.
3900 Essex Ln, Suite 900
Houston, Texas 77027
Main Telephone: 713-650-1425

Contact information for the contractor that will collect and analyze the survey information is:

IMPAQ International, LLC

10420 Little Patuxent Parkway

Columbia, MD 21044

(443) 259-5500

6