Part B Final

Title II of the 2014 Workforce Innovation and Opportunity Act (WIOA; Pub.L. 113-128) provides funding to states for local programs that are designed to help adults obtain the education credentials, knowledge, and skills necessary for employment and economic self-sufficiency.¹ WIOA mandates an independent national evaluation of adult education programs funded under Title II. The national evaluation is being conducted by the U.S. Department of Education’s (ED’s) Institute of Education Sciences (IES). This request for clearance is for the National Study of the Implementation of Adult Education Under WIOA, which is part of the national evaluation. The study will provide Congress, ED, and the field with implementation information on adult education programs and practices.

The study will be descriptive and primarily involve collecting and tabulating data from surveys of adult education state directors and local providers funded under Title II. Some key findings from the provider survey will be compared with findings from the Adult Education Program Survey, an earlier national survey of providers conducted in 2003. This comparison will allow for an assessment of the extent to which adult education programs have evolved since prior to the enactment of WIOA. The study also will include the use of extant data, such as from ED’s National Reporting System (NRS). The NRS collects information on adult education programs funded, populations served, and participant outcomes achieved.

Part B. Description of Statistical Methodology

B.1. Respondent Universe and Statistical Design and Estimation

The National Study of the Implementation of Adult Education Under WIOA includes two surveys: the State Director Survey (Appendix A) and the Provider Survey (Appendix B).

State Director Survey. This survey will be administered to State directors of agencies (n = 57) that disburse Title II WIOA funding to the providers of adult education in their states. This includes entities from all 50 states, the District of Columbia, and six territories and outlying areas (the American Samoa, Guam, Northern Marianas, Palau, Puerto Rico, and the U.S. Virgin Islands).
Provider Survey. This survey will be administered to local adult education providers from the 50 states and the District of Columbia that offer adult education and literacy activities and services funded under Title II of WIOA. All providers of adult education programs in the states and the District of Columbia that received funding in program year 2018–19 (n ~ 2,000) and are still operational in program year 2019–20 will be included. Because most survey items reference program year 2018–19, newly funded providers in program year 2019–20 will not be included.

Adult education program providers in the territories and outlying areas are not included in the data collection to retain comparability with the 2001-02 Adult Education Provider Survey, which was administered to a sample of programs in the states and District of Columbia only. This collection is the first one for state directors, so there are no comparability constraints for the State Director Survey.

Both the State Director Survey and the Provider Survey are universe surveys, not sample surveys. Surveying all State directors will allow the study to provide state-level information for all states, territories and outlying areas, and will increase the precision of the national estimates. Including the universe of adult education providers in the states and the District of Columbia also has benefits including: increased precision in estimates; a more complete universe of providers as a sampling frame for possible future studies (e.g., learner survey, targeted evaluations for best practices); simplified analyses and data processing (i.e., sampling weights not required, although nonresponse adjustments may be added); and unconstrained subgroup analyses.

Exhibit B.1 presents the types of adult education providers that received funding for program year 2017–18, which is the most recent program data available. We expect the distribution of adult education providers in program year 2018–19 to be very similar. Most of these providers are local education agencies, community/junior/technical colleges, and community-based organizations.

Exhibit B.1. Number of Title II WIOA-Funded Adult Education Providers, Program Year 2017–18

Provider agency	Count
Local education agencies	962
Public or private nonprofit agencies
Community-based organizations	313
Faith-based organizations	41
Libraries	18
Institutions of higher education
Community, junior, and technical colleges	537
Four-year colleges and universities	49
Other institutions of higher education	^a
Other agencies
Correctional institutions	53
Other institutions (noncorrectional)	^a
All other agencies	22
Total	Approximately 2,000

^aTo protect the confidentiality of ED data and tabulations containing information about individuals, values from 1 to 5 are suppressed.

Source: National Reporting System for Adult Education Programs, Table 14, program year 2017–18.

The State Director Survey will be administered through a Web-based platform. The Provider Survey will be administered primarily through a Web-based platform but also will include mail and telephone options during nonresponse follow-up. The data collection procedures are described in Section B.2.

The data collection processes described in this document are designed to yield at least an 85 percent response rate from local providers and a 100 percent response rate from state directors.

B.1.1. State Director and Local Provider List Development

The American Institutes for Research (AIR) maintains a list of state directors through its NRS technical support project and will work with ED’s Office of Career, Technical and Adult Education (OCTAE) to ensure that the list is current before fielding the State Director Survey. If a state director’s position is vacant at that time, we will identify the acting director or most knowledgeable person for the study by consulting with ED and contacting the state agency, as needed.

There is no existing national list of adult education providers, so the Provider List will need to be compiled from data from each state director. Because the states fund and receive NRS data from these providers, the state directors will have contact and other information in their records. We will provide state directors with a template to provide information about the local adult education providers funded by Title II of WIOA (see Appendix C). Information will include contact information for the provider director and information about the program, such as name, address, and type of provider.

Although we anticipate a 100 percent response rate, it is prudent to prepare for nonresponse. If we do not receive a list from a state director by the deadline, then we will compile a listing from online, public sources for that state and make sure it is as up-to-date as possible just prior to fielding the Provider Survey.

B.1.2. Nonresponse Bias Analysis

We are not planning to conduct a nonresponse bias analysis for the State Director Survey. Because ED regulations require that recipients of federal funds cooperate in evaluations of grant programs under which they receive funding, we expect all directors to respond. We will conduct a unit nonresponse bias analysis for the Provider Survey using the disposition code (i.e., survey response status) for each provider we attempted to survey and the frame variables with information for all providers in the target population. The nonresponse bias analysis will evaluate whether responding providers differ substantially on the frame variables from nonresponding providers by comparing response rates across subgroups of each frame variable. If the response rate is substantially higher (or lower) for one subgroup relative to other subgroups, the subgroup is overrepresented (or underrepresented) in the respondent sample. There may be a bias in the estimate for survey variables that correlate with this specific frame variable.

We plan on using Chi-Square Automatic Interaction Detection (CHAID) to identify a group of frame variables that best explain the variation in survey participation. The variables considered for use as predictors of response must be available for both survey respondents and nonrespondents. We will therefore rely on variables from the list of adult education providers submitted by the state directors (such as type of provider and enrollment counts) for the CHAID analysis.

The results of the CHAID analysis will be used to statistically adjust estimates for nonresponse. To evaluate the effectiveness of the nonresponse adjustments, we will compare universe distributions from the frame to estimates computed using unadjusted and adjusted weights. The distribution using the adjusted weights should more closely reflect the distribution from the frame, thereby reducing bias.

B.2. Procedures for Data Collection

B.2.1. Procedures for the Provider List

We will request the Provider List via e-mail from each state director and will follow up with nonrespondents by e-mail and telephone, as needed. We will also enlist OCTAE staff, as needed, to e-mail specific state directors and encourage participation.

The protocols for this data collection include the following (see Appendix D):

E-mail request. This e-mail will explain that we need a complete list of adult education providers to conduct the study. State directors will be asked to return an Excel spreadsheet using either the template provided or their own method.
E-mail reminders 1 and 2. The first request will be followed by two e-mail reminders sent about 7 days apart.
Telephone reminder. We will call all state directors who have not returned the list. The purpose of this call is to remind them to complete this step and make sure there are no technical difficulties or other issues preventing completion.
E-mail reminder 3 and 4. The telephone reminders will be followed by additional e-mail reminders sent about weekly.

B.2.2. Procedures for the State Director Survey

The State Director Survey will be administered using a Web-only mode throughout the data collection period. In addition to our personalized e-mails, we will enlist OCTAE staff, as needed, to e-mail or telephone state directors who are not responsive to our efforts. A PDF version of the questionnaire will be made available through the online instrument as a tool to facilitate data collection. An option to collect information over the phone in Spanish will be available for state directors in the territories and outlying territories.

The protocols for this data collection include the following:

Prenotification letter. We will begin data collection with a prenotification letter sent via first-class mail to all state directors of adult education. This letter will explain the purpose of the study and notify participants that they can expect an e-mailed invitation to complete the survey. The letter will be printed on ED’s letterhead and will include a QR code to permit early and easy access to the survey with mobile devices. State directors may scan the QR code with their smartphone or tablet and be directed to the survey.
E-mailed invitation. One week after the prenotification letter is mailed, we will e-mail the directors survey invitations that include unique links to access the survey.
E-mail reminders 1 and 2. The invitation will be followed by two e-mail reminders sent 1 week apart.
Postcard reminder. We will then send a postcard with an eye-catching graphic to capture the attention of nonresponding state directors.
E-mail reminders 3 and 4. This will be followed by two more e-mail reminders sent 1 week apart.
Telephone reminder. Interviewing staff will call all directors who have not yet responded and will remind them to complete the survey.

B.2.3. Procedures for the Updated Provider List

When each state completes the State Director Survey, we will request the Updated Provider List that includes descriptive provider-level information such as program enrollment that states receive from providers each fall and report at the state level to NRS. Because these data will not yet be available for the program year of interest when the Provider Lists are initially collected in the summer of 2019, we will request this additional information in early 2020.

So that state directors recognize that there is one final step, a note at the end of the survey will indicate that we will be requesting an Updated Provider List that includes provider-level NRS data. We will follow-up with an e-mail request as noted in the following protocol:

Thank You E-mail/Request. This e-mail will thank the state director for completing the survey and explain why we are collecting this additional data. State directors will be asked to return an Excel spreadsheet using either the template provided or their own method.
E-mail reminders 1 and 2. The first request will be followed by two e-mail reminders sent about 1-2 weeks apart.
Telephone reminder. We will call all state directors that have not returned the Updated Provider List. The purpose of this call is to remind them to complete this step and make sure there are no technical difficulties or other issues preventing completion.
E-mail reminder 3 and 4. The telephone reminders will be followed by additional e-mail reminders sent about weekly.

B.2.4. Procedures for the Provider Survey

Three data collection modes will be offered to respondents throughout the data collection period for the Provider Survey. The primary mode of data collection will be a Web survey. About 2 months from the start of data collection, we will send providers who have not responded a hard-copy questionnaire to complete. We also will enlist state directors and OCTAE staff, as needed, to e-mail providers and encourage participation. We will conclude data collection by conducting telephone interviews and allowing providers who have not submitted a survey to complete the questionnaire by telephone with a telephone interviewer. Copies of the questionnaire in PDF format will be made available throughout data collection to those who wish to use it as a tool for completing the online instrument.

The protocols for this data collection include the following:

Prenotification letter. The prenotification letter will be sent via first-class mail to all local providers of adult education. This letter will explain the purpose of the study and will notify participants that they can expect an e-mailed invitation to complete the survey. The letter will be printed on ED’s letterhead and will include a QR code to permit early and easy access to the survey with mobile devices.
E-mail survey invitation. One week after the prenotification letter is mailed, we will e-mail the providers survey invitations that include unique links for accessing the survey.
E-mail reminders 1 and 2. The survey invitation will be followed by two e-mail reminders sent 1 week apart.
Telephone reminder. We will call all providers that have not completed a survey. The purpose of this call is to remind the provider to complete the survey and make sure there are no technical difficulties or other issues preventing completion.
E-mail reminder 3 and letter reminder. The telephone reminders will be followed by an e-mail and letter reminders mailed the following week. The letter will be sent by first-class mail and, again, will include a QR code for quick access on mobile devices to the online instrument.
Paper questionnaire. About 1 week after the letter reminder, a paper questionnaire will be mailed to all providers that have not completed the survey. The questionnaire mailing will include a cover letter, frequently asked questions, and a postage-paid return envelope. All questionnaire returns will be logged and data entered to combine responses with results to the online survey.
E-mail reminder 4 and postcard reminder. The questionnaire mailing will be followed by an e-mail reminder and then a postcard, if no response has occurred. The postcard will have an eye-catching graphic to catch the attention of nonresponding providers.
Telephone reminder. We will encourage providers to complete the survey online or by paper. Although this is a reminder operation, we will collect the survey by telephone if the provider would like to complete it immediately.
Telephone interviews. In the final phase of data collection, telephone interviewers will contact all nonrespondents and ask them to complete the survey by telephone. Responses to the telephone survey will be combined in the case management system with responses to the online survey so that we can easily track responses to the survey.

B.3. Methods for Maximizing Response Rates

Securing cooperation begins with raising awareness of the study and its importance before the surveys are fielded. We are conducting outreach efforts at conferences, workshops, and trainings to increase awareness of the surveys among state directors and local providers. We also are using these events to provide states and providers with an understanding of the knowledge to be gained from the surveys and how they will be able to use the data to inform their own programmatic or policy-related decisions. For example, data on challenges experienced in implementing the reforms under WIOA will be useful for identifying areas in which state and local staff need technical assistance.

To maximize response rates during data collection, state directors and local providers will receive survey reminders using multiple methods throughout the survey window to prompt initiating and completing the survey. Our follow-up efforts with local providers also introduce the ability to respond by a paper questionnaire or by telephone for those who do not prefer the online instrument. These options are sequenced so that only the online instrument is offered during the first 9 weeks and then the paper survey is mailed to nonrespondents. Collection by telephone is offered during the final 5 weeks of the collection. In addition to these steps, we will enlist state directors to send e-mails encouraging participation by their local providers.

B.4. Tests of Procedures and Methods

B.4.1. Technical Working Group

In October 2018, we convened a technical working group (TWG) of researchers, state directors of adult education, and local program administrators. The TWG provided input on the data collection instruments under development for this study, including conceptualization and operationalization of research questions and survey terminology. The list of TWG members can be found in Part A, Exhibit A.2.

B.4.2. Testing of the Data Collection Instruments

We used cognitive research methods to pilot test both surveys in March 2019. The key component of the cognitive interview process is the think-aloud interview, in which participants verbalize their thoughts as they respond to the survey questions.

For the State Director Survey, the universe is already very small; therefore, we conducted a smaller number of interviews (n = 6) than for the Provider Survey (n = 8), but we conducted these interviews in two rounds of three interviews each. Respondents for the State Director Survey were recruited in February and March 2019, using personal contacts through our NRS support project. We targeted a set of states that represent different regions and diverse participant populations (e.g., majority Adult Basic Education [ABE], majority English as a second language [ESL], or a mix of ABE/ESL).

For the Provider Survey, two rounds of cognitive interviews of four interviews each (eight total) were conducted to gather feedback on the survey instructions, questions, and response options. We recruited providers in February and March 2019 using a combination of personal contacts with known providers from past AIR studies and referrals from state directors with whom we have regular contact through various TWGs. Respondents were selected to include program directors from a variety of program types (e.g., community colleges, school districts) and represent different regions and diverse participant populations (e.g., majority ABE, majority ESL, or a mix of ABE/ESL).

Interviews for both surveys were conducted virtually by trained cognitive interviewers with extensive experience conducting similar interviews. Revisions to the items based on the findings were discussed with IES while finalizing the surveys.

B.5. Individuals Responsible for Study Design and Performance

AIR is the contractor for the National Study of the Implementation of Adult Education Under WIOA. Safal Partners is the subcontractor providing additional content expertise and will assist in the cognitive testing of the surveys. The structure of key staff is detailed in Exhibit B.2.

Exhibit B.2. Organizations and Individuals Involved in the Study

Responsibility	Organization	Contact name	Telephone number
Contracting Officer Representative	IES	Melanie Ali, PhD	202-245-8345
Associate Commissioner	IES	Marsha Silverberg	202-245-7201
Project director	AIR	Stephanie Cronen, PhD	202-403-5229
Survey task lead	AIR	Deanna Lyter Achorn, PhD	202-403-5390
Senior advisor	AIR	Larry Condelli, PhD	202-403-5331
Subcontractor lead	Safal Partners	Margaret Patterson, PhD	703-399-5948

1 Title II of WIOA also is referred to as the Adult Education and Family Literacy Act.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	AIR Proposal
Subject	AIR Proposal
Author	Sorensen, Diane
File Modified	0000-00-00
File Created	2021-01-15