Part A 5.9.19

p art a

Assessing Evidence of Effectiveness in Adult Education: OMB Data Collection Package

April 25, 2019

Submitted to:

Institute of Education Sciences
550 12th Street, SW

Room 4105
Washington, DC 20004

Project Officer: Melanie Ali
Contract Number: 91990018C0057

Submitted by:

Mathematica Policy Research

P.O. Box 2393
Princeton, NJ 08543-2393
Telephone: (609) 799-3535
Facsimile: (609) 799-0005

Project Director: Peter Schochet
Reference Number: 50728.BY.T03.000.000

CONTENTS

PART A. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION 1

A. JUSTIFICATION 1

A.1. Circumstances necessitating the collection of information 1

A.2. Purposes and uses of the data 4

A.3. Use of technology to reduce burden 5

A.4. Efforts to avoid duplication of effort 5

A.5. Methods to minimize burden on small entities 5

A.6. Consequences of not collecting data 5

A.7. Special circumstances 6

A.8. Federal Register announcement and consultation 6

A.9. Payments or gifts 6

A.10. Assurances of confidentiality 6

A.11. Additional justification for sensitive questions 7

A.12. Estimates of hours burden 7

A.13. Estimates of cost burden to respondents 8

A.14. Estimates of annual costs to the federal government 8

A.15. Reasons for program changes or adjustments 8

A.16. Plan for tabulation and publication of results 8

A.17. Approval not to display the OMB expiration date 9

A.18. Explanation of exceptions 9

REFERENCES 10

appendix a: feasibility study screener and interview protocol

appendix B: MATHEMATICA confidentiality statement

appendix C: advance Email

TABLES

A.1. Adult education approaches for exploration in feasibility study 3

A.2. Questions the feasibility and design study will address 4

A.3 Members of the Technical Working Group 6

A.4. Estimated reporting hour burden 8

PART A. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION

This Office of Management and Budget (OMB) package requests clearance for data collection to support the study Assessing Evidence of Effectiveness in Adult Education. The Institute of Education Sciences (IES) at the U.S. Department of Education (ED) has contracted with Mathematica Policy Research and its subcontractors, Manhattan Strategy Group and Social Policy Research Associates (SPR), to conduct this study (Contract 91990018C0057).

The funding of adult education programs through the Workforce Innovation and Opportunity Act (WIOA) of 2014 is a key component of federal efforts to help low-skilled adults succeed in the workforce and become productive and engaged citizens. However, existing research on adult education provides little guidance on effective approaches for adult learners. As part of the congressionally mandated National Assessment of Adult Education, ED is implementing this feasibility and design study, which is intended to help fill this knowledge gap. The study begins with a feasibility assessment to determine whether specific adult education approaches could be rigorously evaluated at this time. If evaluations that could provide policy- and practice-relevant knowledge are feasible, ED may decide to conduct up to two evaluations in a subsequent phase of the National Assessment of Adult Education.

The study’s initial feasibility assessment, the focus of this clearance package, will determine whether specific adult education approaches prioritized by ED and aligned with WIOA priorities can be evaluated using rigorous study designs. The assessment will draw on interviews with directors of adult education programs that currently implement, or that are interested in implementing, one of the prioritized approaches. If ED determines that impact studies are feasible, revised clearance packages will be submitted for data collections not covered under this request.

A. JUSTIFICATION

A.1. Circumstances necessitating the collection of information

a. Statement of need for a rigorous evaluation of adult education approaches

Title II of the Workforce Innovation and Opportunity Act (WIOA), known as the Adult Education and Family Literacy Act (AEFLA), is the legislation currently governing the federal investment in adult education. Section 242(b) of WIOA mandates that ED carry out rigorous research and evaluation on effective adult education and literacy activities, one component of which is an independent evaluation of Title II programs and activities. Section 242(c) of WIOA also authorizes ED to examine various issues related to the effectiveness of adult education programs and services.

The funding of adult education programs has been a key part of federal efforts to address the nation’s skill deficits. Under Title II of WIOA, the Office of Career, Technical, and Adult Education (OCTAE) awards grants to states to help improve adults’ basic skills. In FY 2018, OCTAE awarded approximately $617 million in AEFLA grants (OCTAE 2018). In turn, states distribute their WIOA Title II funds through awards to service providers. In program year 2017-2018 states primarily distributed these funds to local education agencies (42 percent of funds), community colleges (33 percent), and community-based organizations (11 percent) (calculated from data available in the National Reporting System [NRS] 2018). Adult education providers across the country served approximately 1.4 million participants in program year 2017-2018 (NRS 2018).

Despite the importance of basic skills in determining career and life success, the research on adult education provides no clear consensus on effective approaches for improving the outcomes of adult learners, partly because of the small number of rigorous studies that have been conducted (Condelli 2009; Kruidenier et al. 2010; National Research Council [NRC] 2012). Seven studies that the NRC reviewed used a random assignment design. However, many of these studies found small or insignificant effects possibly because of small samples, lack of a meaningful contrast in services provided between study groups, and weak persistence in the programs.

Against this backdrop, ED is funding this feasibility and design study to help fill the knowledge gap. Through a systematic evidence review and, if feasible, through rigorous effectiveness studies of prioritized adult education program models, activities and/or services, this study offers an important opportunity to provide policy-relevant information not only to states and programs seeking guidance on proven strategies but also to federal policymakers, who can promote AEFLA program improvement in different ways.¹

b. Overview of study design and research questions

The feasibility study will investigate whether specific policy-relevant adult education approaches could be the focus of effectiveness studies involving random assignment or regression discontinuity designs. Plans for the study were shared at national conferences with directors and staff of state adult education offices and local programs, where these staff shared their priorities for approaches that might be tested in a national study. This information, along with a scan of existing research and more in-depth phone conversations with nine state directors, informed the identification of the specific approaches being considered for an impact study.

In selecting the specific approaches being considered for an impact study, priority was given to approaches that are aligned with WIOA performance measures, consistent with approaches emphasized within WIOA, and applicable to large segments of adult learners. Table A.1 contains a description of the eight approaches being considered. These approaches were selected because they reflect one of two WIOA priorities:

• A clear link between adult education and workforce development. Integrated education and training, on-ramp and bridge classes, and career navigation supports are approaches that integrate adult education and literacy activities within the larger context of career pathways and support students’ progression along a career path.

• Effective use of technology. Distance learning, blended learning, and the use of mobile or online learning tools to supplement instruction are approaches that use technology to increase learner access to instruction and to opportunities to practice the content of instruction.

Table A.1. Adult education approaches for exploration in feasibility study

Approach	Description
Integrated education and training (IET) through co-teaching	Programs in which two teachers—one basic skills teacher and one occupational skills teacher—teach integrated curricula (adult education instruction concurrently and contextually with workforce preparation activities and workforce training) focused on a single set of learning goals. The model involves teachers planning together and teaching in the same classroom at the same time.
IET through co-planning and alternate teaching	Programs in which two teachers—one basic skills teacher and one occupational skills teacher—teach integrated curricula (adult education instruction concurrently and contextually with workforce preparation activities and workforce training) focused on a single set of learning goals. The model involves co-planning and alternate teaching, where teachers plan together, but teach during separate class times.
On-ramp and bridge classes focused on occupational skills training	Classes or programs to prepare learners to transition to occupational training by providing basic skill instruction along with occupational content or employment skills instruction.
Bridge classes that focus on college readiness	Classes or programs to prepare learners to transition to college courses by providing basic skill instruction along with college success strategies.
Career navigation supports	Supports available to learners through an adult education provider but delivered outside of class time that are designed to help learners progress on a career path. Supports include career exploration and planning assistance; assistance accessing financial supports for education; and support developing study and work skills.
Distance learning	An approach that delivers all instruction in an entirely virtual format. The technology used can include the Internet, broadcast, closed circuit, cable, wireless communication devices, videos, DVDs, or CD-ROMs.
Blended learning	An approach that delivers instruction through a combination of in-person and virtual instruction, with both components required as part of the course.
Mobile or online learning tools to supplement instruction	Resources that learners can independently access through the internet on a computer or mobile device and that reinforce classroom instruction.

ED may decide to evaluate up to two of these approaches. If ED decides to conduct an evaluation of any of the approaches, a revised clearance package will be submitted, covering the additional data collection.

For the feasibility study, the study team will conduct interviews with adult education program directors to help assess the feasibility of evaluating each of the approaches in Table A.1. Interviews will be conducted with program directors who are implementing or interested in implementing one of these approaches. An initial set of screener questions will be asked to confirm whether program directors are implementing and/or interested in and able to implement one of these approaches. Those who are not implementing or interested will be screened out. The study assumes that through this screening process, an initial sample of 60 program directors will result in a final sample of 50 program directors for the full feasibility interviews. Table A.2 lists key questions that the feasibility and design study will address.

Table A.2. Questions the feasibility and design study will address

What do we know about effective approaches to adult education?

Approach: Systematic evidence review of high quality studies in adult education and related education and workforce settings

1. What is known about the effectiveness of adult education program models, activities, and services in improving learner outcomes such as literacy, numeracy, and functional and workplace skills, English language proficiency, and employment and earnings? Do impacts vary across key subgroups of learners?

2. What key adult education program models, activities, and services have not yet been rigorously evaluated?

^{What do we know about the implementation of the selected adult education approaches?}

Approach: Interviews with 50 adult education program directors

1. What are the key features of the adult education approaches? How do approaches vary across programs?

2. Are the approaches implemented with fidelity across a range of program contexts, or could they be?

3. Are the approaches sufficiently different from what a control group would receive to produce meaningful impacts?

4. Could the approaches be evaluated with random assignment or regression discontinuity designs? What challenges might such evaluations face? Is there evidence of site interest in participating in an evaluation of the approach?

^{Which adult education approaches are feasible for impact studies and what are the strongest design options for evaluating them?}

Approach: Evaluation design experts from the study team synthesize the information gathered from previous tasks to prepare a design options report

1. What is the target population for the study?

2. What services can the control group receive that would differ sufficiently from those received by the treatment group?

3. Is random assignment feasible? If so, should it occur at the learner, classroom, or site level to balance considerations about program recruitment success, statistical power, and the potential for control group contamination?

4. Is a regression discontinuity design feasible (quantitative score and specific cutoff used to determine receipt)?

5. What is an appropriate minimum detectable effect (MDE) for the study to target? What sample sizes of programs, classrooms, and learners are required to achieve this MDE? How many cohorts will be needed?

6. What recruitment methods and incentives will be needed to ensure sufficient site participation?

7. Which variations in site features (e.g., state governance, region, size, service dosage, and provider type) should be considered to allow for policy-relevant, program-related subgroup analyses?

8. What participant intake, consent, and staff training procedures should be developed to minimize staff burden?

9. What technical assistance and other supports will be required to ensure sufficient program demand to form a control group and that the adult education approaches are implemented with fidelity?

10. Which outcomes should be collected based on the conceptual model underlying the approaches? When and how should they be collected? Is a survey needed to measure key outcomes or will less costly administrative data be sufficient?

11. What information should be collected in order to evaluate the cost-effectiveness of the approach?

A.2. Purposes and uses of the data

This clearance package requests permission to conduct screening with up to 60 adult education program directors, continuing with feasibility interviews with up to 50 of the screened directors, to help assess the feasibility of evaluating each of the eight approaches that are in consideration for an impact study (see screening and interview protocols in Appendix A). The screener questions will ensure that the program directors who participate in the feasibility interviews are either implementing or interested in implementing at least one of the priority approaches. The feasibility interviews will help to narrow the range of adult education approaches that might be included in an impact evaluation, shape the study design options (e.g., random assignment, regression discontinuity) for each approach (including data collections that will be proposed in later clearance packages), and gauge program interest in participating in a possible study. This information will be included in a study design report.

A.3. Use of technology to reduce burden

To minimize respondent burden, the data collection plan is designed to obtain information efficiently. When feasible, we will gather information from existing data sources, using the most efficient methods available. For example, we will collect publicly available data on each program by reviewing its website. This will reduce the need to ask every interview question to every program director.

We will interview the program directors over the phone. This mode allows the interviewers to work around respondents’ schedules and to complete the interview over multiple sessions if necessary.

A.4. Efforts to avoid duplication of effort

As part of the feasibility study, we will conduct interviews with adult education program directors to assess whether and how each of the approaches in Table A.1 could be rigorously evaluated. Data to understand the capacity of adult education sites to participate in an effectiveness study is not readily available from other sources.

A.5. Methods to minimize burden on small entities

The adult education sites selected for inclusion in the feasibility study are primarily expected to be education agencies, community colleges, and community-based organizations. If any of the sites included in the feasibility study are small entities, the study is designed to minimize burden by gathering data from publicly available sources and conducting focused interviews of 60 minutes following a 5-minute set of screening questions conducted by phone.

A.6. Consequences of not collecting data

The requested data are needed to determine whether and how the adult education approaches being considered for an impact study (in Table A.1) could be rigorously evaluated. Existing research on adult education programs provides little guidance on effective approaches for serving adult learners. Ultimately, this study aims to help fill this knowledge gap and provide evidence on the effectiveness of important adult education programs and services aligned with WIOA priorities. Failing to conduct this feasibility study would make it impossible both to conduct effectiveness studies and to inform policymakers and practitioners about the effectiveness of potential approaches to improve the education and employment outcomes of low-skilled adults.

Without information from the screening questions, the feasibility interviews might be conducted with some program directors who are not implementing or interested in implementing the priority approaches. The screener questions ensure that the feasibility interviews are only conducted with program directors better positioned to provide useful information to assess the feasibility of implementing an evaluation.

Without information from the feasibility interview, we would be unable to meet the feasibility study’s overall goals and to respond to WIOA’s mandate to assess the effectiveness of adult education practices. The feasibility interview is a critical step in collecting information such as programs’ recruitment and intake processes, enrollment numbers, implementation capacity, and interest in participating in a rigorous study. Without knowing this information, rigorous evaluations of these adult education approaches will not be possible.

A.7. Special circumstances

There are no special circumstances involved with this data collection.

A.8. Federal Register announcement and consultation

a. Federal Register announcement

The 60-day notice to solicit public comments was published in Vol. 83, No. 242, and Page 64813-64814 of the Federal Register on 12/18/2018. We received one comment addressing the content of this study, which was taken into account in developing plans for identifying the sample for this collection.

b. Consultations outside the agency

We have assembled a Technical Working Group composed of consultants with various types of expertise in the areas relevant to this study; members are listed in Table A.3. The Technical Working Group convened on February 13, 2019 to discuss the study scope and design. The group provided input on key approaches in adult education that should be considered as part of the feasibility study. Consultations will continue to take place throughout the study to ensure the technical soundness, the relevance of the study to the field of adult education, and accessibility of the information from the study.

Table A.3 Members of the Technical Working Group

Name	Title, Affiliation
Daphne Greenberg	Professor of Educational Psychology, Special Education, and Communication Disorders, Georgia State University
Sheryl Hart	Deputy Associate Superintendent, Arizona Department of Education, Adult Education Services
Jon Kerr	Director of Adult Education. Washington State Board for Community and Technical Colleges
Judy Mortrude	Senior Policy Analyst, Center for Law and Social Policy (CLASP)
Kathleen Porter	Executive Director, Poway Unified School District
Esther Prins	Professor of Education, Pennsylvania State University
Cory Rayala	Education Programs Consultant, Adult Education Office, California Department of Education
Stephen Reder	Professor of Applied Linguistics, Portland State University
Jamil Steele	Senior Director for Adult Education, Illinois Community College Board

A.9. Payments or gifts

There are no payments or gifts associated with this data collection effort.

A.10. Assurances of confidentiality

The study team has established procedures to protect the confidentiality and security of its data. This approach will comply with all relevant regulations and requirements, in particular the Education Sciences Reform Act of 2002, Title I, Subsection (c) of Section 183, which requires the director of IES to “develop and enforce standards designed to protect the confidentiality of persons in the collection, reporting, and publication of data.”

The study team will protect the full privacy and confidentiality of all people who provide data. The study will use directory information to contact program directors and will not collect personally identifiable information (PII). The study team will ensure that respondents’ names are not identified in publicly available reports or findings, and, if necessary, the study team will mask distinguishing characteristics.

Mathematica uses the following safeguards to protect confidentiality:

• All Mathematica employees sign a pledge that emphasizes the importance of confidentiality and describes their obligation to maintain confidentiality (Appendix B).

• All internal networks are protected from unauthorized access by using defense-in-depth best practices, which incorporate firewalls and intrusion detection and prevention systems. The networks are configured so that each user has a tailored set of rights, granted by the network administrator, to files approved for access and stored on the network. Access to hard-copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.

• Computer data files are protected with passwords, and access is limited to specific users, who must change their passwords on a regular basis and conform to strong password policies.

• Especially sensitive data are maintained on removable storage devices that are kept physically secure when not in use.

A.11. Additional justification for sensitive questions

No questions of a sensitive nature will be included in this study.

A.12. Estimates of hours burden

Table A.2 provides an estimate of the time burden for the data collection activities during a requested two-year clearance period. The total hours cover the screening and feasibility interviews.

Screening. The estimated cost burden for the screening is based on a 5-minute screening call with 60 adult education program directors. To complete the full feasibility interviews with 50 directors, we assume the study team will need to conduct screening with up to 60 directors. Assuming an average wage of $45.80 per hour, the total cost of burden associated with 5 hours to conduct the feasibility screening is $229.00. Annualized over the two-year period, the cost burden associated with 2.5 hours each year is $114.50.

Feasibility interview. The estimated cost burden for the feasibility interview is based on a 60-minute feasibility interview with 50 adult education program directors. Assuming an average wage of $45.80 per hour, the total cost of burden associated with 50 hours to conduct the feasibility interviews is $2,290.00. Annualized over the two-year period, the cost burden associated with 25 hours each year is $1,145.00.

The total burden for this feasibility study collection is 55 hours with a total burden cost of $2,519.00. Annualized over the two-year period, the cost burden associated with 27.5 hours each year is $1,259.50.

Table A.4. Estimated reporting hour burden

Type of instrument	Total number of respondents	Total responses	Number of responses per respondent	Average burden per response (hours)	Total annual hours	Respondent average hourly wagea	Respondent annual cost burden
Screener (Appendix A)	60	60	1	0.08	2.5	$45.80	$114.50
Feasibility interview (Appendix A)	50	50	1	1	25	$45.80	$1,145.00
Total	60	60			27.5		$1,259.50

^aCosts are based on the mean hourly wage for education administrators from the May 2017 National Occupational Employment and Wage Estimates (https://www.bls.gov/oes/current/oes_nat.htm#11-0000).

A.13. Estimates of cost burden to respondents

There are no additional respondent costs associated with this data collection beyond the burden estimated in Section A.12.

A.14. Estimates of annual costs to the federal government

The estimated cost for the two-year base study, including establishing a technical working group, preparing initial OMB clearance forms, conducting a systematic evidence review, identifying possible adult education approaches, exploring evaluation feasibility, and developing a design report, is $1,399,413, or approximately $699,707 per year.

A.15. Reasons for program changes or adjustments

This is a new information collection request.

A.16. Plan for tabulation and publication of results

a. Tabulation plans

Feasibility Memo

A feasibility memo will summarize the findings from the interviews to help ED decide which adult education approaches appear feasible to evaluate through effectiveness studies. For each approach, we will assess the feasibility and anticipated challenges of implementing a random assignment or regression discontinuity design, as well as program directors’ willingness to participate in an evaluation. We also will describe participant characteristics, intended outcomes, and constraints that might affect implementation or adoption of each approach.

b. Publication plans

Design Report

The study team will prepare a design report that draws on key information from the feasibility memo and the study’s other base contract tasks, including convening and ongoing consultation with a technical working group, and a systematic review of existing research. The report will specify designs for up to five rigorous impact studies of adult education approaches. Each design will be guided by a conceptual model of how the approach can affect the targeted learner outcomes. For each design, the report will describe the approach, the likely contrast in services between the study groups, site selection, needed study supports, the random assignment process and unit, the regression discontinuity design (if relevant), required sample sizes, and outcome measures and data collection strategies.

A.17. Approval not to display the OMB expiration date

All data collection instruments will include the OMB expiration date.

A.18. Explanation of exceptions

No exceptions are requested.

REFERENCES

Condelli, L., H.S. Wrigley, and K.W. Yoon. “‘What Works’ for Adult Literacy Students of English as a Second Language. Developing Learners’ Reading Skills in Adult Basic Education Programs.” In Tracking Adult Literacy and Numeracy Skills, edited by S. Reder and J. Brynner. New York: Routledge, 2009 (pp. 132–159).

Kruidenier, J.R., C.A. MacArthur, and H.S. Wrigley. Adult Education Literacy Instruction: A Review of the Research. Washington, DC: National Institute for Literacy, 2010.

National Reporting System for Adult Education. “Policy and Data.” 2018. Available at https://www.nrsweb.org/.

National Research Council. Improving Adult Literacy Instruction: Options for Practice and Research. Committee on Learning Sciences: Foundations and Applications to Adolescent and Adult Literacy, edited by A.M. Lesgold and M. Welch-Ross. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press, 2012.

Office of Career, Technical, and Adult Education. “Adult Education-Basic Grants to States” https://www2.ed.gov/programs/adultedbasic/funding.html

www.mathematica-mpr.com

Improving public well-being by conducting high quality,
objective research and data collection

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Seattle, WA ■ TUCSON, AZ ■ Washington, DC ■ Woodlawn, MD

1 Hereafter, the adult education program models, activities and/or services that the feasibility study will explore are referred to as approaches.