Screener and Interview Protocol
Assessing Evidence of Effectiveness in Adult Education
Appendix A interview protocol final
Screener and Interview Protocol
OMB: 1850-0947
Assessing Evidence of Effectiveness in Adult Education
APPENDIX A.
FEASIBILITY
INTERVIEW PROTOCOL
This appendix contains four versions of the feasibility interview protocol, and each focuses on one or more of the eight priority approaches:
Bridge or on-ramp programs. This version covers the following:
Bridge or on-ramp programs focused on occupational skills training: classes or programs to prepare learners to transition to occupational training by providing basic skill instruction along with occupational content or employment skills training.
Bridge programs focused on college readiness: classes or programs to prepare learners to transition to college courses by providing basic skill instruction along with college success strategies.
Instructional approaches using technology. This version covers the following:
Distance learning: an approach that delivers all instruction in an entirely virtual format. The technology used can include the Internet, broadcast, closed circuit, cable, wireless communication devices, videos, DVDs, or CD-ROMs.
Blended learning: an approach that delivers instruction through a combination of in-person and virtual instruction, with both components required as part of the course.
Mobile or online learning tools to supplement instruction: resources that learners can independently access through the internet on a computer or mobile device and that reinforce classroom instruction.
Integrated Education and Training (IET): This version covers programs that include integrated curricula (adult education instruction provided concurrently and contextually with workforce preparation activities and workforce training) focused on a single set of learning goals, through either “co-teaching” or “co-planning and alternate teaching.”
Career navigation supports: This version covers a range of supports designed to help learners progress on a career path. Supports include career exploration and planning assistance; assistance with connecting to postsecondary institutions and/or occupational training and accessing financial supports for education; and support developing study and work skills.
Feasibility Screener/Interview Protocol—Bridge Programs
Provider name |
|
Provider city and state |
|
Respondent name |
|
Respondent title |
|
Interviewer |
|
Date of interview |
|
Current or potential implementer of focal approach? |
|
Introduction
The U.S. Department of Education is conducting a National Assessment of Adult Education. This assessment is required by Congress and includes activities to understand and potentially expand evidence on the effectiveness of particular adult education program models, activities, and services. For shorthand, we will refer to these as “approaches” throughout our conversation. The Department contracted with Mathematica to explore the feasibility of conducting future large-scale effectiveness studies in adult education. We’re in the process of speaking to adult education providers that are implementing or might implement one or more of the approaches of interest, including bridge and on-ramp programs that prepare learners to transition to college courses or occupational training.
If current implementer: We identified [PROVIDER NAME] as implementing bridge programs.
If potential implementer: We identified [PROVIDER NAME] as a potential site for implementing bridge programs.
We are speaking to providers for three reasons. First, we aim to better understand how adult education approaches like bridge programs [ARE/COULD BE] implemented, and to get a picture of how adult education providers recruit and enroll learners. Second, we want to understand what participant data providers collect. Last, our goal is to understand what would be needed to include a provider like you in a national evaluation and how willing your organization might be to participate in an evaluation.
Information from these interviews will be used to inform the Department’s decision about which adult education program models, activities or services are feasible to study.
Mathematica and its subcontractors, Manhattan Strategy Group and Social Policy Research, follow the confidentiality and data protection requirements of the Institute of Education Sciences at the Department of Education (Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for research purposes. Your participation in this interview is voluntary; if you do not feel comfortable answering any specific question, we can skip that question and move on to the next one. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific individual or organization. The study will not disclose the names of individual study participants, except as required by law. Do you agree to participate in this interview?
I would like to record the interview, just in case I need to check the accuracy of my notes. The recording will not be used for any other purpose or shared with anyone outside the study team. Would it be OK if I record this interview?
According
to the Paperwork Reduction Act of 1995, no persons are required to
respond to a collection of information unless such collection
displays a valid OMB control number. The valid OMB control number
for this information collection is xxxx-xxxx.
We estimate that it will take about 5 minutes to answer the
questions in the screener interview and 60 minutes to answer the
questions in the feasibility interview. If you have any comments
concerning the accuracy of the time estimates or suggestions for
improving this form, please write to: U.S. Department of Education,
Washington, DC 20202-4651. If you have comments or concerns
regarding the status of your individual submission of this form,
write directly to: Institute of Education Sciences, U.S. Department
of Education, 550 12th Street, SW, Washington, DC 20202.
INTERVIEWER NOTE: Skip to Section B for potential implementers.
SECTION A: FOCAL APPROACH QUESTIONS FOR CURRENT IMPLEMENTERS
A1. Screener
INTERVIEWER NOTE: Information available about the provider’s implementation of the approach should be prefilled by the interviewer. Depending on how much information you have about the provider before the interview, you may be able to complete the screener through email while scheduling the interview.
This interview will focus on bridge programs that prepare learners for college or occupational training and how these are implemented by [PROVIDER NAME].
Bridge programs may also be referred to as on-ramp programs. They offer basic skill instruction along with occupational content, employment skills, and/or college success strategies. They are aimed to prepare learners to transition to college courses or occupational training. Before we get started, I want to confirm that [PROVIDER NAME] offers bridge programs or classes. Is that correct?
Is there another term you would use to refer to bridge programs? Do you refer to them as bridge programs or bridge classes?
A2. Interview
Program description
This interview will focus on bridge programs and how they are implemented by [PROVIDER NAME].
What bridge programs does [PROVIDER NAME] offer?
Are these programs focused on preparing learners for transition to occupational training or future jobs, preparing learners to transition to college, or both?
(Some questions below are for programs that focus on occupational training or for programs that focus on college readiness.)
Ask for each bridge program offered:
Does this bridge program use a commercial, off-the-shelf curriculum, or is it developed by the instructors, or a mix of both? If it uses an off-the-shelf curriculum, which one?
Is your bridge program offered as a standalone “bridge class,” or do you offer a course that integrates adult education and literacy content with skill development related to postsecondary education or employment?
If standalone class: Are all participants in the bridge class also enrolled in another adult education class focused on basic skills at the same time? How is this bridge class scheduled relative to other adult education classes?
What basic skills content is taught in the bridge programs? For example, literacy, numeracy, English language acquisition, high school equivalency preparation.
What skill levels is the bridge program targeted at? If ABE or ASE, ABE levels 1–6? If ESL, ESL levels 1–6?
What are other characteristics of learners in bridge programs—for example, levels of education, employment status, and experience?
Is there a range of [skill levels, levels of education, employment status, and experience] across participants? Can you describe the range?
Do participants in bridge programs differ from the participants in your traditional adult education classes? If so, how?
Ask for each bridge program focused on occupational training; skip if focused only on college:
Thinking about how the bridge programs might prepare learners for transition to occupational training or future jobs . . .
Are there ways in which basic skill content is contextualized with content from a specific occupational field? Is there a focus on a specific occupational field?
If multiple occupational fields: In what ways do the bridge programs differ across occupational fields?
(Note to interviewer: If there are differences, probe throughout about differences across occupational fields.)
Are any job shadowing or other job-site learning opportunities offered?
What, if any, employment skills or workforce preparation skills are taught in the bridge programs?
Probe if not mentioned:
Applied math, writing, or science skills
Interpersonal skills, such as team work, leadership, conflict resolution, and customer service
Critical-thinking skills, such as decision making, problem solving, and planning
Communication skills, including verbal communication and reading and writing workplace and business documents
Technology skills
System navigation skills
Resource management skills, including time, money, materials, and personnel
What, if any, career exploration activities are included in the bridge programs?
Would you be interested in adding any of the following components to the bridge programs: employment or workforce preparation skills training, or career exploration activities? Is there a particular approach that you think would work well? Why?
In some areas, bridge programs are part of IET programs, or IET programs are referred to as bridge programs. Do you also offer IET?
What is the relationship between this bridge program and your IET program(s)? How does the content differ? How do the skill levels of learners differ?
If bridge program not part of IET: Is your bridge program part of a career pathway?
What is the relationship between this bridge program and your career pathway program?
Ask for each bridge program focused on college readiness; skip if focused only on occupational training:
Thinking about how the bridge programs might prepare learners for transition to college . . .
What, if any, college success strategies are taught in the bridge programs?
Are other connections to colleges offered through the bridge program—for example, holding classes on a college campus or other types of partnerships with colleges?
Are there other elements of the bridge program that mirror college courses—for example, semester-long instructional sequences?
Would you be interested in adding any new content or activities to your bridge program that would help students prepare to transition to college? Is there a particular approach that you think would work well? Why?
Ask for all each bridge program offered:
Are any supplemental supports offered as part of the bridge program, such as one-on-one career advising, introductions to college admissions processes, assistance with financial aid or course selection?
After participants finish the bridge program, what options are available to them? For example, another adult education class that [PROVIDER NAME] offers, an IET program, a high school equivalency preparation course, developmental education courses at a college, college credit courses, postsecondary education, a job, an apprenticeship, an internship, something else?
After cycling through questions above for each bridge program offered, ask the following questions once, referring to all bridge programs offered.
Does [PROVIDER NAME] offer sequences of bridge classes?
How many weeks or months do bridge classes typically run?
Do you offer bridge classes throughout the year or at specific times during the year?
In how many locations do you provide bridge programs?
If more than one: Where are those located?
Do you also offer other adult education classes at the same locations? What types of classes?
In what ways does the basic skill instruction part of bridge programs differ from basic skill instruction taught in other adult education classes you offer?
Do you partner with other organizations to provide bridge programs?
Can you describe the partnerships, including what they offer your program?
How long has [PROVIDER NAME] been offering bridge programs?
Does [PROVIDER NAME] have plans to expand or change your bridge program offerings? In what ways? When?
What challenges do you face or have you faced in implementing bridge programs?
Probe: Recruiting learners? Retaining learners until completion? Hiring, training, and retaining staff who can implement the program? Staff availability for planning and class time? Having the right material resources, such as a space to offers classes, the software/hardware necessary? Issues with developing or maintaining necessary partnerships?
To what extent are you satisfied with the [approach/name of model, if applicable] you are using?
What works well?
What doesn’t work as well?
Is there anything you would like to change about it?
Recruitment and eligibility
Next, I’d like to understand how you recruit learners in bridge programs.
Can you walk me through how an individual would enroll, starting with how they learn about the bridge programs, contact [PROVIDER NAME], then the intake, screening, and enrollment procedures?
Probe: How are they recruited? For example, word of mouth (family, friends), partner referrals, local media (radio, newspapers), place(s) of worship, place of employment, American Job Centers, libraries.
Probe: Do you conduct any marketing or outreach about bridge programs?
Probe: How does the intake process work? Is it done electronically or on paper? Are there specific individuals who do intake? How many?
Are there any eligibility requirements participants must meet? For example, a pre-test assessment score, certain education functioning levels, a degree of English proficiency.
If there is a pre-test: What is the pre-test assessment?
Is a cutoff score used?
If yes: Is the same cutoff used consistently for all learners?
If yes: How did you determine the cutoff score?
How many applicants do you receive, and how many do not meet the cutoff score?
Do all learners who enroll take this pre-test assessment before starting the bridge program?
How does it differ from the assessments administered to learners in other adult education and literacy classes?
If there is not an eligibility requirement: Do learners who enroll take a pre-test assessment before starting the bridge program?
If yes: What is the assessment?
How does it differ from the assessments administered to learners in other adult education and literacy classes?
If occupational focused bridge program: Are there any industry-related prerequisites? Any aptitude tests?
Are many learners who are interested in the bridge programs turned away because they are not eligible? Which eligibility criteria do these learners typically fail to meet? About how many are turned away due to eligibility?
Are potential learners counseled to enroll, or not to enroll, in bridge programs versus traditional adult education classes? How do counselors determine which learners to encourage to participate in bridge programs?
Is there a cost to participate in bridge programs?
If yes: What is the cost?
To what extent is cost a barrier to participation?
Enrollment capacity
These next questions ask about your current enrollment and the capacity for [PROVIDER NAME] to expand services.
If multiple bridge programs are offered, probe on the following for each one.
How many learners currently participate in bridge programs?
Is this a typical number?
Do you expect it to change in the next few years? If so, how?
Do bridge programs have a managed enrollment period or open enrollment throughout the year?
If managed: What is the length of the typical period (for example, semester, every 6–8 weeks, other)?
Are you currently able to serve all learners who are interested in and meet the requirements for bridge programs?
If no:
Why not?
About what percentage of interested, eligible learners are turned away for these reasons?
Do you have a waitlist? How does the waitlist work? About how long do most learners stay on the waitlist before getting a spot?
Do you refer those who don’t get a spot to another type of class or service you offer or to another provider in the area? To what extent do learners enroll in these alternatives?
What is the maximum number of learners you could enroll in bridge programs at a given time? Does this vary at different times in the year? If so, how does it vary?
What factors limit the number of learners you could enroll?
Probe: Staff, space, or something else?
Data availability and assessment
Next we’d like to understand more generally how your organization collects and records data about learners.
What data do you collect on bridge learners’ demographics and participation?
What type of data do you track about the classes or services an individual participates in? For example, the classes or services learners enroll in or the length of time enrolled?
Do you track attendance?
Do the data you track differ for learners you serve in other adult education classes?
What kind of contact information and other identifying information do you collect?
How often is contact information missing? How often is the Social Security number missing?
Which assessment(s) do you use as a post-test for bridge learners?
Do you administer post-tests after a set amount of time or after a set number of instructional hours?
Does this differ from the assessments administered to learners in other adult education and literacy classes?
For about what percent of learners do you have post-test data?
Do you use any assessments to measure progress during enrollment in bridge programs? Which assessments?
At what time points or after how many instructional hours do you administer these assessments?
Does this differ from the assessments administered to learners in in other adult education and literacy classes?
How do you obtain data on other learner outcomes? Probe: employment measures, credential attainment.
Are any of these outcomes from an administrative data source like unemployment insurance wage records? For what percentage of your learners are you able to match to existing records?
Are the demographic, participation, assessment, and other outcome data maintained in a program management information system?
Do you have a dedicated data manger?
If no: Who manages your program data?
Feedback on random assignment and doing a study
One of the best ways to learn about the effectiveness of bridge programs is an experiment in which we would randomly choose (such as by a coin flip) which learners, classes, or even sites, would receive a bridge curriculum, or a specific enhancement or variation of bridge, such as career exploration activities or direct connections with colleges. Or, if you had more learners interested in and eligible for bridge programs than spots available, we could conduct a lottery to determine which learners would receive these services and which would not.
We’d like to get your reactions to the idea of studying this approach through a study where some locations or some learners participate in bridge programs.
Suppose some learners or sites were assigned by lottery to attend bridge programs and others to a basic skills class without a bridge focus, or to a different bridge program (for example, with different types of occupational content or a different type of career exploration). Do you have any concerns about this way of selecting who would get the services? How do you think your staff and partners would react?
To what extent do learners and instructors from different classes interact, potentially sharing information that they learned or taught in a different class?
We understand that some states have requirements for services that must be offered that might restrict us from being able to assign learners through a lottery. Are you aware of any requirements or restrictions related to bridge programs in [STATE NAME] that might restrict our ability to conduct a study like this? Do you anticipate any such initiatives in the future?
One way to implement the study so that all learners receive the program in some form is to offer different versions of the approach being studied. For example, some learners receive bridge programs with on-the-job training opportunities and some receive bridge programs with occupational instruction in the classroom. Are there variations like these that you would be interested in testing?
What types of supports would be helpful to convince [PROVIDER NAME] to participate in an evaluation like this of bridge programs? For example, funds to support a staff person collecting data or additional staff development. Are there financial or other incentives that would make participation more attractive?
Have any evaluations been conducted on your bridge programs? Do you know of any evaluations of similar bridge programs conducted elsewhere?
INTERVIEWER NOTE: Skip to Section C.
SECTION B: FOCAL APPROACH QUESTIONS FOR POTENTIAL IMPLEMENTERS
B1. Screener
INTERVIEWER NOTE: Information available about the provider’s current offerings should be prefilled by the interviewer. Depending on how much information you have about the provider before the interview, you may be able to complete the screener through email while scheduling the interview.
This interview will focus on bridge programs that prepare learners for college or occupational training and how these could be implemented by [PROVIDER NAME].
Bridge programs may also be referred to as on-ramp programs. They offer basic skill instruction along with occupational content, employment skills, and/or college success strategies. They are geared toward preparing learners to transition to college courses or occupational training.
Before we get started, I want to confirm that [PROVIDER NAME] does not currently offer bridge programs or classes. Is that correct?
Are you planning to implement bridge programs or classes in the next year or two?
Are you far enough along in the planning process for implementation that you could answer some questions in this interview about what implementation of bridge programs or classes would look like, including topics like recruitment, enrollment capacity, and potential implementation challenges?
B2. Interview
Program description
The first set of questions focuses on programs currently offered by [PROVIDER NAME] and how bridge programs might fit into them.
What types of adult education and literacy classes do you currently offer—ABE, ASE, and/or ELA?
We’re interested in whether you offer adult education and literacy alongside any occupational training or topics.
Do you currently offer adult education and literacy classes during the same time period and in the same classroom as instruction with an occupational skills training partner (that is, IET with co-teaching)?
How about adult education and literacy classes that use a shared curriculum as instruction with an occupational skills training partner but are not offered in the same classroom (that is, IET with co-planning and alternate teaching)?
How about adult education and literacy classes offered concurrently with instruction with an occupational skills training partner but without a shared curriculum?
How about adult education and literacy classes offered within the context of occupational topics?
Do you see a need at specific skill levels for a bridge program, focused on either further education or training? If ABE or ASE, ABE levels 1–6? If ESL, ESL levels 1–6?
If you were to add a bridge program, would it be part of a career pathway or IET program?
Would you want to offer it as a standalone class or integrated into an adult education and literacy course?
Is there a specific bridge model or off-the-shelf curriculum you are planning to use? Why did you identify this model or curriculum?
Are you planning to develop a bridge program focused on postsecondary readiness or on a specific occupational area(s), or both? If an occupational area, which one(s)?
Why does this interest you?
What steps have you taken to lay the groundwork for implementing a bridge program?
Do you have sufficient resources to implement a bridge program?
What supports would be necessary for [PROVIDER NAME] to implement a bridge program?
When do you think you might be able to implement a bridge program?
How feasible is this? What might get in the way?
What challenges do you anticipate with implementing a bridge program?
Probe: Recruiting learners? Retaining learners until completion? Hiring, training, and retaining staff who can implement the program? Staff availability for planning and class time? Having the right material resources, such as a space to offer classes and the software/hardware necessary? Issues with developing or maintaining necessary partnerships?
In how many locations do you provide adult education and literacy classes?
If more than one: Where are those located?
Do you offer the same classes at the same levels and with the same content at all locations? If not, how do they differ?
Do you partner with other organizations to provide adult education and literacy classes?
Can you describe the partnerships, including what they offer your program?
Recruitment and eligibility
Next, I’d like to understand how you recruit learners.
Can you walk me through how an individual would enroll, starting with how they learn about the adult education and literacy classes, contact [PROVIDER NAME], then the intake, screening, and enrollment procedures?
Probe: How are they recruited? For example, word of mouth (family, friends), partner referrals, local media (radio, newspapers), place(s) of worship, place of employment, American Job Centers, libraries.
Probe: Do you conduct any marketing or outreach about adult education and literacy classes?
Probe: How does the intake process work? Is it done electronically or on paper? Are there specific individuals who do intake? How many?
If you implemented a bridge program, would the intake process for it use a similar approach?
Are potential learners counseled to enroll, or not to enroll, in specific types of adult education and literacy classes? How do counselors determine which learners to encourage to participate in specific classes?
If you implemented a bridge program, how would learners be counseled to enroll or not enroll in it?
Are there any eligibility requirements participants must meet? For example, a pre-test assessment score, certain education functioning levels, a degree of English proficiency.
If there is a pre-test: What is the pre-test assessment?
Is a cutoff score used?
If yes: Is the same cutoff used consistently for all learners?
If yes: How did you determine the cutoff score?
How many applicants do you receive, and how many do not meet the cutoff score?
Do all learners who enroll take this pre-test assessment before starting adult education and literacy classes?
Does the pre-test differ across types and levels of classes?
If there is not an eligibility requirement: Do learners who enroll take a pre-test assessment before starting the adult education and literacy classes?
If yes: What is the assessment? If you implemented a bridge program, would you use the same eligibility requirements?
Are many learners that are interested in adult education and literacy classes turned away because they are not eligible? Which eligibility criteria do these learners typically fail to meet? About how many are turned away due to eligibility?
Is there a cost to participate in adult education and literacy classes?
If yes: What is the cost?
To what extent is cost a barrier to participation?
If you implemented a bridge program, would the costs to learners be the same as for other classes?
Enrollment capacity
These next questions ask about your current enrollment and the capacity for [PROVIDER NAME] to expand services.
How many learners currently participate in adult education and literacy classes?
Is this a typical number?
How does enrollment vary across types and levels of classes?
Do you expect it to change in the next few years? If so, how?
About how many learners do you think might be at the appropriate skill level and interested in participating in the bridge program you are planning to implement?
Are you currently able to serve all learners who are interested in and meet the requirements for adult education and literacy classes?
If no:
Why not?
About what percentage of interested, eligible learners are turned away for these reasons?
Do you use a managed enrollment period or an open enrollment throughout the year?
If managed: What is the length of the typical period (for example, semester, every 6–8 weeks, other)?
If you implemented a bridge program, would you use the same type of enrollment period?
Data availability and assessment
Next we’d like to understand more generally how your organization collects and records data about learners.
What data do you collect on learners’ demographics and participation?
What type of data do you track about the classes or services an individual participates in? For example, the classes or services learners enroll in or the length of time enrolled?
Do you track attendance?
Do the data you track differ for learners you serve in different types of classes?
What kind of contact information and other identifying information do you collect?
How often is contact information missing? How often is the Social Security number missing?
Which assessment(s) do you use as a post-test for adult education learners?
Do you administer post-tests after a set amount of time or after a set number of instructional hours?
Does this differ across types and levels of classes?
For about what percent of learners do you have post-test data?
Do you use any assessments to measure progress during enrollment in adult education classes? Which assessments?
At what time points or after how many instructional hours do you administer these assessments?
Does this differ across types and levels of classes?
How do you obtain data on other learner outcomes? Probe: employment measures, credential attainment.
Are any of these outcomes from an administrative data source like unemployment insurance wage records? For what percentage of your learners are you able to match to existing records?
Are the demographic, participation, assessment, and other outcome data maintained in a program management information system?
Do you have a dedicated data manger?
If no: Who manages your program data?
Feedback on random assignment and doing a study
One of the best ways to learn about the effectiveness of bridge programs is an experiment in which we would randomly choose (such as by a coin flip) which learners, classes, or even sites, would receive a bridge curriculum, or a specific enhancement or variation of bridge, such as career exploration activities or direct connections with colleges. Or, if you had more learners interested in and eligible for bridge programs than spots available, we could conduct a lottery to determine which learners would receive these services and which would not.
We’d like to get your reactions to the idea of studying this approach through a study where some locations or some learners participate in bridge programs.
Suppose some learners or sites were assigned by lottery to attend bridge programs and others to a basic skills class without a bridge focus, or to a different bridge program—for example, with or without career exploration. Do you have any concerns about this way of selecting who would get the services? How do you think your staff and partners would react?
To what extent do learners and instructors from different classes interact, potentially sharing information that they learned or taught in a different class?
We understand that some states have requirements for services that must be offered that might restrict us from being able to assign learners through a lottery. Are you aware of any requirements or restrictions related to bridge programs in [STATE NAME] that might restrict our ability to conduct a study like this? Do you anticipate any such initiatives in the future?
One way to implement the study so that all learners receive the program in some form is to offer different versions of the approach being studied. For example, some learners receive bridge programs with job-site learning opportunities and some receive bridge programs with occupational instruction in the classroom. Are there variations like these that you would be interested in testing?
What types of supports would be helpful to convince [PROVIDER NAME] to participate in an evaluation like this of bridge programs? For example, funds to support a staff person collecting data or additional staff development. Are there financial or other incentives that would make participation more attractive?
INTERVIEWER NOTE: Continue to Section C.
SECTION C: NON-FOCAL APPROACH QUESTIONS FOR ALL RESPONDENTS
C. Interview non-focal approaches questions
Feasibility of implementing a new approach
We are looking at a number of other models or approaches that could be implemented and evaluated. (Interviewer: The table below will be sent in an email to respondents before the interview; refer to the table and read through definitions of each approach not addressed in previous sections, and ask questions a and b to determine current implementation and interest in future implementation.)
Is your organization currently implementing this approach?
If not, do you have plans to implement this approach, and have you taken steps to lay the groundwork for it that would allow you to launch it in the next year with the resources you currently have available to you? Would you be interested in participating in an impact evaluation of this approach?
Intervention |
Implementing? |
Planning? |
Distance learning: An approach that delivers all instruction in an entirely virtual format. The technology used can include the Internet, broadcast, closed circuit, cable, wireless communication devices, videos, DVDs, or CD-ROMs. |
|
|
Blended learning: An approach that delivers instruction through a combination of in-person and virtual instruction, with both components required as part of the course. |
|
|
Mobile or online learning tools to supplement instruction: Resources that learners can independently access through the internet on a computer or mobile device and that reinforce classroom instruction. |
|
|
IET: Programs that include integrated curricula (adult education instruction concurrently and contextually with workforce preparation activities and workforce training) focused on a single set of learning goals, through either “co-teaching” or “co-planning and alternate teaching.” |
|
|
Career navigation supports: A range of supports to help learners progress on a career path. Supports include career exploration and planning assistance; assistance accessing financial supports for education; and support developing study and work skills. |
|
|
INTERVIEWER NOTE: Ask sections below for (a) all approaches currently implementing and (b) all approaches interested in and able to implement in next year. If there is not time to cover all, ask about the two approaches already implemented or close to implementation, and ask if we can follow up via email if necessary about others.
Distance or blended learning
If currently implementing distance or blended learning:
Do you use [APPROACH] with ABE, ASE, or ELA CLASSES?
Do you use an off-the-shelf curriculum for the distance/blended classes? Are there any commercial software products you use in these classes?
If currently implementing blended: What components of the blended classes are delivered virtually and what components are delivered in person?
Do you have more learners interested in distance or blended learning than you are able to serve? (Probe for distance and blended separately, if implementing both.)
What challenges do you face in implementing distance or blended learning? (Probe for distance and blended separately, if implementing both.)
Are there new curricula, software products, or approaches you would like to try or changes you would like to make to your distance or blended classes?
If planning on implementing distance or blended learning:
Is there an off-the-shelf distance or blended curriculum you are planning to use? A commercial software product you would be interested in using in these classes?
What kind of resources would you need to support implementing this type of approach? Additional space or materials? Additional resources to purchase curricula or software? Additional staff? Training of staff? Other capacity issues?
What challenges would you anticipate with implementing distance or blended learning? (Probe for distance and blended separately, if planning on both.)
Do you anticipate that you would have more learners interested than you would be able to serve?
Supplemental online or mobile tools
If currently implementing supplemental online or mobile tools:
What types of online or mobile tools are used by instructors to supplement material taught in adult education classes? Do instructors connect the material covered in the online or mobile tools to class instruction? Do instructors monitor learners’ use of these tools?
Do you use [APPROACH] with ABE, ASE, or ELA CLASSES?
What challenges do you face in using supplemental online or mobile tools?
Are there new tools you would like to try or changes you would like to make to how tools are used to supplement classes?
If planning on implementing supplemental online or mobile tools:
Is there a specific online or mobile tool you are planning to use to supplement adult education classes?
What kind of resources would you need to support implementing this type of approach? Training of staff? Other capacity issues?
What challenges would you anticipate with using these tools?
IET
If currently implementing IET:
Does your IET program(s) use an off-the-shelf curriculum, or is it developed by the instructors, or a mix of both? If it is an off-the-shelf curriculum, which one?
What skill levels does the IET program target? Does it serve ABE, ASE, or ELA learners?
Do you use a co-teaching approach or a co-planning with alternate teaching approach?
Do you have more learners interested in IET than you are able to serve?
What challenges do you face in implementing IET?
Would you be interested in trying a [co-teaching/co-planning] approach instead? Why or why not?
Are there new curricula you would like to try or changes you would like to make to your IET programs?
If planning on implementing IET:
Is there a specific IET model or off-the-shelf curriculum you plan to use?
Do you plan to use a co-teaching approach or a co-planning with alternate teaching approach, or another approach?
What kind of resources would you need to support implementing this type of approach? Additional space? Additional staff? Training of staff? Other capacity issues? (Probe for resources related to co-teaching vs. co-planning models.)
What challenges would you anticipate with implementing IET? (Probe for challenges related to co-teaching vs. co-planning models.)
Do you anticipate that you would have more learners interested than you would be able to serve?
Career navigation supports
If currently implementing career navigation supports:
What types of career navigation supports are provided to adult learners? Do you provide career exploration and planning assistance? Do you assist learners in making connections to postsecondary systems (for example, connections to the admissions process)? Do you assist learners in making connections to workforce systems? Do you use any navigation or academic tracking software?
Does career navigation include other supports such as assistance accessing financial supports for education or support developing study and work skills?
Who delivers these career navigation supports?
Do you have more learners interested in supports than you are able to serve?
What challenges do you face in offering these supports?
Are there new approaches to delivering supports you would like to try or changes you would like to make to the supports?
If planning on implementing career navigation supports:
What types of career navigation supports are you planning to offer? For example, will you provide career exploration and planning assistance? Assistance in making connections to postsecondary systems, or to workforce systems?
Are you planning to offer other types of support along with these career supports, such as assistance accessing financial supports for education or support developing study and work skills?
Do you have staff who could deliver these types of supports?
What kind of resources would you need to support implementing this type of approach? Additional space? Training of staff? Other capacity issues?
What challenges would you anticipate with offering career navigation supports?
Do you anticipate that you would have more learners interested than you would be able to serve?
Wrap-up
Those are all the questions I have for you. Thank you very much for your time and thoughtful answers. After I share this information with my colleagues, I might have one or two follow-up questions. In the meantime, if you have any follow-up comments or questions, please feel free to contact me at [INTERVIEWER PHONE/EMAIL].
Feasibility
Screener/Interview Protocol—Instructional
approaches
using technology
Provider name |
|
Provider city and state |
|
Respondent name |
|
Respondent title |
|
Interviewer |
|
Date of interview |
|
Current or potential implementer of focal approach? |
|
Introduction
The U.S. Department of Education is conducting a National Assessment of Adult Education. This assessment is required by Congress and includes activities to understand and potentially expand evidence on the effectiveness of particular adult education program models, activities, and services. For shorthand, we will refer to these as “approaches” throughout our conversation. The Department contracted with Mathematica to explore the feasibility of conducting future large-scale effectiveness studies in adult education. We’re in the process of speaking to adult education providers that are implementing or might implement one or more of the approaches of interest, including instructional delivery approaches using technology.
If current implementer: We identified [PROVIDER NAME] as using technology, either for distance or blended classes or with supplemental online or mobile tools.
If potential implementer: We identified [PROVIDER NAME] as a potential site for using technology, either for distance or blended classes or with supplemental online or mobile tools.
We are speaking to providers for three reasons. First, we aim to better understand how adult education providers [ARE/COULD BE] using technology for distance or blended classes or as supplements to reinforce instruction, and to get a picture of how adult education providers recruit and enroll learners. Second, we want to understand what participant data providers collect. Last, our goal is to understand what would be needed to include a provider like you in a large-scale national evaluation and how willing your organization might be to participate in an evaluation.
Information from these interviews will be used to inform the Department’s decision about which adult education program models, activities, or services are feasible to study.
Mathematica and its subcontractors, Manhattan Strategy Group and Social Policy Research, follow the confidentiality and data protection requirements of the Institute of Education Sciences at the Department of Education (Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for research purposes. Your participation in this interview is voluntary; if you do not feel comfortable answering any specific question, we can skip that question and move on to the next. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific individual or organization. The study will not disclose the names of individual study participants, except as required by law. Do you agree to participate in this interview?
I would like to record the interview, just in case I need to check the accuracy of my notes. The recording will not be used for any other purpose or shared with anyone outside the study team. Would it be OK if I record this interview?
According
to the Paperwork Reduction Act of 1995, no persons are required to
respond to a collection of information unless such collection
displays a valid OMB control number. The valid OMB control number
for this information collection is xxxx-xxxx.
We estimate that it will take about 5 minutes to answer the
questions in the screener interview and 60 minutes to answer the
questions in the feasibility interview. If you have any comments
concerning the accuracy of the time estimates or suggestions for
improving this form, please write to: U.S. Department of Education,
Washington, DC 20202-4651. If you have comments or concerns
regarding the status of your individual submission of this form,
write directly to: Institute of Education Sciences, U.S. Department
of Education, 550 12th Street, SW, Washington, DC 20202.
INTERVIEWER NOTE: Skip to Section B for potential implementers.
SECTION A: FOCAL APPROACH QUESTIONS FOR CURRENT IMPLEMENTERS
A1. Screener
INTERVIEWER NOTE: Information available about the provider’s implementation of the approach should be prefilled by the interviewer. Depending on how much information you have about the provider before the interview, you may be able to complete the screener through email while scheduling the interview.
This interview will focus on using technology in adult education and how it is implemented by [PROVIDER NAME].
We are talking today about three different ways that technology might be used in adult education:
Distance education is defined as a method that uses technology to deliver instruction to learners who are separated from the instructor, and to support regular interaction between the learners and the instructor, either with the class going at a common pace or learners going at their own pace. Technology can include the Internet; broadcast, closed-circuit, and cable TV; wireless communication devices; videos; DVDs; and CD-ROMs.
Blended approaches include a combination of in-person time with a teacher in large or small groups and independent learning using technology, with both components required as part of the course.
There are also online or mobile tools that are accessed by learners to supplement or reinforce course instruction.
Before we get started, I want to confirm that [PROVIDER NAME] is using technology in adult education in one of these ways. Does [PROVIDER NAME] offer adult education classes where all instruction is virtual or distance, and there is no in-person instruction? Does [PROVIDER NAME] offer blended adult education classes that combine in-person and virtual instruction? Does [PROVIDER NAME] provide access to or encourage learners to access online or mobile tools to supplement their adult education classes?
Are there any differences that we should know about between the definitions I provided and how [PROVIDER NAME] implements [TECHNOLOGY]?
Is there another term that you would use to refer to [TECHNOLOGY]?
INTERVIEWER NOTE: Throughout, the placeholder “[DISTANCE/BLENDED]” appears in questions. Please ask questions about one approach or the other, or both, depending on what is offered by the provider. If both types of instruction are offered, you might need to ask questions separately for distance and then for blended.
A2. Interview
Program description
This interview will focus on the use of technology in adult education and how it is implemented by [PROVIDER NAME].
If offer distance classes:
Which types of classes—adult basic education, adult secondary education, or English language acquisition—do you offer in a distance format?
What skill levels do the distance classes target? If ABE or ASE, ABE levels 1–6? If ESL, ESL levels 1–6?
What are other characteristics of learners in distance classes—for example, levels of education, employment status, and experience?
Is there a range of [skill levels, levels of education, employment status, and experience] across learners in your distance classes? Can you describe the range?
How do learners in distance classes differ from the learners in your fully in-person adult education classes?
Do most learners live and/or work near [PROVIDER NAME]? Does this differ from your fully in-person adult education classes?
In what ways do distance classes differ across different skill levels or content areas?
What types of virtual resources do learners access to receive instruction? To practice skills? To communicate with the teacher or classmates? Are there other uses of virtual resources?
Are the virtual resources commercial, off-the-shelf products, or developed by the instructors, or a mix of both? If they are commercial products, which ones?
What types of technology—computers, tablets, mobile phones—can learners use to access the virtual resources?
Do they typically access them at home, at work, at the library, in a computer lab at [PROVIDER NAME], or somewhere else?
Can learners progress at their own pace through class materials, or is there content that all learners access each week at the same pace?
How do teachers monitor learners’ use of virtual resources and progress in class?
How many weeks or months do distance classes typically run?
Do you offer distance classes throughout the year or at specific times during the year?
How long has [PROVIDER NAME] been offering distance classes?
Do you provide any training to instructors on how to use the technology needed for distance classes? What does the training cover? How many hours is it?
If offer blended classes:
Which types of classes—adult basic education, adult secondary education, or English language acquisition—do you offer in a blended format?
What skill levels are the blended classes targeted at? If ABE or ASE, ABE levels 1–6? If ESL, ESL levels 1–6?
What are other characteristics of learners in blended classes—for example, levels of education, employment status, and experience?
Is there a range of [skill levels, levels of education, employment status, and experience] across learners in your blended classes? Can you describe the range?
How do learners in blended classes differ from the learners in your fully in-person adult education classes?
Do most learners live and work near [PROVIDER NAME]? Does this differ from your fully in-person adult education classes?
In what ways do blended classes differ across different skill levels or content areas?
What parts of instruction occur in person and what parts occur virtually?
What types of virtual resources do learners access to receive instruction? To practice skills? To communicate with the teacher or classmates? Are there other uses of virtual resources?
Are the virtual resources commercial, off-the-shelf products, or developed by the instructors, or a mix of both? If they are commercial products, which ones?
Do they typically access them at home, at work, at the library, in a computer lab at [PROVIDER NAME], or somewhere else?
Can learners progress at their own pace through class materials, or is there content that all learners access each week at the same pace?
Can the learner ask for and receive assistance from the instructor virtually while using the distance learning tool? Or is assistance only available during the in-class segment?
How do teachers monitor learners’ use of virtual resources and progress in class?
How many weeks or months do blended classes typically run?
Do you offer blended classes throughout the year or at specific times during the year?
In how many locations do you provide blended classes?
If more than one: Where are those located?
Do you also offer fully in-person adult education at the same locations? Are these for similar topics and levels as the blended options?
How long has [PROVIDER NAME] been offering blended classes?
Do you provide any training to instructors on how to use the technology needed for blended classes? What does the training cover? How many hours is it?
If offer supplemental use of technology:
Now I have some questions about online or mobile tools that are used to supplement and reinforce adult education class instruction.
What types of online or mobile tools can learners access or do teachers encourage them to access to supplement adult education classes?
Is the online or mobile content aligned with what they learn in class? Do they typically align with all content taught in a class or specific portions of content?
Are these tools primarily used to provide additional instruction on a topic? To provide activities to practice skills taught in class? To help learners accelerate through class materials? Or another purpose?
Do teachers monitor learners’ use of these tools?
When teachers encourage use during a class, about what proportion of learners use the tools?
How long has [PROVIDER NAME] been using [TOOLS]?
Do you provide any training to instructors on how to use the online or mobile tools? What does the training cover? How many hours is it?
Ask for all providers:
Do you partner with other organizations to provide [DISTANCE/BLENDED] classes?
Can you describe the partnerships, including what they offer your program?
Does [PROVIDER NAME] have plans to expand or change the current [DISTANCE/BLENDED] classes? Expand or change the use of online or mobile tools? In what ways? When?
What challenges do you/have you faced in implementing [DISTANCE/BLENDED] classes? What about with using online or mobile tools as supplements?
Probe: Recruiting learners? Retaining learners until completion? Learner access to Internet and computers outside of the classroom? Hiring, training, and retaining staff who can implement the program? Staff availability for planning and class time? Having the right material resources, such as a space to offers classes, the software/hardware necessary? Issues with developing or maintaining necessary partnerships?
If using off-the-shelf product: To what extent are you satisfied with the [approach/name of model, if applicable] you are using?
What works well?
What doesn’t work as well?
Is there anything you would like to change about it?
Is there a state policy on distance or blended learning? What is it?
Does your program receive funding specifically for distance or blended learning programming? For example, do distance learners generate full-time equivalent funds or learner contact hour funding?
Recruitment and eligibility (for providers of distance or blended classes)
Next, I’d like to understand how you recruit learners to [DISTANCE/BLENDED] classes.
Can you walk me through how an individual would enroll in one of these classes, starting with how they learn about the class and contact [PROVIDER NAME], then explain the intake, screening, and enrollment procedures?
Probe: How are learners recruited? For example, word of mouth (family, friends), partner referrals, local media (radio, newspapers), place(s) of worship, place of employment, American Job Centers, libraries.
Probe: Do you conduct any marketing or outreach about [DISTANCE/BLENDED] classes?
Probe: How does the intake process work? Is it done electronically or on paper? Are there specific individuals who do intake? How many?
Are potential learners counseled to enroll, or not enroll, in [DISTANCE/BLENDED] classes versus fully in-person classes? Are they counseled to enroll, or not to enroll, in distance versus blended?
How do counselors determine which learners to encourage to participate in distance or blended classes?
Are there any eligibility requirements participants must meet for the [DISTANCE/BLENDED] classes? For example, a pre-test assessment score, certain education functioning levels, a degree of English proficiency.
Does this differ by class?
Are there any requirements that are different from fully in-person adult education classes?
If there is a pre-test: What is the pre-test assessment?
Is a cutoff score used?
If yes: Is the same cutoff used consistently for all learners?
If yes: How did you determine the cutoff score?
How many applicants do you receive, and how many do not meet the cutoff score?
Do all learners who enroll take this pre-test assessment before starting [DISTANCE/BLENDED] classes?
How does it differ from the assessments administered to learners in other adult education and literacy classes?
If there is not an eligibility requirement: Do learners who enroll take a pre-test assessment before starting [DISTANCE/BLENDED] classes?
If yes: What is the assessment?
How does it differ from the assessments administered to learners in other adult education and literacy classes?
Are many learners that are interested in [DISTANCE/BLENDED] classes turned away because they are not eligible?
Which eligibility criteria do these learners typically fail to meet?
About how many are turned away due to eligibility?
Is there a cost to learners to participate in [DISTANCE/BLENDED] classes?
If yes: What is the cost?
To what extent is cost a barrier to participation?
Participation in online or mobile tools (only for providers that offer tools as supplements to adult education classes)
Next, I’d like to understand how learners start using online or mobile tools as supplements to adult education classes.
Can you walk me through how an individual would learn about and start using an online or mobile tool, starting with how they learn about the tool, sign up for it, and access it?
Probe: How do they learn about online and mobile tools? For example, do you provide a list of tools available that are related to their skill levels or career goals? Do you provide a learning module or training class on using the tools?
Probe: Do staff at [PROVIDER NAME] help them set up usernames, install software, or anything else needed to access tools on their own devices?
Are learners encouraged to use or not use online or mobile tools by instructors or counselors or someone else? How do staff determine which learners to encourage to use these tools?
Are the typical learners who use online or mobile tools different from those enrolled in adult education classes and not using online or mobile tools?
Probe: Skill levels, levels of education, employment status, and experience?
Do all learners have access to the same online tools, or are they only available for some classes?
Are there specific requirements participants must meet to use online or mobile tools? Does this differ by tool? For example, a pre-test assessment, certain education functioning levels, a degree of English proficiency.
If there is a pre-test: What is the pre-test assessment?
Is a cutoff score used?
If yes: Is the same cutoff used consistently for all learners?
If yes: How did you determine the cutoff score?
How many applicants do you receive, and how many do not meet the cutoff score?
Are there many learners interested in using online or mobile tools who cannot use them because they do not meet certain eligibility criteria? Which eligibility criteria do these learners typically fail to meet? About how many are turned away due to eligibility?
Is there a cost to use online or mobile tools?
[IF YES] What is the cost?
To what extent is cost a barrier to participation?
Enrollment capacity
These next questions ask about your current enrollment and the capacity for [PROVIDER NAME] to expand services.
If offer distance and/or blended classes:
How many learners currently participate in [DISTANCE/BLENDED] classes?
Is this a typical number?
Do you expect it to change in the next few years? If so, how? Why?
Do [DISTANCE/BLENDED] classes have a managed enrollment period or open enrollment throughout the year?
If managed: What is the length of the typical period (for example, semester, every 6–8 weeks, other)?
What is the maximum number of learners you could enroll in [DISTANCE/BLENDED] classes at a given time? Does this vary at different times in the year? If so, how does it vary?
Are you currently able to serve all learners who are interested in and meet the requirements for [DISTANCE/BLENDED] classes?
If no:
Why not?
About what percentage of interested, eligible learners are turned away for these reasons?
Do you have a waitlist? How does the waitlist work? About how long do most learners stay on the waitlist before getting a spot?
Do you refer those who don’t get a spot to another type of class or service you offer or to another provider in the area? To what extent do learners enroll in these alternatives?
What factors limit the number of learners you could enroll?
Probe: Staff, space, or something else?
If offer supplemental use of technology:
How many learners currently have access to [EACH OF THE TOOLS] as supplements to an adult education class?
How many of those currently use [EACH OF THE TOOLS]?
How do you track who has access to and who uses the tools?
Is there a maximum number of learners that could access [EACH OF THE TOOLS] at a given time?
If maximum: What is the maximum? Does this vary at different times in the year? If so, how does it vary?
If maximum: What factors limit the number of learners that could access the tools?
Probe: Staff, space, or something else?
If maximum: Are you currently able to provide access to all learners who are interested in and meet the requirements for using these tools?
If no: Why not?
If no: About what percentage of interested, eligible learners are turned away for these reasons?
If no: Do you refer those who don’t get access to another type of supplemental instruction or activities that you offer or to another provider in the area?
Data availability and assessment
Next, we’d like to understand more generally how your organization collects and records data about learners.
What data do you collect on [DISTANCE/BLENDED] learners’ demographics and participation?
What type of data do you track about the classes an individual participates in? For example, the classes learners enroll in or the length of time enrolled?
Do you receive any information from the technology vendors about usage of websites, software, or tools? What information do they provide?
Do you track attendance and participation in virtual components in other ways beyond what is provided by the technology vendor?
Do the data you track differ for learners you serve in fully in-person adult education classes?
What kind of contact information and other identifying information do you collect on [DISTANCE/BLENDED] learners?
How often is contact information missing? How often is the Social Security number missing?
Do you collect any participation data for learners who access online or mobile tools as supplements to an adult education class? If so, what do you collect?
Do you receive any information from the technology vendors about usage of websites, software, or tools? What information do they provide?
Do you track usage of these tools in other ways beyond what is provided by the technology vendor?
Which assessment(s) do you use as a post-test for [DISTANCE/BLENDED] learners?
Do you administer post-tests after a set amount of time or after a set number of instructional hours?
How do you determine instructional hours completed for learners who access content asynchronously?
Does this differ from the assessments administered to learners in fully in-person adult education classes?
For about what percent of learners do you have post-test data?
Do you use any assessments to measure progress during enrollment in [DISTANCE/BLENDED] classes? Which assessments?
At what time points or after how many instructional hours do you administer these assessments?
How do you determine instructional hours completed for learners who access content asynchronously?
Does this differ from the assessments administered to learners in fully in-person adult education classes?
Is it required that these assessments are administered in person? If so, does this create any barriers for distance learners?
How do you obtain data on other learner outcomes? Probe: Employment measures, credential attainment.
Are any of these outcomes from an administrative data source like unemployment insurance wage records? For what percentage of your learners are you able to match to existing records?
Are the demographic, participation, assessment, and other outcome data maintained in a program management information system?
Do you have a dedicated data manger?
If no: Who manages your program data?
Feedback on random assignment and doing a study
One of the best ways to learn about the effectiveness of using technology in adult education is an experiment in which we would randomly choose (such as by a coin flip) which learners, classes (or even sites) would receive a distance or blended class, or access to specific online tools. Or, if you had more learners interested in and eligible for distance or blended classes (or access to tools) than spots available, we could conduct a lottery to determine which learners would receive these services and which would not.
We’d like to get your reactions to the idea of studying this approach through a study where some locations or some learners participate in distance or blended classes, or have access to an online or mobile tool, and some don’t.
Suppose some learners were assigned by lottery to participate in distance or blended classes, and others would be assigned to a fully in-person class or a different version. Do you have any concerns about this way of selecting who would get the services? How do you think your staff and partners would react?
Suppose some learners were assigned by lottery to access specific online or mobile tools to supplement other instruction, and others were not. Do you have any concerns about this way of selecting who would get the services? How do you think your staff and partners would react?
Suppose it was sites being randomly selected to offer distance or blended classes, or to offer access to online or mobile tools. Do you have any concerns about this way of selecting who would get the services? How do you think your staff and partners would react?
We understand that some states have requirements for services that must be offered that might restrict us from being able to assign learners through a lottery. Are you aware of any requirements or restrictions related to use of technology in adult education, or specifically distance or blended learning in adult education, in [STATE NAME] that might restrict our ability to conduct a study like this? Do you anticipate any such initiatives in the future?
One way to implement the study so that all learners receive the program in some form is to offer different versions of the approach being studied. For example, some learners receive a blended class that provides online discussion boards and interactive online activities, and some learners receive a blended class that includes lectures that can be accessed virtually. Are there variations of blended learning that you would be interested in testing?
Similarly, are there variations of distance learning that you would be interested in testing? For example, variations in real-time versus self-paced learning?
Are there components of online or mobile tools—or ways they are used to supplement instruction—that you would be interested in testing?
What types of supports would be helpful to convince [PROVIDER NAME] to participate in an evaluation of distance or blended learning or of online or mobile tools? For example, funds to support a staff person collecting participation or assessment data or additional staff development. Are there financial or other incentives that would make participation more attractive?
Have any evaluations been conducted on your distance or blended learning programs? What about on the online or mobile tools that you use? Do you know of any evaluations on similar programs or tools conducted elsewhere?
INTERVIEWER NOTE: Skip to Section C.
SECTION B: FOCAL APPROACH QUESTIONS FOR POTENTIAL IMPLEMENTERS
B1. Screener
INTERVIEWER NOTE: Information available about the provider’s current offerings should be prefilled by the interviewer. Depending on how much information you have about the provider before the interview, you may be able to complete the screener through email while scheduling the interview.
This interview will focus on using technology in adult education and how this could be implemented by [PROVIDER NAME].
We are talking today about three different ways that technology might be used in adult education:
Distance education is defined as a method that uses technology to deliver instruction to learners who are separated from the instructor, and to support regular interaction between the learners and the instructor, either with the class going at a common pace or learners going at their own pace. Technology can include the Internet; broadcast, closed-circuit, and cable TV; wireless communication devices; videos; DVDs; and CD-ROMs.
Blended approaches include a combination of in-person time with a teacher in large or small groups and independent learning using technology, with both components required as part of the course.
There are also online or mobile tools that are accessed by learners to supplement or reinforce course instruction. .
Before we get started, I want to confirm that [PROVIDER NAME] is not using technology in adult education in any of these ways. Is that correct?
Are you planning on using technology in adult education in any of these ways in the next year or two?
Are you far enough along in the planning process for implementation that you could answer some questions in this interview about what using technology in adult education would look like, including topics like recruitment, enrollment capacity, and potential implementation challenges?
B2. Interview
Program description
The first set of questions focuses on programs currently offered by [PROVIDER NAME] and how distance instruction, blended learning, or supplemental tools might fit into them.
What types of adult education and literacy classes do you currently offer: ABE, ASE, and/or ELA?
Do you see a need at specific skill levels for [TECHNOLOGY]? If ABE or ASE, ABE levels 1–6? If ESL, ESL levels 1–6?
What type of technology would you be interested in implementing: distance, blended, and/or supplemental tools?
Why do these interest you?
Is there a specific model or off-the-shelf distance or blended curriculum you would be interested in using? Why?
Are there any online or mobile tools you would be interested in using? Why?
Do you have any specific plans to implement distance or blended learning?
What steps have you taken to lay the groundwork for implementing distance or blended learning?
Do you have sufficient resources to implement distance or blended learning?
What supports would be necessary for [PROVIDER NAME] to implement distance or blended learning?
When do you think you might be able to implement distance or blended learning?
How feasible is this? What might get in the way?
What challenges would you anticipate facing with implementing distance instruction? What about blended learning?
Probe: Recruiting learners? Retaining learners until completion? Hiring, training, and retaining staff who can implement the program? Staff availability for planning and class time? Having the right material resources, such as a space to offers classes, the software/hardware necessary? Issues with developing or maintaining necessary partnerships?
In how many locations do you provide adult education and literacy classes?
If more than one: Where are those located?
Do you offer the same classes at the same levels and with the same content at all locations? If not, how do they differ?
Do you partner with other organizations to provide adult education and literacy classes?
Can you describe the partnerships, including what they offer your program?
Recruitment and eligibility
Next, I’d like to understand how you recruit learners.
Can you walk me through how an individual would enroll, starting with how they learn about the adult education and literacy classes, contact [PROVIDER NAME], then the intake, screening, and enrollment procedures?
Probe: How are they recruited? For example, word of mouth (family, friends), partner referrals, local media (radio, newspapers), place(s) of worship, place of employment, American Job Centers, libraries.
Probe: Do you conduct any marketing or outreach about adult education and literacy classes?
Probe: How does the intake process work? Is it done electronically or on paper? Are there specific individuals who do intake? How many?
If you implemented a distance or blended learning program, would this intake process work in the same way for it?
Are potential learners counseled to enroll, or not to enroll, in specific types of adult education and literacy classes? How do counselors determine which learners to encourage to participate in specific classes?
If you implemented a distance or blended learning program, how would learners be counseled to enroll or not enroll in it?
Are there any eligibility requirements participants must meet? For example, a pre-test assessment score, certain education functioning levels, a degree of English proficiency.
If there is a pre-test: What is the pre-test assessment?
Is a cutoff score used?
If yes: Is the same cutoff used consistently for all learners?
If yes: How did you determine the cutoff score?
How many applicants do you receive, and how many do not meet the cutoff score?
Do all learners who enroll take this pre-test assessment before starting adult education and literacy classes?
Does the pre-test differ across types and levels of classes?
If there is not an eligibility requirement: Do learners who enroll take a pre-test assessment before starting the adult education and literacy classes?
If yes: What is the assessment?
If you implemented a distance or blended learning program, would you use the same eligibility requirements?
Are many learners that are interested in adult education and literacy classes turned away because they are not eligible? Which eligibility criteria do these learners typically fail to meet? About how many are turned away due to eligibility?
Is there a cost to participate in adult education and literacy classes?
If yes: What is the cost?
To what extent is cost a barrier to participation?
If you implemented a distance or blended learning program, would the costs to learners be the same as for other classes?
Enrollment capacity
These next questions ask about your current enrollment and the capacity for [PROVIDER NAME] to expand services.
How many learners currently participate in adult education and literacy classes?
Is this a typical number?
How does enrollment vary across type and levels of classes?
Do you expect it to change in the next few years? If so, how?
About how many learners do you think might be at the appropriate skill level and interested in participating in a distance or blended learning program?
Are you currently able to serve all learners who are interested in and meet the requirements for adult education and literacy classes?
If no:
Why not?
About what percentage of interested, eligible learners are turned away for these reasons?
Do you use a managed enrollment period or an open enrollment throughout the year?
If managed: What is the length of the typical period (for example, semester, every 6–8 weeks, other)?
If you implemented a distance or blended learning program, would you use the same type of enrollment period?
Data availability and assessment
Next we’d like to understand more generally how your organization collects and records data about learners.
What data do you collect on learners’ demographics and participation?
What type of data do you track about the classes or services an individual participates in? For example, the classes or services learners enroll in or the length of time enrolled?
Do you track attendance?
Do the data you track differ for learners you serve in different types of classes?
What kind of contact information and other identifying information do you collect?
How often is contact information missing? How often is the Social Security number missing?
Which assessment(s) do you use as a post-test for adult education learners?
Do you administer post-tests after a set amount of time or after a set number of instructional hours?
Does this differ across types and levels of classes?
For about what percent of learners do you have post-test data?
Do you use any assessments to measure progress during enrollment in adult education classes? Which assessments?
At what time points or after how many instructional hours do you administer these assessments?
Does this differ across types and levels of classes?
How do you obtain data on other learner outcomes? Probe: Employment measures, credential attainment.
Are any of these outcomes from an administrative data source like unemployment insurance wage records? For what percentage of your learners are you able to match to existing records?
Are the demographic, participation, assessment, and other outcome data maintained in a program management information system?
Do you have a dedicated data manger?
If no: Who manages your program data?
Feedback on random assignment and doing a study
One of the best ways to learn about the effectiveness of using technology in adult education is an experiment in which we would randomly choose (such as by a coin flip) which learners, classes (or even sites) would receive a distance or blended class, or access to specific online tool. Or, if you had more learners interested in and eligible for distance or blended classes (or access to tools) than spots available, we could conduct a lottery to determine which learners would receive these services and which would not.
We’d like to get your reactions to the idea of studying this approach through a study where some locations or some learners participate in distance or blended classes, or some have access to an online or mobile tool, and some don’t.
Suppose some learners were assigned by lottery to participate in distance or blended classes, and others would be assigned to a fully in-person class or assigned to a waitlist for future distance or blended classes. Do you have any concerns about this way of selecting who would get the services? How do you think your staff and partners would react? Do you think that students interested in distance learning but assigned to an in-person class would actually enroll and attend class, or do most distance students require flexibility and virtual access?
Suppose some learners were assigned by lottery to have access to specific online or mobile tools to supplement instruction, and others were not. Do you have any concerns about this way of selecting who would get the services? How do you think your staff and partners would react?
Suppose it was sites being randomly selected to offer distance or blended classes, or to offer access to online or mobile tools. Do you have any concerns about this way of selecting who would get the services? How do you think your staff and partners would react?
We understand that some states have requirements for services that must be offered that might restrict us from being able to assign learners through a lottery. Are you aware of any requirements or restrictions related to use of technology in adult education, or specifically distance or blended learning in adult education, in [STATE NAME] that might restrict our ability to conduct a study like this? Do you anticipate any such initiatives in the future?
One way to implement the study so that all learners receive the program in some form is to offer different versions of the approach being studied. For example, some learners receive a blended class that uses a specific commercial product as a virtual component, and some learners receive a blended class that includes instructor lectures that can be accessed virtually. Are there variations of blended learning that you would be interested in testing?
Similarly, are there variations of distance learning that you would be interested in testing?
Are there variations of online or mobile tools—or ways they are used to supplement instruction—that you would be interested in testing?
What types of supports would be helpful to convince [PROVIDER NAME] to participate in an evaluation of distance or blended learning? What about an evaluation of online or mobile tools? For example, funds to support a staff person collecting data or additional staff development. Are there financial or other incentives that would make participation more attractive?
INTERVIEWER NOTE: Continue to Section C.
SECTION C: NON-FOCAL APPROACH QUESTIONS FOR ALL RESPONDENTS
C. Interview non-focal approaches questions
Feasibility of implementing a new approach
We are looking at a number of other models or approaches that could be implemented and evaluated. (Interviewer: The table below will be sent in an email to respondents before the interview. Refer to the table and read through definitions of each approach not addressed in previous sections and ask questions a and b to determine current implementation and interest in future implementation.)
Is your organization currently implementing this approach?
If not, do you have plans to implement this approach, and have you taken steps to lay the groundwork for it that would allow you to launch it in the next year with the resources you currently have available? And would you be interested in participating in an impact evaluation of this approach?
Intervention |
Implementing? |
Planning? |
IET: Programs that include integrated curricula (adult education instruction concurrently and contextually with workforce preparation activities and workforce training) focused on a single set of learning goals, through either “co-teaching” or “co-planning and alternate teaching.” |
|
|
Bridge or on-ramp programs: Classes or programs to prepare learners to transition to college courses or occupational training by providing basic skill instruction along with occupational content, employment skills, and/or college success strategies. |
|
|
Career navigation supports: A range of supports to help learners progress on a career path. Supports include career exploration and planning assistance; assistance accessing financial supports for education; and support developing study and work skills. |
|
|
INTERVIEWER NOTE: Ask sections below for (a) all approaches currently implementing and (b) all approaches interested in and able to implement in next year. If there is not time to cover all, ask about the two approaches already implemented or close to implementation, and ask if we can follow up via email, if necessary, about others.
IET
If currently implementing IET:
Does your IET program(s) use an off-the-shelf curriculum, or is it developed by the instructors, or a mix of both? If off-the-shelf curriculum, which one?
What skill levels does the IET program target? Does it serve ABE, ASE, or ELA learners?
Do you use a co-teaching approach or a co-planning with alternate teaching approach?
Do you have more learners interested in IET than you are able to serve?
What challenges do you face in implementing IET?
Would you be interested in trying a [co-teaching/co-planning] approach instead? Why or why not?
Are there new curricula you would like to try or changes you would like to make to your IET programs?
If planning on implementing IET:
Is there a specific IET model or off-the-shelf curriculum you plan to use?
Do you plan to use a co-teaching approach or a co-planning with alternate teaching approach, or another approach?
What kind of resources would you need to support implementing this type of approach? Additional space? Additional staff? Training of staff? Other capacity issues? (Probe for resources related to co-teaching vs. co-planning models.)
What challenges do you anticipate with implementing IET? (Probe for challenges related to co-teaching vs. co-planning models.)
Do you anticipate that you would have more learners interested than you would be able to serve?
Bridge programs
If currently implementing bridge programs:
Does your bridge program(s) use a commercial, off-the-shelf curriculum, or is it developed by the instructors, or a mix of both? If off-the-shelf curriculum, which one?
What skill levels does the bridge program target? Does it serve ABE, ASE, or ELA learners? Is it focused on preparing learners for occupational training or postsecondary education, or both?
Do you have more learners interested in bridge programs than you are able to serve?
What challenges do you face in implementing bridge programs?
Are there new curricula you would like to try or changes you would like to make to your bridge programs?
If planning on implementing bridge programs:
Is there a specific bridge model or off-the-shelf curriculum you plan to use?
What kind of resources would you need to support implementing this type of approach? Additional space? Additional staff? Training of staff? Other capacity issues? (Probe for resources related to bridge for occupational training vs. bridge for postsecondary preparation.)
What challenges would you anticipate with implementing bridge programs? (Probe for challenges related to bridge for occupational training vs. bridge for postsecondary preparation.)
Do you anticipate that you would have more learners interested than you would be able to serve?
Career navigation supports with navigation
If currently implementing career navigation supports:
What types of career navigation supports are provided to adult learners? Do you provide career exploration and planning assistance? Do you assist learners in making connections to postsecondary systems (for example, connections to the admissions process)? Do you assist learners in making connections to workforce systems? Do you use any navigation or academic tracking software?
Does career navigation include other supports such as assistance accessing financial supports for education or support developing study and work skills?
Who delivers these career navigation supports?
Do you have more learners interested in supports than you are able to serve?
What challenges do you face in offering these supports?
Are there new approaches to delivering supports you would like to try or changes you would like to make to the supports?
If planning on implementing career navigation supports:
What types of career navigation supports, such as career exploration and planning assistance or connections to postsecondary and workforce systems, do you plan to offer? For example, will you provide career exploration and planning assistance? Assistance in making connections to postsecondary systems, or to workforce systems?
Are you planning to offer other types of support along with these career supports, such as assistance accessing financial supports for education or support developing study and work skills?
Do you have staff who could deliver these types of supports?
What kind of resources would you need to support implementing this type of approach? Additional space? Training of staff? Other capacity issues?
What challenges would you anticipate with offering career navigation supports?
Do you anticipate that you would have more learners interested than you would be able to serve?
Wrap-up
Those are all the questions I have for you. Thank you very much for your time and thoughtful answers. After I share this information with my colleagues, I might have one or two follow-up questions. In the meantime, if you have any follow-up comments or questions, please feel free to contact me at [INTERVIEWER PHONE/EMAIL].
Feasibility
Screener/Interview Protocol—Integrated
Education and
Training
Provider name |
|
Provider city and state |
|
Respondent name |
|
Respondent title |
|
Interviewer |
|
Date of interview |
|
Current or potential implementer of focal approach? |
|
Introduction
The U.S. Department of Education is conducting a National Assessment of Adult Education. This assessment is required by Congress and includes activities to understand and potentially expand evidence on the effectiveness of particular adult education program models, activities, and services. For shorthand, we will refer to these as “approaches” throughout our conversation. The Department contracted with Mathematica to explore the feasibility of conducting future large-scale effectiveness studies in adult education. We’re in the process of speaking to adult education providers that are implementing or might implement one or more of the approaches of interest, such as Integrated Education and Training (IET).
If current implementer: We identified [PROVIDER NAME] as implementing IET.
If potential implementer: We identified [PROVIDER NAME] as a potential site for implementing IET.
We are speaking to providers for three reasons. First, we aim to better understand how adult education approaches like IET [ARE/COULD BE] implemented, and to get a picture of how adult education providers recruit and enroll learners. Second, we want to understand what participant data providers collect. Last, our goal is to understand what would be needed to include a provider like you in a large scale national evaluation and how willing your organization might be to participate in an evaluation.
Information from these interviews will be used to inform the Department’s decision about which adult education program models, activities, or services are feasible to study.
Mathematica and its subcontractors, Manhattan Strategy Group and Social Policy Research, follow the confidentiality and data protection requirements of the Institute of Education Sciences at the Department of Education (Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for research purposes. Your participation in this interview is voluntary; if you do not feel comfortable answering any specific question, we can skip that question and move on to the next one. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific individual or organization. The study will not disclose the names of individual study participants, except as required by law. Do you agree to participate in this interview?
I would like to record the interview, just in case I need to check the accuracy of my notes. The recording will not be used for any other purpose or shared with anyone outside the study team. Would it be OK if I record this interview?
According
to the Paperwork Reduction Act of 1995, no persons are required to
respond to a collection of information unless such collection
displays a valid OMB control number. The valid OMB control number
for this information collection is xxxx-xxxx.
We estimate that it will take about 5 minutes to answer the
questions in the screener interview and 60 minutes to answer the
questions in the feasibility interview. If you have any comments
concerning the accuracy of the time estimates or suggestions for
improving this form, please write to: U.S. Department of Education,
Washington, DC 20202-4651. If you have comments or concerns
regarding the status of your individual submission of this form,
write directly to: Institute of Education Sciences, U.S. Department
of Education, 550 12th Street, SW, Washington, DC 20202.
INTERVIEWER NOTE: Skip to Section B for potential implementers.
SECTION A: FOCAL APPROACH QUESTIONS FOR CURRENT IMPLEMENTERS
A1. Screener
INTERVIEWER NOTE: Information available about the provider’s implementation of the approach should be prefilled by the interviewer. Depending on how much information you have about the provider before the interview, you may be able to complete the screener through email while scheduling the interview.
This interview will focus on IET and how it is implemented by [PROVIDER NAME].
IET includes three required integrated components with a single set of learning objectives: (1) adult education and literacy activities, (2) workforce preparation activities, and (3) workforce training. Before we get started, I want to confirm that [PROVIDER NAME] is implementing IET or something with a similar approach. Is that correct?
Is there another term you would use to refer to IET programs?
A2. Interview
Program description
This interview will focus on IET and how it is implemented by [PROVIDER NAME]. These first questions ask you to describe how [PROVIDER NAME] implements IET.
Does your IET program(s) use a commercial, off-the-shelf curriculum, or is it developed by the instructors, or a mix of both? If off-the-shelf curriculum, which one?
To what extent are the three required components—(1) adult education and literacy activities, (2) workforce preparation activities, and (3) workforce training—integrated in the curriculum with a single set of learning objectives?
Do the three components occur simultaneously?
Do the three components all use occupationally relevant instructional materials?
Is there a single set of learning objectives that identifies specific activities or content from each of the three components?
How does staffing facilitate integration? For example, do you use a co-teaching approach or an approach that alternates teaching at different times but instructors co-plan together?
If alternate teaching: Is there a reason why you use an alternate teaching approach rather than a co-teaching approach?
If co-teaching: Is there a reason why you use a co-teaching approach rather than an alternate teaching approach?
Do instructors have access to any professional development to facilitate integration of curriculum? Who offers this professional development?
Does [PROVIDER NAME] offer more than one IET program?
If multiple IET programs are offered, probe on the following questions for each one:
What type of workforce preparation skills are taught as part of IET?
Probes:
Basic academic skills
Critical thinking
Digital literacy
Self-management skills
Employability skills related to using resources and information, working with others, understanding systems, and obtaining skills necessary to successfully transition to and complete postsecondary education, training, and employment
Other employability skills to increase preparation for workforce
In what occupational areas do you offer IET?
What workforce training activities are provided as part of IET? How are activities provided? (For example, classroom instruction, hands-on training, practicum, on-the-job-training)
Does the training provide learners the opportunity to earn a recognized postsecondary credential? (For example, these include industry-recognized certificate or certification, license recognized by state or federal government, an associate or baccalaureate degree)
What skill levels does the IET program target? If ABE or ASE, ABE levels 1–6? If ESL, ESL levels 1–6?
What are other characteristics of learners in IET programs—for example, levels of education, employment status, and experience?
Is there a range of [skill levels, levels of education, employment status, and experience] across participants? Can you describe the range?
Do participants in IET programs differ from the participants in your traditional adult education classes? If so, how?
After participants finish the IET program, what do you expect them to transition to? For example, another adult education class that [PROVIDER NAME] offers; a high school equivalency preparation course; developmental education courses at a college; college credit courses; a job; an apprenticeship; something else? (Probe for all the apply.)
Does [PROVIDER NAME] offer sequences of IET classes?
How many weeks or months do IET classes typically run?
Do you offer IET throughout the year or at specific times during the year?
In how many locations do you provide IET?
If more than one: Where are those located?
Do you also offer traditional adult education at the same locations? What types of classes?
Do you partner with other organizations to provide IET?
Can you describe the partnerships, including what they offer your program?
How long has [PROVIDER NAME] been offering IET?
Does [PROVIDER NAME] have plans to expand or change your IET offerings? In what ways? When?
Are there other providers or states that you know of that implement IET with a similar staffing or instructional model?
What challenges do you/have you faced in implementing IET?
Probe: Recruiting learners? Retaining learners until completion? Hiring, training, and retaining staff who can implement the program? Staff availability for planning and class time? Having the right material resources, such as a space to offer classes, the software/hardware necessary? Issues with developing or maintaining necessary partnerships?
If using off-the-shelf model: To what extent are you satisfied with the [name of model] you are using?
What works well?
What doesn’t work as well?
Is there anything you would like to change about it?
Recruitment and eligibility
Next, I’d like to understand how you recruit learners in IET.
Can you walk me through how an individual would enroll, starting with how they learn about the IET program, contact [PROVIDER NAME], then the intake, screening, and enrollment procedures?
Probe: How are they recruited? For example, word of mouth (family, friends), partner referrals, local media (radio, newspapers), place(s) of worship, place of employment, American Job Centers, libraries.
Probe: Do you conduct any marketing or outreach about IET?
Probe: How does the intake process work? Is it done electronically or on paper? Are there specific individuals who do intake? How many?
Are there any eligibility requirements participants must meet? For example, a pre-test assessment score, certain education functioning levels, a degree of English proficiency.
If there is a pre-test: What is the pre-test assessment?
Is a cutoff score used?
If yes: Is the same cutoff used consistently for all learners?
If yes: How did you determine the cutoff score?
How many applicants do you receive, and how many do not meet the cutoff score?
Do all learners who enroll take this pre-test assessment before starting IET?
How does it differ from the assessments administered to learners in other adult education and literacy classes?
If there is not an eligibility requirement: Do learners who enroll take a pre-test assessment before starting IET classes?
If yes: What is the assessment?
How does it differ from the assessments administered to learners in other adult education and literacy classes?
Are many learners that are interested in IET turned away because they are not eligible? Which eligibility criteria do these learners typically fail to meet? About how many are turned away due to eligibility?
Are potential learners counseled to enroll, or not to enroll, in IET versus traditional adult education? How do counselors determine which learners to encourage to participate in IET?
Is there a cost to participate in IET?
If yes: What is the cost?
To what extent is cost a barrier to participation?
Enrollment capacity
These next questions ask about your current enrollment and the capacity for [PROVIDER NAME] to expand services.
If multiple variations of IET are offered, probe on the following questions for each one.
How many learners currently participate in IET?
Is this a typical number?
Do you expect it to change in the next few years? If so, how?
Does IET have a managed enrollment period or open enrollment throughout the year?
If managed: What is the length of the typical period (for example, semester, every 6–8 weeks, other)?
Are you currently able to serve all learners who are interested in and meet the requirements for IET?
If no:
Why not?
About what percentage of interested, eligible learners are turned away for these reasons?
Do you have a waitlist? How does the waitlist work? About how long do most learners stay on the waitlist before getting a spot?
Do you refer those who don’t get a spot to another type of class or service you offer or to another provider in the area? To what extent do learners enroll in these alternatives?
What is the maximum number of learners you could enroll in IET at a given time? Does this vary at different times in the year? If so, how does it vary?
What factors limit the number of learners you could enroll?
Probe: Staff, space, or something else?
Data availability and assessment
Next we’d like to understand more generally how your organization collects and records data about learners.
What data do you collect on IET learners’ demographics and participation?
What type of data do you track about the classes or services an individual participates in? For example, the classes or services learners enroll in or the length of time enrolled?
Do you track attendance?
Do the data you track differ for learners you serve in traditional, non-IET classes?
What kind of contact information and other identifying information do you collect?
How often is contact information missing? How often is the Social Security number missing?
Which assessment(s) do you use as a post-test for IET learners?
Do you administer post-tests after a set amount of time or after a set number of instructional hours?
How do you determine instructional hours completed for learners who access content asynchronously?
Does this differ from the assessments administered to learners in other adult education classes?
For about what percent of learners do you have post-test data?
Do you use any assessments to measure progress during enrollment in IET classes? Which assessments?
At what time points or after how many instructional hours do you administer these assessments?
How do you determine instructional hours completed for learners who access content asynchronously?
Does this differ from the assessments administered to learners in other adult education classes?
How do you obtain data on other learner outcomes? Probe: Employment measures, credential attainment.
Are any of these outcomes from an administrative data source like unemployment insurance wage records? For what percentage of your learners are you able to match to existing records?
Are the demographic, participation, assessment, and other outcome data maintained in a program management information system?
Do you have a dedicated data manger?
If no: Who manages your program data?
Feedback on random assignment and doing a study
One of the best ways to learn about the effectiveness of IET is an experiment in which we would randomly choose (such as by a coin flip) which learners, classes (or even sites) would receive IET, or a specific enhancement or variation of IET, such as alternate teaching or co-teaching. Or, if you had more learners interested in and eligible for IET than spots available, we could conduct a lottery to determine which learners would receive these services and which would not.
We’d like to get your reactions to the idea of studying this approach through a study where some locations or some learners participate in IET.
Suppose some learners or sites were assigned by lottery to receive IET and others would be assigned to a different version or another type of class. Do you have any concerns about this way of selecting who would get the services? How do you think your staff and partners would react?
To what extent do learners and instructors from different classes interact, potentially sharing information that they learned or taught in a different class?
We understand that some states have requirements for services that must be offered that might restrict us from being able to assign learners through a lottery. Are you aware of any requirements or restrictions related to IET in [STATE NAME] that might restrict our ability to conduct a study like this? Do you anticipate any such initiatives in the future?
One way to implement the study so that all learners receive the program in some form is to offer different versions of the approach being studied. For example, some learners receive IET through co-teaching while others receive it through alternate teaching. Are there variations of IET that you would be interested in testing?
What types of supports would be helpful to convince [PROVIDER NAME] to participate in an evaluation like this of IET? For example, funds to support a staff person collecting data or additional staff development. Are there financial or other incentives that would make participation more attractive?
Have any evaluations been conducted on your IET program?
INTERVIEWER NOTE: Skip to Section C.
SECTION B: FOCAL APPROACH QUESTIONS FOR POTENTIAL IMPLEMENTERS
B1. Screener
INTERVIEWER NOTE: Information available about the provider’s current offerings should be prefilled by the interviewer. Depending on how much information you have about the provider before the interview, you may be able to complete the screener through email while scheduling the interview.
This interview will focus on IET and how it could be implemented by [PROVIDER NAME].
IET includes three required integrated components with a single set of learning objectives: (1) adult education and literacy activities, (2) workforce preparation activities, and (3) workforce training. It is also required to connect to a career pathway.
Before we get started, I want to confirm that [PROVIDER NAME] does not currently offer IET programs. Is that correct?
Are you planning on implementing IET in the next year or two?
Are you far enough along in the planning process for implementation that you could answer some questions in this interview about what implementation of IET would look like, including topics like recruitment, enrollment capacity, and potential implementation challenges?
B2. Interview
Program description
The first set of questions focuses on programs currently offered by [PROVIDER NAME] and how IET might fit into them.
What types of adult education and literacy classes do you currently offer: ABE, ASE, and/or ELA?
We’re interested in whether you offer adult education and literacy alongside any occupational training or topics, but without integration as there would be in IET.
Do you offer adult education and literacy classes concurrently with instruction with an occupational skills training partner, but without a shared curriculum?
How about adult education and literacy classes within the context of occupational topics?
Do you see a demand at specific skill levels for an IET program? If ABE or ASE, ABE levels 1–6? If ESL, ESL levels 1–6?
What occupational area(s) do you plan to target with your IET program?
Is there a specific IET model or off-the-shelf curriculum you would be interested in using? Why?
IET can be implemented in different ways, sometimes with co-teaching in the same classroom and sometimes with alternate teaching and co-planning time for instructors. Which approach are you planning to implement? Why?
Do you have any specific plans to implement an IET program?
What steps have you taken to lay the groundwork for implementing IET?
Do you have sufficient resources to implement IET?
What supports would be necessary for [PROVIDER NAME] to implement IET?
When do you think you might be able to implement IET?
How feasible is this? What might get in the way?
What challenges would you anticipate facing with implementing an IET program?
Probe: Recruiting learners? Retaining learners until completion? Hiring, training, and retaining staff who can implement the program? Staff availability for planning and class time? Having the right material resources, such as a space to offer classes, the software/hardware necessary? Issues with developing or maintaining necessary partnerships?
How would the challenges be different if you were implementing IET as a co-teaching model versus an alternate teaching model with co-planning time for instructors?
Do you anticipate any challenges with integrating the three required components—(1) adult education and literacy activities, (2) workforce preparation activities, and (3) workforce training—into the curriculum with a single set of learning objectives?
Do you anticipate particular challenges in offering and integrating the workforce preparation activities? What about the workforce training activities?
In how many locations do you provide adult education and literacy classes?
If more than one: Where are those located?
Do you offer the same classes at the same levels and with the same content at all locations? If not, how do they differ?
Do you partner with other organizations to provide adult education and literacy classes?
Can you describe the partnerships, including what they offer your program?
Recruitment and eligibility
Next, I’d like to understand how you recruit learners.
Can you walk me through how an individual would enroll, starting with how they learn about the adult education and literacy classes, contact [PROVIDER NAME], then the intake, screening, and enrollment procedures?
Probe: How are they recruited? For example, word of mouth (family, friends), partner referrals, local media (radio, newspapers), place(s) of worship, place of employment, American Job Centers, libraries.
Probe: Do you conduct any marketing or outreach about adult education and literacy classes?
Probe: How does the intake process work? Is it done electronically or on paper? Are there specific individuals who do intake? How many?
If you implemented an IET program, would this intake process work in the same way for it?
Are potential learners counseled to enroll, or not to enroll, in specific types of adult education and literacy classes? How do counselors determine which learners to encourage to participate in specific classes?
If you implemented an IET program, how would learners be counseled to enroll or not enroll in it?
Are there any eligibility requirements participants must meet? For example, a pre-test assessment score, certain education functioning levels, a degree of English proficiency.
If there is a pre-test: What is the pre-test assessment?
Is a cutoff score used?
If yes: Is the same cutoff used consistently for all learners?
If yes: How did you determine the cutoff score?
How many applicants do you receive, and how many do not meet the cutoff score?
Do all learners who enroll take this pre-test assessment before starting adult education and literacy classes?
Does the pre-test differ across types and levels of classes?
If there is not an eligibility requirement: Do learners who enroll take a pre-test assessment before starting adult education and literacy classes?
If yes: What is the assessment? If you implemented an IET program, would you use the same eligibility requirements?
Are many learners that are interested in adult education and literacy classes turned away because they are not eligible? Which eligibility criteria do these learners typically fail to meet? About how many are turned away due to eligibility?
Is there a cost to participate in adult education and literacy classes?
If yes: What is the cost?
To what extent is cost a barrier to participation?
If you implemented an IET program, would the costs to learners be the same as for other classes?
Enrollment capacity
These next questions ask about your current enrollment and the capacity for [PROVIDER NAME] to expand services.
How many learners currently participate in adult education and literacy classes?
Is this a typical number?
How does enrollment vary across type and levels of classes?
Do you expect it to change in the next few years? If so, how?
About how many learners do you think might be at the appropriate skill level and interested in participating in an IET program?
Are you currently able to serve all learners who are interested in and meet the requirements for adult education and literacy classes?
If no:
Why not?
About what percentage of interested, eligible learners are turned away for these reasons?
Do you use a managed enrollment period or an open enrollment throughout the year?
If managed: What is the length of the typical period (for example, semester, every 6–8 weeks, other)?
If you implemented an IET program, would you use the same type of enrollment period?
Data availability and assessment
Next we’d like to understand more generally how your organization collects and records data about learners.
What data do you collect on learners’ demographics and participation?
What type of data do you track about the classes or services an individual participates in? For example, the classes or services learners enroll in or the length of time enrolled?
Do you track attendance?
Do the data you track differ for learners you serve in different types of classes?
What kind of contact information and other identifying information do you collect?
How often is contact information missing? How often is the Social Security number missing?
Which assessment(s) do you use as a post-test for adult education learners?
Do you administer post-tests after a set amount of time or after a set number of instructional hours do you administer a post-test?
Does this differ across types and levels of classes?
For about what percent of learners do you have post-test data?
Do you use any assessments to measure progress during enrollment in adult education classes? Which assessments?
At what time points or after how many instructional hours do you administer these assessments?
Does this differ across types and levels of classes?
How do you obtain data on other learner outcomes? Probe: Employment measures, credential attainment.
Are any of these outcomes from an administrative data source like unemployment insurance wage records? For what percentage of your learners are you able to match to existing records?
Are the demographic, participation, assessment, and other outcome data maintained in a program management information system?
Do you have a dedicated data manger?
If no: Who manages your program data?
Feedback on random assignment and doing a study
One of the best ways to learn about the effectiveness of IET is an experiment in which we would randomly choose (such as by a coin flip) which learners, classes (or even sites) would receive a specific enhancement or variation of IET, such as alternate teaching or co-teaching. Or, if you had more learners interested in and eligible for IET than spots available, we could conduct a lottery to determine which learners would receive these services and which would not.
We’d like to get your reactions to the idea of studying this approach through a study where some locations or some learners participate in IET.
Suppose some learners or sites were assigned by lottery to receive IET and others would be assigned to a different version or another type of class. Do you have any concerns about this way of selecting who would get the services? How do you think your staff and partners would react?
To what extent do learners and instructors from different classes interact, potentially sharing information that they learned or taught in a different class?
We understand that some states have requirements for services that must be offered that might restrict us from being able to assign learners through a lottery. Are you aware of any requirements or restrictions related to IET in [STATE NAME] that might restrict our ability to conduct a study like this? Do you anticipate any such initiatives in the future?
One way to implement the study so that all learners receive the program in some form is to offer different versions of the approach being studied. For example, some learners receive IET through co-teaching while others receive it through alternate teaching. Are there variations of IET that you would be interested in testing?
What types of supports would be helpful to convince [PROVIDER NAME] to participate in an evaluation like this of IET? For example, funds to support a staff person collecting data or additional staff development. Are there financial or other incentives that would make participation more attractive?
INTERVIEWER NOTE: Continue to Section C.
SECTION C: NON-FOCAL APPROACH QUESTIONS FOR ALL RESPONDENTS
C. Interview non-focal approaches questions
Feasibility of implementing a new approach
We are looking at a number of other models or approaches that could be implemented and evaluated. (Interviewer: The table below will be sent in an email to respondents before the interview; refer to the table and read through definitions of each approach not addressed in previous sections, and ask questions a and b to determine current implementation and interest in future implementation)
Is your organization currently implementing this approach?
If not, do you have plans to implement this approach, and have you taken steps to lay the groundwork for it that would allow you to launch it in the next year with the resources you currently have available? And would you be interested in participating in an impact evaluation of this approach?
Intervention |
Implementing? |
Planning? |
Distance learning: An approach that delivers all instruction in an entirely virtual format. The technology used can include the Internet, broadcast, closed circuit, cable, wireless communication devices, videos, DVDs, or CD-ROMs. |
|
|
Blended learning: An approach that delivers instruction through a combination of in-person and virtual instruction, with both components required as part of the course. |
|
|
Mobile or online learning tools to supplement instruction: Resources that learners can independently access through the internet on a computer or mobile device and that reinforce classroom instruction |
|
|
Bridge or on-ramp programs: Classes or programs to prepare learners to transition to college courses or occupational training by providing basic skill instruction along with occupational content, employment skills, and/or college success strategies. |
|
|
Career navigation supports: A range of supports to help learners progress on a career path. Supports include career exploration and planning assistance; assistance accessing financial supports for education; and support developing study and work skills. |
|
|
INTERVIEWER NOTE: Ask sections below for (a) all approaches currently implementing and (b) all approaches interested in and able to implement in next year. If there is not time to cover all, ask about the two approaches already implemented or close to implementation, and ask if we can follow up via email if necessary about others.
Distance or blended learning
If currently implementing distance or blended learning:
Do you use [APPROACH] with ABE, ASE, or ELA CLASSES?
Do you use an off-the-shelf curriculum for the distance/blended classes? Are there any commercial software products you use in these classes?
If currently implementing blended: What components of the blended classes are delivered virtually and what components are delivered in person?
Do you have more learners interested in distance or blended learning than you are able to serve? (Probe for distance and blended separately, if implementing both.)
What challenges do you face in implementing distance or blended learning? (Probe for distance and blended separately, if implementing both.)
Are there new curricula, software products, or approaches you would like to try or changes you would like to make to your distance or blended classes?
If planning on implementing distance or blended learning:
Is there an off-the-shelf distance or blended curriculum you plan to use? A commercial software product you would be interested in using in these classes?
What kind of resources would you need to support implementing this type of approach? Additional space or materials? Additional resources to purchase curricula or software? Additional staff? Training of staff? Other capacity issues?
What challenges would you anticipate with implementing distance or blended learning? (Probe for distance and blended separately, if planning on both.)
Do you anticipate that you would have more learners interested than you would be able to serve?
Supplemental online or mobile tools
If currently implementing supplemental online or mobile tools:
What types of online or mobile tools are used by instructors to supplement material taught in adult education classes? Do instructors connect the material covered in the online or mobile tools to class instruction? Do instructors monitor learners’ use of these tools?
Do you use [APPROACH] with ABE, ASE, or ELA CLASSES?
What challenges do you face in using supplemental online or mobile tools?
Are there new tools you would like to try or changes you would like to make to how tools are used to supplement classes?
If planning on implementing supplemental online or mobile tools:
Is there a specific online or mobile tool you are planning to use to supplement adult education classes?
What kind of resources would you need to support implementing this type of approach? Training of staff? Other capacity issues?
What challenges would you anticipate with using these tools?
Bridge programs
If currently implementing bridge programs:
Does your bridge program(s) use a commercial, off-the-shelf curriculum, or is it developed by the instructors, or a mix of both? If off-the-shelf curriculum, which one?
What skill levels does the bridge program target? Does it serve ABE, ASE, or ELA learners? Is it focused on preparing learners for occupational training or postsecondary education, or both?
Do you have more learners interested in bridge programs than you are able to serve?
What challenges do you face in implementing bridge programs?
Are there new curricula you would like to try or changes you would like to make to your bridge programs?
If planning on implementing bridge programs:
Is there a specific bridge model or off-the-shelf curriculum you plan to use?
What kind of resources would you need to support implementing this type of approach? Additional space? Additional staff? Training of staff? Other capacity issues? (Probe for resources related to bridge for occupational training vs. bridge for postsecondary preparation.)
What challenges would you anticipate with implementing bridge programs? (Probe for challenges related to bridge for occupational training vs. bridge for postsecondary preparation.)
Do you anticipate that you would have more learners interested than you would be able to serve?
Career navigation supports
If currently implementing career navigation supports:
What types of career navigation supports are provided to adult learners? Do you provide career exploration and planning assistance? Do you assist learners in making connections to postsecondary systems (for example, connections to the admissions process)? Do you assist learners in making connections to workforce systems? Do you use any navigation or academic tracking software?
Does career navigation include other supports such as assistance accessing financial supports for education or support developing study and work skills?
Who delivers these career navigation supports?
Do you have more learners interested in supports than you are able to serve?
What challenges do you face in offering these supports?
Are there new approaches to delivering supports you would like to try or changes you would like to make to the supports?
If planning on implementing career navigation supports:
What types of career navigation supports do you plan to offer? For example, will you provide career exploration and planning assistance? Assistance in making connections to postsecondary systems, or to workforce systems?
Are you planning to offer other types of support along with these career supports, such as assistance accessing financial supports for education or support developing study and work skills?
Do you have staff who could deliver these types of supports?
What kind of resources would you need to support implementing this type of approach? Additional space? Training of staff? Other capacity issues?
What challenges would you anticipate with offering career navigation supports?
Do you anticipate that you would have more learners interested than you would be able to serve?
Wrap-up
Those are all the questions I have for you. Thank you very much for your time and thoughtful answers. After I share this information with my colleagues, I might have one or two follow-up questions. In the meantime, if you have any follow-up comments or questions, please feel free to contact me at [INTERVIEWER PHONE/EMAIL].
Provider name |
|
Provider city and state |
|
Respondent name |
|
Respondent title |
|
Interviewer |
|
Feasibility Screener/Interview Protocol—Career Navigation Supports
Provider name |
|
Provider city and state |
|
Respondent name |
|
Respondent title |
|
Interviewer |
|
Date of interview |
|
Current or potential implementer of focal approach? |
|
Introduction
The U.S. Department of Education is conducting a National Assessment of Adult Education. This assessment is required by Congress and includes activities to understand and potentially expand evidence on the effectiveness of particular adult education program models, activities, and services. For shorthand, we will refer to these as “approaches” throughout our conversation. The Department contracted with Mathematica to explore the feasibility of conducting future large-scale effectiveness studies in adult education. We’re in the process of speaking to adult education providers that are implementing or might implement one or more of the approaches of interest, including individualized and targeted career navigation supports.
If current implementer: We identified [PROVIDER NAME] as implementing [FILL WITH TYPE OF SUPPORT].
If potential implementer: We identified [PROVIDER NAME] as a potential site for implementing [FILL WITH TYPE OF SUPPORT].
We are speaking to providers for three reasons. First, we aim to better understand how adult education approaches like career navigation supports [ARE/COULD BE] implemented, and to get a picture of how adult education providers recruit and enroll learners. Second, we want to understand what participant data providers collect. Last, our goal is to understand what would be needed to include a provider like you in a large scale national evaluation and how willing your organization might be to participate in an evaluation.
Information from these interviews will be used to inform the Department’s decision about which adult education program models, activities, or services are feasible to study.
Mathematica and its subcontractors, Manhattan Strategy Group and Social Policy Research, follow the confidentiality and data protection requirements of the Institute of Education Sciences at the Department of Education (Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for research purposes. Your participation in this interview is voluntary; if you do not feel comfortable answering any specific question, we can skip that question and move on to the next. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific individual or organization. The study will not disclose the names of individual study participants, except as required by law. Do you agree to participate in this interview?
I would like to record the interview, just in case I need to check the accuracy of my notes. The recording will not be used for any other purpose or shared with anyone outside the study team. Would it be OK if I record this interview?
According
to the Paperwork Reduction Act of 1995, no persons are required to
respond to a collection of information unless such collection
displays a valid OMB control number. The valid OMB control number
for this information collection is xxxx-xxxx.
We estimate that it will take about 5 minutes to answer the
questions in the screener interview and 60 minutes to answer the
questions in the feasibility interview. If you have any comments
concerning the accuracy of the time estimates or suggestions for
improving this form, please write to: U.S. Department of Education,
Washington, DC 20202-4651. If you have comments or concerns
regarding the status of your individual submission of this form,
write directly to: Institute of Education Sciences, U.S. Department
of Education, 550 12th Street, SW, Washington, DC 20202.
INTERVIEWER NOTE: Skip to Section B for potential implementers.
SECTION A: FOCAL APPROACH QUESTIONS FOR CURRENT IMPLEMENTERS
A1. Screener
INTERVIEWER NOTE: Information available about the provider’s implementation of the approach should be prefilled by the interviewer. Depending on how much information you have about the provider before the interview, you may be able to complete the screener through email while scheduling the interview.
This interview will focus on career navigation supports and how they are implemented by [PROVIDER NAME].
By career navigation supports, I mean a range of supports to help learners progress on a career path. Supports include career exploration and planning assistance, connecting learners to postsecondary institutions and workforce training organizations, assistance accessing financial supports for education, and support developing study and work skills. These services are sometimes offered by or coordinated through a navigator or a similar role. Navigators can also be referred to as college or career navigators, coaches, or transition coordinators.
Before we get started, I want to confirm that [PROVIDER NAME] provides supports such as these for adult learners.
Does [PROVIDER NAME] offer career navigation supports? Are there any differences that we should know about between the definition I just provided and how [PROVIDER NAME] implements career navigation supports?
Probe: Is there a term you would use to refer to the learner supports [PROVIDER NAME] provides or the role of the counselor or navigator who provides these supports?
A2. Interview
Program description
These first questions ask you to describe how [PROVIDER NAME] implements career navigation supports for adult learners.
What career navigation supports does [PROVIDER NAME] provide for adult learners beyond what an instructor might provide during class time?
Probe on each if not mentioned:
Career transition support, such as career planning, exploration of career options, interview preparation, connections to workforce training organizations, or other connections that would help learners progress through a career pathway, including connections to postsecondary institutions (such as connecting student to college admission processes or support services offered on campus)
Academic support, such as support with developing study and work skills
Financial support, such as assistance accessing financial support for education
Who provides career navigation supports? What is his or her title?
Does this person only provide career navigation supports or also other types of learner support?
Do you partner with other organizations to provide career navigation supports?
Can you describe the partnerships, including what they offer your program?
On average, how many learners does the [TITLE] support at a given time?
What kind of professional background does [TITLE] have?
Did he or she receive any training related to his or her role?
If yes, probe: What kind of training? How much? Where?
What does a typical career navigation support session look like?
Who typically initiates a session?
Do the learner and [TITLE] meet in person, virtually, or both?
How often do the sessions occur? How long do they last? Are there any requirements for either of these?
What topics might be covered?
Does [TITLE] use any tools to track learner progress and if the learners are meeting certain benchmarks?
Does [TITLE] use any tools to supplement or complement in-person support, such as online advising or navigator software?
How are career navigation supports funded?
To what extent are career navigation support services optional versus mandatory for learners? If optional, probe: What proportion of learners participate in or receive these supports?
Are learners required to participate in any nonacademic support activities, such as courses related to learner success or goal-planning meetings (more intrusive advising models)?
Do instructors, advisors, or the [TITLE] typically reach out to them, or do learners reach out when they need support?
How do you know if a learner needs help?
Do you offer career navigation support services to all adult education learners, or do you target specific learners—for example, based on specific classes, skill levels, or occupational areas?
Are there any requirements participants must meet to receive services? For example, certain education functioning levels or a degree of English proficiency. If ABE or ASE, ABE levels 1–6? If ESL, ESL levels 1–6?
Do learners have to be enrolled in a certain classes, programs, or services to receive career navigation supports?
If no skill level requirements: What skill levels do career navigation support services target? If ABE or ASE, ABE levels 1–6? If ESL, ESL levels 1–6?
How long have you been offering career navigation support services?
Do you plan to expand your career navigation support offerings?
In how many locations do you provide career navigation support?
If more than one: Where are those located?
What challenges do you/have you faced in implementing career navigation support?
Probe: Engaging learners? Hiring, training, and retaining staff who can provide the learner supports? Having the right material resources, such as a space to offers classes, the software/hardware necessary? Issues with developing or maintaining necessary partnerships?
INTERVIEWER: Ask the following questions only if respondent identifies a single role providing multiple services. If there are multiple roles that provide different combinations of services, ask for each one.
You mentioned that the same person provides [SERVICE TYPES]. Can you tell me more about how [TITLE] coordinates services across different types of supports?
What challenges do they face in integrating the different types of services?
Can you tell me how this role came to be?
Was the role intentionally developed this way or did it occur more naturally?
Receipt of career navigation supports
Next I’d like to understand how learners receive career navigation supports.
Can you walk me through how you would determine if a new or currently enrolled learner is eligible to receive career navigation supports?
How do they learn about career navigation support services? For example, their adult education instructor, word of mouth from other learners, marketing at the program site, or some kind of external marketing outside of the program site.
Are there specific steps they need to take to sign up to receive support services?
Probe: How does the support service intake process work? Is it done electronically or on paper? Are there specific individuals who do intake? How many?
Probe: Is there an assessment used to determine eligibility for support services before receiving services (that is, similar to a pre-test for classes)?
If yes: If so, is a cutoff score used?
If yes: If so, is the same cutoff used consistently for all learners?
If yes: How did you determine the cutoff score?
How many applicants do you receive, and how many do not meet the cutoff score?
Are there many learners interested in career navigation support services who are turned away because they are not eligible?
Program enrollment
These next questions ask about your current number of learners served and the capacity for career navigation supports to expand services.
How many learners currently receive career navigation support services?
Is this a typical number?
Do you expect it to change in the next few years? If so, how?
Are you currently able to serve all learners who are interested in and meet the requirements for career navigation support services?
If no: Why not?
If no: About what percentage of eligible learners are turned away?
If no: Do you refer those who don’t get a spot to another type of service you offer or to another provider in the area? To what extent do learners enroll in these alternatives?
What is the maximum number of learners you could provide career navigation support services at a given time? Does this vary at different times in the year? If so, how does it vary?
What factors limit the number of learners you could provide services for?
Probe: Staff, space, or something else?
Data availability and assessment
Next we’d like to understand more generally how your organization collects and records data about learners.
What data do you collect on learners’ demographics and participation?
What type of data do you track about the classes or services an individual participates in? For example, the classes or services learners enroll in or the length of time enrolled?
How often do you collect participation data?
Does this differ for those who receive career navigation support services?
Do you collect demographic, participation, and assessment data on all learners or just those participating in certain components?
What kind of contact information and other identifying information do you collect?
How often is contact information missing? How often is the Social Security number missing?
How do you track services received by a learner and their attendance? Are these data maintained in a program management information system?
Which assessment(s) do you use as a post-test for adult education learners?
Do you administer post-tests after a set amount of time or after a set number of instructional hours?
Does this differ across types and levels of classes?
Do you conduct any assessments after receiving support services? If so, what assessments?
For about what percent of learners do you have post-test data?
Do you use any assessments to measure progress during enrollment in adult education classes? Which assessments?
At what time points or after how many instructional hours do you administer these assessments?
Does this differ across types and levels of classes?
Do you conduct any assessments while receiving support services? If so, what assessments?
How do you obtain data on other learner outcomes? Probe on employment measures, and credential attainment?
Which are from an administrative data source like unemployment insurance wage records? For what percentage of your learners are you able to match to existing records?
Are the demographic, participation, assessment, and other outcome data maintained in a program management information system?
Do you have a dedicated data manger?
If no, probe: Who manages your program data?
Feedback on random assignment and doing a study
One of the best ways to learn about the effectiveness of career navigation support services is an experiment in which we would randomly choose (such as by a coin flip) which learners, classes (or even sites) would receive career navigation support services, or a specific component of career navigation support such as support connecting with postsecondary institutions or processes. Or, if you had more learners interested in and eligible for career navigation support services than spots available, we could conduct a lottery to determine which learners would receive these services and which would not.
We’d like to get your reactions to the idea of studying this approach through a study where some locations or some learners receive career navigation support services and some don’t.
Suppose some learners or sites were assigned by lottery to receive career navigation support services and others would be assigned to a different version, another type of service, or no services. Do you have any concerns about this way of selecting who would get the services? How do you think your staff and partners would react?
We understand that some states have requirements for services that must be offered that might restrict us from being able to assign learners through a lottery. Are you aware of any requirements or restrictions related to career navigation support in [STATE NAME] that might restrict our ability to conduct a study like this? Do you anticipate any such initiatives in the future?
One way to implement the study so that all learners receive the support services in some form is to offer different versions of the services being studied. For example, navigators could receive different types of training, or some learners could receive specific types of career preparation supports while others receive more academic supports. Are there variations like these that you would be interested in testing?
What types of supports would be helpful to convince [PROVIDER NAME] to participate in an evaluation like this of career navigation support or similar models or approaches? For example, funds to support a staff person collecting data or additional staff development. Are there financial or other incentives that would make participation more attractive?
INTERVIEWER NOTE: Skip to Section C.
SECTION B: FOCAL APPROACH QUESTIONS FOR POTENTIAL IMPLEMENTERS
B1. Screener
INTERVIEWER NOTE: Information available about the provider’s current offerings should be prefilled by the interviewer. Depending on how much information you have about the provider before the interview, you may be able to complete the screener through email while scheduling the interview.
This interview will focus on career navigation supports and how it could be implemented by [PROVIDER NAME].
By career navigation supports, I mean a range of supports to help learners progress on a career path. Supports include career exploration and planning assistance, connections to postsecondary institutions and workforce training organizations, assistance accessing financial supports for education, and support developing study and work skills. These services are sometimes offered by or coordinated through a navigator or a similar role. Navigators can also be referred to as college or career navigators, coaches, or transition coordinators.
Before we get started, I want to confirm that [PROVIDER NAME] does not currently offer these types of supports offered by a navigator or similar role, aimed to help learners persist in and complete career pathways programs. Is that correct?
Are you planning on implementing career navigation supports in the next year or two?
Are you far enough along in the planning process for implementation that you could answer some questions in this interview about what implementation of career navigation supports would look like, including topics like recruitment, enrollment capacity, and potential implementation challenges?
B2. Interview
Program description
The first set of questions focuses on programs currently offered by [PROVIDER NAME] and how career navigation supports might fit into them.
What types of adult education and literacy classes do you currently offer: ABE, ASE, and/or ELA?
Does [PROVIDER NAME] currently provide any counseling or guidance for adult learners beyond what an instructor might provide during class time?
If yes:
What types of counseling or guidance?
How is this different from the learner supports I described earlier aimed at helping students persist in and complete career pathways programs?
Who provides the counseling or guidance?
Are there any additional learner supports to support students on a career path that you would like to offer adult learners?
Probe on each if not mentioned:
Career transition support, such as career planning, exploration of career options, interview preparation, and connections workforce training organizations, or other connections that would help them progress through a career pathway, including connections to postsecondary institutions (such as connecting student to college admission processes or support services offered on campus)
Academic support, such as support with developing study and work skills
Financial support, such as assistance accessing financial support for education
Why do these interest you?
Do you see a need at specific skill levels for additional career navigation supports? If ABE or ASE, ABE levels 1–6? If ESL, ESL levels 1–6?
How do learner support needs vary at different skill levels?
Do you have any specific plans to implement new career navigation supports?
What steps have you taken to lay the groundwork for implementing new career navigation supports?
Do you have sufficient resources to implement new career navigation supports?
What supports would be necessary for [PROVIDER NAME] to implement new career navigation supports?
When do you think you might be able to implement new career navigation supports?
How feasible would this be? What might get in the way?
What challenges would you anticipate facing with implementing career navigation supports?
Probe: Recruiting learners? Engaging learners on an ongoing basis? Hiring, training, and retaining staff who can implement the supports? Staff availability? Issues with developing or maintaining necessary partnerships?
In how many locations do you provide adult education and literacy classes?
If more than one: Where are those located?
Do you offer the same classes at the same levels and with the same content at all locations? If not, how do they differ?
Do you partner with other organizations to provide adult education and literacy classes or support services?
Can you describe the partnerships, including what they offer your program?
Recruitment and eligibility
Next, I’d like to understand how you recruit learners.
Can you walk me through how an individual would enroll, starting with how they learn about the adult education and literacy classes, contact [PROVIDER NAME], then the intake, screening, and enrollment procedures?
Probe: How are they recruited? For example, word of mouth (family, friends), partner referrals, local media (radio, newspapers), place(s) of worship, place of employment, American Job Centers, libraries.
Probe: Do you conduct any marketing or outreach about adult education and literacy classes?
Probe: How does the intake process work? Is it done electronically or on paper? Are there specific individuals who do intake? How many?
Are potential learners counseled to enroll, or not to enroll, in specific types of adult education and literacy classes? How do counselors determine which learners to encourage to participate in specific classes?
If any support services currently offered: Are potential learners counseled to participate in or receive, or not participate in, specific types of support services? How do counselors determine which learners to encourage to participate in support services?
If you implemented new career navigation supports, how would learners be counseled to enroll or not enroll in it?
Are there any eligibility requirements participants must meet for adult education and literacy classes? For example, a pre-test assessment score, certain education functioning levels, a degree of English proficiency.
If there is a pre-test: What is the pre-test assessment?
Is a cutoff score used?
If yes: Is the same cutoff used consistently for all learners?
If yes: How did you determine the cutoff score?
How many applicants do you receive, and how many do not meet the cutoff score?
Do all learners who enroll take this pre-test assessment before starting adult education and literacy classes?
Does the pre-test differ across types and levels of classes?
If any support services currently offered: Do you conduct any assessments before receiving support services? If so, what assessments?
Are many learners that are interested in adult education and literacy classes turned away because they are not eligible? Which eligibility criteria do these learners typically fail to meet? About how many are turned away due to eligibility?
Is there a cost to participate in adult education and literacy classes?
If yes: What is the cost?
To what extent is cost a barrier to participation?
Enrollment capacity
These next questions ask about your current enrollment and the capacity for [PROVIDER NAME] to expand services.
How many learners currently participate in adult education and literacy classes?
Is this a typical number?
How does enrollment vary across type and levels of classes?
Do you expect it to change in the next few years? If so, how?
If any support services currently offered: How many of those enrolled in classes receive support services?
If any support services currently offered: What is the maximum number of learners you could provide support services to at a given time? Does this vary at different times in the year? If so, how does it vary?
About how many learners do you think might be interested in receiving additional career navigation supports?
Are you currently able to serve all learners who are interested in and meet the requirements for adult education and literacy classes?
If no:
Why not?
About what percentage of interested, eligible learners are turned away for these reasons?
If any support services currently offered: Are you currently able to serve all learners who are interested in and meet the requirements for support services?
If no:
Why not?
About what percentage of interested, eligible learners are turned away for these reasons?
Do you use a managed enrollment period or an open enrollment throughout the year?
If managed: What is the length of the typical period (for example, semester, every 6–8 weeks, other)?
If any support services currently offered: Do support services follow a similar intake schedule as classes?
If you implemented new learner supports, would you use the same type of intake schedule?
Data availability and assessment
Next we’d like to understand more generally how your organization collects and records data about learners.
What data do you collect on learners’ demographics and participation?
What type of data do you track about the classes or services an individual participates in? For example, the classes or services learners enroll in or the length of time enrolled? (Probe for services provided outside of class time.)
Do you track attendance? (Probe for services provided outside of class time.)
Do the data you track differ for learners you serve in different types of classes or different types of services?
What kind of contact information and other identifying information do you collect?
How often is contact information missing? How often is the Social Security number missing?
Which assessment(s) do you use as a post-test for adult education learners?
Do you administer post-tests after a set amount of time or after a set number of instructional hours?
Does this differ across types and levels of classes?
If any support services currently offered: Do you conduct any assessments after receiving support services? If so, what assessments?
For about what percent of learners do you have post-test data?
Do you use any assessments to measure progress during enrollment in adult education classes? Which assessments?
At what time points or after how many instructional hours do you administer these assessments?
Does this differ across types and levels of classes?
If any support services currently offered: Do you conduct any assessments while receiving support services? If so, what assessments?
How do you obtain data on other learner outcomes? Probe: Employment measures, credential attainment
Are any of these outcomes from an administrative data source like UI wage records? For what percentage of your learners are you able to match to existing records?
Are the demographic, participation, assessment, and other outcome data maintained in a program management information system?
Do you have a dedicated data manger?
If no: Who manages your program data?
Feedback on random assignment and doing a study
One of the best ways to learn about the effectiveness of career navigation support services is an experiment in which we would randomly choose (such as by a coin flip) which learners, classes, or even sites, would receive career navigation support services, or a specific component of career navigation support services such as additional professional development for navigators. Or, if you had more learners interested in and eligible for career navigation support services than spots available, we could conduct a lottery to determine which learners would receive these services and which would not.
We’d like to get your reactions to the idea of studying this approach through a study where some locations or some learners receive career navigation support services and some don’t.
Suppose some learners or sites were assigned by lottery to receive career navigation support services and others would be assigned to a different version, another type of service, or no services. Do you have any concerns about this way of selecting who would get the services? How do you think your staff and partners would react?
We understand that some states have requirements for services that must be offered that might restrict us from being able to assign learners through a lottery. Are you aware of any requirements or restrictions related to career navigation support services in [STATE NAME] that might restrict our ability to conduct a study like this? Do you anticipate any such initiatives in the future?
One way to implement the study so that all learners receive the program in some form is to offer different versions of the approach being studied. For example, navigators could receive different types of training, or some learners could receive specific types of career preparation supports while others receive more academic supports. Are there variations like these that you would be interested in testing?
What types of supports would be helpful to convince [PROVIDER NAME] to participate in an evaluation like this of academic or career services? For example, funds to support a staff person collecting data or additional staff development. Are there financial or other incentives that would make participation more attractive?
INTERVIEWER NOTE: Continue to Section C.
SECTION C: NON-FOCAL APPROACH QUESTIONS FOR ALL RESPONDENTS
C. Interview non-focal approaches questions
Feasibility of implementing a new approach
We are looking at a number of other models or approaches that could be implemented and evaluated. (Interviewer: The table below will be sent in an email to respondents before the interview; refer to the table and read through definitions of each approach not addressed in previous sections, and ask questions a and b to determine current implementation and interest in future implementation.)
Is your organization currently implementing this approach?
If not, do you have plans to implement this approach, and have you taken steps to lay the groundwork for it that would allow you to launch it in the next year with the resources you currently have? And would you be interested in participating in an impact evaluation of this approach?
Intervention |
Implementing? |
Planning? |
Distance learning: An approach that delivers all instruction in an entirely virtual format. The technology used can include the Internet, broadcast, closed circuit, cable, wireless communication devices, videos, DVDs, or CD-ROMs. |
|
|
Blended learning: An approach that delivers instruction through a combination of in-person and virtual instruction, with both components required as part of the course. |
|
|
Mobile or online learning tools to supplement instruction: Resources that learners can independently access through the internet on a computer or mobile device and that reinforce classroom instruction |
|
|
IET: Programs that include integrated curricula (adult education instruction concurrently and contextually with workforce preparation activities and workforce training) focused on a single set of learning goals, through either “co-teaching” or “co-planning and alternate teaching.” |
|
|
Bridge or on-ramp programs: Classes or programs to prepare learners to transition to college courses or occupational training by providing basic skill instruction along with occupational content, employment skills, and/or college success strategies. |
|
|
INTERVIEWER NOTE: Ask sections below for (a) all approaches currently implementing and (b) all approaches interested in and able to implement in next year. If there is not time to cover all, ask about the two approaches already implemented or close to implementation, and ask if we can follow up via email if necessary about others.
Distance or blended learning
If currently implementing distance or blended learning:
Do you use [APPROACH] with ABE, ASE, or ELA CLASSES?
Do you use an off-the-shelf curriculum for the distance/blended classes? Do you use any commercial software products in these classes?
If currently implementing blended: What components of the blended classes are delivered virtually and what components are delivered in-person?
Do you have more learners interested in distance or blended learning than you are able to serve? (Probe for distance and blended separately, if implementing both.)
What challenges do you face in implementing distance or blended learning? (Probe for distance and blended separately, if implementing both.)
Are there new curricula, software products, or approaches you would like to try or changes you would like to make to your distance or blended classes?
If planning on implementing distance or blended learning:
Is there an off-the-shelf distance or blended curriculum you plan to use? A commercial software product you would be interested in using in these classes?
What kind of resources would you need to support implementing this type of approach? Additional space or materials? Additional resources to purchase curricula or software? Additional staff? Training of staff? Other capacity issues?
What challenges would you anticipate with implementing distance or blended learning? (Probe for distance and blended separately, if planning on both.)
Do you anticipate that you would have more learners interested than you would be able to serve?
Supplemental online or mobile tools
If currently implementing supplemental online or mobile tools:
What types of online or mobile tools are used by instructors to supplement material taught in adult education classes? Do instructors connect the material covered in the online or mobile tools to class instruction? Do instructors monitor learners’ use of these tools?
Do you use [APPROACH] with ABE, ASE, or ELA CLASSES?
What challenges do you face in using supplemental online or mobile tools?
Are there new tools you would like to try or changes you would like to make to how tools are used to supplement classes?
If planning on implementing supplemental online or mobile tools:
Is there a specific online or mobile tool you are interested in using to supplement adult education classes?
What kind of resources would you need to support implementing this type of approach? Training of staff? Other capacity issues?
What challenges would you anticipate with using these tools?
IET
If currently implementing IET:
Does your IET program(s) use an off-the-shelf curriculum, or is it developed by the instructors, or a mix of both? If off-the-shelf curriculum, which one?
What skill levels does the IET program target? Does it serve ABE, ASE, or ELA learners?
Do you use a co-teaching approach or a co-planning with alternate teaching approach?
Do you have more learners interested in IET than you are able to serve?
What challenges do you face in implementing IET?
Would you be interested in trying a [co-teaching/co-planning] approach instead? Why or why not?
Are there new curricula you would like to try or changes you would like to make to your IET programs?
If planning on implementing IET:
Is there a specific IET model or off-the-shelf curriculum you plan to use?
Do you plan to use a co-teaching approach or a co-planning with alternate teaching approach, or another approach?
What kind of resources would you need to support implementing this type of approach? Additional space? Additional staff? Training of staff? Other capacity issues? (Probe for resources related to co-teaching vs. co-planning models.)
What challenges would you anticipate with implementing IET? (Probe for challenges related to co-teaching vs. co-planning models.)
Do you anticipate that you would have more learners interested than you would be able to serve?
Bridge programs
If currently implementing bridge programs:
Does your bridge program(s) use a commercial, off-the-shelf curriculum, or is it developed by the instructors, or a mix of both? If off-the-shelf curriculum, which one?
What skill levels does the bridge program target? Does it serve ABE, ASE, or ELA learners? Is it focused on preparing learners for occupational training or postsecondary education, or both?
Do you have more learners interested in bridge programs than you are able to serve?
What challenges do you face in implementing bridge programs?
Are there new curricula you would like to try or changes you would like to make to your bridge programs?
If planning on implementing bridge programs:
Is there a specific bridge model or off-the-shelf curriculum you plan to use?
What kind of resources would you need to support implementing this type of approach? Additional space? Additional staff? Training of staff? Other capacity issues? (Probe for resources related to bridge for occupational training vs. bridge for postsecondary preparation.)
What challenges would you anticipate with implementing bridge programs? (Probe for challenges related to bridge for occupational training vs. bridge for postsecondary preparation.)
Do you anticipate that you would have more learners interested than you would be able to serve?
Wrap-up
That’s all the questions I have for you. Thank you very much for your time and thoughtful answers. After I share this information with my colleagues, I might have one or two follow-up questions. In the meantime, if you have any follow-up comments or questions, please feel free to contact me at [INTERVIEWER PHONE/EMAIL].
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Lisbeth Goble |
| File Modified | 0000-00-00 |
| File Created | 2021-01-16 |
© 2024 OMB.report | Privacy Policy