New Instrument #11 Study Participant Check-in Call

Supporting Statement for OMB Clearance Request

Appendix P: In-Depth Participant Interveiw Guides

Innovative Strategies for Increasing Self-Sufficiency (ISIS) – Follow-up Data Collection

OMB No. 0970-0397

March 2013

Submitted by:

Brendan Kelly

Office of Planning, Research
and Evaluation

Administration for Children
and Families

U.S. Department of Health and Human Services

Initial Contact Letter
Treatment Group

[Date]

[First Name] [Last Name]
[Street Address]
[City], [State] [Zip]

Dear [First name] [Last name],

On behalf of our research team, I want to thank you again for your participation in the Innovative Strategies for Improving Self-Sufficiency (ISIS) project and to ask for your continued participation in this very important project.

We are contacting you now because your name has been selected at random from [NAME OF PROGRAM] participants to participate in an additional part of the ongoing study of ISIS programs and participants. If you choose to participate, one of our researchers would interview you about your experiences in PROGRAM X, your goals for your career, how you balance participation in PROGRAM X with the rest of your life, and other issues. This interview will allow you to talk in detail about your experiences and will help improve programs like PROGRAM X.

This interview will take about 60 minutes, and if you decide to participate, you will receive a token of appreciation in the amount of $40. Researchers plan on visiting [NAME OF CITY] between the dates of X and X, and we will arrange a time that is convenient for you to do the interview.

If you decide to participate, we will also ask you to allow us to contact you to complete a short (10 minute) telephone interview with us next year as well as one other in-person interview as you progress in PROGRAM X.

I will contact you in the next few days to provide more information about this phase of the study, answer any questions you might have, and, if you decide to participate, arrange a time and location to meet.

We look forward to your continued participation in this study. If you have any questions or concerns, please contact me at 734-615-2113 (office) or 734-717-1239 (cell).

Our research is incomplete without you. I look forward to talking with you soon.

The Paperwork Reduction Act Burden Statement: This collection of information is voluntary and will be used to develop a comprehensive understanding of study participant experiences. Public reporting burden for this collection of information is estimated to average 60 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to [Contact Name]; [Contact Address]; Attn: OMB-PRA (0970-0397).

Sincerely,
Kristin Seefeldt
ISIS Study Team Member
University of Michigan

Script for Setting-Up In-Depth Participant Interview

Treatment Group

Hello Mr./Ms.____________. My name is _____________ and I am part of the team that it evaluating Program X. As the letter we recently sent you noted, your name has been selected at random from [NAME OF PROGRAM] participants to participate in an additional part of the ongoing study of ISIS programs.

In this interview we would like to talk with you again in more detail about work, family, and your participation in Program X. I’m calling today because we will be in [name of city] on [dates] and I would like to set up a time to interview you.

The interview can take place at a location and time that works best for you and should take about an hour to complete. If you have any questions about this phase of the study, I’m happy to answer them. Otherwise, can we find a time to do the interview? <arrange date and time>.

Privacy and Consent Terms

Innovative Strategies for Increasing Self-Sufficiency (ISIS)
In-Depth Interviews PRIVACY AND CONSENT TERMS

The following statements describe the research in which you are being asked to participate and tells you your specific rights and our responsibilities:

• You are being invited to participate in an additional component of the evaluation of the ISIS project. This part of the project is currently funded by the U.S. Department of Health and Human Services and is being led by Karen Gardiner of Abt Associates (Bethesda, Maryland) and Kristin Seefeldt of the University of Michigan.

• The purpose of this part of the evaluation is to learn more about how individuals balance work, family, and school responsibilities and how people think about the opportunities available to help them meet their educational and career goals.

• Participation in this phase of the evaluation is completely voluntary.

• If you participate, you will be interviewed and asked questions about your current and recent employment, your family background, your finances, and your participation in any education and training programs. You may consider some of these questions personal and sensitive, but the researchers have taken many steps, outlined below, to protect your privacy.

• You may choose not to answer any question and you can stop your participation at any time; all you have to do is say, “I want to stop.”

• The interview will last about 60 minutes. As a token of our appreciation for your participation, you will receive $40 at the completion of the interview.

• The researchers are planning to conduct one additional in-depth interview next year and one short (10 minute) phone interview before then. If you participate, you agree to be contacted for the next interview. Being contacted in the future does not mean that you have to participate again. You may decline further participation when re-contacted.

• If you participate, you agree to the audio taping of the interview. The recording of your interview will be kept in secured files on secure computer space. Only authorized project staff are allowed access to recorded interviews. The content of the interviews will be typed into a word processing file, with only the identification code, and not your name, used in the file. Once the project is complete, all recordings will be destroyed.

• Your family, any services you may receive, including those from [NAME OF PROGRAM], or your employment will not be affected in any way by participating or choosing not to participate in this aspect of the study.

• You will be asked about your experiences at [NAME OF PROGRAM], but staff at PROGRAM X will not know that you are participating in this interview (unless you tell them), and anything you report about the program will not be attributed to you.

• The researchers have taken steps to minimize the risks of this study. Even so, you may still experience some risks related to your participation, even when the researchers are careful to avoid them. These risks may include providing information that you consider to be personal and sensitive.

• Although you may not benefit directly from this study, your participation will help us understand the issues faced by individuals like you so that the state and federal government can design better programs, policies, and services to help people achieve their career goals.

The research team will take the following steps to protect your privacy:

• Your answers will be kept private. The researchers plan to publish the results of this study. When the researchers present information from the interviews in reports, papers or presentations, it will usually be in an aggregate form, but will never identify you. If the researchers use quotations from this interview or describe your situation, they will use a pseudonym (a different name), and other potentially identifying information about you will be changed in order to protect your privacy.

• There are some reasons why people other than the researchers may need to see information you provided. This includes organizations responsible for making sure the research is conducted safely and properly, including the University of Michigan and government offices, such as the study sponsor, the U.S. Department of Health and Human Services.

• Answers and information you provide in subsequent interviews and surveys may be linked together and used in analysis, reports, and papers. However, individual responses will never be used.

• Your name and contact information will be kept in secure files separate from the answers you provide. The only way to connect your name with your answers is an identification code, which by itself does not identify you. Only the researchers will have access to your name and your identification code during the data collection period. After all of the surveys are completed,
  only the researchers will have access to the identification code.

• You agree to allow the researchers to keep the data you provide for use in data analysis. The researchers may keep the data after the study ends. This data will be kept in secure, password protected files with your name and other identifying information removed.

If you have questions about this study, you may contact the study director, Karen Gardiner at ISIS@abtassoc.com or 301-634-1700 or Kristin Seefeldt, research team member at 734-615-2113
or kseef@umich.edu.

If you have questions regarding your rights as a research participant, or wish to obtain information, ask questions, or discuss any concerns about this study with someone other than the researchers, please contact the University of Michigan Health Sciences and Behavioral Sciences Institutional Review Board, 540 East Liberty Street, Suite 202, Ann Arbor, MI, 48104-2210, (734) 936-0933,
toll free at (866) 936-0933, or via email at irbhsbs@umich.edu

The Paperwork Reduction Act Burden Statement: This collection of information is voluntary and will be used to develop a comprehensive understanding of study participant experiences. Public reporting burden for this collection of information is estimated to average 60 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to [Contact Name]; [Contact Address]; Attn: OMB-PRA (0970-0397).

Signing this consent form indicates that you understand and are willing to participate in this study.

Respondent’s Name _________________________________________

Respondent’s Signature __________________________________________

Date __________________________________________

Script for Check-in Call: Treatment Group

Hello [first name] [last name]

This is XX from the ISIS study team. As you may remember, when we interviewed you in [date], we stated that we would like to conduct one short interview with you to check in on your progress in PROGRAM X. I’m calling today to see if we can complete this interview, which should take about 10 minutes.

Is now an ok time to do this interview?

<If no- arrange different time>

<If yes>:

We would like you to know a few things about this interview. First, your participation is completely voluntary and you may choose not to answer any questions you don’t wish to answer, or end the interview at any time. This interview is for research purposes only and is funded by the U.S. Department of Health and Human Services. Your family, any services you may receive, including those from [NAME OF PROGRAM], or your employment will not be affected in any way by participating or choosing not to participate in this interview. You will be asked about your experiences at [NAME OF PROGRAM], but staff at PROGRAM X will not know that you are participating in this interview (unless you tell them), and anything you report about the program will not be attributed to you.

I’d like to continue now, unless you have any questions.

1. Are you still participating in PROGRAM X?

• <if yes>: How is the program going for you? Tell me a little bit about what you are doing?

• <if no>: When did you finish? What is your overall assessment of PROGRAM X?

2. Are you working , have you changed jobs, or is there anything else going on in your life that you think would be important for the research team to know about as we evaluate PROGRAM X’s effectiveness for people like you? Explain?

3. Since we are hoping to continue following you and your career progression is there anything else you’d like to share about your experiences with PROGRAM X, your goals, or other issues?

Do you have any questions for me?

Thank you so much for participating. We will contact you again in the future to complete a similar interview.

ISIS Treatment Group In-Depth Interview, Time 1

We’d like to start by getting to know a little bit about you.

A. Background/Biography

1. In which city or community did you grow up? How long did you live there (have you lived here)? Have you lived in any other city or community? If yes, where and for how long? If respondent did not grow up in same community as [ISIS Program]: How did you come to live here in [ISIS Program location]?

2. Where did you go to school when you were growing up (names and locations of schools)? What is the highest grade that you completed? During elementary and high school, what was enjoyable and what was challenging during each time period? (Probe for aspects of school that were enjoyable or challenging and why).

3. Please describe your living situation now. Are you married, divorced or single? Has your marital status changed over the past year or two? Do you have children? If yes, how many and what are their ages? Do the children live with you? Please describe your main responsibilities for your children. (If respondent is not married and living with children, ask if respondent lives alone or with someone.)

4. Can you tell me a little bit about where you live, including who lives with you (probe for any people who move in and out), what type of place it is, and the neighborhood? You don’t need to tell me the names of people who live with you, just their relationship to you. How many different places have you lived during the past 5 years?

5. What are your goals for your education?

• Have these goals changed over time? If yes, in what ways have your goals changed?

• What steps have you taken/are you taking to help meet your goals?

• Since high school, have you participated in education services other than (current ISIS services)?

• If yes, what types of education services did you receive and when did you receive them? Did you receive any certificates or credentials from these services? If yes, which certificates or credentials?

• Other than (current ISIS service), have you participated in any job training? If yes, what type of training, who provided it, and when did you participate? Did you receive any certificate or credential from the job training? If yes, what type of certificate (in which occupational field)?

6. Let’s talk about jobs and employment. Have you held any jobs during the past three years? (If respondent is employed, begin with current job. Ask for name of job, type of work, number of hours per week, stability of hours and schedule, and hourly wage for most recent job).

• In thinking about these jobs: have you liked any aspects of your jobs? If yes, what have you liked about which jobs and why? What aspects of each job have you not liked? Why?

• Have you learned anything about your skills and interests from the jobs you have had? If yes, what have you learned?

7. Let’s talk about your career goals.

• What are your current goals for your career—what type of job would you like to have in five years?

• Has your education and work experiences (if respondent has been employed) influenced your career goals? If yes, in what ways? What types of skills do you need to develop to help you meet your goals? Why do you say that?

• Do you expect to have any challenges in meeting your career goals? If yes, what are the challenges? How might you address these challenges?

Thank you for sharing information about your background. Next I’d like to ask you about your experience with [ISIS Program] so far.

B. Motivation for Choosing to Enroll in Program in ISIS Evaluation

8. How did you hear about this program (note: possible sources are through friends, family member, TV, radio, Internet, newspaper, high school counselor, college staff.)?

9. What prompted you to find out about or enroll in this program? Was there anything going on in your life, for example at home or at work, that made you want to enroll?

C. Experiences in Program in ISIS Evaluation

10. We’d like to understand what it’s like to be a participant in [ISIS Program]. Please describe what you do in a typical day in the program, beginning when you arrive (Probe for the types of classes or activities in which respondent participates, length of classes or activities).

• Are all days during the week about the same in terms of the program’s activities in which you participate, or do you do different activities on certain days?

• If your activities change during the week, please describe a typical week.

11. What are the classes or activities in which you have participated thus far?

• Are there things you have found useful in the program to help learn the material? If yes, please explain (probe for type of content taught, methods used to teach content)? If no, why not?

• Have the classes been difficult? If yes, what about the classes has been difficult and why do you think that is (probe for skills required to do the class work, the types of assignments that are required, the types of information that you need to know about the topics being taught)?

• What about the classes has been easy? Please explain.

• Think about the instructors—have they enabled you to learn? If yes, what have they done to help your learning? If not, why haven’t the instructors been effective in helping you learning?

12. Have you had to pay out of pocket to attend [ISIS Program]?

• If yes, what has the cost been (probe for individual course costs, special activities’ costs, books and materials cost)?

• How often do you pay for various costs--by term or semester, or by another period of time? (Ask if the schedule for payment differs by types of costs.)

• Have you received any financial aid to help you pay for courses, activities, books, other items? (Probe for Pell grants, loans, scholarships, financial assistance from family members, employment, or savings. Probe for whether respondent will need to pay back any loans or other type of financial assistance.)

13. How long have you been attending the program?

• Are you working toward a certificate or credential in your educational classes? If yes, what certificate or credential are you working toward? When do you expect to receive this certificate or credential?

• Have you received any credentials to date from this program? If so, what have you received and when did you receive it?

• For participants in programs in which education services are offered outside of the ISIS program—what services are you receiving? How long do you expect to receive these services?

14. What are your next steps in the program?

D. Assistance and Support Provided by ISIS Program

15. Can you describe a typical day for me, from the time you get up to the time you go to bed? If your days are quite different, tell me about what you did yesterday and how that might be different from other days.

16. You’ve talked about the aspects of your life that keep you busy (e.g., work, family responsibilities). What has it been like to fit [ISIS Program] into the rest of your life?

Have you encountered any unexpected challenges? (Probe for issues such as: childcare, transport, difficulty of class, being able to afford tuition or supplies, etc.)

How did you deal with that/how are you dealing with that?

17. Do you receive any assistance from [ISIS Program] staff and instructors on these issues?

• For each source of program support, ask the following questions:

• If yes, who in the program provided assistance?

• How often do you meet with this individual? (e.g., on a set schedule, on demand, etc.)

• What type of assistance does this person provide? Is the assistance helpful? If yes, in what ways? If no, why not?

18. Overall, what have been your biggest challenges to participating in [ISIS Program]? Explain.

19. What types of additional program support would be helpful to your success in this program?

E. Family and Friend Support

20. Do you have any sources of support from family or friends that you can rely on to help you balance the demands of home with the demands of [ISIS Program]? If yes, who provides support? What support is provided? Is this support helpful? If yes, in what ways?

21. How do your family and friends feel about your participation in [ISIS Program]? (Probe as to whether they think participation is a good idea, feasible given the respondents’ circumstances.)

22. Where you live, do you have a specific place where you can study? How easy or difficult is it for you to do homework, study, or do any other preparations for ISIS Program? Explain.

23. How long does it take for you to travel to [ISIS Program]? What type of transportation do you use? Do you ever have difficulties getting to the program? If yes, what are some of the problems that you have encountered?

24. How easy or difficult is it for you (and your family, if relevant) to manage financially while you are in ISIS Program? Why do you say that? (Probe for debt that participant has incurred , use of public program funds for support )

25. How different is your financial situation compared to your situation before you began attending [ISIS Program]? Explain.

26. Besides participating in {ISIS Program] (and your work/family, if relevant), what other major responsibilities do you have right now? How easy or difficult is it to manage these responsibilities? Why?

27. Have you made friends with any of the other participants in [ISIS Program]? Do you ever study or do other things together? (Probe for extent to which respondent is part of a group or participates more solitarily)

F. Looking Forward

28. What is your sense of the economy in your city or area? What do you think are your chances for getting a job when you finish [ISIS Program}? Why do you say that?

29. Are there other any other services, support, or other activities that [ISIS Program] could provide to assist you in:

• Earning a certificate or credential?

• Obtaining a job? To what extent do you feel this program has started you on career track to obtain a certain job? Please explain.

Thank you for participating in this interview. We appreciate all of the time you have taken to help us understand your experience in [ISIS Program].

ISIS Treatment Group In-Depth Interview, Time 2

To start, we’d like to check on some information about your participation in [ISIS Program]:

A. Experiences in Program in ISIS Evaluation

1. Can you remind me, when did you enroll in/start participating in [ISIS Program]?

2. We last talked to you in person on <month/year>. Since then, did you complete the program, stop attending, or are you still enrolled?

• <if completed program> Were you able to complete the program all at once, or did you have to stop and start again? Please explain,

<If completed all at once>: Congratulations on finishing! Some people have challenges getting through programs. What do you think made the difference for you?

• <if had interruptions>: When and for how long did you stop? (probe for multiple spells) Can you tell me a little bit about what happened or what was going on that interrupted or stopped your participation? (probe for dissatisfaction with program, financial issues, employment, family issues)

Was there anything that the [ISIS Program] could have done to help you stay enrolled? Why do you say that?

• <if stopped attending> Did you stop attending altogether, or did stop attending for a short time?

When and for how long did you stop? (probe for multiple spells). Can you tell me a little bit about what happened or what was going on that interrupted or stopped your participation? (probe for dissatisfaction with program, financial issues, employment, family issues, different issues at different times)

Was there anything that ISIS Program could have done to help you stay enrolled? Why do you say that?

• <if still enrolled> Have you had to take breaks from [ISIS Program] or slow down your participation? If yes, when and for how long did you stop? (probe for multiple spells) Can you tell me a little bit about what happened or what was going on that interrupted or stopped your participation? (probe for dissatisfaction with program, financial issues, employment, family issues)

• Was there anything that ISIS Program could have done to help you stay enrolled? Why do you say that?

Let’s talk a little bit more about [ISIS Program]

3. Can you give me an overview of the classes you’ve taken and activities you participated in, particularly any you might have done since we last interviewed you in 2013?

• What did you think of them? (Probe for perceptions of instructor competence/quality, interesting/relevant content etc.)

• What about the class/program has been most useful to you so far? Least helpful? Explain

• What about the class/program has been most difficult for you? What has come easy? Explain

4. What, if anything, have you had to pay to participate in [ISIS Program]?

• If you had to pay out of pocket, how did you pay for it? (Probe for Pell grant, loans, scholarship, financial assistance from family members, employment or savings. Probe for whether R will need to pay back any loans)

• How easy or difficult was/is it to get the money you needed? (in particular, probe for challenges navigating the financial aid system, obtaining loans). Explain.

5. How long were you in the program?

• <if still participating> How much longer do you anticipate you’ll be involved with [ISIS Program]?

B. Assistance and Support Provided by ISIS Program

6. What was it like to fit [ISIS Program] into the rest of your life? Explain.

• Did you encounter any unexpected challenges? (Probe for issues such as: childcare, transport, difficulty of class, being able to afford tuition or supplies, etc.)

7. What were the main sources of assistance and support you used to help you balance the demands of home with the demands of [ISIS Program? Explain.

8. What about assistance and support from [ISIS Program staff and instructors?

• How often did you meet with program support staff? (e.g., on a set schedule, on demand, etc.)

• How easy was to meet with advisors or case managers?

• Note to interviewers: if program has multiple types of participant support staff, please adapt questions for each type of staff.

9. What kinds of things did you talk about when you meet with program support staff? Can you give me an example of what you talked about the last time you met with staff?

10. To what extent did you feel that program staff/instructors understand the fact that you and/or your classmates have responsibilities/competing priorities outside of class? Why do you say that?

11. Overall, what would you say were or have been your biggest challenges to participating in [ISIS Program]? Explain.

12. Since we last talked to you, are you or have you been enrolled in any other classes or training activities or college, through [ISIS Program] or through another place?

<if yes> Please tell me more about where you’ve gone, what type of classes/training you’ve done, when you started and when you finished (if applicable)

Why did you decide to enroll in this program? What was going on that made you want to enroll?

How do you see this training as fitting or continuing with what you did in [ISIS Program]? Why (probe for other skills R wanted and why)

Are you working toward a credential or certificate? What is that?

What about any difficulties or challenges you’ve faced with this training? Can you talk about those?

13. Do you think you’ll go on to take more classes or enroll in further training? Explain.

<if no> Have you thought about enrolling? Why or why not? (probe for satisfaction with what they’ve done so far, time constraints, other issues)

• In the future do you think you’ll go on to take more classes or enroll in further training? Explain.

C. Employment

14. Have you worked for pay at any time since 2013?

• <if yes, go to question 2 below>

• <if no>: When is the last time you worked? What is the reason you left that job?

• Can you talk a little bit about why you’re not working, for example, if you’re looking for a job, if you’ve decided to take time off for any reason, etc. (probe for job search difficulties, decisions to stay home for kids, health problems, time off for education/training etc.)

• In what ways do you think participating in [ISIS Program] will help you find a job? Explain. Do you think it will help you find the type of job that you want? Why or why not? (go to question 3 below)

15. Tell us about any jobs that you have held since 2013, including your current job if you’re now working (probe for hours, stability of hours and schedule, wages, type of work for current or most recent job).

• What do you like most about your current/most recent job?

• What have you not liked about these jobs?

• In what ways did participating in [ISIS Program] help you in your current/most recent job? Would you say that participating in {ISIS Program] helped you get the type of job you wanted? Why or why not?

• <if not currently employed> When is the last time you worked? What is the reason you left that job?

• Can you talk a little bit about why you’re not working, for example, if you’re looking for a job, if you’ve decided to take time off for any reason, etc. (probe for job search difficulties, decisions to stay home for kids, health problems, time off for education/training etc.)

• In what ways do you think participating in [ISIS Program] will help you find a job? Explain. Do you think it will help you find the type of job that you want? Why or why not?

16. What is your sense of the economy in your city or area? How do you see the economy as affecting the types of jobs available to you? Why do you say that?

17. Can you describe a typical day for me, from the time you get up to the time you go to bed? If your days are quite different, tell me about what you did yesterday and how that might be different from other days.

D. Family and Friend Support

18. In general, how supportive are your friends and family regarding your career and other goals? Explain (probe for concrete examples)

• How did your friends and family feel about you being a part of ISIS Program? Why do you say that? (probe for different types of reactions, especially if R has children, how much emotional and financial support received, concrete examples)

• <if completed program> Did your friends’ and family’s feelings change at all during the time you participated? How so? Why do you think that was?

19. During the time you were enrolled in [ISIS Program], how many places did you live?

• <if 1, go to question 3 below>

• <if more than 1> What happened that you moved? When was this? (if multiple moves, ask about circumstances and timing of each)

• Where did you move to? (if multiple moves, ask about each) How was this place different from where you lived before? Explain.

• How do you think your move/moves affected your participation in ISIS Program? Explain. (if not clear, probe for any additional moves since finishing and how that affects any new training they might be in)

20. In your house/apartment, how easy or difficult was it for you to do homework, study, or do any other preparations for ISIS Program or other training? Explain (if moved, probe for differences; if in additional training, ask about differences)

21. Who lives with you? You don’t have to tell me their names, just their relationship to you.

• Has this changed since we last talked to you in 2013, including people who might have only lived with you a short time? Explain

• Have you had any changes with your family, for example, getting married, having a child, losing a family member? Explain.

22. How easy or difficult was it for you and your family to manage financially while you were in [ISIS Program]? Why do you say that? (probe for debt, use of public programs to help make ends meet)

• How different is your financial situation compared to before starting {ISIS Program}? Did you experience any changes during your time in [ISIS Program] Explain (probe to get information as to what was going on to cause change or to maintain stability)

• What about now? How would you describe your financial situation? In what ways is it different? What challenges are you currently facing financially? Explain.

How did you to get to and from [ISIS Program}? Explain (probe for long commutes, uncertain ability to attend)

23. Besides participating in [ISIS Program] (and your work/family-if relevant), what other major responsibilities did you have? How easy or difficult was it to manage these responsibilities? Why? What did you do?

• <if no longer in program> What would you say your major responsibilities are now? How easy or difficult is it to manage these responsibilities? Explain.

24. How well do you know some of the other students in [ISIS Program]? Did you ever study or do other things together?
    (probe for extent to which R is part of a group or participating more solitarily) Are you still in touch with any of them, with staff or [ISIS Program]? Explain (probe for how those connections are maintained and used)

E. Looking Forward

25. Tell me a little bit about the career goals you have for yourself now.

• In what ways has [ISIS Program] changed how you think about these goals?

• How has [ISIS Program] helped you achieve your goals? What else do you think [ISIS Program] could have done to help you meet your goals?

• What kinds of additional skills do you hope to/need to develop to help you meet your goals?

• What could hold you back from meeting your goals?

• <if R has children> What kind of goals do you have for your children in terms of education, work, or anything else that is important? Explain. Is there anything you/they are doing to work toward those goals? Explain.

26. <if enrolled in additional training>

• Earlier, you talked about <additional program/training> How do you see this program as contributing to your future goals?

F. Overall Assessments

27. Overall, what were the most useful aspects of [ISIS Program]? This could be particular classes, teachers or other staff, other participants, or any other part of the program. Explain

28. What do you think [ISIS Program] could have done differently to have improved your experience?

29. In what ways did participating in [ISIS Program] change or affect any other aspects of your life or your family’s life? This could be both positive and negative. Explain

30. As you may know, [ISIS Program] is part of a national evaluation of programs that are trying to help individuals reach their career goals. What would you tell policy makers about the types of services, programs, or other changes that people like you need in order to reach those goals?

31. Given everything we’ve talked about today, what else do you think would be useful for the researchers studying this program to know about your experience?

Thank you for participating in this interview. We appreciate all of the time you have taken to help us understand your experience in [ISIS Program].

Initial Contact Letter: Control Group

[Date]

[First Name] [Last Name]
[Street Address]
[City], [State] [Zip]

Dear [First name] [Last name],

You may recall that [about x months ago] you agreed to participate in a study called ISIS to help us understand how well [NAME OF ISIS PROGRAM] and programs like it work in helping individuals achieve their education, training and employment goals compared to other options in the community. Although you were not randomly selected to participate in [ISIS PROGRAM], you still play a vital role in helping us understand more about programs and services that assist people to achieve their educational and employment goals.

To gain this understanding we would like interview individuals who are not participating in [ISIS Program]. We are contacting you now because your name has been selected at random from the list of individuals who were interested in [NAME OF ISIS PROGRAM] but were not selected. If you decide to participate in this part of our evaluation, one of our researchers would interview you about your background, career goals, and experiences in any other education or training programs. Although you may not benefit directly, this interview will allow you to talk in detail about your experiences and will help improve programs like [NAME OF ISIS PROGRAM].

This interview will take about 60 minutes, and if you decide to participate, you will receive a token of appreciation in the amount of $40. Researchers plan on visiting [NAME OF CITY] between the dates of X and X, and we will arrange a time that is convenient for you to do the interview. If you decide to participate, we will also ask you to allow us to contact you to complete a short (10 minute) telephone interview with us over the course of the next year as well as one other in-person interview.

I will contact you in the next few days to provide more information about this phase of the study, answer any questions you might have, and, if you decide to participate, arrange a time and location to meet.

We look forward to your participation in this study. If you have any questions or concerns, please contact me at 734-615-2113 (office) or 734-717-1239 (cell). I look forward to talking with you soon.

The Paperwork Reduction Act Burden Statement: This collection of information is voluntary and will be used to develop a comprehensive understanding of study participant experiences. Public reporting burden for this collection of information is estimated to average 60 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to [Contact Name]; [Contact Address]; Attn: OMB-PRA (0970-0397).

Sincerely,

Kristin Seefeldt
ISIS Study Team Member
University of Michigan

Script for Setting-Up In-Depth Participant Interview: Control Group

Hello Mr./Ms.____________. My name is _____________ and I am part of the team that it evaluating Program X. As the letter we recently sent you noted, your name has been selected at random from the group of people who took part in Program X’s lottery but were not chosen. However, you still play a vital role in helping us understand more about programs and services that assist people to achieve their educational and employment goals.

In this interview we would like to talk with you in more detail about work, family, and other issues. I’m calling today because we will be in [name of city] on [dates] and I would like to set up a time to interview you.

The interview can take place at a location and time that works best for you and should take about an hour to complete. If you have any questions about this phase of the study, I’m happy to answer them. Otherwise, can we find a time to do the interview? <arrange date and time>.

Script for Check-in Call: Control Group

Hello [first name] [last name]

This is XX from the ISIS study team. As you may remember, when we interviewed you in [date], we stated that we would like to conduct a short interview with you to check in with you. I’m calling today to see if we can complete this interview, which should take about 10 minutes.

Is now an ok time to do this interview?

• <If no- arrange different time>
  <If yes>:

We would like you to know a few things about this interview. First, your participation is completely voluntary and you may choose not to answer any questions you don’t wish to answer, or end the interview at any time. This interview is for research purposes only and is funded by the U.S. Department of Health and Human Services. Your family, any services you may receive, or your employment will not be affected in any way by participating or choosing not to participate in this interview. You will be asked about work and any participation in education or training programs. Anything you report during this interview will not be attributed directly to you.

 I’d like to continue now, unless you have any questions.

1. Are you participating/still participating in an education or training program?

• <if yes>: How is the program going for you? Tell me a little bit about what you are doing?

• <if no>: Did you finish a program?

• <if yes>:What is your overall assessment of this program?

• <if no>: Are you considering enrolling in any sort of program? Why or why not?

2. Are you working , have you changed jobs, or is there anything else going on in your life that you think would be important for the research team to know about as we evaluate education and training programs for people like you? Explain?

3. Since we are hoping to continue following you and your career progression is there anything else you’d like to share about your experiences, your goals, or other issues?

Do you have any questions for me?

Thank you so much for participating. We will contact you again in the future to complete a similar interview.

ISIS Control Group In-Depth Interview Guide, Time 1

We’d like to start by getting to know a little bit about you.

A. Background/Biography

1. In which city or community did you grow up? How long did you live there (have you lived here)? Have you lived in any other city or community? If yes, where and for how long? If respondent did not grow up in same community as currently residing: How did you come to live here?

2. Where did you go to school when you were growing up (names and locations of schools)? What is the highest grade that you completed? During elementary and high school, what was enjoyable and what was challenging during each time period? (Probe for aspects of school that were enjoyable or challenging and why).

3. Please describe your living situation now. Are you married, divorced or single? Has your marital status changed over the past year or two? Do you have children? If yes, how many and what are their ages? Do the children live with you? Please describe your main responsibilities for your children. (If respondent is not married and living with children, ask if respondent lives alone or with someone.) How many places have you lived during the past five years?

4. Can you tell me a little bit about where you live, including who lives with you (probe for any people who move in and out), what type of place it is, and the neighborhood? You don’t need to tell me the names of people who live with you, just their relationship to you.

5. What are your goals for your education?

• Have these goals changed over time? If yes, in what ways have your goals changed?

• What steps have you taken/are you taking to help meet your goals?

• Since high school, have you participated in education services?

• If yes, what types of education services did you receive and when did you receive them? Did you receive any certificates or credentials from these services? If yes, which certificates or credentials?

• Have you participated in any job training? If yes, what type of training, who provided it, and when did you participate? Did you receive any certificate or credential from the job training? If yes, what type of certificate (in which occupational field)?

6. Let’s talk about jobs and employment. Have you held any jobs during the past three years? (If respondent is employed, begin with current job. Ask for name of job, type of work, number of hours per week, stability of hours and schedule, and hourly wage for most recent job.

• <if not currently employed>: When is the last time you worked? (if applicable): What is the reason you left that job?

• Can you talk a little bit about why you’re not working, for example, if you’re looking for a job, if you’ve decided to take time off for any reason, etc. (probe for job search difficulties, decisions to stay home for kids, health problems, time off for education/training etc.)

<for those with any employment>

• In thinking about these jobs: have you liked any aspects of your jobs? If yes, what have you liked about which jobs? What aspects of each job have you not liked?

• Have you learned anything about your skills and interests from the jobs you have had? If yes, what have you learned?

7. Let’s talk about your career goals.

• What are your current goals for your career—what type of job would you like to have in five years?

• Has your education and work experiences (if respondent has been employed) influenced your career goals? If yes, in what ways? What types of skills do you need to develop to help you meet your goals?

• Do you expect to have any barriers or challenges in meeting your career goals? If yes, what are the barriers or challenges? How might you address these barriers?

Thank you for sharing information about your background. Next I’d like to ask you about your experience with any education and training programs you have participated in so far.

B. Motivation for Choosing to Enroll in Program in ISIS Evaluation

8. As you may recall, you went through an enrollment and lottery process for [ISIS Program} in x/xx. How did you hear about {ISIS Program}?

9. What initially interested you/appealed to you about [ISIS Program] and made you want to participate in the lottery? Was there anything going on in your life, for example at home or at work, that made you want to enroll?

C. Experiences in Education and Training Programs

We are interested in whether you are considering or participating in other programs/services in the community. PROBE: Examples of these include: adult basic education (ABE), remedial coursework, occupational training, ESL/ESOL, on-the-job training (OJT), One-Stop services (resume prep, case management), or life skills classes.

10. Are you currently participating in any kind of education, training, or other type of program?

• If No; go to Section C, q7.

• If Yes:

11. What is the name of the program?

• We’d like to understand what it’s like to participate in this program. Please describe what you do in a typical day in the program, beginning when you arrive (Probe for the types of classes or activities in which respondent participates, length of classes or activities).

• Are all days during the week about the same in terms of the program’s activities in which you participate, or do you do different activities on certain days?

• If your activities change during the week, please describe a typical week.

12. What are the classes or activities in which you have participated thus far?

• Are there things you have found useful in the program to help you learn the material? If yes, in what ways (probe for type of content taught, methods used to teach content)? If not, why not?

• Have the classes been difficult? If yes, what about the classes has been difficult and why do you think that is (probe for skills required to do the class work, the types of assignments that are requirement, the types of information that you need to know about the topics being taught)?

• What about the classes has been easy? Please explain.

• Think about the instructors—have they enabled you to learn? If yes, what have they done to help your learning? If not, why haven’t the instructors been effective in helping you learn?

13. Have you had to pay to participate in this program?

• If yes, what has the cost been (probe for individual course costs, special activities’ costs, books and materials cost)?

• How often do you pay for various costs--by term or semester, or by another period of time? (Ask if the schedule for payment differs by types of costs.)

• Have you received any financial aid to help you pay for courses, activities, books, other items? (Probe for Pell grants, loans, scholarships, financial assistance from family members, employment, or savings. Probe for whether respondent will need to pay back any loans or other type of financial assistance.)

14. How long have you been attending the program?

• Are you working toward a certificate or credential in your educational classes? If yes, what certificate or credential are you working toward? When do you expect to receive this certificate or credential?

• Have you received any credentials to date from this program? If so, what have you received and when did you receive it?

15. What are your next steps in the program? (go to section D after answering)

16. Are you interested in participating in education or training programs?

• If yes, what is/are the reason(s) for not participating right now? (Probe for lack of program options, identified a program but unable to enroll and why (financial reasons, time commitments, etc.)

• Please tell me a little bit about the program/services that you are interested in exploring and why. (Probe for type, length, cost, barriers to entry, how they heard about it, etc.)

Go to Section D

D. Assistance and Support for Education and Training

17. We’ve talked about some of the other responsibilities in your life that keep you busy

• <If enrolled in program>: What has it been like to fit this program into your day to day life?

• <if not enrolled> What do you imagine it would be like to fit in an education/training program into your life? Explain.

• <For those who have been/are in a program>:

18. Have/did you encounter unexpected challenges? (Probe for issues such as: childcare, transport, difficulty of class, being able to afford tuition or supplies, etc.) How did you deal with that/how are you dealing with that?

19. Do you receive any assistance or support from program staff and instructors?

• For each source of program support, ask the following questions:

• Who in the program provides support?

• How often do you meet with this individual? (e.g., on a set schedule, on demand, etc.)

• What type of support does this person provide? Is the support helpful? If yes, in what ways? If no, why not?

20. Overall, what have been your biggest challenges to participating in this program? Explain.

21. What types of additional program support would be helpful to your success in this program?

22. Did you complete the program?

• If no: What lead to your decision to leave or stop? What was going on at the time in your life?

• If yes: Sometimes people find it difficult to complete programs like these. What factors contributed to your ability to complete the program?

E. Family and Friends Support

23. Do you have any sources of support from family or friends that you can rely on to help you balance the demands of home with the demands of education or training or work? If yes, who provides support? What support is provided? Is this support helpful? If yes, in what ways?

If participated in education or training:

24. How do your family and friends feel about your participation in education and training? (Probe as to whether they think participation is a good idea, feasible given the respondents’ circumstances.)

25. Where you live, do you have a specific place where you can study? How easy or difficult is it for you to do homework, study? Explain.

26. How long does it take for you to travel to your education/training program? What type of transportation do you use? Do you ever have difficulties getting to the program? If yes, what are some of the problems that you have encountered?

27. How easy or difficult is it for you (and your family, if relevant) to manage financially while you are in the education or training program? Why do you say that? (Probe for debt that participant has incurred , use of public program funds for support )

28. How different is your financial situation compared to your situation before you began attending a program? Explain.

29. Besides participating in a program (and your work/family, if relevant), what other major responsibilities do you have right now? How easy or difficult is it to manage these responsibilities? Why?

30. Have you made friends with any of the other students? Do you ever study or do other things together? (Probe for extent to which respondent is part of a group or participates more solitarily)

If did not participate in education or training:

31. How easy or difficult is it for you (and your family, if relevant) to manage financially while you since the lottery for {ISIS Program}. Why do you say that? (Probe for debt that participant has incurred , use of public program funds for support )

F. Looking Forward

32. What is your sense of the economy in your city or area? What do you think are your chances for getting a job when you your program? Why do you say that?

33. Are there other any other services, support, or other activities that could assist you in:

• Earning a certificate or credential?

• Obtaining a job? To what extent do you feel this program has started you on career track to obtain a certain job? Please explain.

Thank you for participating in this interview.

ISIS Control Group In-Depth Interview Guide, Time 2

1. Participation in Education and Training

We are interested in whether you are participating or have participated in any programs or services in the community. PROBE: Examples of these include: adult basic education (ABE), remedial coursework, occupational training, ESL/ESOL, on-the-job training (OJT), One-Stop services (resume prep, case management), or life skills classes.

1. Since we last talked to you, are you or have you been enrolled in any other classes or education and training activities or college?

• <if yes> Please tell me more about where you’ve gone, what type of classes/training you’ve done, when you started and when you finished (if applicable) (if more than one, ask about each)

• Probe for type, length, components/classes, cost, barriers to entry, how they heard about it, and stage of completion.

• If you participated in any hands-on or work-based learning components (e.g., internships/externships, clinical placements, labs) as part of [ALTERNATE ISIS PROGRAM], please describe these components.

• What, if anything, have you had to pay to participate in this program (Probe for Pell grant, loans, scholarship, financial assistance from family members, employment or savings. Probe for whether R will need to pay back any loans)

• How easy or difficult was/is it to get the money you needed? (in particular, probe for challenges navigating the financial aid system, obtaining loans). Explain.

2. Why did you decide to enroll in this program? What was going on that made you want to enroll?

3. Have you been able to participate continuously or have you had to stop and start or have other disruptions? Explain (if any disruptions, probe for when and what was going on)

4. Did you complete the program?

• <If no:> What kinds of things informed your decision to leave or stop? What was going on at the time in your life?

• <If completed the [program/course/opportunity>: Sometimes people find it difficult to complete programs like these. What factors in your life and in the program contributed to your ability to complete the program?

5. <if currently enrolled> How much longer do you anticipate you’ll be involved in the program?

6. <if currently enrolled> What are you looking forward to in the next steps of the program?

7. In your house/apartment, how easy or difficult was it for you to do homework, study, or do any other preparations for this program? (if moved, probe for differences)

8. How do/did you to get to and from this program? Explain (probe for long commutes, uncertain ability to attend)

9. Do you think you’ll go on to take more classes or enroll in further training? Explain.

<if no> Have you thought about enrolling? Why or why not? (probe for satisfaction with what they’ve done so far, time constraints, other issues)

In the future do you think you’ll go on to take more classes or enroll in further training? Explain.

2. Employment
   
   

10. Now let’s talk about jobs and work. Have you worked for pay at any time since 2013?

• <if yes, go to question 2 below>

• <if no>: When is the last time you worked? What is the reason you left that job?

• Can you talk a little bit about why you’re not working, for example, if you’re looking for a job, if you’ve decided to take time off for any reason, etc. (probe for job search difficulties, decisions to stay home for kids, health problems, time off for education/training etc.)

• How easy or difficult do you think it will be/it is to find a job? Explain. How about the kind of job you want? Explain.

• <if in any training> In what ways do you think participating in education/training will help you find a job? Explain. Do you think it will help you find the type of job that you want? Why or why not?

11. Tell us about any jobs that you have held since 2013, including your current job if you’re now working (probe for hours, stability of hours and schedule, wages, type of work for current or most recent job).

• What do you like most about your current/most recent job?

• What have you not liked about these jobs?

• Is this/are these the kinds of jobs you want or that you hoped to have? Explain.

• <if in any training> In what ways did participating in training/education help you in your current/most recent job? Would you say that participating in training/education helped you get the type of job you wanted? Why or why not?

• <if not currently employed> When is the last time you worked? What is the reason you left that job?

• Can you talk a little bit about why you’re not working, for example, if you’re looking for a job, if you’ve decided to take time off for any reason, etc. (probe for job search difficulties, decisions to stay home for kids, health problems, time off for education/training, etc.)

• <if in any training> In what ways do you think participating in training/education will help you find a job? What about the type of job you want? Why or why not?

12. What is your sense of the economy in your city or area? How do you see the economy as affecting the types of jobs available to you? Why do you say that?

13. Can you describe a typical day for me, from the time you get up to the time you go to bed. If your days are quite different, tell me about what you did yesterday and how that might be different from other days.

14. Tell me a little bit about the career goals you have for yourself.

• How has your work and education experience so far influenced these goals?

• What kinds of skills do you hope to/need to develop to help you meet your goals?

• What could hold you back from meeting your goals?

<For those in any training>

• <if employed> In what ways do you use the skills or other things you’ve learned on your job? What else could the program offer you to help you with your job?

• <if not employed> Did you expect that this program would help you get a job? Why or why not?

• <for all>In what ways has your participation in this/these program helped you with achieving your career goals? What else do you think you need?

C. Living Arrangements

15. Since we last talked to you in 2013, how many places did you live?

• <if 1, go to question 2 below>

• <if more than 1> What happened that you moved? When was this? (if multiple moves, ask about circumstances and timing of each)

• Where did you move to? (if multiple moves, ask about each) How was this place different from where you lived before? Explain

• How do you think your move/moves affected you in terms of work, school, or other parts of your life? Explain

16. Who lives with you? You don’t have to tell me their names, just their relationship to you.

• Has this changed since we last talked to you in 2013, including people who might have only lived with you a short time? Explain

• Have you had any changes with your family, for example, getting married, having a child, losing a family member? Explain.

D. Assistance and Supports for Education and Training

17. What are some of the other responsibilities and things in your life that keep you busy? How easy or difficult is it to manage these responsibilities? Why?

• <If enrolled in program>: What was it like to fit [this program] into the rest of your life.

• <if not enrolled> What do you imagine it would be like to fit in an education/training program into the rest of your life? Explain. <skip to question 18>

<For those who have been/are in a program>:

18. Have/did you encounter any unexpected challenges? (Probe for issues such as: childcare, transport, difficulty of class, being able to afford tuition or supplies, etc.)

19. What are/were sources of support that you can rely on to help you balance the demands of home with the demands of [this program]?

20. What about assistance and support from any instructors or staff in the program/s you were in?

• How often did you meet with program support staff? (e.g., on a set schedule, on demand, etc.)

• How easy was it to meet with advisors case managers, or other staff?

21. What kinds of things did you talk about when you meet with program support staff? Can you give me an example of what you talked about the last time you met with staff?

22. To what extent do you feel that program staff/instructors understood the fact that you and/or your classmates have responsibilities/competing priorities outside of class? Why do you say that?

23. How well do you know some of the other participants? Did you ever study or do other things together? (probe for extent to which R is part of a group or participating more solitarily) Are you still in touch with any of them, with staff X? Explain (probe for how those connections are maintained and used)

24. Overall, what would you say have been your biggest challenges to participating in this program? Explain.

E. Family and Friends Support

25. In general, how supportive are your friends and family regarding your career and other goals? Explain

• <If attended program> How did your friends and family feel about you participating in [programs referenced above]? Why do you say that? (probe for different types of reactions, especially if R has children, how much emotional and other support they’ve received)

• <if completed program> Did your friends’ and family’s feelings change at all during the time you participated? How so? Why do you think that was?

• <if R has children> What kind of goals do you have for your children in terms of education, work, or anything else that is important? Explain. Is there anything you/they are doing to work toward those goals? Explain.

26. How easy or difficult is it for you and your family to manage financially? Why do you say that? (probe for debt, use of public programs to help make ends meet)

• <if participated in program> In what ways did you participation in [program referenced above] affect your financial situation? Explain.

• How different is your financial situation compared to what it was in 2013? Explain

F. Looking Forward

27. As you may know, you are part of a national evaluation of programs that are trying to help individuals reach their career goals. What would you tell policy makers about the types of services, programs, or other changes that people like you need in order to reach those goals?

28. Given everything we’ve talked about today, what else do you think would be useful for the researchers to know about your experiences?

Thank you for participating in this interview. We appreciate all of the time you have taken to help us understand your experiences.