Appendix C
Process Study Protocol
Multi-site Evaluation of Foster Youth Programs
Site Visit Protocol Memorandum
This memorandum 1) identifies the types of respondents we plan to interview during our site visits, 2) describes the data collection methods we plan to use, 3) lists the major topics we plan to address through the site visits, and 4) provides a draft interview protocol with the specific questions we will ask of site visit respondents.
The process analysis, of which the site visits are the primary data collection component, is intended to address the following key research questions:
What features of ILPs are likely to influence their impact on their youth clients? How are the programs structured and administered? What internal evaluation or quality assurance procedures exists? In programs that are privately administered, how do public agencies monitor and assess private agency operations?
How might external community or state‑level variables contribute to outcomes achieved by program youth? How do programs interact with other public and community-based agencies that serve youth?
How do the ILPs identify and implement their logic models? What are the logic models of the selected programs? What specific program features are likely to influence youth outcomes? What specific youth outcomes do program officials believe they have the greatest likelihood of improving? To what extent are programs implemented as specified in these logic models? In sites with multiple programs, how consistently are programs implemented?
Who is being served? How do programs inform potential referral sources of the services they provide and the youth they serve? How do programs determine whether children are eligible to receive services? How, if at all, do programs prioritize among youth who are eligible? How, if at all, are youth who participate in the evaluation different from other youth with the goal of independent living?
How are these services implemented? How are services delivered and by whom? What are the qualifications and training received by key staff? How are the needs of youth assessed? Where are services provided? Is there written documentation about program procedures and operations (e.g., training manuals).
How do programs engage youth? How are youth involved in program design, service delivery? How do programs seek input and/or feedback from youth?
What services are provided? What are the frequencies, duration, and intensity of services provided to the experimental and control groups? Which youth receive what service or service mix? Are services provided in a timely manner? For what services and reasons are youth referred? To which agencies are youth referred?
What are the barriers to implementation? How can they be ameliorated or eliminated? How might the local context of the program affect its success? What barriers do programs face in implementation and how do program staff attempt to address these barriers?
To what extent might these programs be adapted to other locales? What were the social, demographic, economic, fiscal, and political factors that impact on the replicability of the interventions or influence the implementation of the project? To what extent could selected programs be adapted to other locales?
Is the integrity of the random assignment process maintained throughout the study? Do children assigned to the control group access services designed for the experimental group at any time during the study period? Are alternative services consistently (or inconsistently) offered to youth who are assigned to the control group?
Respondents
Based on our preliminary information gathering, it appears that many different types of respondents are involved in the independent living programs we will evaluate. The types of key stakeholders that we will want to interview will vary considerably from site to site. Some of the programs we may evaluate will be publicly administered, while others will be run by a private agency or agencies. Locations will also have different programs that may serve youth in control groups. Thus, the respondents we interview and the content of these interviews will vary considerably across the sites.
We have identified several types of respondents we plan to interview, if applicable, in each site:
Independent living program administrators: Whether the independent living program is public or private, we plan to speak with respondents knowledgeable about the development and day-to-day administration of the program. In addition to the program director, we plan to meet, as appropriate, with task force or advisory group members, budget analysts, training specialists, and agency researchers.
Independent living program supervisors, workers: We plan to meet with the front-line workers and supervisors who interact with foster youth. These interviews will focus on the referral process, service delivery, characteristics of program clients, and interaction with other youth providers.
Youth: We plan to meet with youth who receive services or support from the independent living program. We plan to ask youth for their assessment of the program including how they became clients of the program, the services they received, how services received compare with their needs, and how services received compare with services available through other agencies or providers. So as not to contaminate the sample of youth randomly assigned, we will conduct focus groups with youth who have already been served by the independent living program for a period of time and are not in the evaluation.
Public Child welfare administrators: We will interview child welfare administrators knowledgeable about the development and/or operation of the independent living program. We are particularly interested in understanding how the independent living program fits within any broader strategy for assisting emancipating youth.
Public Child welfare workers: Interviews with front-line child welfare workers will focus on their interaction with the independent living program including the referral process, the types of youth they refer to the program, and the program services they see as most useful for their clients, plus the challenges they identify in working with the program.
Local advocates: In some sites, it appears that local advocates (including youth advisory boards) were heavily involved in the development of the independent living program and in such cases we will interview representatives from these organizations to understand the early history of program development. In addition, we will ask local advocates to assess the broader system for assisting vulnerable youth.
Local service providers: We plan to contact local service providers including other r ILP or youth oriented programs, who may refer youth to the independent living program, receive referrals from these programs, or serve youth that may be in the control group.
We plan to hold semi-structured interviews (individual as well as group) with all respondents. Youth interviews will be conducted via focus groups. Youth will receive fifty dollars for participating in the focus groups. If the independent living program has many workers, we also plan to conduct focus groups with program workers as well. During the site visits, we will also observe the operation of programs serving youth in both the control and experimental groups. The site visits will be supplemented by an analysis of written documentation including program manuals, reports, curricula, and youth assessments.
Below is an outline of the topics we will focus on during our site visit interviews.
Early history
Mission and goals
Service delivery philosophy
Target population
Changes in program design
Technical assistance received
Lessons learned
Contextual factors influencing program implementation
Funding
Eligibility requirements
Description of the youth served
Staffing
Youth engagement
Referral process
Collaboration with other independent living providers
Ongoing strategic planning
Changes in program operation, changes anticipated
Services provided
Financial support
Service referrals
Hours of operation
Reporting requirements
Program strengths and weaknesses
Future plans for the program
Unintended consequences
Lessons learned
Draft Site Visit Protocol
Below we list the detailed questions we plan to ask during our site visits. Following each question we identify the respondents (the numbers in parentheses refer to the respondents number identified above) we expect to address the question. It is important to note that we plan to use the information we obtain prior to our visits to design site-specific protocols. At the end, we identify the questions we plan to ask during the youth focus groups. These questions are broader, allowing youth more flexibility in discussing their experiences and enhancing group discussion. We are hoping to collect considerable written documentation about the programs prior to our visits that will answer most of the factual questions we have leaving the visits for documenting different stakeholders perspectives on how and why the program has evolved and the lessons that we can draw from their experience.
Program Planning
1. Describe the recent history of the program and how it got started. (1, 4)
Who was instrumental in establishing the program? Describe the establishment of the program.
When was the program established ?
Is the program based on (or was it influenced by) an established model? If so, describe the model that the program is based on and why it was chosen?
What stakeholders were involved in the planning and development of the program (Child welfare administrators, caseworkers, other agency administrators serving youth, advocates, others)? To what extent, if any, were youth involved in the early planning of the program? How were different stakeholders involved and what did they envision the program purpose and goals to be? To what extent did the different stakeholders involved in the program planning agree on the mission of the program?
Was anyone left out who in hindsight should have been involved?
To what extent are those who were involved during the planning stages still involved in the program? How so?
What, if any, obstacles arose during the planning stages and how were they resolved?
How long did it take to get the program up and running? When did the program start serving clients? In retrospect was more time needed to establish the program before serving clients?
What obstacles arose in the program development phase? How were these handled?
{Obtain any written documentation about early planning stages including task force reports, meeting minutes, or interagency memos}
2. What is the overall mission of the program? What are the program’s goals and how is the program designed to accomplish these goals (what is the logic model)? What are the issues or problems that this program was intended to address? {Obtain written mission statement or program objectives} (1, 4)
Among the goals of the program, which are of highest priority or most often the goal for the youth you serve?
What was the existing service delivery system for youth and how was this program envisioned to enhance this system (existing services, gaps, assess problems)? What was making it difficult for these issues to be addressed by the child welfare agency? (1, 4, 6, 7)
What specific youth outcomes do program officials believe they have the greatest likelihood of improving?
In what areas has program had the least impact? Why?
3. Has the program articulated a specific service delivery philosophy? If so, please describe. Does the program intend to provide long-term or short term support or both depending upon the youth? How, if at all, does the program seek to involve foster parents in helping youth achieve independence? (1, 2, 4)
{Obtain written service delivery philosophy if available}
4. Who is the intended target population for this program? How, if at all, has this changed since the program began? (1, 4)
How was the target population determined?
How, does the program assess the needs of the target population? (e.g., surveys or focus groups of caseworkers or youth)
5. What major decisions went into the design of the program? (1, 4)
a. How did the agency determine the scope of the services offered?
b. How did the agency determine the eligibility for services?
c. How did the agency determine the duration of services?
6. How has the design of the program changed or evolved over time? How has the scope of the program changed? How, if at all, did the initial program implementation differ from the initial vision for the program and why? What challenges did the program face as the program expanded, evolved? (1, 4)
7. What technical assistance or guidance did the program receive during planning (e.g., from other programs, from state agency officials, from foundation sponsors etc.)? (1, 4)
8. What lessons did you learn in the development of this program? What advice would you pass on to those interested in developing similar programs in other localities or states? (1, 4)
Are there any contextual factors you believe affected the planning and implementation of the program or that now affect the program’s current success (e.g., history of interagency collaboration, political factors, economic influences, etc.)? (1, 4, 6, 7)
Has an evaluation be done on this program?
Operational Aspects
1. How is the program structured and administered? Where is the program housed administratively? If the program is part of a larger agency, how does it fit within this larger organizational structure? If the program has multiple sites, how consistently are programs implemented? (1, 4)
a. What are the roles, responsibilities, authority of the program director and other key program officials?
b. Does the program have an advisory board or other body that helps oversee the operation? If so, who is on the advisory board and how/why were they chosen? What roles does the advisory body play and how often do they meet? What have been the key issues addressed by the advisory board?
c. {For private agencies} What role, if any does (did) the child welfare agency have in program design and operation beyond referring clients? (If there is no/little interaction, why not?)
What other service providers address the needs of foster youth and how, if at all, does the program interact with them? What agreements, if any has the program established with other agencies? (1, 4, 6, 7)
Describe the partnerships that you may have cultivated with other local agencies that serve youth (e.g., mental health; juvenile justice; substance abuse treatment; etc.). How effective are these collaborations? Do you have formal partnerships with them? (e.g., MOUs).
Describe the relationship with county/state child welfare agency. Are child welfare workers supportive of this program? Do they refer on a regular basis? Do you hold ongoing meetings/trainings to keep them apprised of program success? What factors have led to the successful collaboration with them? What could make the relationship better?
Is this program viewed as part of the continuum of child welfare services for foster youth? Should it be seen that way?
{Obtain organizational charts, interagency agreements, …etc.}
2. How is the program funded? (1)
a. What different funding streams does the program rely on and what do the different funds pay for? How was the program funded prior to Chafee? How did the infusion of Chafee funds affect program design, service delivery?
b. What is the total budget for the program and how are funds allocated between different program functions?
c. Is the program able to blend funds from multiple sources or must the different funds be kept separate?
d. How, if at all, does the program’s budget constrain the types of services it can provide or the number of youth that it can serve? If the program was given additional funds, how would they be spent?
e. what is the stability of funding?
{Obtain annual report or budget documentation if available}
3. What are the eligibility requirements for the youth participating in the program? How are eligible youth located or identified? Do requirements vary depending upon the service or types of support? Under what circumstances will programs refuse to serve referred youth and to what extent does this vary? What are the inclusionary and exclusionary criteria for becoming a client? What causes a client to be discontinued from the program? (1, 2)
{Obtain written documentation if available including screening forms used}
4. When space is limited, how does the program prioritize among eligible youth? Does the program maintain a waiting list (for certain services)? If so, explain the waiting list procedures? Do most/all wait listed youth eventually receive services if they still want to? How long is the typical wait for services for youth who eventually get services? (1, 2)
5. Describe the youth you serve. How, if at all, do youth enrolled in the evaluation differ from other youth served by the program? (1, 2)
Ages, sex, presenting problems, education status?
What are their experiences with the child welfare agency? Placement status?
How do the youth served by the program differ from other foster youth with the goal of independence? (4, 5, 6, 7)
6. How is the program staffed? What are the qualifications, responsibilities of staff? (1, 2)
How are youth assigned to a specific worker? To what extent do programs seek to match youth with workers based on sex, race, or other factors?
How many youth does each staff serve (i.e., caseload size)?
What are the roles of staff from the child welfare agency?
What training is provided to staff (pre service, ongoing)? Describe the ongoing supervision that staff receive.
What background and training do staff have?
Has the program had difficulty recruiting/retaining qualified staff? Why or why not? What is the turnover rate? Were staff newly hired for this program or were they reassigned/retrained? Are staff representative of the clientele they serve (race, sex, community, etc.)? How do staff interact with one another and with staff from other agencies?
7. How do youth become clients of this program? (1, 2, 4, 5, 6, 7)
What is the referral process? Do the courts refer, mandate program services? Can youth self-refer? Frequency of self referrals?
How do referring workers identify youth that may be appropriate for the ILP intervention being evaluated? To what extent do they consider a youth’s commitment to following through with the program? Are formal assessment tools used? How much discretion do staff have? Are there any categorical exemptions?
What information about the youth, if any, do child welfare workers (other referral sources) typically provide?
What types of outreach does the program conduct? Has the program experienced any difficulty in identifying, recruiting, and engaging youth to participate in the program?
Are youth typically referred to the program generally or for specific services? If for specific services, for which services are youth most commonly referred?
Which types of youth do workers from these agencies feel are most appropriate for referral? To what other programs do they refer youth?
How, if at all, do child welfare agency staff remain involved with youth they refer to the independent living program (e.g., ongoing monitoring, case management, protective supervision)?
How knowledgeable are child welfare staff about the services offered by the independent living program? How does the program inform child welfare staff and other referral sources about available services? What do they see as the greatest strengths, limitations of the program?
{Obtain referral forms, information provided to perspective referrers, outreach materials, etc.}
8. How does the program seek to engage youth? (1,2)
To what extent does the program have difficulty convincing referred youth to accept services? How does the program initially work with youth to inform them of available services and to gain their interest?
How does the program seek input from youth on program operations/service delivery issues?
How, if at all, do youth participate in operations/service delivery?
How does the program seek feedback from youth?
9. What mechanisms are in place to facilitate ongoing communication and collaboration with other providers serving foster youth? What existing services, service providers did the program anticipate relying on? (1, 6, 7)
10. What, if any, ongoing strategic planning does the program conduct? What types of issues are dealt with? What mechanisms are available for ongoing planning? (1)
11. How has the operation of the program as we have just discussed changed over time and why? How do you anticipate the operation of the program changing in the future? (1)
1. How are the service needs of youth determined? (1, 2)
Describe the intake and assessment process? What specific assessment tools and instruments are used? Are child welfare workers consulted? When does this occur initially and how often, if at all, do follow-up assessments occur? How long after a youth is referred to the program does intake/assessment occur?
Does each youth have a treatment plan? Does the youth participate in generating this plan? Anyone else ? (e.g., foster parent; CWW; mentor; teacher?)
Do youth request specific services?
How do program workers seek to engage youth? How do workers contend with resistant clients?
How, if at all, does the program involve foster parents or other adults who are important to the youth?
{Obtain forms used for assessment}
2. Describe the services your program provides directly to clients. (1, 2)
How many youth can the program serve at any one time, i.e., what is the service capacity of the program? Does this vary by type of service offered?
Do youth typically receive one or multiple services?
Are services provided on-site or off-site? If on-site, by whom? If off-site, where?
How often can services be provided? What is the typical length and frequency of service delivery? Do many/most youth only have contact with the program a few times or is it more common for youth to have an ongoing relationship with the program? What service gaps are perceived?
How do you define successful completion of the program? How do you assess this?
What is your rate of successful graduates of the program? What factors of your program help determine this success rate? How would you define a youth client who is prone to be a successful graduate?
Conversely, what is the failure rate? What factors have contributed to this rate?
Which services have you found to be the most difficult to provide? Why? (e.g., funding, staffing, planning time). How does funding affect the types of services offered?
How, if at all, has the variety of services offered changed over time and why?
Are youth expected to pay a fee for any service (e.g., housing)? Are you expected to match savings?
Do you provide follow-up services to clients? What type? For how long?
Do you have any information on the long-term outcomes of former clients? Do you maintain contact with clients?
What other factors affect client outcomes?
3. What, if any, financial payment does your program provide to youth? (1, 2)
Ongoing payment? How does this compare to the monthly assistance provided to foster parents for children of similar ages?
For how long can youth receive assistance?
Is emergency/one-time assistance provided (moving expenses, education)? Do certain youth receive priority for this type of assistance?
4. What services does the program provide that youth cannot access elsewhere? (1, 2, 3, 4, 5, 6, 7)
What aspects of the services do youth find most, least valuable?
Are other local organizations providing similar services?
Are additional or more intensive services needed? What do these include?
Are any new services currently being developed?
5. For what services does your program refer youth to other agencies? (2, 5)
For what services are youth most often referred?
Describe the referral process (i.e., do program workers help arrange for youth to get services, ensure they do, check up on them etc.).
6. {If appropriate} What are the program’s hours of operation? (1, 2)
How was this determined?
Is any after-hours assistance available?
What barriers, if any, may youth have in accessing services (e.g., transportation)?
Assessment of the Program
1. What types of information does the program obtain about youth and how? Does the program maintain data on services provided, outcomes of youth? How is data stored, analyzed, used? (1)
2. How is the program monitored externally (e.g., by the child welfare agency) What reporting requirements does the program have? (1, 4)
3. How does the program assess its effectiveness? (1)
Does the program assess client satisfaction? If so, how?
Does the program have quality assurance mechanisms?
How effective do you feel the program has been? What are the program’s major strengths?
Have there been formal evaluations? What evaluation strategies have been used, and at what points? By whom? How has feedback been used by the program? What data sources are available for self-evaluation, or evaluation by others?
What challenges does the program face? How have these changed over time?
4. What are the plans for the program over the next few years? What does the program hope to accomplish, improve? Expand? (1)
5. Have there been any unintended consequences, either positive or negative, of the implementation of the program? (1, 4, 6, 7)
6. What lessons have been learned to date about the operation of an independent living program that may be useful to other communities interested in establishing a program? (1)
How were persons who refer youth to the independent living program informed of the random assignment evaluation? What concerns did referring workers raise about the evaluation? (5, 7)
How, if at all, did implementation of the evaluation affect the referral process? (1, 2, 4, 5)
How, if at all, did implementation of the evaluation affect the intake/eligibility determination process? How were intake workers informed of the random assignment evaluation? What concerns did they raise about the evaluation? (1, 2)
Was the program permitted to make exceptions to the random assignment process? How were exceptions determined and how were youth excepted different than youth enrolled? (1, 2, 4, 5)
How is the program ensuring that youth enrolled in the control group did not receive services intended only for the experimental group? (1, 2)
Youth Focus Group
Thank you very much for agreeing to participate in this focus group today. I’m ________ and this is ______ from the Urban Institute/Chapin Hall Center for Children, a non-partisan research organization in Washington, DC/Chicago. I will be leading this session and _____ will be assisting me and also taking notes.
We are conducting a study for the U.S. Department of Health and Human Services looking at how well programs prepare foster youth for life after foster care. We have chosen ____ as one of the program to study, and we will be visiting five other programs nationwide. Today, we would like to ask you some questions about your experiences with the program. We really appreciate your cooperation and hope that you will feel free to be open in our discussions.
We are taking notes and taping the session so that we can make sure to hear all that you tell us. However, everything you tell us is anonymous and confidential; we will never link your name to anything that is said here. Furthermore, we will not tell the program or your CWW any specific answers that you provide. In addition, we ask all participants to not discuss others’ responses to specific questions after you leave the focus group today.
I will be asking questions about your experiences, which are open to the whole group. There are no right or wrong answers here. It's okay if you don't agree with one another in your responses--we want to hear all sides. Also, you don't have to answer every question. Again, we are very pleased to have you here today.
Experimental youth
Intro/referral
1. Please introduce yourself by telling us your first name, your age, and how long you have been in foster care.
2. How did you first hear about this program? Probe: Did you choose/request to receive services from the program or did you child welfare worker encourage you to participate?
Services you received
3. Were you referred for/did you seek out the program for a specific service or for support more generally?
4. What services or assistance have you received (from the program) thus far and how well have they met your needs?
Probe: a. How were your service needs determined?
b. Duration of services
c. Accessibility of services (location, waiting list, pay for services)?
d. Unmet needs
e. Financial payment (regular or emergency, how much, how often)?
5. What services or assistance did the program refer you to other agencies for?
Relationship with Other Agencies
6. What other agencies have you received assistance or services from?
Probe: a. Ongoing relationship with referring agency, esp. child welfare and/or TANF (how has participation in the program changed relationship, requirements, services)
b. What services/assistance did you receive and how well did they meet your needs?
7. What services/assistance does the independent living program offer that you cannot receive from other agencies?
Assessment
What do you think are the strengths and weaknesses of the independent living program? Has the program sought out your opinion on how to improve services?
What services do you wish they had provided?
9. What are your plans for yourself after you exit foster care? How do you think the program can help you obtain these goals? What additional assistance could help? How do you feel about your future?
Control youth
Intro/referral
1. Please introduce yourself by telling us your name, your age, and how long you have been in foster care.
What programs have you heard about that provide services to foster youth? Probe: did you know about the independent living program (under evaluation) and did you want to receive services from the program?
Services you received
For what programs/services, if any, did your child welfare worker refer you?
What services or assistance have you received (from the program) thus far and how well have they met your needs?
Probe: a. How were your service needs determined?
b. Duration of services
c. Accessibility of services (location, waiting list, pay for services)?
d. Unmet needs
e. Financial payment (regular or emergency, how much, how often)?
Why have you not received services from programs you have heard about or were referred to?
Assessment
What do you think are the strengths and weaknesses of the programs available/you have received services from? Has the program sought out your opinion on how to improve services?
What are your plans for yourself after you exit foster care? How do you think the program available/you have received services from can help you obtain these goals? What additional assistance could help?
| File Type | application/msword |
| File Title | Alternative Kinship Care Protocol Draft |
| Author | Jacob Leos-Urbel |
| Last Modified By | KMalm |
| File Modified | 2006-07-28 |
| File Created | 2006-06-05 |