Ssb Il Rise Orb_5.13.13 Final

Permanency Innovations Initiative: Phase 2

OMB Information Collection Request

0970 - 0408

Supporting Statement

Part B

May 2013

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

Project Officer:

Maria Woolverton

B1. Respondent Universe and Sampling Methods

DCFS

The sample for the Illinois Department of Children and Family Services (DCFS) evaluation includes 684 youth from the universe of those ages 11-16 who are placed in traditional, relative, and specialized foster homes throughout the state of Illinois who, upon reaching the 2-year anniversary of entering foster care, are experiencing mental health symptoms and/or have had at least one placement change. DCFS youth will meet the age criteria for participation in the program on a rolling basis. Thus, DCFS will project cases 1-2 months ahead that will soon meet the established eligibility criteria based on age and time in foster care. DCFS will confirm the youth’s eligibility by examining administrative data on the youth’s current mental health and trauma-related symptoms and previous placement changes. DCFS will then consult the caseworker to verify the appropriateness of the youth to participate in the study and will obtain consent from the DCFS Guardian, who has legal authority over all children in foster care in Illinois, for each individual youth to participate in the intervention and the data collection. Biological parents eligible to participate in the study include those: (1) who are the parent of a program-eligible youth and (2) for whom DCFS is still pursuing reunification of the youth and parent. DCFS will make the determination as to the viability of reunification between the youth and the parent. In the rare event that it is not the biological parent but a legal guardian or an adoptive parent who DCFS is seeking to reunify with a study youth, this parent will be eligible to participate in the intervention and data collection. Foster parents are eligible to participate in the study if they are the current caregiver of a program-eligible youth.

DCFS will use a computer program to randomly assign youth to treatment or services-as-usual groups, half in each group. DCFS will assign the foster and biological parents to a condition based on the youth’s assignment. All youth and foster and biological parents will be asked to assent/consent to data collection.

A Westat statistician computed the power analysis using the following assumptions:

1. We anticipate that 684 youth will enroll in the evaluation over a 19-month period (based on historical analyses and the expected timing of securing OMB clearance). Thus, the nominal sample size per treatment or control group is 342.

2. There will be two data collection points for all youth in study: start of enrollment in the study (baseline) and a 6-month follow-up.

3. We expect after 1 year post-baseline, 30 percent of the control group and 40 percent of the treatment group will have exited to permanency. Therefore, the effect size is 10 percent.

4. We assume that some clustering will occur at different levels (e.g., foster home level, sibling level, caseworker level, etc.) in the sample. Thus, we computed the power analysis using a range of design effects including 1 (no clustering, all children are independent), 1.2, 1.5, 2 and 2.5.

The findings suggest that we would have over 80 percent power if we end up with a minimal amount of clustering (1.2 design effect) in our sample. It is also important to note that even if the sample includes a moderate amount of clustering (1.5 design effect), we would still have nearly 80 percent power.

Table B1. DCFS Power Analysis Results
Design Effect	Effective Sample Size per Group	Power
1.0	342	0.916
1.2	285	0.867
1.5	228	0.794
2.0	171	0.688
2.5	137	0.605

RISE

The target indicator for RISE’s ORB intervention is for more than 50 percent of trained workers to show improved competency from pretest to posttest. Assuming that the respondents are statistically independent (i.e., represent a random sample), N=470, and the response rate is 70 percent, then at least 58.0 percent would have to show improved competency to conclude, with 80 percent power (traditionally, the desired power), that more than 50 percent achieved improved competency. In other words, if the proportion of workers that show improved competency from pretest to posttest is at least 58.0 percent, the probability is 80 percent that RISE achieved its target indicator: that over 50 percent had improved competency.

RISE believes they will achieve a 70-percent response rate, but we also conducted a power analysis with a response rate of 50 percent in order to see what proportion would have to show improvement at that response rate. In that case, a slightly higher percent (at least 59.5 percent) of the staff would have to show improved competency to conclude, with 80 percent power, that over 50 percent had improved competency. In other words, if the proportion of workers that show improved competency from pretest to posttest is 59.5 percent, the probability is 80 percent that RISE achieved its target indicator: that over 50 percent had improved competency.

Below is a table of power versus proportion of staff with increased competency. N total = 329 corresponds to 70 percent response rate, and N total=235 corresponds to 50 percent response rate.

Table B2. RISE Power Analysis Results
Proportion	N total	Power
0.570	329	0.713
0.570	235	0.525
0.575	329	0.772
0.575	235	0.586
0.580	329	0.824
0.580	235	0.645
0.585	329	0.867
0.585	235	0.702
0.590	329	0.903
0.590	235	0.753
0.595	329	0.931
0.595	235	0.800
0.600	329	0.953
0.600	235	0.841

B2. Procedures for Collection of Information

DCFS

Informed Consent for Biological Parents: DCFS will send a study FAQ letter to the biological parents (see Attachment C2, DCFS Biological Parent FAQ Letter and Consent). The letter will include informed consent information and will inform potential respondents that if they do not contact DCFS by a certain date, DCFS will release their contact information to Westat so that the study team can tell them more about the study. After the target date, DCFS will provide Westat the Parent Study Contact Form information (see Attachment C3, DCFS Biological Parent Study Contact Form) for all the parents who provided passive consent for their information to be released to Westat. The contact information will be uploaded to Westat’s secure server in a report generated by the DCFS database system. A data collector will then call the parent to discuss the study, review the study consent form, and answer any questions about the study (see Attachment C2 for the DCFS Biological Parent FAQ Letter and Consent). During the consent process, the data collectors will ask prospective respondents to separately consent to: (1) participation in the phone interviews and (2) Westat merging their interview responses with DCFS administrative data on their family. Parents can decline consenting to the latter and still participate in the interviews. The data collectors will audio record all verbal agreements to participate.

Data Collection for Biological Parents: We will invite program-eligible biological parents to participate in two 15-minute telephone interviews: at the start of their enrollment in the study and 6 months later. The interviews will assess their abilities to regulate their emotions and behaviors and the quality of their support system. In the rare event that it is not the biological parent but a legal guardian or an adoptive parent who DCFS is seeking to reunify with a study youth, Westat will invite this parent to participate in the data collection using identical instruments. The instruments administered during both biological parent interviews are the Abbreviated Dysregulation Inventory and the Social Provisions Scale (see Attachment C4, DCFS Biological Parent Interview). Both are standardized instruments.

Informed Consent for Foster Parents: Once a youth is randomly assigned to a treatment or control condition, DCFS will send an FAQ letter and passive consent form to the youth’s foster parent (Attachment C5 shows the DCFS Foster Parent FAQ Letter and Consent Form). The FAQ letters will include informed consent information and will inform potential respondents that if they do not contact DCFS by a certain date, DCFS will release their contact information to Westat so the study team can tell them more about the study. After the target date, DCFS will provide Westat the contact information of all the parents who provided passive consent for their information to be released to Westat. The DCFS Youth and Foster Parent Study Contact Form (Attachment C6), which also includes the contact information of the youth’s caseworker, will be uploaded to Westat’s secure server in a report generated by the DCFS database system. If the foster parent does not agree to their contact information being released to Westat, DCFS will not release their contact information and the foster parent will not participate in data collection. If the foster parent passively agrees to their contact information being released to Westat, DCFS headquarters will upload the youth and foster parent’s name and contact information to Westat’s secure server in a report generated by the DCFS database system. A data collector will then call the foster parent to review the FAQ and determine their interest in participating in the study. Then the data collector will schedule a date to meet with both the youth and the foster parent during a visit to the home. Once at the foster home, the data collector will meet with the potential respondents individually to review the study assent/consent form, answer any questions, and initiate the interview for those who agree to participate (see Attachment C7, DCFS Foster Parent Interview). In addition to consenting to the interviews, the data collector will seek consent from foster parents for Westat to merge their interview responses with DCFS administrative data on the youth’s family. Foster parents can decline the latter and still participate in the interviews.

Data Collection for Foster Parents: Program-eligible foster parents will be invited to take part in a 45-minute Computer-Assisted Self-Interview (CASI) session at the start of their enrollment in the study and 6 months later. A data collector will set up the CASI and be available to assist the parent as needed throughout the CASI session. The CASI includes questions that ask about the youth’s ability to regulate their emotions and behaviors from the parent’s perspective, their parenting skills in responding to youth’s emotions and behaviors, and the quality of their support system. The instruments administered during the foster parent interview are shown in Attachment C7 (DCFS Foster Parent Interview) and include the Abbreviated Dysregulation Inventory-Parent Version, Social Provisions Scale, and the Parenting Practices Chicago Survey- Parent Version. These are all standardized instruments. (See Attachment C10 for the CASI-ACASI Measures and Instruction Screens.)

Informed Assent for Youth: The DCFS Guardian, who has legal authority over all children in foster care in the State of Illinois, will need to provide consent for a foster child under the age of 18 to participate in both the intervention and the data collection. As DCFS identifies eligible youth, they will seek individual consent from the Guardian to enroll each youth in the study. This approval will occur prior to randomly assigning a youth to treatment or control conditions. As previously described, DCFS will send the DCFS Youth FAQ Letter and Assent Form (Attachment C8). The FAQ letter will include informed consent information. A data collector will then call the foster parent to schedule a time to meet with the youth and tell them about the study. Once at the foster home, the data collector will meet with the youth to review the DCFS Youth FAQ Letter and Assent Form, answer any questions, and initiate the baseline interview for those who agree to participate.

Data Collection for Youth: The youth interview (see Attachment C9, DCFS Youth Interview) will be a 45-minute Audio Computer-Assisted Self-Interview (ACASI) session administered at the start of their enrollment in the study and 6 months later. A data collector will set up the ACASI and be available to assist the youth as needed throughout the ACASI session. The ACASI includes questions that examine the youth’s trauma-related and mental health symptoms, their ability to regulate their emotions and behaviors, quality of their relationship with adults, and their capacity to form and maintain relationships. The instruments administered during the youth interview include three standardized instruments (Abbreviated Dysregulation Inventory, Youth Social Support Scale, and the Trauma Symptoms Checklist for Children) and two brief measures modified from non-standardized instruments (the Biological Parent Contact with Youth Instrument and the Youth Emotional Permanency Instrument. (See Attachment C10 for screen shots of the ACASI administration.)

If the foster parent agrees to the release of their contact information to Westat, the youth interview will take place at the youth’s foster home. If the foster parent does not agree to the release of their contact information to Westat, the youth interview will not take place in the youth’s foster home. Instead, the data collector will contact the youth’s caseworker to facilitate scheduling of a meeting with the youth to discuss the study and if the youth assents, have the youth complete the ACASI session. The interviews will occur in a private room at the caseworker’s office.

RISE

Foster Care Staff Consent and Data Collection: Participants in ORB training who were randomly selected to participate in data collection will receive a consent form immediately prior to the training (see Attachment D2, RISE Pretest Consent Form). Staff who consent will complete the RISE Staff Survey (see Attachment D3) to measure LGBTQ competency. Pretest and posttest materials will be labeled with unique IDs prior to data collection, in order to ensure the privacy of participants and to enable follow-up for the posttest. After completing the survey, the data collector will hand out a packet with the RISE Posttest Consent Form (Attachment D4), the RISE Staff Survey (Attachment D3), and a postage-paid envelope addressed to Westat. Staff will be instructed to complete the scale, if they consent, 1 month after their last ORB training session and then use the provided addressed envelope to return the form back to Westat. RISE program staff will send posttest completion reminders to staff near the 1-month anniversary of training completion (see Attachment D6).

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Expected Response Rates

The overall response rate in Kansas (study in operation since 9/1/2012) is about 76 percent, and we expect comparable response rates for DCFS. RISE is aiming for an overall response rate of 70 percent for staff completion of pretest and posttest, and will use tokens of appreciation and reminders to achieve that response rate.

Dealing with Nonresponse

DCFS

In order to increase buy-in, DCFS has enlisted the feedback of key DCFS staff, private service providers, and community stakeholders in deciding on the particular intervention and deciding on instruments and evaluation procedures. DCFS currently operates under a family-centered, trauma-informed, strength-based model of child welfare practice, and expects that this foundation also will increase the buy-in by DCFS staff and private service providers.

To overcome possible nonresponse among interview respondents, the study is using highly trained data collectors who can explain areas of the instrument that may confuse respondents, who know how to re-approach a hesitant participant, and who are available to make multiple attempts at data collection. In addition, respondents will be reassured as to the privacy of their information, and interviews will be conducted at times and places convenient to the respondents.

RISE

For evaluation of the ORB component, the burden of staff participation is low (15 minutes for completion of the staff survey), and participation made convenient by administering the pretest immediately before the first training session, and both giving the posttest to staff at the first training and later mailing them the posttest in case they misplaced the copy they received at the training. Further, during the needs assessment phase, a focus group of foster care staff indicated that the need for LGBTQ competency training was great. RISE anticipates that staff will be eager to participate and will be equally eager to learn whether the training produced demonstrable improvements in competency.

Maximizing Response Rates

Each site is offering a token of appreciation to respondents to encourage them to participate in the data collection (see Section A.9). DCFS is offering a token of appreciation to family participants and RISE is offering a token of appreciation to staff participants.

B4. Tests of Procedures or Methods to be Undertaken

Both grantees are pretesting their instruments and procedures under OMB generic clearance 0970-0355, received on Oct. 4, 2011. Any difficulties we encounter with respect to our procedures, materials, or instruments will be discussed with the sites and the evaluation leadership, and suggested revisions to the evaluation plans will be outlined.

B5. Individual Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The team is led by Maria Woolverton, project officer; Andrea Sedlak, project director for the PII evaluation; and Mark Testa, principal investigator for the evaluation. Additional staff consulted on statistical issues at Westat include John Rogers and Barnali Das, senior statisticians.

7