SSB CAPP-RISE CCT-admin-cost 6 12 14_clean

Permanency Innovations Initiative: Phase 3

OMB Information Collection Request

0970 - 0408

Supporting Statement

Part B

March 2014

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

Project Officer:

Maria Woolverton

B1. Respondent Universe and Sampling Methods

RISE CCT

For the CCT services, a DCFS caseworker with a 5-to-19-year-old LGBTQ or gender nonconforming child or youth receiving in home services or who is placed in out-of-home care, may refer the child or youth to RISE. Eighty¹ children and youth will participate in the evaluation;40 youth will be in the treatment group, and 40 youth will be in the control group. There are anticipated to be 65 youth ages 11-19 in the sample total (see SSA for the specifics regarding what interviews these youth will receive); and 15 children ages 5-10 in the sample total (see SSA for the interviews these children will receive. The evaluation contractor will receive administrative data on all 80 children and youth (see administrative data study).

The evaluation contractor’s statistician computed the power analysis using the following assumptions:

1. Eighty children and youth will enroll in the study over a 6-month period following OMB approval. Thus, the nominal sample size per treatment or comparison group is 40.

2. There will be three data collection points for all youth in the study including those who enroll towards the end of the enrollment period: start of enrollment in the study (baseline), midpoint qualitative interview, and a 1-year follow-up.

3. A one-sided test of differences is appropriate because RISE services should only have a positive effect on the target population’s outcomes.

4. Based on data mining conducted with a near sample, we expect the median survival time for the control group (which does not receive CCT) to be 2.9 years; we expect the median survival time for the treatment group to be 1.5 years.

5. We assume that some clustering will occur at different levels (e.g., foster care services office and treatment team) in the sample. Thus, the power analysis was computed using a range of design effects including 1 (no clustering, all children are independent), 1.2 (mild clustering), and 1.5 (moderate clustering).

The probability that differences will be detected is reduced as the amount of clustering increases and the length of time until follow-up decreases. The effective sample size becomes 33 per group with mild (1.2) clustering and the effective sample size is reduced to 27 per group with moderate (1.5) clustering. As Table B2 shows, to achieve a power of 80 percent, minimal clustering must be present and treatment group exits must be quicker than the projected 1.5 years.

Table B2: Power for detecting one-sided differences in survival time with design effects of 1.2 and 1.5 at RISE
Design Effect of 1.2:
Median Survival Time in Years for Treatment Group	Power for Detecting Difference
0.9	0.776
1.0	0.692
1.4	0.395
1.5	0.339
Design Effect of 1.5:
0.9	0.688
1.0	0.603
1.4	0.337
1.5	0.290

RISE ORB

The evaluation of the RISE ORB component was previously reviewed and approved by OMB (for additional details, please see the information collection request approved in August 2013 under OMB #0970-0408). Please see that OMB package for additional information regarding the power analyses for RISE ORB.

Cost Study

The cost study will collect activity log data from PII project personnel whose use of person-time (labor) and cost directly or indirectly relates to a census of active PII cases. The number of respondents to complete activity log surveys will vary across grantees based on the differences in the number of PII project personnel, caseload sizes, and ratio of supervisors to caseworkers. The number of respondents for four participating grantees is estimated at about 603, which is comprised of 27 focus group participants, 369 case workers that complete weekly logs, 117 supervisors that complete weekly logs, and 90 managers and administrators that complete monthly logs. An 85% response rate is expected for the three logs for each grantee.

We did not conduct a power analysis for the cost study because our analysis of the cost data will be descriptive – e.g., distributions, medians, ranges. We will not have a pre/post comparison or a comparison group and we will not calculate effect sizes.

Administrative Data Study

The respondent universe for the cross-site administrative data study includes all children in the child welfare systems in the grantees’ states. The 6- and 12-month data submissions comprise all of the universe. Data will not be sampled, and since the entire universe will be included, we will not be able to expand the sample. However, we conducted a power analysis in order to assess the power required to detect effects.

There are two tables, one for no clustering and one for highly clustered data. There is high power to detect differences of 5 percent (effect size). For the unclustered data, we will be able to detect differences of 2.5 percent with high power. We only have lower power when we have highly clustered data and effect sizes of 2.5 percent, which would be an extreme scenario.

Table B3: Power analysis for administrative data study, unclustered data (design effect=1, effective sample size per group=7,323)
Effect size	% exiting care within 2 years in the control group
	20.0%	30.0%	40.0%	50.0%	60.0%
10.0%	>0.999	>0.999	>0.999	>0.999	>0.999
5.0%	>0.999	>0.999	>0.999	>0.999	>0.999
2.5%	0.959	0.904	0.867	0.857	0.874

Table B4: Power analysis for administrative data study, clustered data (design effect=2.5, effective sample size per group=2,929)
Effect size	% exiting care within 2 years in the control group
	20.0%	30.0%	40.0%	50.0%	60.0%
10.0%	>0.999	>0.999	>0.999	>0.999	>0.999
5.0%	0.996	0.983	0.972	0.97	0.977
2.5%	0.648	0.541	0.493	0.482	0.502

B2. Procedures for Collection of Information

RISE CCT

Youth Data Collection: Youth ages 11-19 in both the intervention group and the comparison group will be invited to participate in data collection three times: (1) immediately after assignment to the intervention or comparison group (baseline²) to collect paper and pencil survey data, (2) six months after assignment to conduct a qualitative interview, and (3) one year post-baseline (follow-up) with the paper and pencil measures. After a youth is assigned to the intervention or comparison condition, RISE will forward the youth’s and his/her current caregiver’s contact information to the evaluation contractor. The evaluation contractor’s data collector will contact the current caregiver to arrange a home visit for the purposes of collecting data. At the youth interview (see Attachment B2), the data collector will administer the youth assent using the youth assent script. If the youth assents, the data collector will administer the interview.

When it is time for the six-month qualitative interview (see Attachment B3), a data collector will again contact the current caregiver to arrange a home visit for the purposes of collecting data. The data collector will administer the youth qualitative assent using the youth assent script. If the youth assents, a second data collector, specially trained to collect qualitative data, will administer the protocol via a webcam.

Procedures for the follow-up interview begin with verifying the contact information for the youth one year after the baseline interview. Then the data collector will contact the youth’s current caregiver to arrange a home visit for the purposes of data collection, the data collector will administer the interview using the same instruments as for the baseline (Attachment B2).

CCT Facilitators: For children ages 5-10 in the intervention group, a parent, attorney, or judge will provide consent for participation in the program and in data collection. The party responsible for consent will depend on the status of the child’s parental rights (terminated or not terminated) and on the child’s level of comfort with his or her parents knowing about his or her LGBTQ status. Once children are authorized for evaluation participation and receiving RISE CCT services, CCT facilitators will administer the CCT Facilitator Interview (see Attachment B4). For all children and youth in the intervention group (ages 5-19), the CCT facilitators will complete the CCT Facilitator Survey (Attachment B5). In addition, the CAFAS is administered as part of case planning, and RISE will provide the CAFAS data to the evaluators (Attachment B6). All these instruments will be administered both at baseline and 1 year later (follow-up).

Intervention Group: Permanency Resource and Current Caregiver Collection of Data: Permanency resources for youth ages 11-19 in the intervention group will be invited to participate twice in data collection for the purposes of collecting paper and pencil survey data. (A permanency resource is an adult involved in the CCT process who has expressed a willingness to provide a permanent home for the child or youth.) RISE will notify the evaluation contractor when a permanency resource has been identified; this person could be identified at any time during the CCT services, and their baseline and follow-up data collection times will depend on when they were identified. Once the contact information is available, the data collector will contact the permanency resource to arrange a home visit for the purposes of collecting data. The data collector will administer the permanency resource consent. If the permanency resource consents, the data collector will administer the interview (see Attachment B7 for the consent and interview). Procedures for the follow-up interview begin with verifying the contact information for the permanency resource approximately 1 year after the baseline interview. Then the data collector will contact the current permanency resource to arrange a home visit for the purposes of data collection, the data collector will administer the interview using the same instruments as for the baseline.

Current caregivers of intervention group youth ages 11-19 years will be invited to participate in data collection twice, after enrollment (baseline) for the purposes of collecting paper and pencil survey data, and again 1-year post-baseline (follow-up) on the paper and pencil measures (see Attachment B8). After no more than one month of CCT service provision to the youth, RISE will inform the evaluation contractor whether it is appropriate to involve the caregiver in data collection. If in RISE’s clinical judgment a caregiver is still hostile about the sexual orientation and/or gender identity of the youth in their care, the evaluation contractor will not invite the caregiver into data collection. If RISE authorizes contact, the data collector will contact the current caregiver to arrange a home visit for the purposes of collecting data. Then the data collector will administer the caregiver consent. If the caregiver consents, the data collector will administer the interview (see Attachment B8 for the consent and interview). Procedures for the follow-up interview begin with verifying the contact information for the caregiver one year after the baseline interview. Then the data collector will contact the current caregiver to arrange a home visit for the purposes of data collection and will administer the interview using the same instruments as for the baseline.

RISE ORB

Some procedures for the RISE ORB evaluation are slightly different from the previously approved OMB package (OMB #0970-0408, which received approval in August 2013). The pretest administration procedures will not change; data collectors will administer the hard-copy consent form and pretest instrument (Attachment B9a) immediately prior to the beginning of ORB training. However, the plans for the posttest now are that the posttest will be administered immediately after the ORB training concludes (Attachment B9b). A data collector or RISE staff member will administer the hard-copy consent form and posttest instrument (the instrument is the same as the pretest instrument, Attachment B9a). The posttest instrument will have a perforated bottom on which respondents will write their name and email address. Then the evaluation contractor will use this information to administer a web-based follow-up survey (Attachment B10) two months after training. The evaluation contractor will send email and print invitations (included in Attachment B10) to respondents asking them to go to the study URL (via a link in the invitation) to complete the follow-up survey. After the survey is completed, the incentive procedure will be initiated.

Cost Study

Cost study data collection includes: (1) preparation for a focus group of case workers and supervisors; (2) participation in a focus group to draft data collection logs tailored for the grantee; (3) post-focus group trial administration of logs; (4) case worker activity log; (5) supervisor activity log; and (6) manager/administrator activity log. The cost study research matrix is shown in Attachment C1.

Preparation for the focus group: We will first administer a written consent form to those who agree to participate in the focus group preparation, focus group itself, and post-focus group trial administration of logs (Attachment C2). The consent form will describe all activities involved in the focus group. We will review the contents of the consent forms with the prospective participants and have them sign the consent form prior to their participation. After informed consent is obtained and before the focus group meeting, focus group participants will be invited to a 30-minute, pre-meeting telephone conference. The focus group process will be described and participants will be asked to review a preliminary listing of casework and supervision activities compiled by the PII Cost Evaluation Team (Attachment C2). The preliminary listing will be sent via email to focus group participants and will include the name and a brief definition of each activity. A preliminary estimate of expected person-time utilization (high, medium, or low in relation to other activities) will be included for each activity. As appropriate to their position, case workers and supervisors will be asked to consider the names, brief definitions, and amounts of person-time needed to complete each listed activity. If a focus group participant wants to suggest a previously unlisted activity or a revised name, definition, or time estimate, she/he will be instructed to indicate the suggested change in a “notes” column for the given activity.

Focus Groups of Case Workers and Supervisors: A single focus group will be conducted on-site at each participating PII grantee (see Attachment C3 for the focus group guide). Each focus group will be comprised of four to six case workers with at least 60 days of experience providing PII services and four to six supervisors with at least 60 days of experience supervising case workers that provide PII services. The Project Director at each grantee will nominate up to 15 case workers and 15 supervisors for focus group participation. The final selection of participants will be made by PII cost evaluation team. The participants will be informed that the focus group to which they are invited to participate is a key part of the PII cost study. The purpose of the focus group is to define activities to be included in a Weekly Case Work Activity Log and a Weekly Supervision Activity Log and establish an estimate of the person-time (labor) typically required to complete selected key PII project casework, supervision, and management and administration activities.

The focus group meeting time will be divided into four exercises: three devoted to case work and supervision activities and one focused on the structure, content, and procedures for completing the Weekly Case Work Activity Log and the Weekly Supervision Activity Log. The four exercises address topics of importance to the cost evaluation as follows:

• Exercise A: Names and Definitions of [name of PII project] Casework and Supervision Activities (60 minutes)

• Exercise B: Estimates of Amounts of Person-time Used to Conduct [name of PII project] Case Work and Supervision Activities (60 minutes)

• Exercise C: Names, Definitions, and Person-time Estimates for [name of PII project] Management and Administration Activities (45 minutes)

• Exercise D: Review and Discussion of Weekly Casework and Supervision Logs (45 minutes)

Post-focus group trial administration of logs: After the meeting, focus group participants will be asked to spend 90 minutes completing a draft version of a weekly activity log and participate in a de-briefing call conducted by the focus group facilitators. During the post-meeting phase, focus group participants will conduct a limited trial of the weekly case worker and supervision activity logs. Within 3 weeks of the focus group meeting, participants will receive an email prompt to complete the weekly log for their position (casework or supervision). After completing the log, each individual will participate in a de-briefing call conducted by the focus group facilitators. (See Attachment C4 for outline of information the participants compile.)

Data Collection for Case Workers³: PII caseworkers will be informed in the introduction to the instrument (see Attachment C5) that their log entries will be kept private and will be asked to consent by electronically recording agreement with a statement of informed consent. After consent is obtained, the statement of informed consent will be deleted from the log introduction for that caseworker. Only the research team will have access to this information. Log entries will not be shared with anyone at the PII project or any other agency. In research reports, the information provided by case workers will not be attributed to individuals.

Case workers will be asked to complete a log for each week of involvement in the PII project. A separate log entry will be completed for each open PII case. After a log for one case is completed, case workers will be asked to complete the same set of questions for each additional PII case. If time is spent on PII cases during the reporting week, the log should take approximately 24 minutes to complete for all cases. The home page for the web-based case work log will include the following options: weekly activity log; view report of logs; and exit and return later. After clicking on the weekly activity log, case workers first will be asked to select a case by client name or other case identifier from a drop-down menu, after which the weekly activity log for that case will be loaded for first case selected. For privacy purposes, the client's name or other case identifier is not kept on the same file with responses about the case. This is the last time the client's name or other case identifier will be displayed. Case workers will be asked to click on the “Next” button to be transferred to the secure log. At the secure log, they will be asked to select from a pop-up calendar the calendar week for which they are completing the log for the selected case. After completing the log for the selected case, the case worker will be prompted to return to select another case.

Data Collection for Supervisors: PII supervisors will be informed in the introduction to the instrument (see Attachment C6) that their log entries will be kept private and will be asked to consent by electronically recording agreement with a statement of informed consent. After consent is obtained, the statement of informed consent will be deleted from the log introduction for that supervisor. Only the research team will have access to this information. Log entries will not be shared with anyone at the PII Project or any other agency. In research reports, the information provided by supervisors will not be attributed to individuals.

A separate log entry will be completed for each PII case worker under their supervision. After a log for one case worker is completed, supervisors will be asked to complete the same set of questions for each additional PII case worker. Supervisors will be asked to view a drop-down menu and then select the name of the PII case worker for which they are reporting. The weekly activity log will load for the first case worker name or other identifier selected. For privacy purposes the case worker's name or other identifier is not be kept on the same file with supervisor’s log entries for that individual. This is the last time the case worker's name or other identifier will be displayed. Supervisors will be asked to click on the “Next” button to be transferred to the secure log. At the secure log, they will be asked to select from a pop-up calendar the calendar week for which they are completing the log for the selected case worker. After completing the log for the selected case, the case worker will be prompted to return to select another case worker.

Data Collection for Managers and Administrators: PII managers and administrators will be informed in the introduction to the instrument (see Attachment C7) that their log entries will be kept private and will be asked to consent by electronically recording agreement with a statement of informed consent. After consent is obtained, the statement of informed consent will be deleted from the log introduction for that manager or administrator. Only the research team will have access to this information. Log entries will not be shared with anyone at the PII project or any other agency. In research reports, the information provided by managers and administrators will not be attributed to individuals. Managers and administrators will be asked view a drop-down menu and then select the week of the reporting month for which they are reporting. The monthly activity log will load for the first week of the reporting month selected. After completing the log for the selected week, the manager or administrators will be prompted to return to select another week.

Administrative Data Study

Data are transmitted from grantee states to the PII evaluation team over secure channels and stored on secure servers.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Expected Response Rates

The overall response rate in Kansas (study in operation since 9/1/2012) is about 76 percent for caregivers (only respondent group), for Washoe is 56 percent for caregivers (only respondent group), and for Illinois is 84 percent for youth, 72 percent for foster parents, and 71 percent for biological parents. We expect a response rate for RISE CCT in the middle of that range, or about 70 percent. RISE is aiming for an overall response rate of 70 percent for staff completion of pretest, posttest, and followup, and will use tokens of appreciation and reminders to achieve that response rate. The expected overall response rate for the cost study is about 85 percent. For the administrative data study, the data comprise the population, so response rates are 100 percent.

Dealing with Nonresponse

To overcome possible nonresponse at RISE CCT, the contractor is using highly trained data collectors who: 1) can explain the informed consent process and areas of the instrument that may confuse respondents; 2) know how to re-approach a hesitant participant; and 3) are available to make multiple attempts at data collection. Respondents will also be assured as to the privacy of their information, and interviews will be conducted at times and places convenient to the respondent. Places convenient to the respondent may include their home, child welfare agency offices, or alternative locations. For the evaluation of ORB, the burden of staff participation is low (15 minutes for completing the instrument), and participation made convenient by administering the pretest and posttest immediately before and after the training session, and emailing participants about the 2-month followup. Further, during the needs assessment phase, a focus group of foster care staff indicated that the need for LGBTQ competency training was great. RISE anticipates that staff will be eager to participate and will be equally eager to learn whether the training produced demonstrable improvements in knowledge and practice.

In order to increase timely completion of weekly and monthly activity logs, the cost study team will solicit feedback from PII personnel on log instruments and procedures as part of log orientation. To overcome possible non-response among PII personnel, the cost evaluation team will ensure that response is convenient by allowing respondents to choose the day and time in which the logs are completed within a timely completion window that will extend 3 days after the end of the reporting week or month. Daily reminders will be sent during the completion window and overdue reminders will be sent after the completion window closes. Overdue reminders will indicate that the grantee requires the completion of logs as part of regular duties. A “help desk” will be available and feedback on the home grantee completion rates will be provided respectively for caseworkers, supervisors, and managers and administrators. Respondents will be reassured as to the privacy of their information, and interviews will be conducted at times and places convenient to the respondents.

Maximizing Response Rates

RISE is offering a token of appreciation to respondents to encourage them to participate in the data collection (see Section A.9). Individual respondents for the cost study and administrative data study will not receive incentives.

Administrative Data Study

The data comprise the population, so response rates are 100 percent.

B4. Tests of Procedures or Methods to be Undertaken

RISE CCT conducted a very limited pretest (fewer than 10 administrations) on their procedures and methods. The results were used to improve the instruments and procedures. For ORB, RISE has been piloting the pretest and posttest measures as approved in October 2012 under the ACF/OPRE generic clearance, Pre-testing of Evaluation Surveys (0970-0355). Based on their experience with the pilot data collection, they made changes to the pretest/posttest instrument and added a followup.

As part of the post-meeting phase of the cost study focus group, difficulties encountered with respect to the logs procedures, materials, or instruments will be discussed with the grantee and the evaluation leadership, and suggested revisions will be instituted.

Regarding the administrative data study, original data are acquired into nationally standardized fields in state administrative data systems; in the event that data fields outside the SACWIS, AFCARS, or NCANDS specifications are used for PII cross-site analysis, the correct coding and interpretation of those fields will be verified with the state data contacts.

B5. Individual Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The team is led by Maria Woolverton, project officer; Andrea Sedlak, project director for the PII evaluation; and Mark Testa, principal investigator for the evaluation. Additional staff consulted on statistical issues at Westat include John Rogers and Barnali Das, senior statisticians.

1 The CCT sample size is relatively small because the CCT is a newly developed intervention and is being conducted with a target population that is extremely vulnerable to repercussions related to their sexual orientation/gender identity. The program staff must safely and carefully conduct their program with a sample that is as representative as possible and still manageable so that implementers can scrutinize each step of the respondents’ involvement and ensure that respondents’ placements are not jeopardized through participation in the CCT.

2 For the intervention group, baseline data collection will occur when a youth enrolls in RISE CCT. For the comparison group, baseline data collection will occur when a youth is referred to RISE and there are no available CCT slots. Baseline data collection does not necessarily occur at the beginning of a foster care placement or upon initiation of child welfare services; at the time of the baseline, the youth will have been in foster care and/or receiving child welfare services for varying amounts of time.

3 The term “case worker” will have to be tailored to each site, based on who is providing the PII services.

9