PII Supporting Statement Part A for OMB Approval_FINAL OMB response revised

Supporting Statement Part A for OMB Approval

Permanency Innovations Initiative (PII) Evaluation: Phase I

August 2012

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

PART A. JUSTIFICATION

The Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS), requests permission to collect data for an evaluation of the Permanency Innovations Initiative (PII). This 5-year initiative, funded by the Children’s Bureau (CB) within ACF, will build the evidence base for innovative interventions that enhance well-being and improve permanency outcomes for particular groups of children and youth who are at high risk for long-term foster care and who experience the most serious barriers to timely permanency. A major emphasis of the PII is the design of rigorous evaluations that will provide credible evidence and replicable interventions for achieving faster permanency for children and youth in foster care.

The CB has funded six grantees to identify local barriers to permanent placement and to develop and implement innovative strategies that mitigate or eliminate those barriers and reduce the likelihood that children will linger in foster care. The first year of the initiative, designed as a planning year, focused on conducting enhanced analyses of local data in order to most effectively target grantees’ proposed intervention programs. In addition, evaluation plans were developed to support rigorous site-specific and cross-site studies to document the implementation and effectiveness of the grantees’ interventions and the initiative overall.

The proposed PII Evaluation includes multiple components: a cross-site evaluation, including implementation study and administrative data study; six site-specific impact evaluations; and a cost study. This clearance package requests approval for Phase I of data collection for the evaluation. This first phase includes the cross-site implementation study (covering all six sites) and the site-specific impact evaluations for two grantees (the State of Kansas and Washoe County, Nevada) that will begin full implementation of their interventions in July 2012. A second phase will include the site-specific impact evaluations for the four grantees that will begin full implementation of their interventions at a later point in time, as well as collection of data for the cost study and administrative data study.

Data for the cross-site implementation study will be collected from grantee staff and other key informants through (1) telephone surveys; (2) web-based instruments; (3) interviews (on-site and telephone); and (4) submission of aggregate fidelity data tracked in grantee data systems.

Data for the site-specific impact evaluations in Kansas and Washoe County will be collected through (1) caregiver/family assessments; and (2) clinical assessments of children and families completed directly by caseworkers or provided via caseworker interviews.

A.1. Circumstances Making the Collection of Information Necessary

Although the Adoption and Safe Families Act of 1997 included provisions focused on moving children and youth quickly into permanent families while maintaining their safety, many jurisdictions continue to experience growing populations of children who age out of foster care without achieving permanency. The PII grantees are developing and implementing innovative interventions to address site-specific barriers in order to achieve timely permanency for more children and youth. The site-specific impact evaluations are designed to test the causal link between grantee interventions and key outcomes of interest. The cross-site evaluation will gather information that speaks to grantees’ implementation capacity and effectiveness. Using a mix of research methods, the various evaluation components will inform the federal government about the effectiveness of the PII interventions and provide information to help other child welfare agencies develop, implement, and strengthen interventions in the future.

Applicants responded to solicitation HHS-2010-ACF-ACYF-CT-0022 (CFDA No. 93.648), which specified that the purpose of the initiative is to “...fund demonstration projects that support the implementation and test the effectiveness of innovative intervention strategies to improve permanency outcomes of subgroups of children that have the most serious barriers to permanency....” External reviewers rated the applications and awarded points based on applicant’s knowledge of and justification for the proposed target population(s); innovativeness of proposed intervention(s); rigor of evaluation plan; plans for partnerships and collaboration; cultural sensitivity; expertise with child welfare systems; and reasonableness of budget. Funded projects were expected to build the evidence base for innovations that improved permanency outcomes for particular groups of children and youth who were at high risk for long-term foster care. Organizations eligible for the funding included public child welfare agencies, nonprofit organizations, institutions of higher education, and Native American Tribal governments or Tribal Organizations. This approach resulted in six grantees with very different grantee structures, target populations, developmental nature of the interventions, and existing evidence for the proposed interventions. Four of the six grantees are public child welfare agencies, while one is an institution of higher education with close existing ties with the public child welfare agency (Kansas, one of the grantees covered in this Supporting Statement), and one is a community nonprofit organization with some limited experience working with the local public child welfare agency (not included in this Supporting Statement). As a result, there is considerable variation in evaluation design and our ability to collect consistent primary data across all six sites is limited. However, in PII Phase 2 we will acquire child welfare administrative data that will give us some consistent measures related to permanency outcomes. Note that the method of selection of programs for this initiative precludes the generalization of the evaluation results to programs with different characteristics.

A.2. Purpose and Use of the Information Collection

Current Information Collection Request

Cross-site implementation study. The implementation study will document the status of grantees’ implementation of their planned interventions and address questions related to whether implementation status mediates or moderates the achievement of proximal outcomes and/or a reduction in long-term foster care, and whether grantees’ implementation capacity improves over the course of the grant period. The Survey of Organization/System Readiness will be administered one time by telephone to approximately 30 individuals in each of the six sites, including individuals such as the PII program manager, steering committee members, supervisors, caseworkers, and practitioners. The Implementation Drivers¹ Web Survey will be administered every 6 months, to approximately 25 individuals per site who are active in the PII organizational structure. Information for the Grantee Case Study Protocol will be collected through site visits and telephone calls with each grantee, during which interviews will be conducted with key informants. Intervention fidelity data will be reported quarterly (for a period of two years) through an Implementation Quotient Tracker. More information on these data collection activities is provided in Statement B, Section B.2. Attachment 1 to this Statement shows a matrix for the implementation study, linking instruments, description of instruments, frequency of administration, target populations, sample sizes, burden, and research question being addressed.

Kansas site-specific impact evaluation. The Kansas grantee (University of Kansas Center for Research) is implementing an intervention – the Kansas Intensive Permanency Project (KIPP) – to target children with Serious Emotional Disturbance (SED) ages 3-16 who are in foster care and their birth parents to prepare the families for reunification. The focus of the intervention is risk factors in the parents that may impede reunification; the intervention includes an evidence-based, behavioral parent management training model that has been tailored to address permanency barriers of parents of children with SED. The site-specific impact evaluation will employ a randomized design to determine the extent to which the intervention achieves its proximal goal of improving parenting skills, enhancing child functioning, and enhancing readiness for reunification among the families who received the services. Long-term outcomes of the intervention (e.g., increase in reunification rates, placement stability, decrease in long-term foster care rates, and maintenance of child safety) will be evaluated using existing State administrative data.

Trained data collectors (KIPP Data Liaisons) will administer an assessment battery with the child’s parent/caregiver that includes interview and observational components. The following measures are part of the family assessment battery: Social Skills Improvement System (SSIS) and the Family Interaction Task (FIT). During the assessment, data liaisons will also gather information from parents in order to complete the North Carolina Family Assessment Scale for General Services and Reunification (NCFAS – G+R) following the interview.

The data liaisons will review each family’s case file and have discussions with caseworkers to verify information recorded on the NCFAS – G+R. Caseworkers will complete a clinical assessment of the child – either the Child and Adolescent Functional Assessment Scale (CAFAS) or the Preschool and Early Childhood Functional Assessment Scale (PECFAS). More information on Kansas’ data collection activities is provided in Statement B, Section B.2.

Data collection will take place at two time points - pre- and post-intervention. The pre-intervention data collection will occur at the time of intake into KIPP services for the treatment group and, for the comparison group (which will receive services as usual), at the time of assignment to the group. The post-intervention data collection will occur at the conclusion of KIPP services for the treatment group, which is expected to be approximately 6 months. For the comparison group, the post-test will be administered at 6 months.

Attachment 2 shows a matrix for Kansas, linking instruments, description of instruments, frequency of administration, target populations, sample sizes, burden, and research question being addressed.

Washoe County, Nevada site-specific impact evaluation. Washoe County is implementing an intervention known as Safe-FC that engages families in a change-focused phase of care to enhance caregiver protective capacities, improve safety of children, and prevent or shorten foster care placement. The intervention is being adapted to support the needs of two target populations: (1) new cases involving children aged 17 ½ or younger coming into the system, who are deemed unsafe, are living with a caregiver, and at risk of foster care placement; and (2) families with children who have been in foster care at least 12 months and who have one or more of the identified risk characteristics for long term foster care at time of placement (i.e., parental substance abuse, homelessness/inadequate housing, single parent households, or parental incarceration), a goal of adoption or guardianship, and an available caregiver. The evaluation includes random assignment of caseworkers and of cases.

A baseline family assessment battery² will be administered to both the intervention group (receives Safe-FC) and the control group (receives services as usual) prior to delivery of services. Change scores will be generated through administering the instruments at 6-month intervals until the case has been closed. All cases will have an exit assessment at the time of case closure. The assessments will be administered through a Computer-Assisted Self Interview (CASI) format. Key proximal outcomes of interest include increase in caregiver readiness for change, improvements in caregiver mental health, and decrease in child behavior problems. Long-term outcomes of the intervention (e.g., decreases in time to permanency, reduced recurrence of child abuse and neglect, decrease in re-entries into foster care) will be evaluated using existing State administrative data. More information on these data collection activities is provided in Statement B, Section B.2.

Attachment 3 shows a matrix for Washoe, linking instruments, description of instruments, frequency of administration, target populations, sample sizes, burden, and research question being addressed.

Future Information Collection Requests

A future information collection request will cover remaining components of the PII evaluation. These include: data collection for site-specific impact evaluations in the other four PII sites; a cost study; and an administrative data study. The cost study will involve grantee submission of common programmatic cost data, using a spreadsheet to disaggregate, record, and automatically tally program expenditures. Grantees will be asked to submit spreadsheets semi-annually. The first categorization of program costs will correspond roughly to budget line items, including personnel, space, utilities, travel, and supplies. The second will comprise the components of personnel (labor) costs, typically represent the largest proportion. For these components, we will distinguish various types of program staff activities, such as direct client services and administrative activities. The administrative data study does not require instruments for measurement and will make use of data currently reported by States under separate OMB clearances for the Adoption and Foster Care Analysis and Reporting System (AFCARS)(OMB Control # 0980-0267) and the National Child Abuse and Neglect Data System (NCANDS) (OMB Control # 0980-0229), as well as data maintained in State Automated Child Welfare Information Systems (SACWIS). The administrative data will serve as the key source of information on the long-term outcomes of importance to PII (e.g., permanency-related outcomes).

The future information collection request will present further details on each of these data collection activities. We are requesting that the 60-day comment periods be waived for future information collection requests. The public will still have the 30-day OMB comment period.

A.3. Use of Improved Information Technology and Burden Reduction

The cross-site implementation study includes web-based data collection to help reduce the burden on respondents. Whenever possible, data for the cross-site evaluation will be collected from existing documentation or administrative data sources in order to reduce burden on the grantee staff. State administrative data will also be utilized to examine long-term outcomes of each grantee’s intervention. To reduce burden on the families, both Kansas and Washoe County plan to offer to administer assessment batteries in the families’ homes and at times convenient to them. The family assessment battery in Washoe County will be administered in a computer-assisted self-interview (CASI) format, which reduces data entry burden and can be easier for respondents to complete, although it is not expected to reduce response time for the respondents. In Kansas, the Family Interaction Task (FIT) will be videotaped for later coding by trained coders affiliated with the intervention developer.

A.4. Efforts to Identify Duplication and Use of Similar Information

For the cross-site implementation study, all of the proposed instruments are new and not currently being used in the grantee sites. Each respondent will be carefully selected to provide a particular perspective on the local PII initiative, in order to be inclusive of the layers and levels within the PII grantee’s organizational structure. This will help to ensure that a variety of perspectives are captured in the implementation study. In some cases, the same informant will respond to more than one instrument (e.g., the program director and a major partner service provider respond to both the Survey of Organization/System Readiness and the Implementation Drivers Web Survey, as they have in-depth knowledge of the grantee organization and implementation infrastructure.) However, the same question will not be asked of a respondent more than once.

For the Washoe County impact evaluation, none of the proposed instruments are currently being used in the child welfare system. For the Kansas impact evaluation, the proposed instrument battery is not currently used in its entirety by all four private foster care agencies that will participate in the project, although components of it are used in some of the agencies for certain families, as follows:

• Social Skills Improvement System (SSIS) – not in use by Kansas foster care agencies

• Family Interaction Task (FIT) – not in use by Kansas foster care agencies

• North Carolina Family Assessment Scale (NCFAS) – in use by two of five regions

• Child and Adolescent Functional Assessment Scale (CAFAS)/ Preschool and Early Childhood Functional Assessment Scale (PECFAS) – used by foster care agencies in three of five regions with some children (e.g., instrument may be used to determine appropriate placement or to justify a higher level of care).

In order to streamline the data collection process and reduce duplicative efforts, we reviewed all measures to confirm that no questions are repeated and the instruments are not redundant. Many of the measures are standardized and are being used in accordance with manuals, but when possible and appropriate for our research purposes we selected brief or shortened versions of the measures. For example, we are using the short form of the Parenting Stress Index, abbreviated versions of the Readiness for Parenting Change Scale and the North Carolina Family Assessment Scale, and only selected questions regarding home stability. Additionally we reviewed all measures to ensure that they measured discrete concepts of interest for this research study. No two measures target the exact same concept. In addition, we will be using extant administrative data to measure distal outcomes such as achievement of permanency.

Thus, in order to obtain consistent data from all families in the evaluations, this new collection of information in Kansas and Washoe County is necessary. None of the cross-site information will be available except through this information collection.

A.5. Impact on Small Businesses or Other Small Entities

No small businesses are impacted by the data collection in this project.

A.6. Consequences of Collecting the Information Less Frequently

The cross-site implementation study is designed to begin collecting implementation information early in the grantees’ timelines in order to track change over time in areas such as organizational readiness. Kansas and Washoe County have data collection schedules that produce the information necessary for tracking families’ progress on the key outcomes of interest. In order to track the progress of the families in both treatment and control groups, the data need to be collected on the specified schedules.

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances requiring deviation from these guidelines.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

The Federal Register Notice soliciting comments on the PII proposed information collection was posted in the Federal Register, Vol. 77, No. 35, pp. 10531-10532, on Feb. 22, 2012. No comments were received during the 60-day comment period. The notice also included a request for comments on waiving future 60 day notices for later phases of the study.

The following experts were consulted on methodological issues concerning the cross-site evaluation and grantee-specific evaluation issues:

• Patti Chamberlain, Ph.D. - expert on implementation and efficacy research, particularly implementation of parent-mediated interventions and scaling up best practices models.

• John Landsverk, Ph.D. - expert in cost calculation and implementation research in child welfare and mental health interventions.

• Andrew Barclay, M.S. - expert in database design, data analysis, and statistics, particularly as applied to child welfare.

• Linda Collins, Ph.D. - expert on optimization of behavioral interventions, particularly on adaptive designs for prevention.

A.9. Explanation of Any Payment or Gift to Respondents

The PII evaluation project will not provide payments or gifts to respondents. In Kansas, however, parents will be offered a small monetary incentive (a $10 gift card) for taking part in the family assessment. Local agencies involved in the Kansas project made a decision to provide incentives from their own budgets. The same incentive is available for older youth for participation in the Family Interaction Task portion of the assessment battery. No incentives are being provided to participants in the Washoe County evaluation or for completion of cross-site instruments.

A.10. Assurance of Privacy Provided to Respondents

All consent forms include assurance of privacy under the Privacy Act. Kansas included a full range of assurances for privacy in their submission to the University of Kansas IRB, which granted approval for the PII project in November 2011. Washoe County has obtained approval from the IRB of the contractor who is conducting the PII evaluation (Westat), and this includes a similar full range of assurances for privacy. The evaluation contractor has received a Certificate of Confidentiality from the National Institute for Health for all six PII grantees.

The assurances, which will be included in the IRB applications for all six PII grantees, include:

• Respondents receive a written informed consent form that will explain the evaluation process and assure them that their information will be private to the extent permitted by law and securely stored.

• Strict policies and procedures for respondents’ confidentiality are followed by all project staff.

• All hard copies of documents are secured behind two locks (e.g., locked file cabinet in locked room).

• All electronic content is stored on secure servers. The server is set with privileges that allow access only by specific individuals who have a username and password.

• All project data are reported and presented at the aggregate level in order to prevent the identification of any individual respondent.

A.11. Justification for Sensitive Questions

The data collection batteries for both Kansas and Washoe County impact evaluations include questions that might be perceived as sensitive. The information is required for monitoring changes in the families and measuring outcomes as part of the evaluation. For example, in the Kansas battery one question to the parent that might be considered sensitive is: In your home, how often does your child see bad behavior that you do not want him/her to copy? And another part of the battery includes a comprehensive assessment of multiple domains of family functioning. The assessments identify, gather, and weigh information from caseworkers and case files to understand the significant factors affecting a child’s safety, permanency, and well-being; parental protective capacities; and the family’s ability to assure the safety of their children. A number of items in both Kansas and Washoe batteries could involve sensitive information, such as the status of parents’ physical health, mental health, and disabilities; parental use of alcohol and/or drugs; disciplinary practices; intimate relationships with other adults who live in or come into the home; potential for physical or verbal violence in the home; anything that could affect a child’s safety and well-being in this family; and family’s income and employment status. Information gathered through the batteries is necessary to monitor critical issues related to safely returning a child to his or her family. Respondents will be informed that there may be sensitive questions and that their participation in all aspects of data collection is voluntary.

A.12. Estimates of Annualized Burden Hours and Costs

Table A.1 contains the estimated burden hours for each type of respondent. The total annual burden for these evaluation activities is expected to be 2922 hours.

TABLE A.1

ESTIMATED ANNUAL RESPONSE BURDEN AND ANNUAL COST

Instrument	Number of Respondents	Number of Responses Per Respondent	Average Burden Hours per Response	Total Burden Hours	Average Hourly Wage	Total Annual Cost
CROSS-SITE IMPLEMENTATION STUDY:
Survey of Organization/ System Readiness	60	1	0.3	18.0	28.70		516.60
Implementation Drivers Web Survey	150	2	0.8	240.0	28.70		6888.00
Grantee Case Study Protocol	30	4	2.0	240.0	28.70		6888.00
Fidelity Data (Implementation Quotient Tracker)	2	8	1.5	24.0	28.70		688.80
Cross-Site Estimated Total	--	--	--	522.0	--		14981.40
KANSAS:
Caregiver Initial Information Form	300	1	0.1	30.0	22.88		686.40
Family Assessment Battery	300	2	1.5	900.0	22.88		20592.00
CAFAS/PECFAS	45	14	1.0	630.0	28.70		18081.00
Caseworker discussions for NCFAS-G&R completion	45	14	0.5	315.0	28.70		9040.50
Kansas Estimated Total	--	--	--	1875.0	--		48399.40
WASHOE COUNTY:
Family Assessment Battery	175	2	1.5	525	22.88		12012.00
Washoe Estimated Total	--	--	--	525	22.88		12012.00
OVERALL TOTAL	--	--	--	2922.0	--		75392.8

To compute the total estimated annual cost for the Kansas and Washoe data collection from caregivers, the total burden hours were multiplied by the overall average hourly wage ($22.88) according to the Bureau of Labor Statistics, Current Employment Statistics Survey, 2011. To compute the total estimated annual cost for the cross-site data collection from grantee staff and stakeholders, as well as caseworker interviews in Kansas, the total burden hours were multiplied by the average hourly wage for full-time employees over age 25 with a bachelor’s degree or higher ($28.70) according to the Bureau of Labor Statistics, Current Employment Statistics Survey, 2011.

A.13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There are no additional costs to respondents.

A.14. Annualized Cost to Federal Government

The total annualized cost to the federal government for gathering the information in each of the three years is estimated to be $535,556, including direct and indirect costs.

A.15. Explanations for Program Changes or Adjustments

This is a new information collection.

A.16. Plans for Tabulation and Publication and Project Time Schedule

The cross-site implementation study schedule is as follows:

1. Survey of Organization/System Readiness. A cross-site report on this survey will be completed in December 2012.

2. Implementation Drivers Web Survey. Grantee-specific reports will be generated 45 days after each survey administration. Reports synthesizing the survey data will be completed in September 2013 and 2014. A report on a cross-site analysis of mediators (including implementation driver variables) will be completed in September 2015.

3. Grantee Case Study. Site-specific case study reports for all grantees will be completed by September 30 in 2012, 2013, 2014, and 2015. A cross-site implementation report including information from the case studies will be completed in September of 2012, 2013, and 2014.

4. Fidelity Data. Cross-site reports on Implementation Quotients (IQ’s) will be completed in September of 2012, 2013, and 2014.

Kansas: Pending OMB clearance, Kansas plans to begin implementing the intervention and collecting evaluation data in late July 2012 or as soon as OMB clearance is received. Currently Kansas is pilot testing (OMB generic clearance 0970-0355 received on Oct. 4, 2011).

Washoe: Pending OMB clearance, Washoe County plans to begin implementing the intervention in late July 2012 or as soon as OMB clearance is received.

A.17. Display of Expiration Date for OMB Approval

The OMB number and expiration date will be displayed on each cross-site instrument and on the parent/caregiver consent forms for Kansas and Washoe.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this data collection.

1 An “implementation driver” is a component of implementation, such as training, that promotes high fidelity behavior from practitioners.

2 The following measures are part of the family assessment battery: Child Behavior Checklist (CBCL), Parenting Stress Index (Short Form), Brief Symptoms Inventory, Adult-Adolescent Parenting Inventory (AAPI2), Readiness for Parenting Change Scale Abbreviated, Social Provisions Scale, Adverse Childhood Experiences (ACE), Helping Relationship Inventory, Home Stability Measure (selected items), and Resiliency Attitudes Scale.