Attachment A1 Research Matrix for Implementation, KS, and Washoe Studies

Instrument	Description	Frequency	Target Population	Annual Number of Respondents	Research Question
Survey of Organization/ System Readiness	Investigates the extent to which PII grantees are positioned through experience and organizational culture to make adjustments that support PII’s two main goals: To improve permanency outcomes for children who have the most serious barriers to permanency through the development of evidence-based, replicable child welfare service interventions To conduct rigorous evaluations to produce high-quality, credible evidence that the PII interventions are effective in decreasing the duration of stays in foster care and potentially replicable by future adopters. The survey will gauge whether grantee organizations support the PII goals by exploring individual respondents’ willingness to use and interest in using evidence-based interventions to address barriers to permanence for children and youth most at risk of LTFC. In addition, the questionnaire will explore respondents’ perceptions of organizational climate as it relates to readiness to change and individual and organizational interest in supporting rigorous evaluation.	The survey will be administered once to participants in all six grantee sites.	Staff and partners employed by direct service providers for the PII initiative	60	What organizational and systemic factors are associated with reaching and sustaining full implementation?
Implementation Drivers Web Survey	Will track processes that sites use to implement interventions. While the intervention and size/scope of implementation varies across sites, there are implementation processes and strategies that are common across projects. Grounded in the National Implementation Research Network framework of organizational and systemic factors that facilitate successful implementation, these measures track progress building implementation infrastructure capacity. The driver assessment survey comprises eight sections that correspond to implementation driver domains that have been the focus of training and technical assistance provided to grantees to strengthen implementation capacity. These domains are: practitioner selection, training, supervision/coaching, performance assessment, decision support data systems, facilitative administration, systems intervention, and leadership.	The Implementation Driver Assessment will be completed for the first time six months after the first clients who will be receiving the full intervention are served by the first trained workers. The survey will be administered every six months thereafter to determine the progress made toward the development of the drivers.	Grantees, partners, and stakeholders who are a part of the PII leadership and implementation structure	150	What organizational and systemic factors are associated with reaching and sustaining full implementation?
Grantee Case Study Protocol	The protocol allows for a qualitative examination of the key implementation activities; interim products and milestone events that occur during exploration, installation, and initial implementation; and the stages of implementation that set a foundation for achievement of full implementation. In addition, the implementation case study allows for examination of the context in which a PII intervention is being developed and implemented: How do external factors and broad systems issues influence implementation progress and success? In addition, the case study will explore the resources and organizational structure that support the implementation of PII. In essence, the implementation case study allows PII-ET to document the individualized story of each grantee’s implementation experience including enablers and barriers encountered during the implementation process.	The case study will be completed for year 1 and 2 and then updated on a yearly basis.	The Evaluation Team will complete the case study using existing documentation. Once completed, the ET will meet with grantee staff to address missing information and to ensure the case study accurately reflects site specific PII experiences.	30	What organizational and systemic factors are associated with reaching and sustaining full implementation? What costs were incurred in reaching and sustaining full implementation?
Fidelity Data (Implementation Quotient Tracker)	A reporting format used to collect data on the percent of caseworkers/practitioners that are implementing the prescribed intervention as intended and with model integrity. This tool captures the primary dependent variable: the proportion of caseworkers/practitioners at a given point in time that are meeting performance assessment criteria. For each practitioner allocated to the program the following will be answered: Is the practitioner in position; Has the practitioner been trained; Has the practitioner being coached as intended; Did the worker meet the performance assessment standards for the most recent assessment; Has the practitioner met the performance assessment standards over the last 4 quarters.	Administered for the first time six months after summative evaluation had begun, then quarterly thereafter. Enables quantification of a site’s overall progress towards full implementation of the PII initiative.	The site-specific evaluation liaison will compile the information for each allocated practitioner using data already available for site specific evaluation and/or implementation requirements	2	To what extent have grantees reached and sustained full implementation? How many calendar months were needed to reach and sustain full implementation?
Family Assessment Battery	(1) Families are videotaped and experts rate them on dimensions such as problem solving, planning a fun activity, teaching, monitoring, and parenting processes; (2) parent-informant items assess social skills, problem behaviors, and academic competence of children and youth.	Completed twice: baseline for all cases and again when parents have completed the PMTO curriculum (6 months in most cases) or at 6 months for control cases	Children age 3-16 with serious emotional disturbance and their parents	300 families	Do children age 3-16 in foster care who meet criteria for SED achieve more timely and stable permanence if their families receive early, intensive home-based parent management training compared to children in foster care whose families receive usual services?
North Carolina Family Assessment Scale (NCFAS)	Used by caseworkers to assess and rate family functioning (problems and strengths) in the domains of Environment, Parental Capabilities, Family Interactions, Family Safety, Child Well-being, Caregiver/Child Ambivalence, Social and Community Life, Self-Sufficiency, Family Health, and Readiness for Reunification. It is used at intake and again at case closure and the change score (the difference between the Intake and Closure ratings) indicates the amount of change achieved during the service period.			45 caseworkers respond for an average of 7 study cases each
Child and Adolescent Functional Assessment Scale (CAFAS)/ Preschool and Early Childhood Functional Assessment Scale (PECFAS)	Used to assess the day-to-day functioning (impairments and strengths) of children ages 5-19 in the domains of School/Work Role Performance, Home Role Performance, Community Role Performance, Behavior Toward Others, Moods/Emotions, Self-Harmful Behavior, Substance Use, Thinking, and Risk Behaviors. The PECFAS is a version used for younger children. The assessor completes the scale based on information collected during a typical clinical assessment.
Family Assessment Battery	Rates a child's problem behaviors and competencies; reports stresses related to parenting; reports measures of mental health symptoms in various domains; assesses parenting and childrearing attitudes of parents; assesses whether parents are at risk of child abuse or neglect; assesses caregiver’s readiness to accept change; assesses resiliency on eight subscales; examines how respondent’s social relationships provide social support; and provides indication of how stable a caregiver’s housing situation is. At baseline only (not at 6-month follow-up), assesses a caregiver’s history of abuse and trauma in childhood. At 6-month follow-up only, assesses the strength of the helping relationship between social workers and their clients.	Baseline and at 6-month intervals until the case is closed. All cases will have an exit assessment at the time of case closure.	Two target populations: (1) new cases involving children aged 17 ½ or younger coming into the system, who are deemed unsafe, are living with a caregiver, and at risk of foster care placement; and (2) families with children who have been in foster care at least 12 months and who have one or more of the identified risk characteristics for long term foster care at time of placement (i.e., parental substance abuse, homelessness/inadequate housing, single parent households, or parental incarceration), a goal of adoption or guardianship, and an available caregiver.	175	Are there changes in the following over time, and are changes different for children whose caregivers receive the SAFE-FC intervention compared to those who receive business as usual? Caregiver parenting stress Caregiver mental health Caregiver attitudes about child-rearing Caregiver readiness to accept change Caregiver resiliency Caregiver housing stability. Does the relationship between a caregiver and their caseworker change over time and how does that relationship relate to permanency for the child? Are caregivers’ histories of abuse and trauma associated with children’s length of time in foster care?