Attachment 1 - Matrix implementation study

ATTACHMENT 1: IMPLEMENTATION STUDY MATRIX
Instrument	Description	Frequency	Target Population	Annual Number of Respondents	Annual Burden Hours	Research Question
Survey of Organization/ System Readiness	Investigates the extent to which PII grantees are positioned through experience and organizational culture to make adjustments that support PII’s two main goals: To improve permanency outcomes for children who have the most serious barriers to permanency through the development of evidence-based, replicable child welfare service interventions To conduct rigorous evaluations to produce high-quality, credible evidence that the PII interventions are effective in decreasing the duration of stays in foster care and potentially replicable by future adopters. The survey will gauge whether grantee organizations support the PII goals by exploring individual respondents’ willingness to use and interest in using evidence-based interventions to address barriers to permanence for children and youth most at risk of LTFC. In addition, the questionnaire will explore respondents’ perceptions of organizational climate as it relates to readiness to change and individual and organizational interest in supporting rigorous evaluation.	The survey will be administered once to participants in all six grantee sites.	Staff and partners employed by direct service providers for the PII initiative	60	18	What organizational and systemic factors are associated with reaching and sustaining full implementation?
Implementation Drivers Web Survey	Will track processes that sites use to implement interventions. While the intervention and size/scope of implementation varies across sites, there are implementation processes and strategies that are common across projects. Grounded in the National Implementation Research Network framework of organizational and systemic factors that facilitate successful implementation, these measures track progress building implementation infrastructure capacity. The driver assessment survey comprises eight sections that correspond to implementation driver domains that have been the focus of training and technical assistance provided to grantees to strengthen implementation capacity. These domains are: practitioner selection, training, supervision/coaching, performance assessment, decision support data systems, facilitative administration, systems intervention, and leadership.	The Implementation Driver Assessment will be completed for the first time six months after the first clients who will be receiving the full intervention are served by the first trained workers. The survey will be administered every six months thereafter to determine the progress made toward the development of the drivers.	Grantees, partners, and stakeholders who are a part of the PII leadership and implementation structure	150	240	What organizational and systemic factors are associated with reaching and sustaining full implementation?
Grantee Case Study Protocol	The protocol allows for a qualitative examination of the key implementation activities; interim products and milestone events that occur during exploration, installation, and initial implementation; and the stages of implementation that set a foundation for achievement of full implementation. In addition, the implementation case study allows for examination of the context in which a PII intervention is being developed and implemented: How do external factors and broad systems issues influence implementation progress and success? In addition, the case study will explore the resources and organizational structure that support the implementation of PII. In essence, the implementation case study allows PII-ET to document the individualized story of each grantee’s implementation experience including enablers and barriers encountered during the implementation process.	The case study will be completed for year 1 and 2 and then updated on a yearly basis.	The Evaluation Team will complete the case study using existing documentation. Once completed, the ET will meet with grantee staff to address missing information and to ensure the case study accurately reflects site specific PII experiences.	30	240	What organizational and systemic factors are associated with reaching and sustaining full implementation? What costs were incurred in reaching and sustaining full implementation?
Fidelity Data (Implementation Quotient Tracker)	A reporting format used to collect data on the percent of caseworkers/practitioners that are implementing the prescribed intervention as intended and with model integrity. This tool captures the primary dependent variable: the proportion of caseworkers/practitioners at a given point in time that are meeting performance assessment criteria. For each practitioner allocated to the program the following will be answered: Is the practitioner in position; Has the practitioner been trained; Has the practitioner being coached as intended; Did the worker meet the performance assessment standards for the most recent assessment; Has the practitioner met the performance assessment standards over the last 4 quarters.	Administered for the first time six months after summative evaluation had begun, then quarterly thereafter. Enables quantification of a site’s overall progress towards full implementation of the PII initiative.	The site-specific evaluation liaison will compile the information for each allocated practitioner using data already available for site specific evaluation and/or implementation requirements	2	24	To what extent have grantees reached and sustained full implementation? How many calendar months were needed to reach and sustain full implementation?