Grantee Case Study Protocol

OMB# 0970-XXXX    

Expiration Date: XX/XX/2015

Instructions: The implementation case study enables qualitative examination of the key implementation activities; interim products and milestone events that occur during exploration, installation, and initial implementation; and the stages of implementation that set a foundation for achievement of full implementation. In addition, the implementation case study allows for examination of the context in which a PII intervention is being developed and implemented: How do external factors and broad systems issues influence implementation progress and success?

The case study will rely heavily on existing, readily available documentation to minimize burden on grantees. PII-ET staff members will do the following to facilitate the completion of the case studies:

1. Utilize available documentation to address as many topics of the case study protocol as possible;

2. Consult with PII-TTAP staff members to help fill in gaps in the chronology related to the implementation process and

When the information available through PII-ET and PII-TTAP is exhausted, work with grantees to ensure the case study is complete.

IMPLEMENTATION CASE STUDY PROTOCOL

Name of Grantee:

Date of Completion:

1. Context: The purpose of this domain is to describe the larger political, historical, economic, and organizational context of the child welfare system in which the PII grantee operates, as well as the political and organizational context specific to the PII jurisdictions at the time of proposal submission.

1. Describe the administrative structure of the child welfare system in the state in which the PII grantee operates (ie: state funded-county administered, state funded state administered).

2. Describe how the grantee operated within or in conjunction with the child welfare system described above.

3. Describe the administration structure within the jurisdiction/grantee implementing the PII intervention.

4. How many children currently reside in the foster care system?

• Statewide?

• PII locations?

Answers to questions 4-15 should reflect the environment at the time the application for PII was prepared and submitted to the Federal government.

5. Describe any political or historical issues that may have impacted the implementation of PII.

6. Describe any existing policies that may support or hinder the implementation of the intervention designed to reduce long term stays in foster care?

7. Describe recent trends in child welfare funding that have impacted overall child welfare service delivery and services directed specifically to the target population (eg: caseload size, access to specific services).

8. Describe the applicant’s relationship with key PII partners (individuals/organizations that will receive payment through contractual relationships with the applicant).

9. Describe key relationships with agencies or individuals other than key partners that may impact service delivery to the PII target population.

10. Describe key issues that must be resolved to ensure service provision to the PII target population (ie: Medicaid waivers).

11. To what extent is the proposed innovation aligned with the agency’s mission, priorities, and values?

12. Describe the statewide and/or local focus on permanence for children in foster care.

13. Describe statewide and/or local experience with the implementation of EBP

14. Describe statewide and/or local experience with the use of rigorous evaluation plans.

15. Describe statewide and/or local experience with the use of implementation science.

16. Have there been any changes to the answers in questions 4-10 that have impacted the implementation of PII since the grant was awarded.

2. Structure: The purpose of this domain is to capture the organizational/governance structure for the PII grant.

General Information

1. What is the role of the grantee organization?

___ Administrative oversight

___ Implementation of program

___ Implementation of program and evaluation

___ Other

2. What “type” of grantee is this?

___ State government ___ Native American tribe

___ County government ___ For-profit organization

___ City/municipal government ___ Non-profit organization

___ Township ___ Other

___ Interstate entity ___ Inter-municipal entity

Pre-Grant Award Structure

3. Was there a champion(s) that made the project/ proposal happen for the organization or the community?

1. Who was that person(s)?

2. What was/ is their role in the organization (How much authority in the organization did/ does this person have?)

3. What roles did they play in conceptualizing the initiative pre-award?

3. Was there a work group or committee structure in place for grant development?

   1. Describe the organizational structure.

   2. Who were the participants?

   3. What were the roles and responsibilities associated with the structure or key individuals.

Post –Award Grant Structure

3. Has the role of Champion changed over time? (Is there now someone else who works to keep the program going in the agency or community?)

   1. Who is that person(s)

   2. When did they take on this role

   3. Why did the change happen

   4. How did the change happen?

   5. What is actively being done to share information about the progress of PII with the champion(s)?

   6. What is the role of the champion(s) in spreading information about the project to others?

4. Is there a work group or organizational structure in place for implementation of the grant (attach any available diagrams or organizational charts)?

Committee/key individual	Roles and Responsibilities	Terms of Reference Exist ( Y or N)	Participants	Relationship to PII (Partner or Internal Stakeholder)*
			Don’t complete will use participant lists	Will add to participant list

*Partners are individuals/organizations that will receive payment through contractual relationships with the applicant. Internal stakeholders are committee members or advisors who may not be paid for their time, but are or have been members of a formal committee as defined by the grantee to operate the cooperative agreement or whose support is essential to program operations.

3. Has a linking-communication protocol been developed to support communication among those working on the grant a (linking communication protocol is a written document or documents that describe the way in which issues should be referred to the appropriate group of people to address the issue; a regular way of reporting information up and down the chain of command)? Describe.

4. To what extent do relevant parties seem to be involved in decision making?

3. Resources:

The purpose of this domain is to capture the dollars allocated to both labor and non-labor activities related to the implementation of PII.

1. Original Budget

PII award amount (federal dollars) Non-Federal (in-kind) Total Budget

Year 1:

Year 2:

Year 3:

Year 4:

Year 5:

Total :

2. Was the budget modified in year 1?

   1. If yes, please indicate why a budget revision was required.

   2. Did the re-budget change the total federal or applicant share of the grant?

   3. Please describe the significant shifts in line item dollars

3. How were year 1 dollars allocated?

   1. Describe how dollars were spent to support labor (what positions in the organizational structure were supported).

   2. Describe how dollars were spent to support non-labor activities

4. Have any dollars beyond what was awarded in the federal grant been used to support the implementation of PII in year 1?

   1. How many dollars have been allocated?

   2. What entity provided the additional dollars?

   3. Describe any parameters guiding the use of the additional dollars.

5. Were any in-kind donations used to support the PII initiative in year 1?

   1. Who or what agency provided the in-kind donations?

   2. What positions and/or activities did the in-kind donations support?

6. Where the dollars awarded sufficient to meet the expectations of the grantor?

   1. Describe the impact that budgetary constraints or surplus had on PII during year 1.

7. Was there a carry-over request for unused dollars from year 1 to year 2?

8. Was the budget modified in year 2

   1. If yes, please indicate why a budget revision was required.

   2. Did the re-budget change the dollars allocated by the federal government or the applicant during year 2.

   3. Please describe the significant shifts in line item dollars

9. How were year 2 dollars allocated?

   1. Describe how dollars were spent to support labor (what positions in the organizational structure were supported).

   2. Describe how dollars were spent to support non-labor activities

10. Have any dollars beyond what was awarded in the federal grant been used to support the implementation of PII in year 2?

    1. How many dollars have been allocated?

    2. What entity provided the additional dollars?

    3. Describe any parameters guiding the use of the additional dollars.

11. Were any in-kind donations used to support the PII initiative in year 2?

    1. Who or what agency provided the in-kind donations?

    2. What positions and/or activities did the in-kind donations support?

12. Where the dollars awarded sufficient to meet the expectations of the grantor?

    1. Describe the impact that budgetary constraints or surplus had on PII during year 2.

13. Overall, have economic constraints limited the scope of the intervention, describe?

14. How many dollars have been allocated for each year of the local evaluation?

15. Have economic constraints limited the scope of the local evaluation.

4. Key Activities/Achievement of Milestones:

The purpose of this domain is to document the key implementation, evaluation, and dissemination activities. The key activities for implementation and evaluation will be categorized by the PII stages of implementation and evaluation respectively.

For each of the key activities/milestones answer the following:

1. The date work started on the key activity

2. The date the key activity was completed.

3. The sub-tasks associated with the key activity.

4. The grantee level person with primary responsibility for the key activity.

5. The part of the organizational structure responsible for overseeing the activity.

6. The number and positions of staff involved in the activity.

7. Breakthroughs, philosophy, or logic resulting from the activity.

8. The product or products resulting from the key activity.

9. Challenges associated with completing the activity.

10. Facilitators associated with conducting the activity.

11. The technical assistance provided to the grantee by the PII-ET or TTAP to accomplish the key activity and the impact of the TA on the completion of the task.

Implementation Activities

1. Exploration Stage

1. Develop a theory of change

   1. Describe the theory of change.

2. Complete Intervention Template

   1. What interventions were reviewed by the grantee?

   2. Describe the process by which interventions were reviewed.

   3. Describe the core components of the intervention that was chosen. If an existing EBP was not chosen, why was this decision made?

   4. Describe how the chosen intervention matches the needs of the target population in relation to need, fit, resources, strength of evidence, and readiness for replication, and capacity to implement.

      1. Which components will work well with your target population?

      2. Which components might not work as well with your target population?

      3. Will implementing the intervention allow you to be more productive when working with clients? Please describe the impact the intervention will have on your workload?

      4. Does the use of this intervention support the priorities of the agency?

      5. To what extent, if any, does the proposed intervention need to be modified to fit provider preferences, organizational priorities, community needs, cultural norms and values?

      6. Does the agency’s culture reinforce the principles of the PII intervention? Explain.

   5. Describe how the services provided by the intervention differ from those currently received by the target population.

   6. Did the intervention change after the intervention template was approved by CB?

3. Complete and gain approval of the Implementation Plan

2. Installation Stage

1. Adapt selected interventions to meet the needs of the identified population or develop an intervention to meet the needs of the population.

2. Develop a practice profile or evidence of operationalized intervention.

3. Develop program referral procedures and policies.

4. Build a collaborative governance/leadership structure

5. Development of linking-communication protocols

6. Develop communication plan

7. Develop terms of reference

8. Form an implementation team

9. Establish decision support data systems

10. Identify and make structural and functional changes needed for implementation (eg: policies, schedule, space, time, materials, re-allocation of roles and responsibilities.

11. Develop selection protocols for first group of staff such as selection criteria, interview protocols, job descriptions, and recruitment methods including practitioners, coaches, supervisors, etc.

12. Develop intervention consent procedures.

13. Select first group of staff.

14. Identify and develop training resources and logistics.

15. Develop coaching and support plans for practitioners.

16. Conduct training of first group of staff.

    1. Topics covered:

    2. Who provided training

    3. Method of training

    4. Length of training:

    5. Frequency of training (initial and booster sessions?)

    6. Dates of training

17. Develop a Performance Assessment Protocol

18. Complete plan for procedures for performance assessment data collection

.

3. Initial Implementation

1. Develop a usability testing plan.

2. Conduct initial usability testing (PDSA cycles).

3. Completed usability report and usability statement.

4. Data system finalized/developed for measuring and reporting performance/fidelity.

5. Complete driver assessment

6. Complete action planning

Evaluation Activities

1. Pre-Implementation

1. Conduct/engage in data mining activities

2. Conduct research reviews

1. Determine the target population/complete the population template:

1. Describe the characteristics and needs of the population.

5. Complete Comparison Template

6. Complete Outcomes Template/Identify Outcomes (proximal outcomes, distal outcomes, logic model)

7. Complete Evaluation Plan

8. Develop data collection plans

9. Create/find measures for primary data collection.

10. Adapt secondary data collection measures.

11. Develop evaluation consent procedures.

12. Develop plan for delivering data to the evaluation team.

13. Prepare and submit OMB application

14. Prepare and submit IRB applications-

15. a. What were the human subject considerations?

16. Develop MOU

17. Promote buy in for the evaluation from grantee and stakeholders

2. Implementation

1. Identify critical proximal outcomes for formative testing

2. Data system developed for measuring and reporting outcomes

3. Complete data sharing agreement

4. Submit AFCARS/NCANDS data

5. Submit SACWIS data

6. Data Quality Improvement Plan standards established and met

7. Develop discrete case-level service cost measures

Dissemination

1. Are there staff members allocated to working on dissemination (Dissemination includes both internal and external communication about the PII initiative)?

1. I f yes, describe who has been involved and in what capacity?

Document specific dissemination activities

Activity	Purpose	Products*	Target Audience*	Reach*	Date

*

• List specific products that were created, conferences attended, journals published, etc. by name.

• Target audiences will be broad groups of people and organizations categorized by policy, practice, research, or cross-cutting.

• Reach is the number of people who received a message through the specified activity..

5. PII National Support Activities: This domain will explore the impact that non-site specific PII activities have had on site specific PII implementation and evaluation.

1. Benchmarks and Incentives:

   1. Please describe how the Benchmark structure has impacted the PII way of work or PII implementation/evaluation process.

Please describe how the incentive structure has impacted the PII way of work or PII implementation/Evaluation process.

2. Grantee Meetings:

   1. Were the grantee meetings helpful in providing the “big picture” of the PII approach?

   2. How was the information provided at the grantee meeting to foster implementation and evaluation?

   3. Which aspects of the grantee meetings were most helpful?

   4. Which aspects of the grantee meetings were least helpful?

3. Evaluation Liaison Meetings:

   1. Were the Evaluation Meetings helpful in providing the “big picture” of the evaluation?

   2. How was the information provided at the evaluation meeting used to foster program evaluation development?

   3. Which aspects of the evaluation meeting were most helpful?

   4. Which aspects of the evaluation meeting were least helpful?

4. PII-TTAP Support:

   1. Describe the impact of the assistance provided by TTAP on each stage of the implementation process (exploration, installation, initial implementation)

   2. Indicate if there was a stage of implementation at which TTAP support was most needed.

   3. Describe the impact of the assistance provided by TTAP on the evaluation process.

   4. What aspect of the TA was most helpful/significant to the implementation process (action planning, usability testing, development of communication and linking protocols, terms of reference, and theory of change, development of implementation plan, etc.)?

   5. What aspect of the TA was least helpful to the achievement of key implementation activities?

5. Peer to Peer Learning:

   1. Indicate which sessions of Peer TA had grantee representation.

   2. Overall, did the Peer to Peer TA sessions provide information that assisted with the implementation and/or evaluation of PII?

   3. Which sessions were most helpful?

   4. Which sessions were least helpful?

6. Evaluation Support:

   1. Describe the impact of the work done by the Evaluation Team on the evaluation process (research reviews, data mining, PICO and logic model development, formative testing, and summative evaluation).

   2. Describe the impact of the work done by the Evaluation Team on the implementation process.

   3. What aspect of the Evaluation Team support was most helpful?

   4. What aspect of the Evaluation Team support was least helpful?

7. Dissemination

   1. Describe the impact of dissemination efforts on implementation and evaluation?

2. Describe the impact of the dissemination policies on implementation and evaluation activities?

8. A goal of PII is to identify, develop, refine, and successfully implement interventions that are effective in reducing long-term foster care (LTFC) and have a high likelihood of producing similar reductions in LTFC when fully replicated in other jurisdictions. Has being a part of a larger initiative designed to create replicable interventions impacted implementation and evaluation activities? If yes, describe how implementation and evaluation have been impacted by the goals of PII.

6. Implementation Outcomes: This domain will connect the events documented in the previous domains and relate them, as appropriate, to a consensus set of implementation outputs (ie: appropriateness, adoption, acceptance, feasibility, fidelity, sustainability, penetration).

Appropriateness: What activities are associated with addressing the perceived fit of the intervention with practice?

Adoption: What activities address the uptake or intention to try the intervention?

Acceptability: What activities were completed to ensure that the intervention was agreeable, palatable, and satisfactory?

Fidelity: What activities were performed related to the adaptation of the EBD, staff development, supervision, coaching, performance assessments, and decision support data systems?

Feasibility: What activities were used to promote the actual fit of the intervention for everyday use?

Implementation Cost: what activities have been undertaken to determine the cost effectiveness of the program.

Penetration: What activities have promoted access to services?

Sustainability: What activities has the grantee engaged in to ensure continuation of the program?

13