Cross-Center Evaluation: Outcomes of and Satisfaction with Tailored Services Survey (Intensive projects) - input from team members

PURPOSE: This survey will be used to assess child welfare staff perceptions of the outcomes of intensive courses of tailored services and their satisfaction with those services. (Intensive services typically last 9 or more months and involve 20 or more hours of service from a Center.) The survey will be administered at the close of intensive service projects and will be completed by the project team lead with input from the rest of the team.

OMB #: 0970-0XXXX

Expiration Date: XX/XX/XXXX

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN: This collection of information will be used to assess changes in organizational capacity and satisfaction with tailored services among those receiving services from the Capacity Building Collaborative. Public reporting burden for this collection of information is estimated to average 15 minutes per response by the team lead, plus 10 minutes per additional team member providing input. These estimates include the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: 0970-XXXX, Exp: XX/XX/XXXX. If you have any comments on this collection of information, please contact Chereese Phillips, ACF, Administration on Children, Youth and Families by e-mail at Chereese.Phillips@acf.hhs.gov.

Evaluation of the Child Welfare Capacity Building Collaborative

Outcomes of and Satisfaction with Tailored Services Survey

Thank you for the time you took to coordinate a response to this survey. The Children’s Bureau funded James Bell Associates (JBA) and ICF to conduct a cross-center evaluation of the Capacity Building Collaborative. We are asking your team to complete this survey because [Jurisdiction] is a recipient of capacity building services from the [Center] and has received services around [component name]. We realize how limited your time is; the survey should take an average of 15 minutes to complete, after obtaining input from your team members. The survey consists of [3/4] sections:

• Section 1: Change Management Knowledge and Skills

• Section 2: Changes in Capacity

• Section 3: Satisfaction with Center Services

• [CBCS only] Section 4: Center for States Practice Model Questions

Your team’s participation in the survey is voluntary. You may refuse to take part in the evaluation or exit the survey at any time. You may decline to answer any question you do not wish to answer. There are no risks involved in participating in the survey. While you will not receive any direct benefits from participating in this survey, your responses will provide information about the outcomes of these services and your satisfaction, which will support Center improvements. It is not recommended to take this survey on a mobile device as questions may not display properly.

Your survey responses will be sent to the Qualtrics survey platform where data will be stored in a password protected electronic format. Only JBA team members will be able to access survey data. Your name or any other personally identifying information will not appear in any report of the information provided in the survey so your responses to this survey will remain private to the extent permitted by law. Data will be shared with the [Center]. Survey data will be kept by JBA through MONTH YEAR.

If you have questions or concerns about the survey or the evaluation, you may contact Project Director Jim DeSantis at DeSantis@jbassoc.com.

Please select your choice below. You may print a copy of this consent form for your records. Clicking on the “Agree” button indicates that:

• You have read the above information;

• You voluntarily agree to participate;

• You are 18 years of age or older.

Agree

Disagree [If selected, exit survey]

Section 1: Change Management Knowledge and Skills

Below is a list of skills and approaches that people sometimes use when solving problems or trying to make improvements to their agency’s services or functioning. No team is expected to have or develop all these skills.

Please think about the team of people from your [agency/Court Improvement Program (CIP)/tribal child welfare program] who are working with [Center Name] on [name of project from CapTRACK]. For each item listed, please describe the team’s current level of knowledge, and your understanding of the team’s level of knowledge prior to the start of this project. 

CM Knowledge & Skills	Current knowledge level						Knowledge level prior to the start of this project
	Not at all	Slightly	Moderately	Very	Extremely	D/K or N/A	Not at all	Slightly	Moderately	Very	Extremely	D/K or N/A
When exploring a problem, or making an improvement in your services, how knowledgeable is your team about ...
Identifying who (in your dependency court system/in your agency/among the group responsible for child welfare services in your tribe) you should involve, and why.
Identifying which external partners/community members you should involve, and why.
Identifying data sources and types of information that you have, or can collect, to explore the issue.
Assessing how widespread or prevalent an issue is.
Identifying groups that are most and least impacted by the issue.
Generating theories and ideas based on your data and information about what causes or contributes to the issue.
6
When considering ways to improve your services, how knowledgeable is your team about...
Finding research and/or peers with expertise to help you think about how you might make improvements.
Considering whether strategies fit your (agency’s/court’s/tribe’s) values and needs.
Assessing your (agency’s/court’s/tribe’s) current capacity and determining whether it is feasible you to implement a strategy.
When selecting or designing strategies, programs, or interventions to make improvements, how knowledgeable is your team about...
Identifying the core activities that make up your strategy and how these activities must be performed in order for your strategy to work.
Identifying specific behaviors that will let you know whether your strategy is being performed as intended.
When planning to implement a strategy, program, or intervention, how knowledgeable is your team about...
Developing capacities that will need to be in place so you can successfully implement what you intend.
Considering whether to pilot or conduct a phased implementation of an improvement.
Monitoring implementation and identifying and solving problems as they arise.
When evaluating the improvements you are trying to make, how knowledgeable is your team about...
Developing indicators and outcomes that let you know whether the core activities of your strategy are being implemented as intended.
Developing indicators and outcomes that let you know whether the problem or issue you are working on is improving.
Using the results of analyses to make adjustments to your activities.

Section 2: Changes in Capacity

These next (insert number) questions are about areas of organizational capacity that your team worked on according to your project workplan. For each item, please think about your (agency, CIP, tribal child welfare programs)’s current level of capacity and your your (agency, CIP, tribal child welfare programs)’s level of capacity prior to the start of this project. A definition of each organizational capacity that your team targeted is provided for your reference.

NOTE: This section has one survey item per sub-capacity. We will only administer the survey items that measure the project team’s targeted sub-capacities. This means most projects will get 2-4 items from this section, not all 29.

Assessment of organizational capacities	Current capacity level								Capacity level prior to the start of this project
	1 Area of clear weak-ness	2	3	4	5	6	7 Area of clear strength	Don’t Know	1 Area of clear weak-ness	2	3	4	5	6	7 Area of clear strength	Don’t Know
Organizational resources
How would you rate your agency’s capacity level in staffing resources? Staffing (people) refers to the stability and adequacy of staff numbers/FTEs to perform job duties related to organization’s desired level of practice. Staff have adequate time to participate in planning and/or designing innovation, staff have time to develop expertise/learn how to apply practice, adequate numbers of staff for service delivery.
How would you rate your agency’s capacity level in fiscal resources? Fiscal (money) refers to sources of revenue and funding mechanisms to implement and sustain improved practices/innovations.
How would you rate your agency’s capacity level in facilities, equipment, and technology resources? Facilities, Equipment, & Technology (things) refers to the space, materials, and technology required for service delivery, materials required to implement innovation, e.g., office space, database for data collection activities.
How would you rate your agency’s capacity level in informational resources and materials? Informational Resources & Materials (content) refers to best practice/evidence-informed, innovation, and/or program-related materials, peers and external specialists with best practice/innovation experience and expertise.
Organizational infrastructure
How would you rate your agency’s capacity level in governance/decision-making structures? Governance/Decision-Making Structures refers to strategic plans, governance structures, decision-making and approval processes. These include structures created to facilitate decision making related to selection, design, and implementation of initiatives to improve practice, including creating multi-disciplinary implementation teams.
How would you rate your agency’s capacity level in administrative structures? Administrative Structures refers to organizational and service delivery structures, including financial and accounting systems, contracting and procurement systems. Structures that are created or revised to facilitate desired practice, including revisions of budgeting, contracting, and procurement systems.
How would you rate your agency’s capacity level in policies, operating procedures, protocols? Policies, Operating Procedures, Protocols refers to standard operating procedures/guidelines, manuals, protocols, and policies. Creation or revision of policies, operating procedures, and protocols that operationalize desired practice.
How would you rate your agency’s capacity level in human resources, recruitment, and staff selection? Human Resources/Recruitment/Staff Selection refers to recruitment and hiring, job descriptions and qualifications, performance evaluation processes, retention efforts, performance incentives. Selection process to identify staff to implement innovations, revision of job requirements and performance evaluations to support desired practice.
How would you rate your agency’s capacity level in staff training? Training System refers to staff learning and training processes, pre-service, ongoing learning structures, and continuing education. Structures created to build staff knowledge of new practice, delivery of curriculum specific to innovation; development or revision of pre-service and in-service trainings to increase knowledge of innovation.
How would you rate your agency’s capacity level in supervising/coaching? Supervisory/Coaching System refers to development or refinement of a supervisory system, development of a system to provide support, consultation, coaching, and feedback to employees to promote skill-building and consistent practice. Creation of processes intended to build staff’s ability to apply innovation-specific practice in context.
How would you rate your agency’s capacity level in its service array/service delivery system? Service Array/Service Delivery System refers to service provision structure and options for children, youth and families, including structures for matching services to address individual needs, and networks of local organizations that coordinate and integrate to increase the safety, permanency, and well-being of children, youth and families.
How would you rate your agency’s capacity level in its information system/data supports? Information System/Data Supports refers to data collection, management and cleaning of information related to organizational systems (human resources, administrative, billing, case management). Data protocols, programming, coding; and ongoing procedures to collect and store data on operational and service indicators.
How would you rate your agency’s capacity level in CQI/evaluation? CQI/Evaluative System refers to processes for internal review of data that allow for fidelity/performance assessment and assist the organization in taking action steps in response to findings. Development of monitoring systems to assess fidelity of innovation and outcomes, and analysis of results to make adjustments to services.
How would you rate your agency’s capacity level in internal communication systems? Internal Communication Systems refers to communication structures and processes meant to inform staff, resolve issues, and improve services through information flow up and down levels and across divisions within the organization. Used to facilitate and structure work, and create feedback loops to support innovation.
How would you rate your agency’s capacity level in external communication systems? External communications Systems refers to processes that govern relationships with systems outside of the organization. Development or revision of structures that support communication with systems outside of the organization including MOUs, written agreements, and contracts.
Organizational knowledge and skills
How would you rate your agency’s capacity level in child welfare practice knowledge and skills? Child Welfare Practice Knowledge and Skills refers to awareness, understanding, and knowledge related to practice, application of knowledge to work with children, youth and families. Understanding new practice/innovation, ability to apply new knowledge in context, development of high levels of competency in desired practice.
How would you rate your agency’s capacity level in analytic/evaluative knowledge and skills? Analytic/Evaluative Knowledge and Skills refers to problem assessment, critical reflection, data analysis, development of measurable outcomes, and data presentation. Evaluating the effect and advantage of innovation, posing research questions, specifying target population, constructing a theory of change, evaluation of implementation and outcomes.
How would you rate your agency’s capacity level in policymaking/administrative knowledge and skills? Policymaking/Administrative Knowledge and Skills refers to efficiently organizing people, fiscal, and material resources, directing organizational activities toward common goals and objectives, crafting policies that clearly explain required processes and how to accomplish these processes to achieve organizational goals.
How would you rate your agency’s capacity level in leadership/management knowledge and skills? Leadership/Management Knowledge and Skills refers to strategy development and execution, developing and communicating vision and mission, setting goals and objectives, and monitoring performance. Planning, organizing, coordinating, prioritizing, delegating, and decision making. Skills in team formation, meeting management, and work planning.
How would you rate your agency’s capacity level in workforce development/supervision knowledge and skills? Workforce Development/Supervision Knowledge and Skills refers to developing others, adult learning strategies and techniques. Abilities related to design, development, and delivery of curriculum and training. Knowledge and skills related to successfully recruiting and retaining employees, effective supervisory and coaching strategies.
How would you rate your agency’s capacity level in cultural competence/humility knowledge and skills? Cultural Competence/Humility Knowledge and Skills refers to understanding of cultural issues and cultural groups, skills and techniques that reflect worker’s knowledge and respect of cultural group. Selection and/or adaption of culture-specific service approaches, tailoring of practice/innovation to more effectively serve cultural communities.
How would you rate your agency’s capacity level in change management/implementation knowledge and skills? Change Management/Implementation Knowledge and Skills refers to understanding of change management processes, including intervention selection and/or development, stages of implementation. Knowledge of how to support and facilitate change, including understanding of capacities that can support implementation and sustainment of change.
Organizational culture and climate
How would you rate your agency’s capacity level in clear organizational norms/values/purpose? Organizational Norms/Values/Purpose refers to norms and expectations regarding how people behave and how things are done in the organization; the values, goals, purpose and mission of the organization, and the fit of an innovation with the organization’s goals, values, norms, and needs.
How would you rate your agency’s capacity level in positive workforce attitudes/morale/motivation/buy-in? Workforce Attitudes/Morale/Motivation/Buy-In refers to workers’ commitment to the organization, staff clarity related to roles and responsibilities, workforce openness to change, workforce motivation and buy-in, perception of importance, and belief that resources are available to implement and sustain practice innovations.
How would you rate your agency’s capacity level in leadership vision/commitment? Leadership Vision/Commitment refers to leadership’s openness to change, commitment to new practice, alignment of innovation with leadership priorities, leadership’s communication of importance of intended changes to internal and external stakeholders, leadership’s dedication and protection of resources to implement/sustain changes.
Organizational engagement and partnership
How would you rate your agency’s capacity level in internal stakeholder engagement? Internal Stakeholder Engagement refers to connections and productivity of internal partnerships, identifying and engaging staff from multiple levels within the organization (executive, middle management, supervisors, frontline) to support improvement of practice, including selecting, designing, and/or implementing innovations.
How would you rate your agency’s capacity level in external stakeholder engagement? External Stakeholder Engagement refers to the number, breadth, and diversity of partners, identifying and engaging partners external to the organization (e.g. courts, tribes, private agencies, health providers, law enforcement, local government) to support service delivery, including selecting, designing, and/or implementing innovations.
How would you rate your agency’s capacity level in family and youth engagement/participation/buy-in? Family and Youth Engagement/Participation/Buy-in refers to partnerships with family and youth stakeholders who can provide input and feedback on the organization’s practice. Seeking input from family and youth who will experience new practices, involving youth and family in selecting, designing and/or assessing identified practice/innovation.
How would you rate your agency’s capacity level in community and cultural group engagement/participation/buy-in? Community & Cultural Group Engagement/Participation/Buy-in refers building trust, working relationships, shared vision and understanding between child welfare and the community. Understanding of, outreach to, and connections with cultural groups in the community. Seeking input to enable culturally-sensitive and community responsive child welfare practice

47. Has your work with the Center resulted in any changes in [your agency’s practice with children and families /your tribal child welfare program’s practice with children and families/ court practices]? If so, please describe those changes and be as specific as possible. (open ended)

Section 3: Satisfaction with Center Services

This next section asks about your satisfaction with the services you received from [Center].

Please indicate the extent to which you agree or disagree with each of the following statements.	Strongly Disagree	Disagree	Somewhat Disagree	Neither Agree nor Disagree	Somewhat Agree	Agree	Strongly Agree	Dont Know
Center representatives provided the subject-matter expertise needed to support our work.
Our team was satisfied with how Center representatives integrated capacity building and implementation activities into our discussion.
Center representatives had a good understanding of our [agency/ team] needs.
Center representatives communicated with us in ways that respected our values and the context of our [agency/court].
Our ideas and preferences about the service delivery were respected.
The Center's process was flexible enough to meet the needs of our [agency/team].
Center representatives engaged our [agency/team] in exploring and/or selecting strategies, activities, and tools that are aligned with our [agency/court] needs.
The knowledge and skills our [agency/team] acquired while partnering with the Center are directly applicable to our work in other areas.
The Center helped us to identify and set milestones and measurable outcomes to track progress and success.
The Center helped us to gather, identify and/or use information/data to document progress, and make decisions and improvements as needed.
The Center's services were well aligned with the needs of our [agency/tribe/team].
My [agency/Court Improvement Program (CIP)] is considering using a similar change process/implementation process in other initiatives.
I would recommend working with the Center to other [states/tribes/CIPs].
Overall, our team was satisfied with the services we received.

NOTE: Items in red will be tailored based on which center respondents received services from. CBCC respondents will see the terms “court” and “CIP.” CBCS respondents will see the terms “agency” and “state.” CBCT respondents will see “agency” and “tribe.”

Could your experience with the Center have been improved? YES NO

[Display only if Yes, is selected in previous question] Please describe how your experience with the Center could have been improved.

Section 4: Center for States Practice Model Questions

This last section includes items the Center for States would like your opinion on.

Please indicate the extent to which you agree or disagree with each of the following statements.	Strongly Disagree	Disagree	Somewhat Disagree	Neither Agree nor Disagree	Somewhat Agree	Agree	Strongly Agree	Dont Know
Center representatives conducted outreach to our agency in a variety of ways that capitalized on previous engagements and efforts.
Center representatives provided appropriate, timely responses to our agency communications
Center representatives brought clarity to conversations about capacities, processes, and services.
The Center helped create a team of representatives from all partner agencies that actively participated and collaborated to promote successful engagement.
Center representatives ensured the team understood the project scope and specific team member roles and responsibilities
The team operated in a way that maximized effectiveness of individual and collective efforts.
The team operated cohesively in setting and accomplishing tasks, providing constructive feedback, and decision-making.
Team members engaged in communications and supported the team’s continuous learning by sharing their own knowledge and experiences.
The partnership fostered mutual exploration of expectations through open dialogue and creation of a realistic scope of work for achieving shared goals.
Center representatives provided an appropriate amount of contact and support during the tailored service process.
Center representatives encouraged an atmosphere of individual and organizational self-reflection and pursued opportunities for continuous quality improvement.
Center representatives helped our agency increase our ability to monitor progress and make data-driven decisions.
Center representatives worked with the team to ensure reflective conversations about the project occurred regularly, including discussion of data about progress.
Center representatives facilitated discussions in an open environment that encouraged all team members to share their perceptions of initiative progress.
Center representatives worked with the team to ensure our agency was focused on planning the transition from the beginning of the process.
Center representatives helped our agency to reach a consensus on transitioning and closing services, while being respectful of individual opinions.
Center representatives supported our agency in identifying strategies for sustaining and applying what we have achieved through the project, and the commitments needed for that sustainment.
The transition process included a reflective discussion about lessons learned, strengths and challenges, and suggestions for improvement.
If services ended prematurely, or inconsistently with the proposed plan, Center representatives discussed the possibility of reengagement and supported an amicable closure.
Our agency had a strong working relationship with the Center team.
Center representatives built credibility by being well-informed about effective practices in child welfare.

Closing:

Thank You

1