Next Generation of Enhanced Employment Strategies Project (SECOND SUBMISSION)

attachment B MATHEMATICA

Next Generation of Enhanced Employment Strategies Project

Brainstorming Meeting Guide

We would like to engage you, a small group of [INTERVENTION NAME] staff and leadership, in a discussion about [INTERVENTION NAME]. Our goal is to engage you in some activities to [if not already documented] document [INTERVENTION NAME]’s theory of change. The activities will also uncover strengths on which to build as well as challenges to program success, and identify solutions to address these challenges. We will use the information from this activity to determine what, if any, technical assistance you might need before being ready to participate in a rigorous evaluation. We will provide a short write-up of our findings to you regardless of whether you eventually are selected and agree to participate in the evaluation.

Your participation in this session is voluntary. Our discussion today will take about 3 hours. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0356 and the expiration date is June 30, 2021. During our discussions, anything you say will be kept private. We won’t attribute anything said here to any specific person or job title.

Before we begin, do you have any questions about the project or this activity?

Icebreaker activity (10 minutes)

Let’s start off by introducing ourselves. We have distributed some index cards and markers. We would like you to take a couple of minutes to draw a picture or write a short phrase on the card that describes something important to you. It can be in your personal or professional life. [Allow three minutes.]

Now, let’s go around and introduce ourselves and what you wrote on your card. [Allow seven minutes.]

Stakeholder mapping (10 minutes)

For all the activities we will do together today, we want to be sure we keep in mind the perspectives of key stakeholders. Stakeholders are people or groups of people who have an interest in [INTERVENTION NAME]. They include clients and staff, but might also include community partners, funders, and others.

With that in mind, our first exercise is called stakeholder mapping. Stakeholder mapping helps us understand and visualize the complex environments in which programs operate and decisions are made. During this exercise, we will collaborate to come up with a comprehensive list of the key stakeholders for [INTERVENTION NAME] and how they relate to each other.

Please take two minutes to jot down as many stakeholders as you can think of. Then we will share our lists, try to group the stakeholders into types, and add some arrows indicating the relationships among them.

[Allow two minutes for brainstorming. Then, the lead facilitator invites sharing from the group. Co-facilitator documents stakeholders on a white board/projected screen so all participants can see. Lead facilitator verbally summarizes the key stakeholder types and their stake in the intervention, indicating their stake through directional arrows with a short description. Allow eight minutes for report out and summary. The picture below illustrates the output from stakeholder mapping.

Image copyright of LUMA Institute]

Intervention theory of change (50 minutes)

In this exercise, we are going to work together to describe [INTERVENTION NAME]’s theory of change, which we call a roadmap for change: the activities you do, the attitudes and behaviors of clients, staff, and other stakeholders that you expect to change, and how those attitudes and behaviors should lead to improved client outcomes. A well-documented roadmap for change that clearly articulates the causal links between what you do and how client outcomes should change as a result is important to have in place before participating in an evaluation. It helps us better understand your program, and it helps you put stakes in the ground around what success looks like for your program. This can become a recipe for strong and consistent implementation over time.

We will use the activities in this session to document your roadmap for change and set the stage for a deeper dive into implementation successes and challenges in the next session. [Allow 5 minutes for introduction, instructions, set up.]

[Co-facilitator divides white board/projected screen into three sections, labelling the top of each section with “Activities,” “Changes in attitudes, behaviors, and skills,” and “Changes in outcomes.”]

1. Activities

First, let’s start with what activities [INTERVENTION NAME] is currently engaging in. Use the stakeholder types we just identified to think comprehensively about the activities. For example, services offered to clients are one type of activity. But also think about activities that target staff or other key stakeholders.

Please take three minutes to jot down as many activities [INTERVENTION NAME] engages in as you can think of. Then we will share our lists.

[Allow three minutes for brainstorming. Then, the lead facilitator invites sharing from the group. Co-facilitator documents activities on the part of the white board/projected screen that says “Activities” so all participants can see. Lead facilitator verbally summarizes the activities and probes the group if no activities were identified for some of the stakeholder types. Allow twelve minutes for report out and summary.]

2. Changes in outcomes

Now let’s talk about what client outcomes you are hoping to change through those activities you just identified. What changes in client outcomes would indicate to you that your program is working?

Please take three minutes to jot down as many client outcomes as you can think of, keeping in mind that a variety of stakeholder types might have influence over those client outcomes. Then we will share our lists.

[Allow three minutes for brainstorming. Then, the lead facilitator invites sharing from the group. Co-facilitator documents client outcomes on the part of the white board/projected screen that says “Changes in outcomes” so all participants can see. Lead facilitator verbally summarizes the outcomes and probes for additional details as needed. Allow twelve minutes for report out and summary.]

3. Changes in attitudes, behaviors, and skills

Now it’s time to link the activities we just talked about to the changes in attitudes, behaviors, and skills that would be needed to get to the desired outcomes. For example, if providing soft skills training to clients is an activity and finding employment is an outcome, what client skills would need to change as a result of training to lead to them finding employment? Remember to think not just about changes in client attitudes, behaviors, and skills, but also what changes would be needed from other stakeholders.

[The lead facilitator invites sharing from the group, co-facilitator documents changes in attitudes, behaviors, and skills on the part of the white board/projected screen that says “Changes in attitudes, behaviors, and skills” so all participants can see. Lead facilitator verbally summarizes how the articulated activities link to outcomes through changes in attitudes, behaviors, and skills and probes the group if no strategies are linked to changes in outcomes or vice versa. Allow fifteen minutes.]

Identifying and unpacking successes and challenges (50 minutes)

We are going to turn now to identifying successes that you have experienced and challenges that you have faced while delivering [INTERVENTION NAME]. Successes present bright spots or strengths on which to build, while challenges present opportunities for improvement. For this activity, we will divide into two groups.

[Lead facilitator divides the attendees into two groups of about three people each for the next exercise. Lead facilitator and co-facilitator each facilitate one small group. Allow 5 minutes for introduction, instructions, getting into groups.]

1. Discuss successes and challenges

[In small groups]. Let’s spend a couple of minutes discussing [INTERVENTION NAME]’s successes and challenges. From our roadmap exercise, it seems that the successes you have experienced include [TAILOR AS APPROPRIATE USING ROADMAP EXERCISE: recruitment and enrollment; quality and consistency of service delivery; availability of services to address client needs; relationships with contracted service providers or other partners]. And some of the challenges include [TAILOR AS APPROPRIATE USING ROADMAP EXERCISE: recruitment and enrollment; quality and consistency of service delivery; availability of services to address client needs; relationships with contracted service providers or other partners].

Which challenge seems the highest priority for you to address right now? [Allow up to 4 minutes for discussion.]

2. Problem tree analysis

Now we’re going to use an analytic technique called a problem tree analysis to examine the causes (roots) and effects (branches) of [IMPLEMENTATION CHALLENGE]. Let’s start with the roots. Take a few minutes to jot down on a piece of paper all the causes you can think of that contribute to [IMPLEMENTATION CHALLENGE].

[Allow 3 minutes for independent brainstorming. Then, facilitators invite individuals to share in their small group, using a white board/projected screen to document the roots of the problem. Allow 4 minutes for sharing.]

Now let’s turn to the effects. What are some of the effects of [IMPLEMENTATION CHALLENGE]? It might help to think back to your stakeholder map, to be comprehensive.

[Allow 3 minutes for independent brainstorming. Then, facilitators invite individuals to share in their small group, using a white board/projected screen to document the effects of the problem. Allow 4 minutes for sharing.]

3. Bulls-eye diagram

We’ll spend the next part of our time together coming up with potential solutions to the [IMPLEMENTATION CHALLENGE]. Before we do that, let’s prioritize the root causes we just identified. [Facilitator indicates a bulls-eye diagram with three concentric circles on white board/projected screen. The circles are labelled “most important,” “somewhat important,” and “a little important.”]

Work together for the next seven minutes to rank all of the causes identified in the previous activity by their importance in causing the challenge. By this we mean their relative contribution to causing the challenge. As you rank them, write them on the board/type them on the computer. [Allow 7 minutes for the activity.]

4. Full Group Report Out

[One person from each small group will summarize the implementation challenge the group identified, the root causes and effects, and which causes the group prioritized. Allow ten minutes for each group, for a total of 20 minutes.]

Develop potential solutions (50 minutes)

In the final portion of this meeting, we will begin developing innovative solutions to the root causes you just identified as being the most important.

1. Brainstorming

[In small groups.] You have in front of you some sticky notes. Take a few minutes to jot down ideas for how to address the most important root causes we just talked about. Put one idea on each sticky note, and then we will share our ideas and cluster them by themes. Try to think creatively, as if you had unlimited resources. [Allow 4 minutes for individual brainstorming. Then, facilitators invite individuals to share in their small group by placing their sticky notes of ideas on a white board or other flat, vertical surface. Facilitators cluster ideas that are similar. Allow 14 minutes for reporting out and clustering.]

2. Impact to difficulty matrix

[In small groups.] Each person select two of their favorite ideas and take the sticky notes with the idea written on them. It doesn’t have to be one of your ideas—it can be someone else’s idea. Now, as a group, take a few minutes to rank the ideas on how much of an impact you think they will have in addressing your implementation challenge. You can arrange them along a horizontal axis from low on the left to high on the right. [Allow 5 minutes for discussion.]

Now, move the ideas vertically to indicate their relative difficulty, while keeping them in the same horizontal plane. The idea that is the least difficult to implement should stay where it is; the idea that is the most difficult should be the highest up on the wall. [Allow 5 minutes for discussion. Then, facilitators draw (if white board) or otherwise indicate four quadrants, labelled as in the figure below.]

In small groups, facilitators lead a discussion about each matrix. Start with the ideas the group identified as “Quick Wins” in the high impact, low effort quadrant. These efforts should be prioritized first. Next turn to “Can Dos”, the relatively low impact, but also low effort ideas. These should be prioritized second. Next turn to Long-term Investments—things that could have high impact, but also require high effort. Planning for these now might be a good use of time, but recognize that the payoff might not be realized for quite some time. Determine whether to move forward on these. Finally, discuss the “Luxuries,” which would be nice to do, but because they are relatively low impact and would require relatively high effort, they are likely not worth pursuing at this time. Allow 8 minutes for discussion.]

3. Full Group Report Out

[One person from each small group will summarize the solutions the group prioritized. Allow a total of 14 minutes.]

Recap and Next Steps (10 minutes)

We have covered a lot of ground during this session. As the next step, we will document the output of our work here so that you can use it to inform your programming. We will also be having further conversations with program leadership about potentially including [INTERVENTION NAME] in the NextGen Project. If it is included in the project, we might be able to help you think more about how to implement some of these activities and see whether they are working, through what we call “formative evaluation.”

Before we go, would you like to share any reflections from this exercise, or anything else we have covered today?

3