Child Care Research and Evaluation Capacity Building Center Needs Assessment

Instrument 2: Focus Group Protocol

Virtual Focus Group Protocol

Needs Assessment

Child Care Research and Evaluation Capacity Building Center

This document is the virtual focus protocol for the Needs Assessment within the Child Care Research and Evaluation Capacity Building Center. In each section, we indicate both what the participant(s) will see as part of the virtual focus groups, and we indicate the text the facilitator will read to the participant(s). The text directly underneath each slide or image is read aloud while participants view that slide or image. Any facilitator text or text in slides in italics and/or in brackets ([ ]) are directions for the facilitator, or are places where we will enter information in advance of the focus group. Those locations will either be adjusted prior to the focus group, once that information is known, or will not be read aloud.

In each virtual focus group, we will go through the first 3 slides with participants from each Lead Agency individually, then we will go through slides 4-6 with the full group from all Lead Agencies.

SLIDE 1:

What participants will see:

What the facilitator will read:

[Start the session and admit participants from one Lead Agency into the main meeting room, continue through slide 3, and repeat until all participants are screened]

Welcome! Thank you so much for agreeing to participate today.

I am [Name and Title of Urban Institute Staff Member], your facilitator for today. I work for the Urban Institute in Washington, DC, a nonprofit, nonpartisan research organization. My colleague [Name of note- taker] is here to take notes. My colleague [Name of Tech support person] is here to support everyone in fully participating in the session. If you are having trouble hearing or seeing, then please send an email to or call [Name of Tech support person]. He/she will help you get reconnected. We are all members of the team supporting the work of the Child Care Research and Evaluation Capacity Building Center.

Everyone invited to participate today are staff in Child Care Development Fund (or CCDF) Lead Agencies. Your agency participated in the Child Care ECB Center Survey although each of you may or may not have contributed to filling out that survey directly. That survey and this focus group are part of an information gathering effort funded by the Office of Planning, Research, and Evaluation (OPRE) which is a research unit inside the U.S. Administration for Children and Families within the U.S. Department of Health and Human Services. Gathering the information is one task of the Child Care Research and Evaluation Capacity Building Center that OPRE contracted with the Urban Institute and Mathematica, an independent research organization with offices across the country, to operate. We are gathering information from the CCDF State, Territory and Tribal Lead Agencies about research and evaluation perspectives, strengths and challenges so that we can better understand what resources and supports may help you and your agencies use and do research.

Our session today will last for about 90 minutes (or until [provide time that is 90 minutes after the planned start time])

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number and expiration date for this collection are OMB 0970-0356, Exp: 02/29/2024.

[Advance Slide]

SLIDE 2:

What participants will see:

What the facilitator will read:

This focus group is a private meeting. By that I mean, only the people invited to attend should be logged in to the meeting and/or able to hear you and your fellow participants.

We need your help in keeping the meeting private because we can’t tell if you are sitting in a room with other people who were not invited to the meeting. If you are, then we ask that you please pick up your handset or put on a headset or ask others to leave the room so that only invited participants can hear what other participants are saying.

Are you in a private space where no one else can see your screen, see what others are writing in chat, or hear what other participants are saying?

[If someone says no, discuss whether they can fix the situation. If no, explain that we will need for them to exit the meeting. We will follow up after the meeting to let them know if it may be possible for them to join another meeting.]

[If yes, then continue] [Advance Slide]

SLIDE 3:

What participants will see:

What the facilitator will read:

The names of the states, territories, and tribes participating in the focus groups will be anonymized to in our report to OPRE, and shared in ACF. The information you share will be combined with the information shared in the survey to help us better understand our agency’s needs, interests, and strengths. Findings from the study will inform ACF activities and may be incorporated into documents or presentations that are made public, but we will not attribute information to a specific individual, state, territory or tribe in public materials.

We will be combining what we hear in this focus group with the four other focus groups that we are conducting. In each focus group, there are three-four agencies participating, so about 15 to 20 agencies in all. We will not connect your individual name with anything you say out loud or through the “chat” box. We are using what we learn in the focus group to help us better interpret the data from the survey responses, and to better understand the context of the survey answers.

Everyone who works on this study has signed a Staff Confidentiality Pledge prohibiting disclosure of anything you say during the focus group that would allow someone outside the study team, including government staff and officials, to identify you. However, because identities of CCDF Administrators are public information and you may interact with some ACF staff regularly, they may figure out who you are from the remarks that we report. The only exception is a researcher may be required by law to report suspicion of immediate harm to yourself, children, or others.

Your participation is voluntary, but OPRE and the Child Care ECB Center can better help you if we know more about what your agency needs and what you and your team think will be most helpful. The goal of this study is to get information that will help ACF better support CCDF Lead Agencies. Your responses to questions in the focus group will be anonymized. We believe the risks of participating are small, however, other participants in the focus group may talk about what they hear in the focus group sessions. In addition, if ACF staff are very familiar with agencies, they may be able to deduce participating agencies by disaggregated themes or particular quotes. If you choose to participate, you can decline to answer specific questions.

The study has been approved by Office of Management Budget [insert official OMB language]. The study team, OPRE, and other ACF Departments and Technical Assistance Centers will use the information provided to suggest and develop resources and supports for CCDF Lead Agencies to help you do and use research.

Do you agree to participate in the focus group?

[Wait for response from each person. If yes, move them into the breakout room. If no, thank them for their time and exit them from the meeting.]

We take your privacy seriously. We will be taking notes as you talk and recording the discussion, but we will only share the recording and the raw notes with members of the study team to help us make sure we have captured what you said correctly. The notes and recording will not be shared with our funder or anyone outside of the study team. Once the project is complete, we will destroy the raw notes and recording to maintain the privacy of your individual perspectives.

We will only record the discussion if that is OK with everyone participating. Are you OK with us recording the discussion?

[Wait for and write down the answer. If anyone says no, then the meeting must not be recorded.]

Thanks again for participating today. Now we are going to move you to another meeting room where you can mingle with other meeting participants. [Either, You are the first group and others will join you shortly OR Others are already waiting.] We will be asking each team these same questions and then admitting them to the mingle room. Once we have spoken to each team, then we will move you back to the main meeting room and begin the focus group.

[Repeat slides 1-3 until all individuals have been given the privacy and consent information]

The slides and images from this point on will be viewed by all participants in each focus group as a group of Lead Agencies.

SLIDE 4:

What participants will see:

What the facilitator will read:

[Announce in the Mingle Room that we are bringing everyone back to the main room to begin the focus group. Close the breakout room to bring everybody back into the main session]

Welcome to the main focus group session. We are almost ready to start asking you questions. First, we’d like each of you to introduce yourselves.

I will say your first name and the name of the [state, territory or tribe] you are representing. I will start with [name of state, territory or tribe] and go in alphabetical order. You should tell us how you’d like us to refer to you during the focus group, and the name of your agency.

[Name of second participants in Name of State/Territory/Tribe]. [Repeat script from above for each participant].

For the remaining 60 minutes, I am going to be asking you questions. Each question is for anyone in the group to answer. I will put each question in the chat so you can remember what it is. We won’t have the slides up during our discussion because we want you all to be able to see each other so it feels like we are in the room together.

After I ask the question, I will pause so that if you want to answer the question, you can raise your actual hand [Facilitator raise your hand so they can see what you mean] or just start talking. If more than one of you begins at once, then I will call on one of you to start.

Remember that answering questions is voluntary. You do not have to answer any questions that you don’t feel comfortable answering.

But, I will call on each agency team so that you have the space to provide an answer if you want to do so. If you don’t want to do so, then you can just say “Not at this time.” We are not trying to get consensus or agreement from the people in the group. We want to hear your own perspectives about your own experiences.

I may have to interrupt some answers if we are running long on a particular question. I want to be sure that we ask you all the questions. If I must interrupt and we have time at the end, I will offer the opportunity for you to finish your answer then.

Does anyone have any questions?

[Advance Slide]

SLIDE 5:

What participants will see:

What the facilitator will read:

In this virtual focus group, we will ask you to respond to questions about your agency’s interest in research, your agency’s current strengths and challenges in doing and using research, and what you think could best support your agency to do or use more research in the future. These questions are about your experiences and perceptions. There are no right or wrong answers. We have asked you to participate so that we can learn from YOU!

We want this to be like a discussion so you can add on to what someone else has said. We know that everyone’s experience is a little bit different. You should not feel like everyone needs to agree. It is helpful for us to hear where your experiences are similar or different from each other, and why you think that might be the case.

Does anyone have any questions? [Advance Slide]

SLIDE 6:

What participants will see:

What the facilitator will read:

Today our questions will use the words research, evaluation and data analysis. When we say these words, we want you to be thinking of them in this way.

Think of research, evaluation, and data analysis inclusively… Some examples that might support your CCDF operational and policy decisions are:

You might examine the data to help in making operational decisions about how many licensing consultants you need

You might map data to better understand child care supply and gaps

You might collect or contract with someone to collect data about the schools and agencies that offer education and professional development to child care teachers in your state to understand needs and gaps You might contract with a consultant or a research organization to help you understand why some parents are not participating in the child care subsidy program

You might design and conduct an evaluation of a program you developed to help improve the quality of child- teacher interactions

In other words, we are including in this focus group discussion, analysis of every day data for every day CCDF decisions and collection of data to answer specific CCDF policy questions. These could be data analysis or research activities you do inside your agency with agency staff or that you contract with others to help you do.

Does anyone have any questions before I start asking you the focus group questions?

OK, great. I’m going to take down the slides now so we can more easily see each other for discussion. I am putting the first question in the chat right now.

[Take down the slides]

From this point on, the slides will be taken down. The following questions and text will be asked one at a time through the Zoom Chat box to the focus group participants, as displayed on pp. 12-22:

Question 1:

• What are some examples of research that you found that helped you address an issue or learn about a program in your state, territory, or tribal CCDF Agency?

• What was frustrating in your search for research?

• For those of you that found helpful research, where did you find it and what was helpful about it?

Question 2a:

• Do any of your agencies help staff learn more about how to do or use research? How do you do that?

Question 2b:

• Does your agency encourage staff to regularly ask questions about data or research?

Question 3:

• What are some current agency priorities that you are interested in investigating through more data analysis or research? What are some questions that you are interested in or have been trying to answer recently?

• What research or analyses have you begun in these areas?

Question 4a:

• What are the strengths in your agency that have helped you answer the questions you have?

• What have been some lessons learned?

Question 4b:

• What are the barriers you faced that made it hard to answer the questions?

• What have been some lessons learned?

Question 5:

• What information or supports could help you overcome your barriers?

Question 6:

• What could help your agency or other agencies build the strengths that have helped some agencies succeed in answering their policy or operational questions?

Question 7:

• What are some examples of ways that your agency has used the data analysis or research it has conducted to help in decision making? Think about both operational and policy decisions.

• Is the data analysis or research your agency conducts usually used in decision making?

  • If yes, why do you think that is?

  • If no, why do you thing that is?

Question 8:

• Does anyone have any other comments you want to add about resources you think would be useful, challenges you have faced, or strengths your agency has in doing or using research?

Thank you!

• If you have any questions, please don’t hesitate to contact me, [name] at [email] and [phone number].

The images displayed until the end of this document are screenshots of what participants will generally see on their screen during the virtual focus group. It shows the setup of Zoom and how each question will be shown to the participants through the Zoom Group Chat (on the right side of each image). A difference is that where it says “Focus Group” with the black background on each image, participants will instead see video of other participants, or the facilitator, depending if they have chosen to have their video on. We want to create a setting like they are in a room together. Also, we will add the questions into the Chat one at a time, but given how Chat works, sometimes more than one question will appear to the viewer. And, because participants could also add text into Chat, exactly what is on the screen at any moment may look a little bit different from what it looks like in this document.

Question 1:

What participants will see:

What the facilitator will read:

We are going to start by reflecting on research that other people have conducted that your agency may have found helpful, how you found the research or frustrations you had in trying to find it, and what was useful about what you did find.

What are some examples of research that you found that helped you address an issue or learn about a program in your state, territory, or tribal CCDF Agency?

What was frustrating in your search for research?

For those of you that found helpful research, where did you find it and what was helpful about it?

[Put next question in chat; remember, you can go in order or you can change the order based on the flow of the conversation]

Question 2a:

What participants will see:

What the facilitator will read:

Thanks, it is helpful to know what your experience has been in using or trying to use existing research.

Not everyone gets training to use or do research when they are in school. Many folks who work in CCDF Agencies came up through program services or went to school to learn how to serve children or families, and not specifically how to examine data or research.

So, we were curious about how much internal training or support your agencies provide to help staff learn about analyzing data, conducting research, or using other people’s research?

[Put next question in chat; remember, you can go in order or you can change the order based on the flow of the conversation]

Question 2b:

What participants will see:

What the facilitator will read:

What about encouraging staff to ask questions about data or research?

• Does your agency encourage that?

• How do they encourage it?

[Put next question in chat; remember, you can go in order or you can change the order based on the flow of the conversation]

Question 3:

What participants will see:

What the facilitator will read:

Now, I’d like you to think about current agency priorities that have you thinking about questions you’d like to answer related to your administration of CCDF in your state, territory, or tribe. For example, you might have questions related to child care subsidy, child care access, child care quality, workforce development, or other issues related to children, families, or child care providers.

Tell us what your priorities and questions are (or were), and what analyses or research you have been pursuing or you are thinking about pursuing to answer them.

What supports would help you answer the questions you have?

I’m going to ask a follow up question about this in a minute.

[Put next question in chat; remember, you can go in order or you can change the order based on the flow of the conversation]

Question 4a:

What participants will see:

What the facilitator will read:

Following up on the last question, for those of you who have been able to answer some of the questions you have, what are some of the strengths in your agency that have helped you do that? What are some lessons you have learned along the way that may be helpful to other organizations?

[Instruction to Facilitator, if not mentioned, probe on the following]

• Staff skills

• Research partnerships

• Staff dedicated to research

• Access to data

• Data systems

• Data tools

• Space/time for research and policy staff to interact

• Funding devoted to research/analysis

[Put next question in chat; remember, you can go in order or you can change the order based on the flow of the conversation]

Question 4b:

What participants will see:

What the facilitator will read:

For those of you who faced challenges in answering your questions, what have been the biggest barriers that have made it harder for you to answer your questions?

[Instruction to Facilitator, if not mentioned, probe on the following]

• Staff skills

• Research partnerships

• Staff dedicated to research

• Access to data

• Data systems

• Data tools

• Space/time for research and policy staff to interact

• Funding devoted to research/analysis

[Put next question in chat; remember, you can go in order or you can change the order based on the flow of the conversation]

Question 5:

What participants will see:

What the facilitator will read:

Wow, that is some really great information to help us understand some questions you are interested in answering, the challenges you have faced, and what has helped you in overcoming them or gotten in the way so far.

Reflecting on your barriers, what information or supports could help you overcome those barriers?

Are there other barriers you didn’t mention before that you think information or supports could help you overcome?

[Instruction to facilitator: If they ask what you mean by information or supports, ask them first to just say what they are thinking. If they only say more money, then give them examples like topical briefs, national webinars on the subject, a help line, peer support networks, etc.]

[Instruction to facilitator: probe on

• Modes of support as indicated in the list above

• Topics for support “how to….”

[Put next question in chat; remember, you can go in order or you can change the order based on the flow of the conversation]

Question 6:

What participants will see:

What the facilitator will read:

Thanks, I appreciate you sharing those ideas.

Earlier, some of you talked about strengths that your agency has that helps you succeed in doing research. You mentioned [list out items they mentioned and put list in chat].

For those of you that have those strengths in your agency, how do you think we could help other agencies build those same strengths?

If you don’t feel like your agency has those strengths, do you feel like there are strategies that could help you build them?

These might be the same strategies you talked about when we discussed barriers, but thinking about strengths might give you some different ideas.

[Put next question in chat; remember, you can go in order or you can change the order based on the flow of the conversation]

Question 7:

What participants will see:

What the facilitator will read:

Thanks for sharing these ideas!

We are almost finished.

Now, I’m going to ask you to reflect on the ways that your agency has used the data analysis or research it has conducted.

Please share some examples of whatever comes to mind.

[Instruction to Facilitator, if no one is mentioning examples, bring up the questions they were interested in before and ask about those.]

• Is the data analysis or research your agency conducts usually used in decision making?

  • If yes, why do you think that is?

  • If no, why do you think that is?

[Instruction to Facilitator, if not mentioned, probe on the following as possible facilitators or barriers]

• Staff skills

• Research partnerships

• Staff dedicated to research

• Access to data

• Data systems

• Data tools

• Space/time for research and policy staff to interact

• Funding devoted to research/analysis

[Put next question in chat; remember, you can go in order or you can change the order based on the flow of the conversation]

Question 8:

What participants will see:

What the facilitator will read:

Thanks for sharing.

Does anyone have any other comments you want to add about resources you think would be useful, challenges you have faced, or strengths your agency has in doing or using research?

[Put thank you in chat]

Thank you!:

What participants will see:

What the facilitator will read:

We so appreciate you spending the last 90 minutes with us sharing your experiences and perspectives. We have learned a lot from you already. It will be a big help in considering how the Child Care Research and Evaluation Capacity Building Center can best help all CCDF Lead Agencies do more of the research, evaluation, and data analysis they want to do.

Please respect the privacy of your peers as you leave the meeting. It will be natural to talk about what you learned or heard in the meeting, but please don’t share information attached to a particular person’s or agency’s name.

If you have any follow up questions, please don’t hesitate to contact me, [name] at [email] and [phone number].

2