Appendix
D
Afterschool Center Director Interview
Instructions for interviewers: Throughout the discussion, text in italics denotes instructions for interviewers. Text in brackets should be customized for the individual center director.
Thank you for taking the time to talk with me today. The U.S. Department of Education is collaborating with the 21st Century program office to conduct a national study that aims to support the 21st Century program’s continuous improvement.
As part of the study, my organization, Research for Action, is interviewing [center directors/site coordinators] to better understand how their centers approach quality improvement. Specifically, we would like to learn more about any improvement goals at your center for the [2021–2022 or 2022–2023] school year, training and coaching you received, and practices implemented by your staff for supporting students’ social and emotional skills. We would also like to collect some background information about your center to better understand your particular context.
We expect this interview will take about 60 minutes. You may choose not to respond to any of the questions or stop the interview at any point. All of your responses are confidential. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district, program, afterschool center, or person. If you have any questions about the study, you may contact Kristina Rall at KRall@mathematica-mpr.com or (202) 264-3468. If you have any questions about your rights as a research volunteer, contact HML IRB toll free at 1-202-753-5040 and reference IRB number 844MATH21.
I would like to record our discussion so I can listen to it later when I write up my notes. No one outside of our study team will listen to the recording, and it will be destroyed after I finish writing up my notes. If you want to say anything that you do not want recorded, please let me know and I will be glad to pause the recorder. Do you have any objections to my recording our discussion?
Do you have any questions before we begin?
I’d like to start by learning more about how you assess the quality of your center’s afterschool program and set improvement goals.
During the [2021–2022 or 2022–2023] school year, did your center’s afterschool program use any specific frameworks or standards for defining program quality?
If yes: Please describe the framework(s) or standards.
During the [2021–2022 or 2022–2023] school year, did you use any of the following methods or tools to collect information on the quality of staff practices at your center:
Observations of staff practices by completing a rubric, checklist, or other tool?
If yes: Which observation tool(s) did you use?
Observations of staff practices without completing a rubric, checklist, or other tool?
Staff members’ descriptions of their own practices?
Feedback from parents/guardians?
Feedback from students?
Any other methods or tools to collect information on the quality of staff practices?
Did your center identify any improvement goals for the [2021–2022 or 2022–2023] school year? Improvement goals are written statements that identify staff practices, student experiences or outcomes, or other aspects of the center that staff at the center are trying to improve.
If the center identified improvement goals: How many improvement goals did your center identify for the [2021–2022 or 2022–2023] school year?
If more than three improvement goals: What were the top three improvement goals that your center identified?
If three or fewer improvement goals: What were the improvement goals that your center identified?
If the center identified improvement goals: We have a few questions about how your center identified these improvement goals for the [2021–2022 or 2022–2023] school year.
Did information on the quality of staff practices have a major role, minor role, or no role in informing the improvement goals? If major or minor role: What type of information? Did the information show areas in need of improvement? Were these areas the same areas in which you set improvement goals?
Did input from staff have a major role, minor role, or no role in informing the improvement goals?
If major or minor role: How was the input collected? Was it collected at a meeting, through a survey, or in another way?
Did input from students or parents have a major role, minor role, or no role in informing the improvement goals?
If major or minor role: How was the input collected? Was it collected at a meeting, through a survey, or in another way?
Did center leadership play a major role, minor role, or no role in identifying the improvement goals?
If major or minor role: Please briefly describe how.
Did staff from your regional or district afterschool program play a major role, minor role, or no role in identifying the improvement goals?
If major or minor role: Please briefly describe how.
If the center identified improvement goals: We have a few questions about what role the improvement goals had in determining the center’s decisions.
Did the improvement goals have a major role, minor role, or no role in determining the types of training offered to staff? If major or minor role: Please briefly describe how.
Did the improvement goals have a major role, minor role, or no role in determining the types of one-on-one feedback provided to staff? If major or minor role: Please briefly describe how.
Did the improvement goals have a major role, minor role, or no role in determining staff hiring and retention decisions? If major or minor role: Please briefly describe how.
Did the improvement goals have a major role, minor role, or no role in determining activities offered to students? If major or minor role: Please briefly describe how.
Did the improvement goals have a major or minor role in determining any other decisions by the center? Please briefly describe how.
I’d like to learn a bit more about any training and coaching you may have received on quality improvement strategies during the [2021–2022 or 2022–2023] school year as part of your work for your afterschool center. Quality improvement strategies include systems, tools, frameworks, or procedures for assessing the quality of your center and planning and carrying out improvements. Did you receive any training or coaching on quality improvement strategies during the [2021–2022 or 2022–2023] school year?
If no, skip to question 8.
If yes:
In the table below, please mark an X to indicate the quality improvement strategies that were a topic of any training you received and the quality improvement strategies that were a topic of any coaching you received. Training is a learning session for staff in which all participants learn the same content, skills, or practices. Coaching is a conversation in which a coach provides individualized feedback to you or helps you reflect on your practices. Some of these conversations may be formal—for example, scheduled in advance—and others may be informal. Training or coaching may be provided by staff from your regional or district afterschool program or someone from outside the program.
Quality improvement strategy |
Mark this column with an X if the strategy was a topic of any training you received |
Mark this column with an X if the strategy was a topic of any coaching you received |
Afterschool program quality standards |
|
|
What to look for when assessing quality in an afterschool program |
|
|
Types of data that can help assess quality in an afterschool program |
|
|
Procedures for assessing quality in an afterschool program |
|
|
How to use data to identify center needs and areas to improve |
|
|
How to identify appropriate trainings to address center and staff needs |
|
|
How to coach center staff to improve practice |
|
|
Other quality improvement strategy (please specify): |
|
|
Other quality improvement strategy (please specify): |
|
|
Other quality improvement strategy (please specify): |
|
|
If training was marked for any topic: During the [2021–2022 or 2022–2023] school year, as part of your work for your afterschool center, approximately how many total hours of training did you receive on quality improvement strategies? Your best estimate of the number of hours of training is fine.
If training was marked for any topic: Which type of trainer provided the majority of the training you received: (1) staff from your regional or district afterschool program or (2) a trainer from outside of your regional or district afterschool program?
If coaching was marked for any topic: During the [2021–2022 or 2022–2023] school year, thinking about the coaching you received on quality improvement strategies as part of your work for your afterschool center, approximately how many times did you participate in a coaching conversation with a coach? How long was a typical coaching conversation?
If coaching was marked for any topic: Which type of coach provided the majority of the coaching you received: (1) staff from your regional or district afterschool program or (2) a coach from outside of your regional or district afterschool program?
Next, I’d like to learn about any training and coaching that you may have received on specific practices to promote students’ social and emotional skills during the [2021–2022 or 2022–2023] school year as part of your work for your afterschool center. Later, I’ll ask about any training and coaching you received on other practices or topics. If needed, refer back to definitions of training and coaching in question 7.
During the [2021–2022 or 2022–2023] school year, as part of your work for your afterschool center, did you receive any training or coaching on specific practices for promoting students’ social and emotional skills?
If no, skip to question 9. If yes:
Practice for promoting students’ social and emotional skills |
Mark this column with an X if the practice was a topic of any training you received (mark all that apply) |
Mark this column with an X if the practice was a topic of any coaching you received (mark all that apply) |
Encouraging safe and respectful student behavior |
|
|
Developing trusting, supportive relationships between afterschool center staff and students |
|
|
Fostering positive and inclusive relationships between students |
|
|
Conducting activities in a well-paced, organized manner |
|
|
Promoting student engagement in afterschool center activities |
|
|
Giving students a voice in shaping afterschool center activities |
|
|
Encouraging students to take on and fulfill challenging responsibilities or leadership roles |
|
|
Helping students make responsible decisions that consider the well-being of others and themselves |
|
|
Supporting students’ development of skills for handling emotions |
|
|
Helping students understand the emotions and experiences of other individuals |
|
|
Supporting students’ development of skills for working to achieve goals |
|
|
Other practice for promoting students’ social and emotional skills (please specify): |
|
|
Other practice for promoting students’ social and emotional skills (please specify): |
|
|
Other practice for promoting students’ social and emotional skills (please specify): |
|
|
If training was marked for any topic: During the [2021–2022 or 2022–2023] school year, as part of your work for your afterschool center, approximately how many total hours of training did you receive on specific practices for promoting students’ social and emotional skills? Your best estimate of the number of hours of training is fine.
If training was marked for any topic: Which type of trainer provided the majority of the training you received: (1) staff from your regional or district afterschool program or (2) a trainer from outside of your regional or district afterschool program?
If coaching was marked for any topic: During the [2021–2022 or 2022–2023] school year, thinking about the coaching you received on specific practices for promoting students’ social and emotional skills as part of your work for your afterschool center, approximately how many times did you participate in a coaching conversation with a coach? How long was a typical coaching conversation?
If coaching was marked for any topic: Which type of coach provided the majority of the coaching you received: (1) staff from your regional or district afterschool program or (2) a coach from outside of your regional or district afterschool program?
I’d also like to learn about any other training or coaching you received as part of your work for your afterschool center that you have not already reported earlier. This training or coaching could cover practices or topics not related to quality improvement strategies or promoting students’ social and emotional skills.
During the [2021–2022 or 2022–2023] school year, as part of your work for your afterschool center, approximately how many total hours of other training did you receive that you have not already reported earlier? Your best estimate of the number of hours of training is fine. What were the main topics covered by the training?
During the [2021–2022 or 2022–2023] school year, thinking about other coaching you received as part of your work for your afterschool center that you have not already reported earlier, approximately how many times did you participate in a coaching conversation with a coach? How long was a typical coaching conversation? What were the main topics covered by the coaching?
Now, please think about any positions you held outside of the afterschool center. I’d like to learn about any training or coaching you received on specific practices for promoting students’ social and emotional skills as part of those positions.
During the [2021–2022 or 2022–2023] school year, as part of any positions you held outside of the afterschool center, did you receive any training on specific practices for promoting students’ social and emotional skills? If yes:
Approximately how many total hours of this training did you receive? Your best estimate of the number of hours of training is fine.
During the [2021–2022 or 2022–2023] school year, as part of any positions you held outside of the afterschool center, did you receive any coaching on specific practices for promoting students’ social and emotional skills? If yes:
How many times did you participate in a coaching conversation with a coach?
How long was a typical coaching conversation?
Now, I’d like to spend a few minutes learning more about which practices for promoting students’ social and emotional skills your staff focused on most during the [2021–2022 or 2022–2023] school year.
Please think back on your staff’s practices for promoting students’ social and emotional skills in the [2021–2022 or 2022–2023] school year. In the table below, please mark an X in up to three rows to indicate up to three practices center staff focused on most:
Practice for promoting students’ social and emotional skills |
Mark this column with an X if the practice was among the three practices staff focused on most (mark no more than 3 practices) |
Encouraging safe and respectful student behavior |
|
Developing trusting, supportive relationships between afterschool center staff and students |
|
Fostering positive and inclusive relationships between students |
|
Conducting activities in a well-paced, organized manner |
|
Promoting student engagement in afterschool center activities |
|
Giving students a voice in shaping afterschool center activities |
|
Encouraging students to take on and fulfill challenging responsibilities or leadership roles |
|
Helping students make responsible decisions that consider the well-being of others and themselves |
|
Supporting students’ development of skills for handling emotions |
|
Helping students understand the emotions and experiences of other individuals |
|
Supporting students’ development of skills for working to achieve goals |
|
Other practice for promoting students’ social and emotional skills (please specify): |
|
Other practice for promoting students’ social and emotional skills (please specify): |
|
Other practice for promoting students’ social and emotional skills (please specify): |
|
Now I’d like to learn more about any challenges that affected staff efforts to make improvements in these practices. For the three practices that staff focused on the most during the [2021–2022 or 2022– 2023] school year, please describe the extent to which the following factors posed challenges to making improvements:
Identifying specific strategies for improvement: was this a major challenge, a minor challenge, or not a challenge?
Providing enough opportunities to build staff skills for implementing strategies (for example, through training or coaching): was this a major challenge, a minor challenge, or not a challenge?
Identifying opportunities to build staff skills that are appropriate for your staff’s background: was this a major challenge, a minor challenge, or not a challenge?
Obtaining staff buy-in to the need for improvements in these areas: was this a major challenge, a minor challenge, or not a challenge?
Dealing with staff turnover: was this a major challenge, a minor challenge, or not a challenge?
Were there other factors that posed challenges to making improvements that I did not mention? If yes, for each factor: would you describe this as a major challenge or a minor challenge?
For each challenge the respondent described as a major challenge: How did you address [name of challenge]? Please describe any strategies you implemented to address this challenge. Which of those strategies, if any, would you recommend that other centers in your situation consider?
For treatment centers only: Next, I’d like to talk more about the continuous quality improvement system you are implementing as part of this study with support from the Weikart Center. We’d like to learn more about the approaches you used to implement that system, any challenges you encountered when implementing that system, and any strategies you used to address those challenges during the [2021–2022 or 2022–2023] school year. [If they aren’t sure -- A continuous quality improvement system is an improvement strategy that involves assessing staff practices, planning improvements, and carrying out improvements in a repeating cycle.]
During the [2021–2022 or 2022–2023] school year, did your center implement a continuous quality improvement system with support from the Weikart Center?
If response to Question 13 is no, skip to Question 24.
In the next set of questions, we are going to ask you to very briefly describe your approaches to implementing some of the key components of the continuous quality improvement system. You only need to provide a few sentences about your approach to each component. [For each question, if more information is needed, ask the respondent if they can speak a bit more about their approach or the challenges they encountered and the strategies they used to address them.]
Preparing to implement the continuous quality improvement system
Please briefly describe the approach you used to identify site team members, including any selection criteria.
For each criterion below that the respondent does not address in the initial response, ask the corresponding question to confirm whether the center implemented the criterion. Only ask about strategies that the respondent does not address in the initial response.
To confirm I understand your approach, I’d like to ask you about [a/some] potential [criterion/criteria] you may have used to identify site team members:
Did you consider staff’s availability and interest in the role?
Did you consider staff’s ability to collaborate with colleagues?
Did you consider staff’s experience with assessing or improving program quality?
Please indicate whether identifying site team members was a major challenge, a minor challenge, or not a challenge.
If described as a major challenge: Please describe any strategies you implemented to address this challenge.
Please briefly describe the approach you used to ensure that all staff understand the study’s continuous quality improvement system and their roles in carrying it out. Please include any specific communication approaches you used and how often you used those approaches.
For each strategy below that the respondent does not address in the initial response, ask the corresponding question to confirm whether the center implemented the strategy. Only ask about strategies that the respondent does not address in the initial response.
To confirm I understand your approach, I’d like to ask you about [a/some] potential [strategy/strategies] you may have used:
Did you hold office hours for staff to ask questions? If so, how often?
Did you share information on the system via email? If so, how often?
Did you reserve time during a regular staff meeting to discuss staff’s questions or concerns? If so, how often?
Please indicate whether ensuring that all staff know the system and their roles in carrying it out was a major challenge, a minor challenge, or not a challenge.
If described as a major challenge: Please describe any strategies you implemented to address this challenge.
Conducting self-assessment activities
Please briefly describe the approach you used to find time for staff to conduct the observations and other activities associated with conducting the self-assessment. (If needed: The self-assessment activities included conducting center observations using the YPQA measure, administering the staff survey on students’ social and emotional skills, and holding a scoring meeting.)
For each strategy below that the respondent does not address in the initial response, ask the corresponding question to confirm whether the center implemented the strategy. Only ask about strategies that the respondent does not address in the initial response.
To confirm I understand your approach, I’d like to ask you about [a/some] potential [strategy/strategies] you may have used:
Did you have staff observe for shorter time increments of less than 20 minutes?
Did you change center staffing assignments or activity schedules on days when staff needed to conduct observations?
Did you hold the scoring consensus meeting during a regularly scheduled staff meeting time?
Please indicate whether finding time for staff to conduct the center’s self-assessment was a major challenge, a minor challenge, or not a challenge.
If described as a major challenge: Please describe any strategies you implemented to address this challenge.
Planning for improvement
Please briefly describe the approach you used to find time for the site team to develop and refine improvement goals and plans.
For each strategy below that the respondent does not address in the initial response, ask the corresponding question to confirm whether the center implemented the strategy. Only ask about strategies that the respondent does not address in the initial response.
To confirm I understand your approach, I’d like to ask you about [a/some] potential [strategy/strategies] you may have used:
Did you conduct improvement planning during a regularly scheduled staff meeting time?
Did you limit or remove other professional development requirements for staff?
Please indicate whether finding time for the site team to develop and refine improvement goals and plans was a major challenge, a minor challenge, or not a challenge.
If described as a major challenge: Please describe any strategies you implemented to address this challenge.
Please briefly describe the approach you used to obtain staff buy-in and support for improvement goals and action steps to achieve those goals.
For each strategy below that the respondent does not address in the initial response, ask the corresponding question to confirm whether the center implemented the strategy. Only ask about strategies that the respondent does not address in the initial response.
To confirm I understand your approach, I’d like to ask you about [a/some] potential [strategy/strategies] you may have used:
Did you share improvement goals with all center staff and ask for their feedback on the goals?
Did you show all center staff the data motivating the improvement goals?
Please indicate whether obtaining staff buy-in and support for improvement goals and action steps was a major challenge, a minor challenge, or not a challenge.
If described as a major challenge: Please describe any strategies you implemented to address this challenge.
Engaging in improvement activities
Please briefly describe the approach you used to schedule trainings to accommodate staff’s availability.
For each strategy below that the respondent does not address in the initial response, ask the corresponding question to confirm whether the center implemented the strategy. Only ask about strategies that the respondent does not address in the initial response.
To confirm I understand your approach, I’d like to ask you about [a/some] potential [strategy/strategies] you may have used:
Did you schedule trainings on nights or weekends?
Did you use time set aside for other professional development activities?
Did you find substitutes for staff attending trainings?
Please indicate whether scheduling trainings to accommodate staff’s availability was a major challenge, a minor challenge, or not a challenge.
If described as a major challenge: Please describe any strategies you implemented to address this challenge.
If you experienced staff turnover during the school year, please indicate whether finding time to train new staff on the continuous quality improvement system was a major challenge, a minor challenge, or not a challenge.
If described as a major challenge: Please describe any strategies you implemented to address this challenge.
Please briefly describe the approach you and your co-lead used to find time to provide coaching to staff.
For each strategy below that the respondent does not address in the initial response, ask the corresponding question to confirm whether the center implemented the strategy. Only ask about strategies that the respondent does not address in the initial response.
To confirm I understand your approach, I’d like to ask you about [a/some] potential [strategy/strategies] you may have used:
Did you redistribute other responsibilities between you and your co-lead?
Did you allocate some of you or your co-lead’s work to other staff?
Please indicate whether finding time to provide coaching to staff was a major challenge, a minor challenge, or not a challenge.
If described as a major challenge: Please describe any strategies you implemented to address this challenge.
Please briefly describe the approach you and your co-lead used to find time to participate in the study’s trainings and professional learning community sessions.
For each strategy below that the respondent does not address in the initial response, ask the corresponding question to confirm whether the center implemented the strategy. Only ask about strategies that the respondent does not address in the initial response.
To confirm I understand your approach, I’d like to ask you about [a/some] potential [strategy/strategies] you may have used:
Did you attend sessions on nights or weekends rather than during normal work hours?
Did you and your co-lead alternate attendance at sessions?
Please indicate whether finding time to participate in trainings and professional learning community sessions was a major challenge, a minor challenge, or not a challenge.
If described as a major challenge: Please describe any strategies you implemented to address this challenge.
Please briefly describe the approach you used to help staff apply the strategies covered by the study’s training or coaching in actual interactions with students.
For each strategy below that the respondent does not address in the initial response, ask the corresponding question to confirm whether the center implemented the strategy. Only ask about strategies that the respondent does not address in the initial response.
To confirm I understand your approach, I’d like to ask you about [a/some] potential [strategy/strategies] you may have used:
Did you ask staff to share tips or experiences during staff meetings?
Did you conduct follow-up observations and coaching on strategies identified as priorities?
Did you search for additional resources to supplement training or coaching?
Please indicate whether helping staff apply the strategies covered by the study’s training or coaching in actual interactions with students was a major challenge, a minor challenge, or not a challenge.
If described as a major challenge: Please describe any strategies you implemented to address this challenge.
Were there any factors we have not mentioned that made implementing the study’s continuous quality improvement system challenging? (Probe: For example, balancing the implementation of the system with other center needs or priorities.)
For each challenge the respondent described as a major challenge: Please describe any strategies you implemented to address [name of challenge].
What was the most important way in which the COVID-19 pandemic affected your center’s implementation of the study’s continuous quality improvement system?
Finally, I’d like to learn more about your center and school context and your professional background. [Before the interview, use the study team’s information to identify the school(s) that are served by this center. If the center serves multiple schools, use “schools” in the following questions. If the center serves a single school, use “the school.”]
Thinking about the staff who worked at your center in [spring 2022 or spring 2023], about how many staff were:
Paid employees of the center?
Unpaid volunteers for the center?
Staff from contractors, vendors, or other partners?
Thinking about the paid employees who worked at your center in [spring 2022 or spring 2023], about how many of them were:
Certified teachers?
Other school-day instructional staff, such as instructional aides or teaching assistants?
Individuals with no school-day instructional positions, such as other youth workers, community members, college students, etc.?
Thinking about the paid employees who worked at your center in [spring 2022 or spring 2023], about how many of them had at least one full year of prior experience working with students in a school, afterschool center, or other youth program?
Thinking about the paid employees who were working for your afterschool center at the beginning of the [2021-2022 or 2022-2023] school year, what percentage of them were still working for your center in [spring 2022 or spring 2023]?
Did your center hold staff meetings during the [2021-2022 or 2022-2023] school year?
If yes: How often did your center hold staff meetings during the [2021-2022 or 2022-2023] school year?
During a typical school year before this one, did your center hold staff meetings? If so, how often did your center hold staff meetings in that year?
Thinking about the school(s) that your center serves, do the school(s) and your center share information on outcomes of individual students, such as test scores, grades, behavior, or progress toward other goals?
If yes: Approximately how often do [the school / each of the schools] and your center share information on outcomes of individual students: once a year, a few times over the year, a few times per month, more than once a week, or daily?
If yes: Please describe some ways in which the school(s) and your center share information.
[Does the school / do the schools] provide input into the center’s curricula and activities?
If yes: Approximately how often does [the school / each of the schools] provide input into the center’s curricula and activities: once a year, a few times over the year, a few times per month, more than once a week, or daily?
If yes: Please describe the ways in which the [school provides / schools provide] input.
Thinking about how your center delivered programming during the [2021-2022 or 2022-2023] school year, was there any periods of time during which your center delivered only virtual programming instead of in-person programming? If yes: For approximately what percentage of the year did your center deliver only virtual programming instead of in-person programming?
I’d like to wrap up with a couple of questions about your background.
In the [2021-2022 or 2022-2023] school year, how many hours per week do you work for the center?
How many years have you served as center director at [name of afterschool center]? Please round up to the nearest whole number and include the current school year.
I’d like to learn about any other roles you had prior to serving as center director at [name of afterschool center], and how many years you served in those roles, including the current school year. Please round up the number of years to the nearest whole number.
Have you served as center director at any other afterschool center? If so, for how many years?
Have you served as some other type of paid staff member or volunteer at [name of afterschool center]? If so, for how many years?
Have you served as some other type of paid staff member or volunteer at any other afterschool center or youth program? If so, for how many years?
Have you served as a regular full-time teacher in a school? If so, for how many years?
Have you served as some other school-day instructional staff member (such as an instructional aide or teaching assistant)? If so, for how many years?
Do you describe yourself as male, female, transgender, nonbinary, or another gender identity? If you do not feel comfortable answering this question, please let me know, and we can skip it.
Do you identify as Hispanic or Latino?
With which of the following racial designations do you identify? Please let me know if you identify as more than one.
Black or African American
Asian
Native Hawaiian or Pacific Islander
American Indian or Alaska Native
White
Those are all the questions I have for you. Thank you so much for taking the time to speak with me today and contributing to this important study.
Do you have any questions for me?
If you have any follow-up comments or questions, please feel free to contact me at [contact information].
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
Author | Mathematica |
File Modified | 0000-00-00 |
File Created | 2022-05-15 |