National Center Interview

Appendix C: Primary Data Collection Instruments

Exhibit C-3 National Center Director Interview Protocol

Notes for reviewers: This interview protocol will be administered to Comprehensive Center (CC) Director in the National Center (NC). The interview will take up to 60 minutes and will focus on details of projects the Centers conducted, services provided, capacity developed, and collaborations with other Centers and RELs. Interviewers will ask the questions in an open-ended manner, and note-takers will code responses into pre-specified categories to the extent possible.

• Purple font indicates instructions to the interviewer.

• Red font indicates skip patterns / programming notes.

• Blue boxes are instructions or transitions that the interviewer will read to respondents.

• Radio buttons indicate only one answer should be selected.

• Checkboxes indicate the interviewer should check all that apply.

Introduction

Thank you for making time to speak with us today – My name is [say your NAME] and I’m joined by my colleague [say note taker’s name]. Our organization, [Abt Associates/AnLar], was contracted by the U.S. Department of Education’s Institute of Education Sciences to conduct a study of the Comprehensive Centers. We are very interested in learning more about how the Comprehensive Centers are developing partnerships, building capacity, and providing training and technical support to state and local education agencies, including successes and challenges. The results of this study will be summarized in a report to Congress and are also expected to inform the next Comprehensive Center cycle, beginning in 2024. The call should take no longer than 60 minutes. Before we get started, we would like to know if you have any time constraints. Is your time open after this call, or do you have a hard stop after 60 minutes?

Prior to our call, we reviewed your responses to the pre-interview survey, as well as Annual Service Plans and Annual Evaluation Reports that you shared with the Department of Education. We also reviewed initial analyses of responses to a survey intended to capture technical assistance recipients’ experiences working with the Comprehensive Centers. I am following up to gather additional information. During our conversation, I may ask a question that you may have already addressed in other existing documentation. Should this be the case, just let me know. At the end of our conversation, we will review a list of these documents to determine the easiest way for me to access them.

It’s important that you know your responses today will be used only for research purposes. None of our study reports will name you as an individual. Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183).

With your permission, we would like to record this call to ensure that our notes are as accurate as possible. The recording will only be used for notetaking purposes and will not be shared with anyone outside of the study team. The recording will be destroyed after the interview notes are finalized. Do you consent to being recorded? [if not, don’t record the interview].

All answers should be recorded verbatim. To avoid signaling to respondents that they should be doing a particular activity, do not use response categories as follow-up prompts. If respondents answer a question prior to its appearance on the protocol, record their answer when given and skip the question when you come to it on the protocol.

Educational Topics and Services Provided

First, we would first like to discuss the different types of services Comprehensive Centers provide to TA recipients.

We understand that during Year 2 (which overlaps with the 2019-20 school year), your Center had X active projects addressing X topic areas. In this part of the interview, we are interested in finding out how your Center designs these projects.

1. How does your Center ensure projects are providing the right assistance to meet an SEA’s priorities?

Probe: do stakeholder needs drive the project design, or is it more of an iterative process?

• SEA proposes the majority of ideas or drives the design of a project

• CC proposes the majority of ideas or drives the design

• The design is an equally distributed effort between the CC and SEA

• Other: _____________________________

2. When the Center takes the lead on a process or activity as opposed to supporting TA recipients, what typically is the reason for doing so?

• TA recipients did not have skills or knowledge needed to lead the process or activity

• TA recipients did not have time to lead the process or activity

• TA recipient did not have access to resources needed (such as access to research databases)

• TA recipients requested CC lead process or activity

• CC is unaware of underlying reason

• Other: _____________________________

3. What proportion of projects do you think TA recipients would be capable of conducting without Center support?

Proportion: ________________

The next question references part of the Center’s logic model. I’ve pulled it up on the screen so you can review it while we talk. [share screen to show logic model and/or list below so they can see the processes]

4. Is the proportion higher or lower for certain processes in the Center logic model?

• Conduct a needs assessment

• Develop a logic model

• Select evidence-based practices (EBPs) / interventions

• Plan implementation of EBPs / interventions

• Support implementation of EBPs / interventions

• Evaluate results

RQ2.1 What are the most common services that Comprehensive Centers provide to TA recipients?

5. The Centers’ logic model references evidence-based practices. What features of your Center’s products or resources encourage uptake of evidence-based practices among your clients?

• Brevity

• Accessible (such as limiting use of jargon and acronyms)

• Visually appealing

• Tailored to audience

• Freely available

• Interactive (such as allowing users to sort or filter information)

• Provides detailed instructions for implementation

• Other: ________________________________________

6. Did any TA recipients request assistance in high-priority topic areas that your Center was not able to address? Probe for topic if they do not specify.

• No (GO TO Q7)

• Yes

Note-taker: use the drop-down list to indicate which topic.

Include drop-down options for note-taker to indicate which topic. Topics should include:

• Accountability & Assessment (Achievement Gaps, Assessment, Data Use, Formative Assessment, Student Outcome Measures)

• College & Career Readiness (Career & Technical Education, Dropout Prevention, Dual Enrollment/Dual Credit, Graduation, Pre-college Preparation, Work-based Learning)

• Curriculum & Instruction (Curriculum & Development, Resource Use, Standards Alignment, Literacy, STEM + Computer Science)

• Diverse Learners (Students of Color (American Indian/Alaskan Native, Asian, Black, Hispanic), English Learners, Low Income Students, Students with Disabilities, Students Experiencing Homelessness, Students in Migrant Families, or Students in Foster Care, LGBTQ+ students)

• Early Learning (Kindergarten Readiness, Head Start)

• Educational Equity (Inequities in School Funding, Inequities in Access to High-Quality Instruction, Inequities in Access to Support Services, Catch-up Strategies)

• Planning, Evaluation & Management (Internal Communication, Compliance Monitoring, Family & Community Engagement, Financial Management, Needs Assessment, Program Development & Implementation, Program Evaluation, Stakeholder Engagement, Strategic Planning)

• Policy & Legislation (Local or State Education Agency Compliance, Every Student Succeeds Act, Policy Development, State Education Agencies, Other Federal Policy)

• Return to School (Strategies to Support Physical Distancing, Addressing Learning Loss during COVID-19, Supporting Students’ Emotional Needs during COVID-19, Communicating Reopening Policies to Families)

• Schools & Classrooms (Instructional Technology, School Choice, School Climate, School Improvement)

• Social-Emotional & Behavioral Learning (Social-Emotional Learning,

Holistic Student Supports, Wrap-around Services, Growth Mindset)

• Teachers & Leaders (Certification & Licensure, Educator Evaluation, Educator Preparation, Leadership, Professional Development for Teachers or Leaders, Recruitment & Retention)

• Teaching & Learning (Culturally Responsive Practices, Evidence-Based Practices, Individualized Learning, Multi-Tiered Systems of Support, Online/Distance Learning, Rural Education, Special Education, Student Engagement)

• Other, specify _____________________________

1. Why have these topics not been addressed? [check all that apply]

Agency Challenges

• Turnover among TA recipient staff

• Changing priorities at TA recipient agency (including COVID-related changes)

• State policy shifts

• Turnover in chief state school officer and/or other leadership

• Meeting diverse needs across TA recipients (and/or regions)

• Cuts in TA recipient staffing and/or budgets

• Difficulty establishing relationships with the TA recipients

CC Challenges

• Lack of Center financial resources

• Project timeline constraints

• Lack of communication within the TA recipient agency

• Mismatch between stated TA recipient needs and Center federal priorities

• Center lacked appropriate skills

• Center lacked appropriate content expertise

• Turnover among Center staff

• Center focused on different priorities

• Access to data or information

• COVID disruptions

• Other: _____________________________

Dimensions of Capacity

The next set of questions is about the Comprehensive Centers’ efforts to increase TA recipients’ capacity to identify, implement, and sustain effective evidence-based practices.

7. How does your Center determine which capacities (human, organizational, resource, policy) TA recipients need to develop?

• The CC needs assessment helps identify needed capacities

• TA recipients request support in building specific capacities

• The capacities to be developed depend on the project topic

• The capacities to be developed depend on the phase of implementation involved (needs assessment, logic model, select EBPs, etc.)

8. What are the key challenges in developing TA recipients’ capacity?

• Stakeholder engagement

• Staff buy in

• Leadership buy-in

• Stability in TA recipient staff (staff retention)

• Preexisting skills

• Preexisting content expertise

• Communication among different departments (silos)

• Leadership invested in using evidence

• Other: __________________

9. How can you tell whether a TA recipient has developed capacity?

• TA recipients take more ownership for services previously provided by CC

• TA recipients train others in their agency

• TA recipients use CC-provided resources and tools independently

• TA recipients are able to adapt resources and tools to apply to new projects

• TA recipients indicate they no longer need CC assistance

• Other: _______________________________

10. What are the most effective methods the Center has used to build capacity?

• Consultation

• Meeting facilitation

• Brokering stakeholder connections

• Webinars or expert panels

• Provide products or tools

• Data analysis

• Provide lit review / references

Changes due to COVID-19

We would now like to turn your attention to how Center work shifted in response to COVID-19.

11. Were new projects, products, or tools initiated to address COVID-related needs?

• No (GO TO Q12)

• Yes

1. Briefly describe the focus of the new work [do not read the list below, but for any focus area mentioned, probe for targeted groups: Students of Color (American Indian/Alaskan Native, Asian, Black, Hispanic), LGBTQ+ Students, English Learners, Low Income Students, Students with Disabilities, Students Experiencing Homelessness, Students in Migrant Families, or Students in Foster Care]

Focus of new work	Targeted group(s)
Addressing achievement gaps
Technology access for students, including strategies to support remote learning for students with limited Internet access
Teaching strategies for remote instruction
Strategies to support continuity of learning
Supporting students’ social, emotional, and mental health needs
Plans to reopen schools, including strategies to support physical distancing and other health or safety-related procedures
Engaging students and families
Strategies for developing attendance policies
Strategies for developing graduation and grading policies
Guidance for meal delivery services
Other: ___________________

12. In what ways was your Center able to be responsive to rapidly changing needs? Were there any barriers or challenges to being responsive? [probe for whether Center program structure made it difficult to be responsive]

• Able to transition to virtual meetings

• Maintained open lines of communication with TA recipients

• Maintained open lines of communication with CC program at U.S. Department of Education

• Other: _____________________

Barriers / Challenges

• Sustaining stakeholder engagement during a crisis

• Need to address new topics or high-leverage problems

• Lack of guidance regarding when and to what extent approved projects could be adapted

• Other: _____________________

Collaboration Successes and Challenges

Next, we would like to discuss the experiences your Center has had collaborating with the Comprehensive Center program and with external partners, including the Regional Educational Laboratories.

13. What types of collaboration, if any, does your Center engage in with the Regional Centers?

• Use RC products / tools or attend RC events

• Jointly facilitate meetings

• Jointly produce services and/or products, such as needs assessments or logic models

• Other: _____________________

• Does not engage with RCs

14. What challenges, if any, have you experienced in working with the Regional Centers?

• Too many meetings

• Duplication of effort

• Logistics / scheduling difficulties

• Lack of clarity regarding roles and responsibilities

• Lack of coordination

• Not sufficiently response to timelines

• None

• Other: _____________________________

15. What are the benefits in collaborating with other Centers?

Benefits:

• Opportunities for collaborative implementation processes (such as common trainings)

• Opportunities to share materials that speed up implementation

• Access to existing tools addressing a common priority area

• Opportunities to leverage content expertise

• Opportunities to leverage skills

• None

• Other: _____________________________

16. Were you involved in the previous round of Comprehensive Centers, which ran from 2012 to 2019?

• No (GO TO Q19)

• Yes

a) In what capacity were you involved in the previous round?

• CC Director at a different Center

  • If different Center, which one?

• CC staff at different Center

• Other: __________________

17. The 2019 cycle revised the Centers program in several ways. In addition to adding the National Center, the number of RCs increased from 15 to 19. Has this been beneficial or not, and why?

• Beneficial

• Not beneficial

Benefits

• More tailored to local needs

• More resources for regional support

• Improved coordination and information sharing

• None

• Other: _____________________________

Challenges

• Fewer resources for regional support

• Less information sharing/coordination

• Divided regions in ways that undermined continuity of efforts

• None

• Other: _____________________________

18. Also in the 2019 cycle, the Content Centers were eliminated. What benefits and challenges were associated with this change?

Benefits

• Centralization of expertise within one NC instead of multiple Content Centers

• Increased efficiency

• Enhanced ability to build capacity

• None

• Other: _____________________________

Challenges

• Reduced access to in-depth content expertise

• Duplication of effort across CCs

• Lack of clarity regarding roles and responsibilities

• Lack of coordination

• Less timely access to content expertise

• None

• Other: _____________________________

19. Has your Center collaborated with RELs beyond the joint needs sensing work to co-develop presentations, co-develop resources or tools, or provide joint service delivery?

• Co-develop presentations (such as webinars, expert panels, or events)

• Co-develop resources or tools

• Joint service delivery

• Other: _____________________________

20. Are there some types of Center services that would benefit from more collaboration between the REL and your Center?

• No (GO TO Q21)

• Yes

1. Which services would benefit the most from such collaboration?

• Conduct a needs assessment

• Develop a logic model

• Select evidence-based practices (EBPs), interventions, or state-wide efforts

• Plan implementation of EBPs, interventions, or state-wide efforts

• Support implementation of EBPs, interventions, or state-wide efforts

• Evaluate results

• Other: ______________________

21. What challenges do you experience in collaborating with RELs?

• REL is part of a competitor organization

• Overlap / duplication of services

• Work with different stakeholders or offices

• Insufficient resources / time to collaborate

• Misalignment of priorities

• Other: _____________________________

• None

22. Has the joint needs sensing between the Centers and the Regional Educational Laboratories in their region that was introduced in the 2019 cycle helped you avoid duplication of effort or resulted in increased collaboration?

• Reduced duplication of effort

• Increased collaboration

• Other: _____________________________

• Has not been successful

23. Did you refer any TA recipients to other ED-funded TA Centers? [Equity Assistance Center/Other]

• No (GO TO Q24)

• Yes

Tell us about the reasons for this referral.

• To avoid duplication of effort

• Other TA Center had more hours/staff available to meet needs

• Other TA Center had content area or methodological expertise needed to meet needs

• TA recipient requested support outside the scope / strategic goals of the Center

• TA recipient requested support with minimal return on investment

24. Reflecting across your projects, what is the biggest remaining challenge in working with partners—including the National Centers, the RELs, other Regional Centers, or the partners we just discussed—to assist TA recipients?

• Identifying the right partners for content knowledge

• Identifying the right partners for technical / methodological expertise

• Coordinating staff time / availability

• Coordinating responsibilities for different parts of the project

• Establishing and adhering to deadlines

• Adjusting to changing demands

• Other: ______________________

Closing Statement

Thank you so much for providing all this information. We really enjoyed learning more about your Comprehensive Center and appreciate your time!

If any documents were mentioned that we need to gather: Earlier you mentioned [documents], which would be helpful for me to have to better understand your Center. What would be the easiest way for me to access these documents?

Abt Associates Inc. Appendix C: Primary Data Collection Instruments  ▌pg. C-1