Appendix A - SRCL Grantee Interview

Appendix A: Grantee Interview Protocol and Consent Form

Comprehensive Literacy Program Evaluation

SRCL Grantee Interview Protocol, Years 2 and 3

Information for the Interviewer

The Striving Readers Comprehensive Literacy (SRCL) Program is a competitive grant program with the goal of advancing the literacy skills, including preliteracy skills, reading, and writing, of children from birth through Grade 12, with a special focus on improving outcomes for disadvantaged children, including low-income children, English language learners, and learners with disabilities. SRCL is designed to achieve these goals for children by awarding grants to State Education Agencies (SEAs) that, in turn, use their funds to support subgrants to Local Education Agencies (LEAs) or nonprofit early-learning providers to implement high-quality literacy instruction in schools and early childhood education programs. Ultimately, this enhanced literacy instruction is the mechanism through which student literacy and reading achievement are expected to be improved.

Important interview topics:

• Technical assistance SEAs are providing to subgrantees

• SEA processes and supports for continuous improvement

• SEAs’ perceptions of how subgrantees are implementing the program

By the end of this interview, you should have details and examples for each of these topics.

Introduction

Thank you for your willingness to participate in this interview about the Striving Readers Comprehensive Literacy (SRCL) Program.

As you know, this interview is being conducted as part of an evaluation of the FY2017 SRCL program commissioned by the U.S. Department of Education. [If applicable: We greatly appreciate your participation in the last interview in September 2018.] The purpose of this interview is to learn more about how [State]’s SRCL grant is progressing, including how [State] is carrying out grant activities such as technical assistance and continuous improvement processes.

During our meeting today, we will take notes and, with your permission, we’d also like to audiotape the conversation so that we can review at a later time for points that we might miss during the conversation. Is it okay if we audiotape today’s conversation?

IF YES: Thank you.
IF NO: We will take notes but will not record today’s conversation.

Also, I wanted to let you know that the information you share with us today may be included in one or more reports we produce as part of our evaluation; however, we will not identify you by name in any reports.

Do you have any questions before we begin?

SECTION I. SUBGRANTEE AWARDS

[ASK IF GRANTEE INDICATED IN YEAR 1 INTERVIEW THAT THEY WERE AWARDING SUBGRANTS IN MULTIPLE YEARS]

1. In [2019-20 (Year 2) or 2020-21 (Year 3], how many subgrants did you award to single districts? To consortia?

2. How many years are these subgrant awards for?

3. Are these subgrantees required to serve children across the full age span from birth through grade 12?

   1. If no: What are the grant requirements about ages of students to be served? What proportion of subgrants serve the full age range of students from birth through Grade 12? What other age configurations are served by subgrantees?

4. How many total schools will these subgrants serve? How many are: Elementary schools? Middle schools? High schools? Early learning programs?

SECTION II. Technical Assistance for LEAs

First, I’d like to ask you about the technical assistance (TA) that you are providing to subgrantees as part of the SRCL program. We’d like to focus on the TA you offered between October 2018 and now – that is, since our last interview.

5. Did you provide TA to SRCL subgrantees in this grant year (September 2018 until present)?

1. What topics did this TA cover?

   1. How did [State] choose these topics (e.g., guidance from U.S. Department of Education [ED], subgrantee requests, weaknesses of subgrants)?

2. Through what channels — or in what formats — did you offer this TA
   (e.g., in-person training, webinar, coaching/other tailored supports, online tool)?

3. How much TA did you offer? (frequency, number of hours)

   1. How much of this TA did subgrantees choose to attend? Probe: Attendance issues, take-up rates

4. Have you evaluated the usefulness of this TA? If so, how (e.g., teacher satisfaction surveys, observations to determine changes in teachers’ practices)?

5. Who was responsible for developing and/or providing this TA? What are their qualifications?

5. As part of the SRCL program, have you conducted any site visits for TA or continuous improvement and monitoring purposes?

1. If yes: Who conducted these site visits? Who participated in these site visits? Probe: Number of subgrantees, participants (e.g., teachers, administrators).

2. What was the purpose of the visits? What did the visits entail?

SECTION II. Monitoring and Continuous Improvement

Now I want to ask some questions about how [State] is monitoring and evaluating subgrantees’ progress as part of the SRCL grant.

5. What forms of regular communication do you have with your subgrantees? Phone calls – how often? With whom? Progress reports from subgrantees – how often? What is included? In-person meetings with all or subsets of subgrantees? Anything we’ve missed?

6. In [the previous interview, if applicable; otherwise, your SRCL application to ED], you indicated that you were planning to collect the following data from subgrantees about implementation of their SRCL-funded interventions/practices: [prepopulate data prior to interview]. Which of these data have you been able to collect on a consistent basis from your subgrantees?

1. How have you used these data to support subgrantees’ improvement or monitor or assess their implementation of the SRCL-funded interventions/practices?

2. How are you reporting back to subgrantees on their progress? How are subgrantees using this information?

3. Have you had to take any action because of data suggesting a subgrantee is implementing an intervention poorly or is at risk of weak implementation? If so, what action have you taken?

5. In [the previous interview, if applicable; otherwise, your SRCL application to ED], you indicated that you would have the following data about outcomes for students in classrooms using SRCL-funded interventions/practices: [prepopulate data prior to interview]. Which of these data do you have on a consistent basis for your subgrantees?

1. How have you used these data to support subgrantees’ improvement or to monitor or assess the outcomes of SRCL schools or teachers?

2. How are you reporting back to subgrantees on their outcomes? How are subgrantees using this information?

3. What are you learning about the extent to which your grant is meeting the SRCL program’s overarching goal of improving students’ literacy outcomes? Probe: Evidence of improvements in students’ reading achievement, evidence of effectiveness of particular literacy interventions/practices?

4. Have you had to take any action because of data suggesting that a subgrantee is not achieving the desired improvements in students’ reading outcomes? If so, what action have you taken?

5. Overall, do you think your state has been able to execute a strong continuous improvement process? Why?

10. What challenges has your state had in operating your continuous improvement process?

SECTION III. Subgrantees’ Activities

5. What aspects of the SRCL program have been hardest for subgrantees to implement fully? Why? Probe: Providing professional development (PD), hiring the right literacy support staff, implementing selected interventions/practices, aligning curricula across ages/grades, serving the youngest kids (i.e., birth to age 3)

6. Have any subgrantees asked to change their SRCL program or how they are spending SRCL funding, from what they proposed in their application to [State]? If yes: In what ways have they proposed to change their plan? What are the reasons subgrantees have asked to change their plans?

7. How would you describe the extent to which subgrantees have been able to meet the goals you or ED set for them?

8. Please describe some features of subgrantees that are doing a particularly good job implementing the SRCL program as well as features of subgrantees that are particularly struggling to implement the program, e.g. other literacy initiatives in district, district leadership around literacy, types of literacy-related activities being implemented, quality of local literacy plan.

9. To what extent do you think your subgrantees are operating programs that align with key elements of your state’s comprehensive literacy plan? Tailored to question 9 in Yr 1 grantee protocol about key elements of state literacy plan.

SECTION IV: Wrap Up

5. Is there anything we didn’t ask but you think we should have, or is there anything you think we should know?

Striving Readers Implementation Evaluation Interview:
Informed Consent

Purpose

American Institutes for Research (AIR) is the independent and external evaluator contracted by the U.S. Department of Education’s Institute of Education Sciences (IES) to conduct the Striving Readers Comprehensive Literacy (SRCL) program evaluation. The purpose of this study is to provide information to policymakers, administrators, and educators regarding the following: grant award procedures, technical assistance, continuous improvement procedures, and literacy interventions at the school level. The study team will employ multiple data collection strategies including interviews, surveys, collection of student records, and site visits.

In an effort to gather insight about SRCL implementation, we are conducting one-on-one phone and/or in-person interviews with state-level grantee administrators and surveys with a sample of principals, reading specialists, and teachers from SRCL subgrantee school districts. Interviews will focus on the award process, literacy interventions’ evidence of effectiveness, strategies for serving the largest number of students, technical assistance from states and districts, professional development plans, and use of assessments to identify student needs, inform instruction, and monitor progress.

The U.S. Department of Education expects SRCL grantees to participate in these interviews as part of grantees’ cooperation with the national evaluation of the SRCL program. AIR and its other contractors will make every effort to minimize the burden on grantees, including scheduling interviews and other data collection efforts at a convenient time for grantees.

Risks and Discomfort

There are no anticipated or known risks in participating in this evaluation. Your responses will have no adverse effect on your employment. This is NOT an evaluation of you personally but an attempt to understand how states, districts, and schools implement the SRCL program. If there is any question that you do not want to answer or you feel uncomfortable about, please let me know and we can skip it.

Benefits

Your participation in the evaluation will contribute to an understanding of the implementation processes associated with a large-scale SRCL initiative. Findings from the interviews will help policymakers and educators to improve the support of federal grant programs and literacy instruction.

Privacy

We will keep the information you share during this interview private. Any documents or audio recordings from this interview will be kept in secure data files to be accessed only by staff on the research team. Audio recordings will be kept until the end of the project and then permanently deleted. Because you have a state-level position as the director of a federal grant program, readers may be able to determine that you participated in the study, but we will not refer to you by name or attribute any quotes to you personally.

More Information

If you would like more information about this evaluation, you may contact the Project Director, Jessica Heppen at AIR at 202-403-5488 or at jheppen@air.org. For questions regarding your rights as a subject participating in this research, please contact the Institutional Review Board (IRB) at IRBChair@air.org or toll free at 1-800-634-0797.