Evaluating the DC Opportunity Scholarship Program After the 2017 Reauthorization

OMB Supporting Statement Evaluation of the DC OSP After the 2017 Reauthorization

Appendix F-1 Program Operator Interview Protocol

Information for Reviewers

The study team for the Evaluation of the DC Opportunity Scholarship Program (OSP) will administer this interview protocol to collect data about the program operator’s activities and challenges. This will be a semi-structured interview. This document includes proposed closed-end and open-ended items.

Reviewers should take note of the following:

Respondent sample and interview timing

• The study team will conduct a 60 minute group interview with the program operator’s leadership team, including the Executive Director, Assistant Director, and Enrollment Coordinator.

• The interview will take place in-person in fall 2021.

Formatting guide

This document uses the following conventions:

[ALL CAPS]

Indicates programming logic that respondents will not see but which drive instrument skip patterns.

Interviewer guidelines

This interview protocol includes likely response categories that the interviewer can use to classify (code) responses. However, the interviewer will not read these categories aloud as part of the initial question, unless otherwise noted. The interviewer will first allow the respondents to answer any open-ended question and then use response categories as follow-up probes unless otherwise noted.

Introduction

My name is [NAME] and this is my colleague [NAME], and we’re here ask you about the DC Opportunity Scholarship Program (OSP). We’d like to learn more about [PROGRAM OPERATOR NAME]’s specific activities and challenges as well as hear about your perspective on families’ and schools’ challenges participating in the OSP. We expect that this conversation will last approximately 60 minutes.

We work at Abt Associates, an independent research firm contracted by the U.S. Department of Education to study the OSP.

During our meeting today, we will take notes, and with your permission, we’d also like to audiotape the conversation so that we can supplement our notes.

We will share information we collect during this interview with our study team for analysis. Information you provide may be included in reports of study findings. However, these reports will not contain information that could directly identify you personally by name or position. The name of your organization will likely be included in any reports.

If you allow us to audiotape the conversation, we will not share the audio recording with the Department, or with anyone else outside our study team. We will transfer the digital audio recording to a non-portable, encrypted electronic database and we will erase the original copy saved on the portable recording device.

Protecting your privacy is very important to us. As with any study involving collection of data, there is some minimal possibility of breach of confidentiality of data. We will take every precaution possible to secure the information you provide to ensure confidentiality.

Your participation is voluntary, and you may choose not to discuss any topic or to end the discussion at any time. Is it ok if we audiotape our conversation? [ASK EACH INDIVIDUAL RESPONDENT AND RECORD RESPONSE ON AUDIO AND BELOW]

Interview respondent #1: Name Yes No

Interview respondent #2: Name Yes No

Interview respondent #3: Name Yes No

Do you have any questions before we begin?

Section A: Program Operator’s Activities

First, we are going to discuss the work [PROGRAM OPERATOR] does and any challenges faced while doing this work.

1. In our previous conversation, [PROGRAM OPERATOR] mentioned three main activities: (1) running the OSP application process, (2) outreach to families about the OSP and to participating private schools, and (3) working with private schools to verify schools’ OSP eligibility and make scholarship payments. Additionally, [PROGRAM OPERATOR] staff mentioned two supplemental activities: (1) tutoring OSP students and (2) assisting families with private school applications. What other responsibilities do you have? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Annual collection of information about participating schools and OSP students

• Annual reporting activities (annual activities report to Secretary)

• Conducting school site visits

• Conducting survey of OSP students’ parents

1a. [IF ANNUAL COLLECTION OF SCHOOL INFORMATION IS MENTIONED]: Which of the following specific information do you collect each year from OSP private schools?

READ ALOUD AND MARK EACH OPTION ENDORSED

• Student test scores:

• for OSP students only  for all students

• High school graduation rates:

• for OSP students only  for all students

• College admission rates:

• for OSP students only  for all students

• Accreditation status

• Valid certificate of occupancy

• Proof of financial sustainability

• Credentials of core subjects teachers

• Proof of appropriate financial systems

• Proof of annual parent communications about academic achievement, safety of the school, accreditation status

2. When we spoke in Spring 2019 we learned about some of the outreach activities you conduct during the year to provide information about the OSP to families. Some of the activities mentioned were:

• Community outreach events at libraries, private schools, and ESA centers.

• Spring event & Winter Carnival

• School fairs

• Christmas event

• Social media

What other ways does [PROGRAM OPERATOR] work to recruit families to apply for the scholarship?

3. Can you walk us through how you recruit families to attend the outreach events?

3a. How does [PROGRAM OPERATOR] identify families to invite them to these events?

4. Thinking specifically about the OSP application, can you describe what types of support [PROGRAM OPERATOR] provides to families? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Reminder(s) about the deadline to apply for the DC OSP scholarship

• Reminder(s) about missing documents

• Help using the online Family Portal

• Help with filling out the scholarship application

• Help obtaining required proof of residency or income

• Information about private schools that accept students with an OSP scholarship

• Guidance on how to select a private school

4a. [IF REMINDERS SENT:] Which families receive reminders?

• Families who attended an outreach event (potential applicants)

• Families that have started an application but not finished

5. How do you communicate information about supports to complete the OSP application to families? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Serving Our Children’s website

• Email announcement

• Mailed announcement

• Communications with individual families

6. What are some specific ways that you assist families with private school applications: DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Provide information about private schools that accept students with an OSP scholarship

• Help families understand the private school application and admissions process

• Provide families with advice about how to choose which schools to apply to

• Translate school application materials into families’ language

• Help families keep track of important school application deadlines

• Help families get letters of recommendation, transcripts (grades) for child

• Give families advice about how to write an essay or prepare for a school interview or visit

• Help prepare students for admissions tests

• Suggest ways families could prepare child for an admissions test

• Suggest that families ask schools to waive their application fees

• Show families how to get reimbursed for school application, admissions, or test fees

• Help families fill out financial aid forms

• Suggest other scholarships families could apply for

7. How do you communicate information about supports to complete private school applications to families? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Serving Our Children’s website

• Email announcement

• Mailed announcement

• Communications with individual families

8. After students are admitted to private schools, what types of support do you provide to families to help with private school enrollment? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Help families choose a private school if their child is accepted to more than one

• Help families understand financial aid package if offered by private school

• Help families complete enrollment forms

• Notify families of important private school dates and deadlines (class registration, orientation, ordering uniforms, etc.)

9. How do you communicate information about supports to enroll in private school to families? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Serving Our Children’s website

• Email announcement

• Mailed announcement

• Communications with individual families

10. What types of support do you provide to families once students have enrolled in a private school? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Financial support (books, uniforms, fees)

• Tutoring

11. Can you walk us through the tutoring program? [IF INFORMATION NOT OFFERED UNPROMPTED ASK EACH BULLETED QUESTION BELOW, AS NECESSARY]

• What entity provides tutoring to OSP students?

• Where is tutoring provided to OSP students?

• How many students receive tutoring on a weekly (monthly) basis?

• How long do students typically receiving tutoring services? (number of sessions per week, for how many weeks?)

• How is information about the tutoring program communicated to families?

• How is information about the tutoring program communicated to schools?

12. How do you communicate information about how to get additional financial support (other than the OSP scholarship itself) to families? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Serving Our Children’s website

• Email announcement

• Mailed announcement

• Communications with individual families

13. What types of help or support do you provide families with renewing their voucher? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Reminder(s) about the deadline to renew the OSP scholarship

• Reminder(s) about missing documents

• Help using the online Family Portal

• Help with filling out the renewal application

• Help obtaining required proof of residency or income

14. Do you provide any additional or different support to families who have renewed the OSP scholarship to help them remain enrolled in private school in subsequent years?

• Yes [PROBE: “What kind of support?”]

• No

14a. [IF YES:] What types of help or support do you provide to families who renew their scholarship but want to (or need to) change private schools?

15. If [PROGRAM OPERATOR] had a larger budget, more staff or other additional resources, what other information or support would you consider providing to families, related to:

• Applying to the OSP?

• Renewing the OSP?

• Learning about private schools?

• Applying to private schools?

• Choosing a private school for enrollment?

• Succeeding in private school?

Now, let’s discuss the support you provide to OSP-participating private schools as well as any outreach to non-participating private schools.

16. How many DC private schools are not currently participating in the OSP?

NUMBER: _________

16a. How many of these schools were participating in the OSP last year?

17. How do schools not already participating learn about the OSP? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• [PROGRAM OPERATOR] attends a private school association meeting and follows-up with non-participating schools

• [PROGRAM OPERATOR] actively reaches out to nonparticipating private schools each year to encourage participation

• Non-participating private schools reach out to [PROGRAM OPERATOR]

17a. [IF ATTENDS PRIVATE SCHOOL ASSOCIATION MEETING:] Which association?

18. How do you encourage ongoing participation from private schools? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Hold meetings with participating private schools to discuss participation in the upcoming school year

• Hold individual calls with private school administrators each year to discuss participation in the upcoming school year

19. What does [PROGRAM OPERATOR] do to help OSP-participating private schools address challenges they face participating in the OSP? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Help with communications to OSP families

• Send reminders to parents to sign biannual OSP checks

20. If [PROGRAM OPERATOR] had a larger budget, more staff or other additional resources, what would you consider doing to encourage more private schools to participate in the OSP?

21. We are interested in learning more about how resources are allocated to each of your activities. What proportion of the budget is allocated towards each of [PROGRAM OPERATOR]’s main activities? What staff are allocated for each of these activities? READ ALOUD EACH ROW

Activity	Proportion of Budget	Staff FTE
Outreach to families about the OSP and organizing information sessions for families
Outreach to nonparticipating private schools
Running the OSP application process
Working with private schools to verify schools’ OSP eligibility
Working with private schools to verify enrollments for scholarship payments
Tutoring supports
Other 1
Other 2

We are interested in learning more about the challenges you face in carrying out the program.

22. During our previous discussion you described some challenges that you have operating the program.

22a. One of these challenges was misalignment between the OSP application deadlines and the private school application deadlines. Is this still a challenge?

• Yes

• No

22b. What have you have done to address this challenge?

22c. Another challenge mentioned was verifying parents’ income and residency eligibility. Is this still a challenge?

• Yes

• No

22d. What have you have done to address this challenge?

22e. A third challenge mentioned was limited parent awareness about supplemental financial aid available from private schools. Is this still a challenge?

• Yes

• No

22f. What have you have done to address this challenge?

22g. A fourth challenge mentioned was private schools restricting the supply of seats for OSP students. Is this still a challenge?

• Yes

• No

22h. What have you have done to address this challenge?

23. What other challenges do you face in operating the OSP program? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Staffing

• Funding tutoring

• Communicating with families

Section B: Families’ Challenges

Next we are going to discuss challenges you think that families face during each part of the process of participation in the OSP—applying for the scholarship, applying to private schools, enrolling in private schools, and renewing their scholarship.

24. A few minutes ago we discussed how [PROGRAM OPERATOR] reaches out to eligible families and supports them in applying for an OSP scholarship. Through these communications and interactions with families, what challenges have you learned that families experience applying for the OSP scholarship? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Lack of awareness about the OSP

• Difficulty using the Family Portal

• Difficulty getting a paper application

• Difficulty finding time to finish the application

• Confusion over the deadlines

• Difficulty obtaining or submitting income proof

• Difficulty obtaining or submitting proof of DC residency

• Difficulty because the scholarship application was not available in families’ home language

• Difficulty getting answers to questions or help from us

25. During your communications and interactions with families, what challenges have you learned that families experience in applying to private schools? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Difficulty keeping track of deadlines for private schools

• Difficulty finishing all of the application requirements (forms, tests, interviews or school visits, letters of recommendation) for each school

• Difficulty affording the application or test fees

• Difficulty completing schools’ financial aid applications

• Difficulty getting answers to questions or help from us

• Difficulty getting answers to questions or help from private schools themselves

• Difficulty because private school application materials were not available in families’ home languages

26. What information does [PROGRAM OPERATOR] collect from families who use and do not use the scholarship about their experiences applying for the OSP scholarship and applying to private schools? For example, do you administer a survey to parents?

• Survey

• Informal conversations

• Focus groups

27. Why do you think some families who are offered a scholarship do not use it to enroll in a private school? DO NOT READ OPTIONS TO START. FIRST MARK ITEMS MENTIONED WITHOUT PROMPTING OR RECORD ANY OTHER REASONS NOT LISTED. FOR EACH REASON GIVEN ASK WITHOUT PROMPT, ASK, “For how many families is that a reason for not using the scholarship? A few (up to one-third of families), some (between one- and two-thirds of families) or many (more than two-thirds of families)?” AFTER RECORDING ALL UNPROMPTED REASONS, ASK FOR EACH OPTION, “How many families did not use an OSP scholarship because they… [READ OPTION]: None, a few (up to 1/3), some (1/3 to 2/3), or many (more than 2/3)?”

Without prompt	(mentioned without prompt, probe): For how many families is this a reason for not using an OSP scholarship? (for each option not mentioned, ask): How many families did not use an OSP scholarship because they…	None	Few (up to 1/3)	Some (1/3 to 2/3)	Many (more than 2/3)
	Moved out of DC				
	Couldn’t find a private school in a convenient location				
	Got a spot in a preferred a public school				
	Accepted to a preferred private school that does not accept OSP scholarships				
	Decided to home-school				
	Child wanted to stay at their current school with their friends				
	Child was not admitted to the private schools(s) they wanted				
	Thought they could not afford private school				
	Needed better transportation options				
	Needed free or affordable breakfast or lunch that their preferred private school did not offer				
	Needed services for their child’s learning, physical, or developmental disability or other special needs that their preferred private school did not offer				
	Needed English language support for their child that their preferred private school did not offer				
	Needed academic support for their child that their preferred private school did not offer				
	Wanted before- or after-school care that their preferred private school did not offer				
	Wanted sports or another activity that their preferred private school did not offer				
	Wanted accelerated learning opportunities that their preferred private school did not offer				

28. What do you think are the most common reasons families do not use the OSP scholarship to enroll in a private school?

29. Once families are enrolled in the OSP, what communications does [PROGRAM OPERATOR] have with families besides scholarship renewal application reminders/deadlines?

29a. [IF PROGRAM OPERATOR COMMUNICATES WITH ENROLLED FAMILIES] What information does [PROGRAM OPERATOR] collect from families about their experiences having their child attend a private school?

30. A minutes ago we talked about challenges families experience when initially applying for the OSP; do you think families experience similar challenges renewing the OSP scholarship?

• Yes

• No

30a. IF NO: What challenges do you think families face renewing their scholarship? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• Difficulty using the Family Portal (online renewal application)

• Difficulty getting a paper renewal application

• Difficulty finding time to finish the renewal application

• Confusion over deadlines for renewing the scholarship

• Difficulty obtaining or submitting income proof

• Difficulty obtaining or submitting proof of DC residency

• Difficulty because the scholarship renewal application was not available in families’ home language

• Difficulty getting help from us with the scholarship renewal application

31. Do you communicate with families when they stop using the OSP scholarship to attend a private school?

• Yes

• No

31a. If so, what do these communications consist of?

32. Why do you think some families do not continue to use their scholarship? FIRST MARK ITEMS MENTIONED WITHOUT PROMPTING OR RECORD ANY OTHER REASONS NOT LISTED. FOR EACH REASON GIVEN ASK WITHOUT PROMPT, ASK, “For how many families is that a reason for not continuing to use the scholarship? A few (up to one-third of families), some (between one- and two-thirds of families) or many (more than two-thirds of families)?” AFTER RECORDING ALL UNPROMPTED REASONS, ASK FOR EACH OPTION, “How many families did not continue to use an OSP scholarship because they… [READ OPTION]: None, a few (up to 1/3), some (1/3 to 2/3), or many (more than 2/3)?”

Without prompt	(mentioned without prompt, probe): For how many families is this a reason for not continuing to use an OSP scholarship? (for each option not mentioned, ask): How many families did not continue to use an OSP scholarship because they…	None	Few (up to 1/3)	Some (1/3 to 2/3)	Many (more than 2/3)
	Moved out of DC				
	Moved within DC and their private school was no longer conveniently located				
	Got a spot in a preferred a public school				
	Got accepted to a preferred private school that does not accept OSP scholarships				
	Decided to home-school				
	child wanted to be with friends in a school that doesn’t accept OSP scholarships				
	Did not approve of child’s friends at their private school				
	Child was bullied, threatened or hurt at their private school				
	Child was unable to participate in desired sports or other activity at their private school				
	Child was expelled or asked not to return to their private school				
	Child graduated from their private school (transitioned from elementary to middle or middle to high school)				
	Could not continue to afford private school				
	Needed free or affordable breakfast or lunch that their private school did not offer				
	Needed services for their child’s learning, physical, or developmental disability or other special needs that their OSP participating private school did not offer				
	Needed English language support for their child that their private school did not offer				
	Needed academic support for their child that their private school did not offer				
	Wanted before- or after-school care that their private school did not offer				
	Wanted accelerated learning opportunities that their private school did not offer				

Section C: Private Schools’ Challenges

Finally, we’d like to go back to thinking about how and why private schools participate in the OSP and any challenges they face.

33. Earlier we discussed how [PROGRAM OPERATOR] reaches out to participating private schools. Through these communications and interactions with schools, have you learned about any challenges that these schools face with the OSP?

34. What communications occur between [PROGRAM OPERATOR] and a private schools that decides to stop participating in the OSP?

35. Why do you think private schools stop participating in the OSP? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• There is already more demand at those schools than available seats

• Concerns about discipline and school safety if OSP students were admitted

• Timing of OSP scholarships is too late in the school’s admissions process

• DC Opportunity Scholarship amount is not large enough

• School officials had concerns that participating in the OSP would reduce the school’s independence

• Concerns that participation in the OSP would reduce the rigor of the school’s admissions policies

• Concerns that participation in the OSP would reduce the rigor of the school’s academic standards

• Concerns about regulations that might come with participation in the OSP

• Concerns about the OSP evaluation

• Concerns about the amount of paperwork and reports that are required of participating schools

• Concerns about potential future testing requirements

• Uncertainty regarding whether the OSP would continue

• Other: ______________________
  
  

36. What information does [PROGRAM OPERATOR] collect from private schools that do not participate such as reasons for not participating?

37. Thinking about private schools that have not ever participated or have not in the last 5 years, why do you think they choose not to participate? DO NOT READ OPTIONS BUT USE TO CODE AND PROBE

• There is already more demand at those schools than available seats.

• Concerns about discipline and school safety if DC Opportunity Scholarship students were admitted

• Timing of OSP scholarships is too late in the school’s admissions process

• DC Opportunity Scholarship amount is not large enough

• School officials had concerns that participating in the OSP would reduce our school’s independence, distinct character or identity

• Concerns that participation in the OSP would reduce the rigor of the school’s admissions policies

• Concerns that participation in the OSP would reduce the rigor of the school’s academic standards

• Concerns about regulations that might come with participation in the OSP

• Concerns about the evaluation

• Concerns about the amount of paperwork and reports that are required of participating schools

• Concerns about potential future testing requirements

• Uncertainty regarding whether the OSP would continue

• Other: _____________________

Those are all of our questions. Thank you again for taking time out of your schedule to talk with us. We greatly appreciate your help in supporting the study.

Abt Associates Appendix F-1. Program Operator Interview Protocol ▌1