2013-14 Civil Rights Data Collection (CRDC) Improvement Study Cognitive Interviews

Draft Protocol for CRDC Cognitive Interviews March 28, 2014

2013-14 Civil Rights Data Collection (CRDC)

Improvement Study Cognitive Interviews Protocol

Introduction:

Thank you for taking the time to speak with us today. My name is__________, and my colleague, ___________, and I are members of the Sanametrix and the American Institutes for Research (AIR) project team that is conducting interviews on behalf of the National Center for Education Statistics (NCES) to improve the Civil Rights Data Collection (CRDC) data collection process. NCES has recently assumed responsibility for designing a new data collection tool for the CRDC in coordination with the Office of Civil Rights.

You were chosen to participate in this interview because we value your opinion and would like to understand from you what ways we might be able to improve definitions of key CRDC terms, table layouts, or organization of the survey to make it easier for you to report your data. We recognize that everyone has different experiences with the CRDC and want to stress that there are no right or wrong answers. In order for us to advise NCES on how to improve the CRDC process, we need to hear your thoughtful and honest feedback.

I will be asking the questions, and _______________ will be taking notes. We would like to record our conversation to make sure that we catch all of the important information that you will share with us and to serve as our back up to the notes that _______________ will take today. NCES will have access to the recordings. Is it okay for me to record you?

During our time today, we would like to focus on the CRDC survey rather than the way you store and report the data for the CRDC. I am particularly interested in hearing about definitions, instructions, and terminology that you think are confusing. We also want to hear about table layouts that you think are awkward or difficult to interpret, as well as any suggestions for improvement. Did you get a chance to review the draft survey we sent you?

Do you have any questions before we get started?

Notes to interviewer:

• If the respondent has not reviewed the draft survey prior to the phone call, ask only the questions in the gray cells.

• If the respondent has reviewed the draft survey prior to the phone call, ask ALL questions.

• Listen for and probe on words, instructions, and terms that are confusing.

• Listen for and probe on table layouts that are confusing.

INTRODUCTION
Q1. Let’s begin with the Introduction. Did you read this information?
If yes –Can you tell me in your own words what this page tells you about the CRDC? Is there information you think should be provided in the Introduction that is not? PROBE for why.
If no – Is there any particular reason you decided not to read this information?
PART I SCHOOL FORM: Fall Snapshot Data – General Instructions
Q2. Let’s move on to the next page. Did you read the general instructions for Part 1 here?
If yes – What is your understanding of what the general instructions are explaining/asking for? What about the second and third bullets? How would you include or exclude students from the race/ethnicity counts? Is there information you think should be provided in the general instructions that is not? PROBE for why.
If no – Is there any particular reason you decided not to read this information?
SECTION I: SCHOOL CHARACTERISTICS
Q3. Let’s look at data tables 1 through 11 on school characteristics. In your own words, what data are these tables asking for? Generally, can you suggest how the tables might be improved?
If yes - Tell me about them. Let’s start with the first one you encountered.
If no – Ask Q4 and Q5.
Q4. Now turn to the first table in Section 1: Grades Offered (#1). Take a minute to look at this now and tell me, what does bullet one mean to you? How would you determine whether to respond you or no? Let’s look at question 3. Do you have any ungraded students? If yes, How would you respond to this question? Did you have any problems or questions about these tables? What about question 7? Is there enough information to determine whether you offer a gifted and talented education program? Can you suggest how the instructions, definitions, and/or table might be improved?
Q5. Does your district/school offer preschool? Since preschool can be offered in a variety of ways, such as through private programs, Head start, state-operate programs, and public programs operate by school districts, can you tell me why you answered yes or no? If yes, let’s turn to the next table on preschool enrollment (#2). In your own words, what data is this table asking for? Do you have any problems or comments on this table? Can you suggest how the instructions, definitions, and/or table might be improved? Does the definition of preschool match how you define preschool in your district?
Q6. Let’s move to the questions on the Advanced Placement program and Dual Enrollment program (# 8- 10). Can you tell me in your own words, what data are these tables asking for? Was there anything in these instructions or definitions you found confusing or have questions about? PROBE for why. Is there any clarification we could provide to make it easier? What about Dual Enrollment Programs? Do you have a dual enrollment/dual credit programs? [If yes], please describe. Would you have any problems reporting these data? PROBE for why.
SECTION II: STUDENT ENROLLMENT
Q7. Let’s move on to Section II: Student Enrollment. There are a couple of instructions and a few key definitions at the beginning of this section. Did you read them? [If not,] Why not? What is your understanding of what the general instructions and key definitions are explaining/asking for? Is there anything in these instructions and definitions that you noted as unclear or confusing? PROBE for why. Is there information you think should be provided in the instructions and key definitions that is not? PROBE for why.
Q8. Take a look at data tables on Overall Student Enrollment, Preschool Enrollment, LEP Enrollment And Students With Disabilities Enrollment (#12-15). Do you have any problems or comments on these tables? Can you suggest how the instructions, definitions, and/or tables might be improved? Let’s look at the table on LEP students and LEP students enrolled in LEP programs; can you tell me what students you would include (or exclude) in the first and second row? Is there information you think should be provided that is not? PROBE for why
Q9. What about the table on Students With Disabilities Enrollment (#15)? Can you look over this table and tell me in your own words what data it is asking for? What data would you include and exclude for this table? Can you suggest how the table might be improved? For section 504 students, let’s take a look over the instructions. What students would you include or exclude? PROBE for why.
SECTION III: CLASSES AND PROGRAM ENROLLMENT
Q10. Okay. Let’s move on to Section III: Classes, Course Enrollment and Program Enrollment. Can you suggest how the instructions, definitions, and/or tables might be improved?
Q11. Turn now to the General Instructions in Section III Classes and Program Enrollment. Let’s take a look at the first bullet under general instructions? How would you determine the count date? Was there anything confusing about this bullet? What about the 3rd bullet? What do you think they mean by duplicated and unduplicated counts?
Q12. What about the Algebra I and Advanced Math/Science tables? Is there anything in these instructions and definitions that you noted as problematic, unclear or confusing? PROBE for why. Would there be any Algebra I classes or students that you would exclude? If so, why? Let’s look at question 22, number of advanced mathematics classes. Can you look over this table and tell me in your own words what data it is asking for?
Q13. How would you respond to question 31 on single-sex classes? Can you look over this table and tell me in your own words what data it is asking for?
SECTION IV: SCHOOL STAFF
Q14. Let’s move on to Section IV: School Staff. What problems or comments do you have in this section? Can you suggest how the instructions, definitions, and/or tables might be improved?
Q15. The first bullet in the instructions in this section defines an FTE. Did you refer to this information when looking at this section? What instructions did you find useful? Is there information you think should be provided in the instructions that is not? PROBE for why. Let’s look over the teacher definition and think about the teachers in your school/district. Are there teachers that you would not know whether to include or exclude after reading this definition?
Q16. Are there any types of staff listed that would be difficult for you to report on? Can you explain? Is there anything in the staff definitions that is problematic or unclear?
Q17. Let’s look specifically at the optional for 2013-2014 table on Security Staff (#49). Take a minute to review this and in your own words, tell me what security staff you would include in the data counts. What type of security personnel work at the schools in your district? Are they employed on your payroll or do they contract for services?
PART 2 SCHOOL FORM: Cumulative or End-of-year data
Q18. Now let’s turn to Part 2. This section asks for you to report the cumulative number of students for the entire school year. How would you explain what that means to someone you needed to train to report for the CRDC?
SECTION I: ALGEBRA I PASSING
Q19. Let’s look at this first set of tables in Part 2 on Algebra I passing. What does Algebra I passing mean to you? What students would you include or exclude in this table? Can you explain why? How does it differ from reporting Algebra enrollment data?
SECTION II: CREDIT RECOVERY
Q20. Now let’s look at the section on Credit Recovery. Please take a minute to read over the definition provided and the table instructions. Do you think this information is easy or difficult to understand? PROBE for why.
SECTION III: SAT REASONING, ACT AND ADVANCED PLACEMENT EXAM-TAKING AND RESULTS
Q21. Let’s move on to the next section on the SAT, ACT, and AP exam taking. Did you have any problems or comments on this section? Let’s look at the first and second rows in question 60, Students who received a qualifying score on AP exams during 2013-14 school year. What students would you include or exclude here? Would you count any students in both rows? Why? Can you suggest how the instructions, definitions, and/or tables might be improved?
SECTION IV: CHRONIC STUDENT ABSENTEEISM
Q22. Okay, let’s turn to the section on Chronic Student Absenteeism. What problems or comments did you note about this section? Take a moment to read the definition and instructions in this section. What is helpful or unhelpful about the definition and instructions? Can you suggest how the instructions, definitions, and/or tables might be improved? Is there information you think should be provided in the instructions or definitions that is not? PROBE for why. Would reporting this data for the CRDC be easy or difficult for your district? Why?
SECTION V: STUDENT RETENTION
Q23. The next section is on Student Retention. What problems or comments did you note about this section? Can you suggest how the instructions, definitions, and/or tables might be improved? Do you have transitional kindergarten? If yes, could you talk about how you would determine whether and how you would count students in those programs in the retention tables?
SECTION VI: SINGLE SEX INTERSCHOLASTIC ATHLETICS
Q24. Okay. Let’s move on to the next section on Single-sex Athletics. In your own words, what data are being requested? Can you tell me in your own words what the third row in question 64 means to you? What additional information do you need to know to determine whether or not to include certain teams to complete this table? PROBE for why.
SECTIONS VII-IX: STUDENT DISCIPLINE, OFFENSES, BULLYING & HARASSMENT
Q25. The next sections on Student Discipline and Bullying and Harassment are being expanded. There is a lot of information to cover in this section. I’d like to turn specifically to the Optional Tables starting with number 69and ending with number 78. Please take a few minutes to review these tables and share your thoughts about these tables with me. I am particularly interested in hearing about terms, definitions, and instructions that are unclear. Can you suggest how the instructions, definitions, and/or tables might be improved? IF TIME ALLOWS ASK FOLLOWING QUESTION. Do you have any comments in general on your district’s ability to provide this type of information for the CRDC?
Q26. Let’s look at the table of discipline for students with disabilities (#68). Can you tell me in your own words what data this table is collecting? The column headers in this table (#68) refer to total IDEA students, can you talk about how you would report Section 504 only students? What about rows three and four in table 68? What do you think they mean by only one and more than one out-of-school suspensions? Would you include any students in both rows? Why? What do you think they are collecting in rows five through seven in the table (expulsions)? Can you describe how these rows are different from one another? Do you have any recommendations or comments on how this table can be improved?
Q27. Let’s turn to table on transfer to alternative school or regular school for students without disabilities (#74), what does transfer for disciplinary reasons mean to you? Which students would you include or exclude here?
Q28. Let’s look at the section on Offenses and specifically to the table on Offense – Number of Incidents (table #23). Please take a few minutes to review the table and share your thoughts about the categories. Are these categories difficult for your district to report? Is there anything about this table that could be improved to make it easier for your district to report this information?
Q29. And let’s turn to the Bullying/ Harassment definitions and tables (80-83), and tell me in your own words what data are being asked for? Can you suggest how the definitions or instructions could be improved? Let’s look at question 80, what does allegation mean to you? Let’s look at table 81, what data are being asked for?
Section IX: RESTRAINT AND SECLUSION
Q30. Take a moment to read the definition and instructions in Section IX Restrain and Seclusion. What is helpful or unhelpful about the definition and instructions? Is there information you think should be provided in the instructions or definitions that is not? PROBE for why. Let’s take a look at question 86, what does instance mean to you?
SECTION XII: SCHOOL EXPENDITURES
Q31. The last section I want to cover is Section XI on School Expenditures. The beginning of this section includes instructions for School Staff Salaries. Please take a moment to review these and let me know if anything is unclear. Can you suggest how the instructions might be improved? How do you track high mobility staff? What about Non-Personnel Expenditures (Tables 90 & 94)? Is there anything we should do to improve the information provided to define these expenditures? Are these categories difficult for your district to report? Is there anything about this table that could be improved to make it easier for your district to report this information?
Q32. Those are all the questions I have for you about the draft survey. Is there anything else you would like to add or any other concerns you would like share with NCES?
Q33. IF TIME ALLOWS ASK FOLLOWING QUESTION. Section XI: Teacher Absenteeism Let’s go back to section XI. Take a moment to read the definition and instructions in this section. What is helpful or unhelpful about the definition and instructions? Can you tell me in your own words what you would report? Can you suggest how the instructions, definitions, and/or tables might be improved? Is there information you think should be provided in the instructions or definitions that is not? PROBE for why.
DRAFT GROUPING OF DATA ELEMENTS
Q34. IF TIME ALLOWS ASK FOLLOWING QUESTION. We sent you a Draft Grouping of Data Elements. Do you have any feedback for us about how we’ve grouped the different data elements? Would it be easier or more problematic to reorder the questions in the survey by these topic areas? Are there other topic areas, or different ways to group the data elements that would make it easier for you to submit your data? Would it be problematic to group questions that ask for beginning of school year data and end of school year data by topic area within the survey? Is it easier or more problematic to keep LEA-level questions and school-level questions about the same topic areas separate? Would it be better to group the LEA-level and school-level questions about the same topic area together? IF NOT ENOUGH TIME, ask respondent to provide feedback via email.

CONCLUDE INTERVIEW:

Thank you for your time. If you think of anything else you would like to share, please give me a call or email me at ___________________..

8