Cognitive Testing Respondent Materials

Participant Recruitment and Screener for NCES Teacher Follow-up Survey

Current and Former Teacher Questionnaire

Telephone Interviews

January 2012

Recruiting Procedures and Goals

Subjects will be recruited from a purchased e-mail list and from an internal database at ICF. Potential subjects will be sent an e-mail with information about the study, and asked to contact ICF if they are interested in participating. Once participants contact ICF, they will be screened and, if appropriate, will work with our recruiter to schedule an interview.

• A minimum of 15 and maximum of 20 participants will be recruited for telephone interviews. They will include 7-10 current teachers (with a subset that have recently changed schools) and 8-10 former teachers.

• For geographic diversity, no more than 3 participants will be recruited from one state.

• At least 3 participants will be recruited from each school level: elementary, middle, and high.

• Participants will receive $30 for participating; the money will be sent to them following the interview.

• Participants will be recruited by email, so they will either be emailing or calling us in response to the email they received.

Recruitment E-mail

Recruiting Teachers and Former Teachers for Paid Research Study

Dear <<insert name>>,

The U.S. Department of Education’s National Center for Education Statistics and the Census Bureau is looking for current and former K-12 classroom teachers from across the country to participate in telephone interviews.  The goal of these interviews is to test new items for a national survey that is administered to educators periodically. The question items to be tested relate to the reasons teachers change schools or leave teaching, how they are evaluated, and how much time they spend on teaching and other school-related activities. Participants will be asked to answer items from the survey and provide feedback.

• Teachers at public and private schools may be eligible to participate.

• We are especially interested in recruiting the following:

• Former teachers who have left teaching the in the past three years.

• Current teachers who have changed schools in the past three years.

Participants will receive $30 in appreciation for completing an interview.

The telephone interviews will take about 45 minutes.  Participants can be interviewed from any phone, at home or at work.

Your participation is voluntary and the information you provide us will be combined with the responses of others in a summary report that does not identify anyone as an individual. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002), 20 U.S.C., § 9573]

If you are interested in participating in this study, please respond to this email or call 1-800-287-1581.

On behalf of the Department of Education and the Census Bureau, we thank you for your help!

Joyce Clingan

Recruiting Coordinator

ICF International

11785 Beltsville Drive, Suite 300

Calverton, MD 20705

www.icfi.com

Response to an Incoming Call

Thank you for your interest in participating in this study. We are conducting a series of interviews with [teachers/former teachers] so we can provide feedback to the U.S. Department of Education’s National Center for Education Statistics and the Census Bureau on the clarity of questions they are planning to include on an educational survey. The question items relate to the reasons teachers change schools or leave teaching, how they are evaluated, and how much time they spend on teaching and other school-related activities.

Each interview will last approximately 45 minutes, and we can schedule it at a time convenient to you, either during or outside of the work day. In order to make sure that you are eligible for this study may I ask you a few questions about your professional background? Continue with Q1.

O utgoing Call in Response to Contact from a Teacher

Hello Mr./Ms./Dr.______________, my name is_______________ and I am responding to your email/phone message stating your interest in participating in a research study for the U.S. Department of Education’s National Center for Education Statistics and the Census Bureau. As you may recall from the email you received, the Department of Education is holding a series of telephone interviews with [teachers/former teachers] in order to get feedback on educational surveys. The questions relate to the reasons teachers change schools or leave teaching, how they are evaluated and how much time they spend on teaching and other school-related activities. Participants who qualify to participate in these interviews will receive $30. Would you be interested in participating in one of these interviews?

• No  Record reason (not interested, not available on that date, etc.) and thank respondent for their time.

• Yes  Continue

I’m glad you are interested in participating. Each interview will last approximately 45 minutes, and we can schedule it at a time convenient to you either during or outside of the work day. In order to make sure that you are eligible for this study may I ask you a few questions about your professional background?

Screening Questions

Q1. Are you currently a K-12 classroom teacher?

• Yes

• No Go to Q3

Q2. Have you changed school in the past three years?

• Yes  Assign to Mover Group. (At least 5)

• No Assign to Current Non-Mover Group

Q3. In what year did you leave K-12 teaching?

_______________

If left teaching within the last three years, assign to Former Teacher Group

If left teaching more than three years ago, terminate phone interview.

Q4. What state do you live/teach in?

• _______________(record answer) (maximum of 3 per state)

Q5. At what school level do you currently teach?

• Elementary (at least 3)

• Middle /Junior High (at least 3)

• High/Senior High (at least 3)

• Other (specify)_____________

Quota not reached  Based on your responses you are eligible to participate in this study. I would like to schedule a time to conduct the interview. What is the best time to contact you to conduct this interview? Record appointment and the number we should contact them on for the interview. To what address should the stipend be mailed? Record mailing address for stipend.

I would like to e-mail you a confirmation letter as well as a copy of the survey items that will be discussed during the interview. Could you please give me an e-mail address where I could send this information? Record email address.

So, just to confirm, you are currently scheduled for an interview on [date] at [time]. We will call you at [number]. If for some reason you will not be able to participate in the scheduled interview please call Joyce Clingan at 1-800-287-1581 as soon as possible. We look forward to the interview on [date] at [time]. Your stipend will be mailed following the interview.

*Quota reached Based on your responses you are eligible to participate in this study. Unfortunately, the group for which you qualify is currently filled. May I put you on our waiting list in the event an interview slot opens up in that group?

• Yes  Thank you. If a spot opens up I will contact you. What is the best way to reach you? (record this information)

• No  Thank you for your time and interest. Terminate interview

C onnected to Teacher Voicemail

Mr./Ms./Dr. _____________________, my name is _____________________ and I am responding to your email/phone message stating your interest in participating in a research study for the U.S. Department of Education’s National Center for Education Statistics and the Census Bureau. If you are still interested in participating, please call me at 1-800-287-1581. Thank you, and I look forward to hearing from you soon.

Name:__________________________________________________________

Address: ________________________________________________________

Title:____________________________________________________________

Email:___________________________________________________________

Phone:___________________________________________________________

Interviewer:_______________________________________________________

Date:________________________ Time:_______________________________

Verbal Informed Consent Statement

Thank you for agreeing to assist us with this project. My name is _____________, and I work for ICF International, a research company that has been hired to conduct this study for the Department of Education’s National Center for Education Statistics and the Census Bureau.

The purpose of this interview is to test potential items for the Teacher Follow-up Survey, a national survey that is conducted by the National Center for Education Statistics and the Census Bureau. Before they make any changes to the survey items, the researchers always test them with potential participants to make sure that the items are as easy to understand as possible. In today’s interview, I am going to have you answer some of these items just as you would if you were really taking the survey so that we can make sure that they are clear and that they are soliciting the information that the survey writers intended. The question items relate to the reasons teachers change schools or leave teaching, how they are evaluated and how much time they spend on teaching and other school-related activities.

Before we begin, here are some things you should know:

• Your participation is totally voluntary. You don’t have to answer any questions that you do not want to answer.

• Your name will not be used in any reports about this interview. We will be taking notes on what you say during the interview, but we will not attach your name to any of your comments. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002), 20 U.S.C., § 9573].

• You will receive a $30 incentive for participating in this interview.

• The interview will last about 45 minutes.

• This interview has been approved by the Office of Management and Budget (OMB# 1850-0803).

• If you have any questions or concerns, you can contact the Census Bureau Director of this project. His name is Michael Long, and his telephone number is (301) 572-0945.

Based on this information, would you still like to participate in this interview?

Current Teacher Questionnaire Interview Protocol

Introduction

“Thank you for agreeing to assist us with this project. My name is _____________, and I work for ICF, a research company that is conducting this study for the Department of Education’s National Center for Education Statistics and the Census Bureau. Before we begin, I’d like to ask whether you have copies of the questions that we are going to be talking about today.

Review Verbal Informed Consent

The purpose of this interview is to test potential items for the Current Teacher Questionnaire of the Teacher Follow-up Survey, a national survey that is conducted by the National Center for Education Statistics and the Census Bureau. Before they make any changes to the survey items, the researchers always test them with potential participants to make sure that the items are as easy to understand as possible. The question items relate to the reasons teachers change schools or leave teaching, how they are evaluated, and how much time they spend on teaching and other school-related activities. In today’s interview, I am going to have you answer some of these items just as you would if you were really taking the survey so that we can make sure that they are clear and that they are soliciting the information that the survey writers intended.

Before we begin, here are some things you should know:

• Your participation is totally voluntary. You don’t have to answer any questions that you do not want to answer.

• Your name will not be used in any reports about this interview. We will be taking notes on what you say during the interview, but we will not attach your name to any of your comments. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002), 20 U.S.C., § 9573].

• You will receive a $30 incentive for participating in this interview.

• The interview will last about 45 minutes.

• This interview has been approved by the Office of Management and Budget (OMB# 1850-0803).

• If you have any questions or concerns, you can contact the Census Bureau Director of this project. His name is Michael Long, and his telephone number is (301) 572-0945.

Based on this information, would you still like to participate in this interview?

As you answer each item I’d like you to ‘think aloud.’ In other words, I would like you to say aloud what you are thinking as you read each question, as you consider the answer choices, and as you finally answer the question. For example, if you are trying to decide which answer to select, please explain why you are unsure. This will help us determine whether the question is truly being understood as it is intended.

Do you have any questions before we begin?

Directions for Interviewer:

For each item, do the following:

1. Ask the participant to read the item, consider the potential answer choices, and then select the most appropriate choice—just as if he or she were actually completing this survey. As they do, remind them to “think aloud.” If they are quiet for a period of time, ask them what they are thinking about. When they select an answer, mark it on the answer sheet for that participant.

2. Allow the participant to answer a complete series (as marked on the protocol) before asking probing questions. Try not to ask any probing or prompting questions as they give their initial answer; if they are having trouble understanding the item or choosing an answer, ask them to describe exactly what they are struggling with.

3. For each item, record three things:

1. Their final answer(s)

2. Notes on any follow-up questions

3. Any other notes on issues that they brought up with the item, anything they struggled with, or anything else relevant that they said while “thinking aloud.”

Questionnaire Items

ITEM 1

Do you CURRENTLY TEACH any regularly scheduled class(es) in any of grades pre-K–12?

(Regularly scheduled classes are those taught at least once per week for a full term.)

* If you teach a particular specialty either within or outside of a regular classroom (e.g., you are a special education teacher, an English as a Second Language teacher, or a reading specialist teaching reading), please answer ‘Yes.’

* If you work in some other capacity at the school (e.g., principal, library media specialist/librarian or school counselor) and occasionally teach a single lesson or unit of instruction, please answer ‘No.’

__ Yes

__ No  Please STOP now and return this questionnaire to the U.S. Census Bureau. You will be sent a different form to complete.

Think-Aloud Notes

Probes

• Is there anything about Item 1 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• What subjects do you currently teach? Verify that the response to the item is consistent with interviewee’s position. If you were a special education teacher that did not have your own classroom, but instead usually pulled students out of class to work with them, how would you answer this question? If you were a librarian at the school, but sometimes came into classes to work with students, how would you answer this question?

• In your own words, what is meant by “regularly scheduled classes?”

ITEM 2

Are you currently teaching in the SAME SCHOOL as you were last year (2010-11)?

__ Yes  GO TO item 6.

__ No

Think-Aloud Notes

Probes

• Is there anything about Item 2 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

ITEM 3

Did you change schools involuntarily (e.g., contract not renewed, laid off, school closed or merged)?

__ Yes  GO TO item 6.

__ No

Think-Aloud Notes

Probes

• Is there anything about Item 3 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

ITEM 4

Indicate the level of importance EACH of the following played in your decision to leave LAST YEAR’S SCHOOL.

* Mark (X) one box on each line.

*If any of the reasons for leaving last year’s school do not apply to you, mark 1 for ‘Not at all important.’

I left last year’s school –

Personal Life Factors	Not at all important	Slightly important	Somewhat important	Very important	Extremely important
Because I moved or wanted to take a job more conveniently located.	___	___	___	___	___
Because of other personal life reasons (e.g., health, pregnancy/childcare, caring for family).	___	___	___	___	___
Because I decided to retire or receive retirement benefits from last year’s school system.	___	___	___	___	___
Salary and Other Job Benefits	Not at all important	Slightly important	Somewhat important	Very important	Extremely important
Because I wanted or needed a higher salary.	___	___	___	___	___
Because I needed better benefits than I received at last year's school.	___	___	___	___	___
Because I was concerned about my job security at last year's school.	___	___	___	___	___
Assignment and Classroom Factors	Not at all important	Slightly important	Somewhat important	Very important	Extremely important
Because I was dissatisfied with my job description or assignment (e.g., responsibilities, grade level, or subject area).	___	___	___	___	___
Because I did not have enough autonomy over my classroom at last year's school.	___	___	___	___	___
Because I was dissatisfied with the large number of students I taught at last year's school.	___	___	___	___	___
Because I felt that there were too many intrusions on my teaching time at last year's school.	___	___	___	___	___
School Factors	Not at all important	Slightly important	Somewhat important	Very important	Extremely important
Because I wanted the opportunity to teach at my current school	___	___	___	___	___
Because I was dissatisfied with workplace conditions (e.g., facilities, classroom resources, school safety) at last year's school.	___	___	___	___	___
Because student discipline problems were an issue at last year's school.	___	___	___	___	___
Because I was dissatisfied with the administration at last year's school.	___	___	___	___	___
Because I was dissatisfied with the lack of influence I had over school policies and practices at last year's school.	___	___	___	___	___
Because there were not enough opportunities for leadership roles or professional advancement at last year’s school.	___	___	___	___	___
Because I did not have the level of support I needed as a teacher (e.g. mentoring, good professional development, coaching).	___	___	___	___	___
Because my opinion about the primary purpose of the school did not match the school’s central mission.	___	___	___	___	___
Because I did not like the way in which the performance of teachers was formally evaluated at last year’s school.	___	___	___	___	___
Student Performance Factors	Not at all important	Slightly important	Somewhat important	Very important	Extremely important
Because I was dissatisfied with how student assessments and school accountability measures impacted my teaching or curriculum at last year’s school.	___	___	___	___	___
Because I was dissatisfied with how some of my compensation, benefits, or rewards were tied to the performance of my students at last year's school.	___	___	___	___	___
Because I was dissatisfied with the support I received for preparing my students for student assessments at last year’s school.	___	___	___	___	___
Other Factors	Not at all important	Slightly important	Somewhat important	Very important	Extremely important
Because of other factors not included in the previous items a-v. – Please specify  __________________________________	___	___	___	___	___

Think-Aloud Notes

Probes

• Is there anything about Item 4 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• Item C: In your own words, what does this item mean? In what situation would someone indicate that this was an important reason that they left?

• Item P: What is an example of a “leadership role?” In your own words, what is meant by “professional advancement?”

• Item Q: What does this item mean, in your own words? In what situation would someone indicate that this was an important reason that they left?

• Item R: In your own words, what does Item R mean?  If you were frustrated that the department head at last year’s school did not seem to like your teaching style, would this be covered under this item?  Why or why not? Did you consider your formal or informal evaluation when you answered this item?

• Can you think of any other reasons for leaving a school that are not on this list?

ITEM 5

What is the LOWEST teaching salary, not including benefits, you would accept to return to last year’s school in the same K-12 teaching position?

*Report in whole dollars.

$ |__|__|__| , |__|__|__|.00 Per year

□ I would not return to the same position in last year’s school regardless of salary.

Think-Aloud Notes

Probes

• Is there anything about Item 5 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• Imagine that you really did not like teaching at last year’s school, and you probably would not return for any amount of money. How would you answer this question?

• Obviously, there are some situations in which someone really doesn’t want to return to a teaching job, but if they were offered a huge figure, like $900,000, they would do so. If that were the case, how would you answer this question?

ITEM 6

Were you formally evaluated for your work as a teacher last school year (2010-11)?

__ Yes

__ No  GO TO item 10.

Think-Aloud Notes

Probes

• Is there anything about Item 6 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• What do you think is meant by a “formal evaluation?” How is this defined in your school? Do you think other teachers in your school might give a different answer to this question? If so, why? Do you think any of them would have a different understanding of what a “formal evaluation” is? At your school is this different from an informal evaluation?

ITEM 7

To what extent do you agree or disagree with each of the following statements about the formal evaluation of your work as a teacher last school year (2010-11)?

*Mark (X) one box on each line.

	Strongly agree	Somewhat agree	Somewhat disagree	Strongly disagree
My work as a teacher was assessed fairly in the formal evaluation.	___	___	___	___
My strengths and weaknesses as a teacher were effectively identified in the formal evaluation.	___	___	___	___
I received feedback from the formal evaluation that was helpful in the development of my work as a teacher.	___	___	___	___
I was satisfied with the formal evaluation process.	___	___	___	___

Think-Aloud Notes

Probes

• Is there anything about Item 7 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• Item A: What do you think this item means, in your own words? Please describe the formal evaluation process you went through last year. In your own words, what is meant by “fairly?” If you thought that the informal input and criticism that you received was fair, but that the formal evaluation you received was not fair, how would you answer this question? Why?

• Item B: If you thought that the person evaluating you had done a good job of identifying your strengths but a bad job of identifying your weaknesses, how would you answer this question? Why?

• Item C: Who provided feedback in your formal evaluation? Is it clear from the question that it is referring to feedback from this person? How would you answer this question if you had received feedback through your formal evaluation that you didn’t find helpful? How would you answer this question if you hadn’t received any feedback at all through your formal evaluation? Notice that this item does not specify who provided the feedback to you. Does this make the item unclear at all?

• Item D: Did you have any difficulty choosing a response? Did you consider your formal or informal evaluation when you answered this item? If you liked the way that you were formally evaluated but were frustrated with some of the informal feedback that you received at your last school, how would you answer this question?  Why?

ITEM 8

Did you receive a single overall rating as part of that formal evaluation?

__ Yes

__ No  GO TO item 10.

Think-Aloud Notes

Probes

• Is there anything about Item 8 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• In your own words, what is this question asking?

If interviewee said “no:” Please describe the kind of rating you received (e.g., no rating, or multiple ratings)?

ITEM 9

Which of the following best describes the single overall rating you received for your work as a teacher last school year (2010-11)?

*Mark (X) only one box.

__ Excellent / Outstanding / Highly effective

__ Satisfactory / Effective

__ Unsatisfactory / Not that effective

Think-Aloud Notes

Probes

• Is there anything about Item 9 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• Would other teachers at your school have difficulty answering this question for any reason?

• How likely do you think it is that teachers would be reluctant to answer this question?

• How likely do you think it is that teachers would provide an answer that is not truthful—that is, indicate that they received a rating of “Excellent” when in fact their rating was “Satisfactory?”

• Can you think of any other single overall rating not on this list of responses?

ITEM 10

Last year, how effective do you think you were as a teacher?

*Mark (X) only one box.

__ Very effective

__ Effective

__ Not that effective

Think-Aloud Notes

Probes

• Is there anything about Item 10 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• In your own words, what is this question asking?

• You responded X. When you chose that answer did you consider last year only, or multiple years?

• In your own words, what is meant by “effective” in Item 10?

• If a teacher was described as “overall a good teacher,” would you consider that teacher to be an effective teacher? Please explain why or why not?

• Can you think of any other response options that are not on this list?

ITEM 11

Overall, how would you describe yourself as a teacher?

*Mark (X) only one box.

__ I am an excellent teacher

__ I am a good teacher

__ I often struggle with teaching

Think-Aloud Notes

Probes

• Is there anything about Item 11 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• In your own words, what is this question asking? How is Item 10 different from Item 11?

• How did you come up with this answer? You answered X. Did you think about your entire teaching career to provide a response?

• Can you think of any other response options that are not on this list?

ITEM 12a

How many hours a week are you paid to deliver INSTRUCTION to a class of students at your current school?

(Example: If your base contract requires you to work 40 hours a week, with 30 of those hours for delivering instruction and 10 hours for planning, monitoring students outside of class time, etc., you would report 30 hours.)

*"PULL-OUT" or "PUSH-IN" TEACHERS: Please include the number of hours you instruct individual students or small groups of students.

|__|__|__| Total WEEKLY hours spent delivering instruction

Think-Aloud Notes

Probes

• Is there anything about Item 12a that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• On Item 12a you answered X hours, how did you come up with this number? Is this the actual number or an estimate?

ITEM 12b

How many hours a week are you paid to deliver INSTRUCTION to a class of students at your current school?

* Count instructional hours included in base contract and exclude hours spent on planning, monitoring students outside of class time, etc.

*"PULL-OUT" or "PUSH-IN" TEACHERS: Please include the number of hours you instruct individual students or small groups of students.

* Report total weekly hours to the nearest whole hour.

If you do not know what is stated in your base contract, please use the following worksheet to calculate your paid instructional hours.

List the number of classes by length of instructional period in a typical week:

__ __ number of classes x __ __ __ instructional minutes = __ __ __ __ minutes

__ __ number of classes x __ __ __ instructional minutes = __ __ __ __ minutes

__ __ number of classes x __ __ __ instructional minutes = __ __ __ __ minutes

__ __ number of classes x __ __ __ instructional minutes = __ __ __ __ minutes

__ __ number of classes x __ __ __ instructional minutes = __ __ __ __ minutes

Sum of minutes of each entry above __ __ __ __ all classes

Convert to hours: ÷ 60

Total WEEKLY hours spent delivering instruction |__|__|__|

Think-Aloud Notes

Probes

• Is there anything about Item 12b that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

Check for consistency in the response for 12a and 12b?

If answer to 12a is different from answer to 12b: Which question was less difficult to answer? Why?

ITEM 13

How many hours are you required to work to receive BASE PAY during a typical FULL WEEK at your current school?

(This would be base contract hours, or the equivalent, NOT including stipends or extra pay for extra duty.)

|__|__|__| Total WEEKLY hours required for BASE PAY

Think-Aloud Notes

Probes

• Is there anything about Item 13 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

ITEM 14

Including hours spent during the school day, before and after school, and on the weekends, how many hours do you spend on ALL teaching and other school-related activities during a typical FULL WEEK at your current school?

|__|__|__| Total WEEKLY hours spent on all teaching and school-related activities

Think-Aloud Notes

Probes

• Is there anything about Item 14 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• In your own words, what is this question asking?

• On Item 14 you answered X hours, how did you come up with this number? Is this the actual number or an estimate? Did you include paid and unpaid hours in your answer?

• The intent of Item 14 was to capture all hours spent on any school-related activities, whether paid or unpaid. Is that clear from the question? If not, how could the question be made clearer?

ITEM 15

During a typical FULL WEEK at your current school, how many hours do you work on teaching and school-related activities that are unpaid; that is, hours that are NOT covered by your base contract hours or by stipends or extra pay?

* Include any unpaid hours worked on teaching and school-related activities during the school day, before and after school, and on the weekends.

* For example, if your base contract includes 10 hours for planning, but you typically spend 25 hours on planning, then you would count 15 hours for unpaid work.

|__|__| Total weekly hours that are UNPAID

Think-Aloud Notes

Probes

• Is there anything about Item 15 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• In your own words, what is this question asking?

• On Item 15 you answered X hours, how did you come up with this number? Is this the actual number or an estimate?

• Comparing items 14 and 15, which question was easier to answer? Why?

• If you were asked for the number of hours worked for all duties for which you receive extra pay or a stipend, how easy or difficult would it be to determine that number?

Former Teacher Questionnaire Interview Protocol

Introduction

“Thank you for agreeing to assist us with this project. My name is _____________, and I work for ICF, a research company that is conducting this study for the Department of Education’s National Center for Education Statistics and the Census Bureau. Before we begin, I’d like to ask whether you have copies of the questions that we are going to be talking about today.

Review Verbal Informed Consent

The purpose of this interview is to test potential items for the Former Teacher Questionnaire of the Teacher Follow-up Survey, a national survey that is conducted by the National Center for Education Statistics and the Census Bureau. Before they make any changes to the survey items, the researchers always test them with potential participants to make sure that the items are as easy to understand as possible. The question items relate to the reasons teachers change schools or leave teaching, how they are evaluated, and how much time they spend on teaching and other school-related activities. In today’s interview, I am going to have you answer some of these items just as you would if you were really taking the survey so that we can make sure that they are clear and that they are soliciting the information that the survey writers intended.

Before we begin, here are some things you should know:

• Your participation is totally voluntary. You don’t have to answer any questions that you do not want to answer.

• Your name will not be used in any reports about this interview. We will be taking notes on what you say during the interview, but we will not attach your name to any of your comments. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002), 20 U.S.C., § 9573].

• You will receive a $30 incentive for participating in this interview.

• The interview will last about 45 minutes.

• This interview has been approved by the Office of Management and Budget (OMB# 1850-0803).

• If you have any questions or concerns, you can contact the Census Bureau Director of this project. His name is Michael Long, and his telephone number is (301) 572-0945.

Based on this information, would you still like to participate in this interview?

As you answer each item I’d like you to ‘think aloud.’ In other words, I would like you to say aloud what you are thinking as you read each question, as you consider the answer choices, and as you finally answer the question. For example, if you are trying to decide which answer to select, please explain why you are unsure. This will help us determine whether the question is truly being understood as it is intended.

Do you have any questions before we begin?

Directions for Interviewer:

For each item, do the following:

2. Ask the participant to read the item, consider the potential answer choices, and then select the most appropriate choice—just as if he or she were actually completing this survey. As they do, remind them to “think aloud.” If they are quiet for a period of time, ask them what they are thinking about. When they select an answer, mark it on the answer sheet for that participant.

3. Allow the participant to answer a complete series (as marked on the protocol) before asking probing questions. Try not to ask any probing or prompting questions as they give their initial answer; if they are having trouble understanding the item or choosing an answer, ask them to describe exactly what they are struggling with.

4. For each item, record three things:

   1. Their final answer(s)

   2. Notes on any follow-up questions

   3. Any other notes on issues that they brought up with the item, anything they struggled with, or anything else relevant that they said while “thinking aloud.”

Questionnaire Items

ITEM 1

Do you CURRENTLY TEACH any regularly scheduled class(es) in any of grades pre-K – 12?

(Regularly scheduled classes are those taught at least once per week for a full term.)

* If you teach a particular specialty either within or outside of a regular classroom (e.g., you are a special education teacher, an English as a Second Language teacher, or a reading specialist teaching reading), please answer ‘Yes.’

* If you work in some other capacity at the school (e.g., principal, library media specialist/librarian or school counselor) and occasionally teach a single lesson or unit of instruction, please answer ‘No.’

__ Yes

__ No

Think-Aloud Notes

Probes

• Is there anything about Item 1 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• If you were a special education teacher that did not have your own classroom, but instead usually pulled students out of class to work with them, how would you answer this question? If you were a librarian at the school, but sometimes came into classes to work with students, how would you answer this question?

• In your own words, what is meant by “regularly scheduled classes?”

ITEM 2

Did you leave your K-12 teaching position involuntarily (e.g., contract not renewed, laid off, school closed or merged)?

__ Yes  GO TO item 5.

__ No

Think-Aloud Notes

Probes

• Is there anything about Item 2 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

ITEM 3

Indicate the level of importance EACH of the following played in your decision to leave the position of a K-12 teacher.

* Mark (X) one box on each line.

* If any of the reasons for leaving teaching do not apply to you, mark 1 for ‘Not at all important.’

I left the position of a K-12 teacher –

Personal Life Factors	Not at all important	Slightly important	Somewhat important	Very important	Extremely important
Because I moved or wanted to take a job more conveniently located.	___	___	___	___	___
Because of other personal life reasons (e.g., health, pregnancy/childcare, caring for family).	___	___	___	___	___
Because I decided to retire or receive retirement benefits from last year’s school system.	___	___	___	___	___
Salary and Other Job Benefits	Not at all important	Slightly important	Somewhat important	Very important	Extremely important
Because I wanted or needed a higher salary.	___	___	___	___	___
Because I needed better benefits than I received at last year's school.	___	___	___	___	___
Because I was concerned about my job security at last year's school.	___	___	___	___	___
Career Factors	Not at all important	Slightly important	Somewhat important	Very important	Extremely important
Because I decided to pursue a position other than that of a K-12 teacher.	___	___	___	___	___
Because I decided to take courses to improve career opportunities WITHIN the field of education.	___	___	___	___	___
Because I decided to take courses to improve career opportunities OUTSIDE the field of education.	___	___	___	___	___
Because I was dissatisfied with teaching as a career.	___	___	___	___	___
Because there were not enough opportunities for leadership roles or professional advancement at last year’s school.	___	___	___	___	___
Because I did not like the way in which the performance of teachers was formally evaluated at last year’s school.	___	___	___	___	___

3. Continued – Indicate the level of importance EACH of the following played in your decision to leave the position of a K-12 teacher.

* Mark (X) one box on each line.

* If any of the reasons for leaving teaching do not apply to you, mark 1 for ‘Not at all important.’

I left the position of a K-12 teacher –

Assignment and Classroom Factors	Not at all important	Slightly important	Somewhat important	Very important	Extremely important
Because I was dissatisfied with my job description or assignment (e.g., responsibilities, grade level, or subject area).	___	___	___	___	___
Because I did not have enough autonomy over my classroom at last year's school.	___	___	___	___	___
Because I was dissatisfied with the large number of students I taught at last year's school.	___	___	___	___	___
Because I felt that there were too many intrusions on my teaching time at last year's school.	___	___	___	___	___
School Factors	Not at all important	Slightly important	Somewhat important	Very important	Extremely important
Because I was dissatisfied with workplace conditions (e.g., facilities, classroom resources, school safety) at last year's school.	___	___	___	___	___
Because student discipline problems were an issue at last year's school.	___	___	___	___	___
Because I was dissatisfied with the administration at last year's school.	___	___	___	___	___
Because I was dissatisfied with the lack of influence I had over school policies and practices at last year's school.	___	___	___	___	___
Because my opinion about the primary purpose of the school did not match the school's central mission.	___	___	___	___	___

3. Continued – Indicate the level of importance EACH of the following played in your decision to leave the position of a K-12 teacher.

* Mark (X) one box on each line.

* If any of the reasons for leaving teaching do not apply to you, mark 1 for ‘Not at all important.’

I left the position of a K-12 teacher –

Student Performance Factors	Not at all important	Slightly important	Somewhat important	Very important	Extremely important
Because I was dissatisfied with how student assessments and school accountability measures impacted my teaching or curriculum at last year’s school.	___	___	___	___	___
Because I was dissatisfied with how some of my compensation, benefits, or rewards were tied to the performance of my students at last year's school.	___	___	___	___	___
Because I was dissatisfied with the support I received for preparing my students for student assessments at last year’s school.	___	___	___	___	___
Other Factors	Not at all important	Slightly important	Somewhat important	Very important	Extremely important
Because of other factors not included in the previous items a-x. Please specify  ­________________ ______________________________	___	___	___	___	___

Think-Aloud Notes

Probes

• Is there anything about Item 3 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• Item C: In your own words, what does this item mean? In what situation would someone indicate that this was an important reason that they left?

• Item K: What is an example of a “leadership role?” In your own words, what is meant by “professional advancement?”

• Item L: In your own words, what does Item L mean?  If you were frustrated that the department head at last year’s school did not seem to like your teaching style, would this be covered under this item?  Why or why not? Did you consider your formal or informal evaluation when you answered this item?

• Item U: What does this item mean, in your own words? In what situation would someone indicate that this was an important reason that they left?

• Can you think of any other reasons for leaving teaching that are not on this list?

ITEM 4

What is the LOWEST teaching salary, not including benefits, you would accept to return to last year’s school in the same K-12 teaching position?

*Report in whole dollars.

$ |__|__|__| , |__|__|__| . 00 Per Year

□ I would not return to the same position in last year’s school regardless of salary.

Think-Aloud Notes

Probes

• Is there anything about Item 4 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• Imagine that you really did not like teaching at last year’s school, and you probably would not return for any amount of money. How would you answer this question?

• Obviously, there are some situations in which someone really doesn’t want to return to a teaching job, but if they were offered a huge figure, like $900,000, they would do so. If that were the case, how would you answer this question?

ITEM 5

Were you formally evaluated for your work as a teacher last school year (2010-11)?

__ Yes

__ No  GO TO item 9.

Think-Aloud Notes

Probes

• Is there anything about Item 5 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• What do you think is meant by a “formal evaluation?” How is a formal evaluation different from an informal evaluation?

ITEM 6

To what extent do you agree or disagree with each of the following statements about the formal evaluation of your work as a teacher last school year (2010-11)?

* Mark (X) one box on each line.

	Strongly agree	Somewhat agree	Somewhat disagree	Strongly disagree
My work as a teacher was assessed fairly in the formal evaluation.	___	___	___	___
My strengths and weaknesses as a teacher were effectively identified in the formal evaluation.	___	___	___	___
I received feedback from the formal evaluation that was helpful in the development of my work as a teacher.	___	___	___	___
I was satisfied with the formal evaluation process.	___	___	___	___

Think-Aloud Notes

Probes

• Is there anything about Item 6 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• Item A: What do you think this item means, in your own words? Please describe the formal evaluation process you went through last year. In your own words, what is meant by “fairly?” If you thought that the informal input and criticism that you received was fair, but that the formal evaluation you received was not fair, how would you answer this question? Why?

• Item B: If you thought that the person evaluating you had done a good job of identifying your strengths but a bad job of identifying your weaknesses, how would you answer this question? Why?

• Item C: Who provided feedback in your formal evaluation? Is it clear from the question that it is referring to feedback from this person? How would you answer this question if you had received feedback through your formal evaluation that you didn’t find helpful? How would you answer this question if you hadn’t received any feedback at all through your formal evaluation? Notice that this item does not specify who provided the feedback to you. Does this make the item unclear at all?

• Item D: Did you have any difficulty choosing a response? Did you consider your formal or informal evaluation when you answered this item? If you liked the way that you were formally evaluated but were frustrated with some of the informal feedback that you received at your last school, how would you answer this question?  Why?

ITEM 7

Did you receive a single overall rating as part of that formal evaluation?

__ Yes

__ No  GO TO item 9.

Think-Aloud Notes

Probes

• Is there anything about Item 7 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• In your own words, what is this question asking?

If interviewee said “no:” Please describe the kind of rating you received (e.g., no rating, or multiple ratings)?

ITEM 8

Which of the following best describes the single overall rating you received for your work as a teacher last school year (2010-11)?

* Mark (X) only one box.

__ Excellent / Outstanding / Highly effective

__ Satisfactory / Effective

__ Unsatisfactory / Not that effective

Think-Aloud Notes

Probes

• Is there anything about Item 8 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• Would other teachers at your school have difficulty answering this question for any reason?

• How likely do you think it is that teachers would be reluctant to answer this question?

• How likely do you think it is that teachers would provide an answer that is not truthful—that is, indicate that they received a rating of “Excellent” when in fact their rating was “Satisfactory?”

• Can you think of any other single overall rating not on this list of responses?

ITEM 9

Last year, how effective do you think you were as a teacher?

* Mark (X) only one box.

__ Very effective

__ Effective

__ Not that effective

Think-Aloud Notes

Probes

• Is there anything about Item 9 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• In your own words, what is this question asking?

• How likely do you think it is that teachers would be reluctant to answer this question?

• You responded X. When you chose that answer did you consider last year only, or multiple years?

• In your own words, what is meant by “effective” in Item 9?

• If a teacher was described as “overall a good teacher,” would you consider that teacher to be an effective teacher? Please explain why or why not?

• Can you think of any other response options that are not on this list?

ITEM 10

Overall, how would you describe yourself as a teacher?

* Mark (X) only one box.

__ I am an excellent teacher

__ I am a good teacher

__ I often struggle with teaching

Think-Aloud Notes

Probes

• Is there anything about Item 10 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• In your own words, what is this question asking? How is Item 9 different from Item 10?

• How did you come up with this answer? You answered X. Did you think about your entire teaching career to provide a response?

• How likely do you think it is that teachers would be reluctant to answer this question?

• Can you think of any other response options that are not on this list?

ITEM 11

Indicate how important each factor would be in influencing your decision to return to the position of a K-12 teacher.

* Mark (X) one box on each line.

	Not at all important	Slightly important	Somewhat important	Very important	Extremely important
Ability to maintain your teacher retirement benefits	___	___	___	___	___
State certification reciprocity (a state’s acceptance of teacher certifications from other states)	___	___	___	___	___
An easier and less costly way to renew/earn certification	___	___	___	___	___
Smaller class sizes	___	___	___	___	___
Availability of full-time teaching positions	___	___	___	___	___
Availability of part-time teaching assignments	___	___	___	___	___
Forgiveness of your student loans	___	___	___	___	___
Housing incentives (e.g., subsidies, rent assistance, low interest loans, relocation assistance)	___	___	___	___	___
An increase in salary	___	___	___	___	___
Availability of suitable childcare options	___	___	___	___	___

Think-Aloud Notes

Probes

• Is there anything about Item 11 that could be confusing or unclear?

• Did you have any difficulty answering this question for any reason?

• Item C: In your own words, what does this item mean? In what situation would someone indicate that this was an important reason that they left?

• Item D: In your own words, what does this item mean? In what situation would someone indicate that this was an important reason that they left?

• Do you think there is a difference between “teaching position” in ‘e’ and “teaching assignment” in “f”? Please explain.