ECLS-K:2011 Spring 4th Grade Cognitive Interviews and Timing Tests

NCES Cognitive, Pilot, and Field Test Studies System

Appendix C ECLS-K-2011 Spring 4th Grade Cog Lab - Cognitive Interview Protocols

ECLS-K:2011 Spring 4th Grade Cognitive Interviews and Timing Tests

OMB: 1850-0803

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1850-0803 can be found here:

Document [docx]

Download: docx | pdf

Appendix C:

Protocols for the Cognitive Interviews with Teachers and Parents

Case ID: T_

Date:

Time:

Interviewer:

Note taker:

Spring 2014

RLA Teacher Questionnaire Protocol

Fourth-Grade Telephone-Based RLA Teacher Cognitive Interview Protocol

First let me start by saying thank you for talking with me today. We would like your input on some questions that may be included in a questionnaire for fourth-grade reading and language arts teachers participating in the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11. The study is a longitudinal study of elementary school students conducted by the National Center for Education Statistics of the U.S. Department of Education.

1. Before we go any further, I just need to identify which questionnaire you received in the mail to review. Does your questionnaire cover refer to reading, math or science? (Circle one)

Reading 1 (Continue)

Math 2 (Skip to 1c)

Science 3 (Skip to 1c)

1a. And can I just confirm that you currently teach this subject to 4^th graders?

Yes 1 (SKIP TO #2)

No 2 (CONTINUE)

1b. Which of these subjects do you teach to 4^th graders? (READ RESPONSE OPTIONS)

Math 1

Science 2

Other/Don’t teacher 4^th graders 3 (Discontinue: Thank you for agreeing to participate, but at this time we are only conducting interviews with fourth-grade teachers who teach reading and language arts, math, or science.)

(IF CODES 2 or 3 in #1 OR CODES 1-2 IN 1b)

1c. We will need to send you the correct questionnaire and then reschedule this interview. I will make sure that the correct questionnaire is sent to you. (SUSPEND INTERVIEW AND RESCHEDULE FOR SOMETIME IN NEXT 5-7 DAYS)

2. Do you have the questionnaire in front of you now?

Yes 1 (CONTINUE)

No, but can go retrieve it 2 (ASK R TO RETRIEVE IT)

No, cannot find it/did not receive it 3 (ASK R FOR EMAIL OR FAX AND SEND IT IMMEDIATELY. RESCHEDULE INTERVIEW IF NECESSARY)

There are a few things you should know before we start. First, the interview today will take approximately one hour. I’ll be taking notes, so I may pause momentarily or you may hear me writing (or typing). (IF APPROPRIATE: [NAME] is also here with me to take more detailed notes.) Even though I am taking notes, in a detailed interview like this, we often want to go back and verify our notes.

3. So, with your permission I would like to audio-record this interview. Is that okay?

Yes 1 (TURN ON RECORDER, CONTINUE)

No 2 (SAY: “OK, WE WILL ONLY TAKE NOTES THEN” AND SKIP TEXT IMMEDIATELY BELOW)

(ONCE RECORDER IS ON) Just to confirm on the recording - now that I’ve turned on the recorder - can you please confirm you have consented to participate in this interview and that I have your permission to record our discussion?

[Respondent confirms]

Great!

I would like to get your feedback on some of the items in the questionnaire. Many of the questions ask you to think about a specific child by referencing “the child on the cover of the questionnaire.” So I’d like you to have a particular student from your 4^th grade reading class in mind as we go through the questionnaire. You don’t need to tell me that student’s name, but please keep that particular child in mind as you answer the questions.

Do you have someone in mind? (IF YES: Great!)

(IF NO: Please think about the first student, alphabetically, in your first reading class of the day.)

Also, we are not going to go through the entire questionnaire – I will ask you to jump to different points in the questionnaire and ask for your feedback on particular items. I will ask you to read the questions out loud and talk about how you would go about answering them. In some cases, that may mean just talking about the mental steps you would take to arrive at the answer. After talking about the mental steps you would take to answer a question, if you are able to provide an answer, I would like to know the response you would write in if you were answering the questionnaire. However, for some questions, it may be that you would need to look up documentation or talk to someone else to be able to provide an answer. I would like to know what you would do to get the answer to a question like that. After each questionnaire item that you talk about, I will ask some follow-up questions to find out if there is anything we can do to make the question easier to understand and answer.

(NOTE TO COGNITIVE INTERVIEWER: RECORD ALL RESPONSES ON THE READING TEACHER COGNITIVE INTERVIEW DATA COLLECTION FORM.)

Please turn to page 18 of your questionnaire and the first item we will look at is item E1.

Question E1

Please read this question out loud and say what you are thinking about or what you would do to answer each of the items in this question. Also, for each item, please tell me what answer you would mark. I will ask you some follow-up questions about each item as we go through this question. Also, when it asks about the “child identified on the cover of this questionnaire,” please think about the student I asked you to keep in mind earlier.

If respondent does not indicate a response, please probe: What would you mark on your questionnaire (if you were answering this question as part of the survey)?

Ask the following probes after each subitem a-g:

(ASK ONLY AFTER ITEM A) Did you interpret the text after the word “including” to be identifying the only activities we want you to consider when answering this item? Or did you interpret it as identifying examples of activities included in that skill area and think that when answering you should consider all the activities you do that fit within that skill area?

Are the examples shown here what comes to mind when you think about what 4^th graders are taught in your school regarding this skill? If not, what does come to mind when you think about teaching this skill to 4^th graders?

Were any of the examples in this particular item redundant or confusing?

Additional probes after all items have been asked about (ONLY ASK PROBES THAT HAVE NOT ALREADY BEEN COVERED IN DISCUSSION OF INDIVIDUAL ITEMS):

Are any of these the kinds of skills that are not taught to 4^th graders? Which ones?

How would you answer about a particular skill area if you teach some of the example activities more often than other example activities? For example, in item g, if you spend more time teaching about roots than you spend on studying words and their context.

How would you answer the question if you do not teach one of the example activities within a particular skill area at all? For example, in item c, if you compare different texts, but do not ever compare texts to movies?

Is it difficult to answer using the numeric categories that were provided? Would a different scale be better?

Now please turn to page 19 of your questionnaire, item E2.

Question E2

Please read this question out loud and then just focus on items e through m. Say what you are thinking about or what you would do to answer each item. Also, for each of these items, please tell me what answer you would mark. We will pause after each item so I can ask you some follow-up questions. (If needed: You don’t need to read items a-d.)

If respondent does not indicate a response, please probe: What would you mark on your questionnaire (if you were answering this question as part of the survey)?

(ASK ONLY FOR ITEM i:) How did you interpret the phrase “when given specific questions”? Would this item have been easier or more difficult if this was asking you about a specific or a typical child rather than all the children in your classroom?

Additional probes after all items have been asked about:

Please take a moment to read all the items in the list. [PAUSE FOR RESPONDENT TO READ.] Overall, how well do you think the activities in this question align with the fourth-grade curriculum at your school?

Are any of these activities used in other grades at your school, and NOT in 4^th grade? Which ones, and in what grades are they used?

The next question we will talk about is on page 10 of your questionnaire, item C1.

Question C1

Please read this question out loud and say what you are thinking about or what you would do to answer this question. Also, for each item, please tell me what answer you would mark. When the question asks you to think about the child identified on the cover of this questionnaire, please think about the child you were asked to think about earlier. This time, you will read and answer the entire set of items, and I’ll ask you some follow-up questions after you answer item i.

Additional Probes:

How easy or difficult would it be for you identify the group of friends that each child in your class associates with?

(IF NOT MENTIONED) If the students a particular child associates with are not students of yours, how easy or difficult would it be to answer this series of questions?

Do any of these items seem to have the same meaning or seem repetitive to you? Which ones?

How did the available response options work for this question? In particular, for item e, are you able to rate how often these kids are often in trouble using the scale?

Now please turn to page 12 of your questionnaire and look at item C5.

Question C5

Please read this question out loud and say what you are thinking about or what you would do to answer this question. Also, for each item, please tell me what answer you would mark. [If needed, remind the respondent to answer thinking about the selected child.] We’ll talk about the items after you have read through all of them.

Additional Probes:

Would there be any children in your class for whom you’d have difficulty answering these items? Why?

Are any of these items difficult to understand or rate? If yes, which ones and why?

Looking at item b, what is an example of a situation in which you might observe a child interpreting what a peer is trying to do?

Looking at item c, in your opinion, what does it mean to generate a “good quality” solution to an interpersonal problem?

And looking at item d, what is an example of a child being aware of the effects of his or her behavior on others?

CLOSING QUESTIONS

After having reviewed the questionnaire briefly before this call, what do you think about the overall length of this questionnaire?

What is the most amount of time you would want to spend answering this type of questionnaire? In the national study, we expect that teachers will be given an honorarium of $20, plus an additional $7 for each of these subject questionnaires they complete. Knowing that, does that change your opinion about the amount of time you would be willing to spend completing this questionnaire?

Some teachers in the national study will be asked to complete the questions in part 1 for several children in their class. How would you feel about answering those questions for several children?

Those are all of the questions that I have for you. Thank you for participating. Your feedback is extremely valuable and will help us to develop a questionnaire that is easier for other teachers to complete. As a thank you, we will mail you a $25 check.

[TURN OFF THE RECORDING]

Did you have a chance to return the check information form?

If not, please fill out and return that form in the postage-paid envelope and we will mail a check out to you.

[If the form is lost say]: Please tell me how you would like your name to appear on the check and the best address to send the check to.

[Verify the spelling of the respondent’s name. Record information on the check information form and Return form]

Again, thank you for your time and input!

Case ID: T_

Date:

Time:

Interviewer:

Note taker:

Spring 2014

Math Teacher Questionnaire Protocol

Fourth-Grade Telephone-Based Math Teacher Cognitive Interview Protocol

First let me start by saying thank you for talking with me today. We would like your input on some questions that may be included in a questionnaire for fourth-grade mathematics teachers participating in the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11. The study is a longitudinal study of elementary school students conducted by the National Center for Education Statistics of the U.S. Department of Education.

1. Before we go any further, I just need to identify which questionnaire you received in the mail to review. Does your questionnaire cover refer to reading, math or science? (Circle one)

Reading 1 (Skip to 1c)

Math 2 (Continue)

Science 3 (Skip to 1c)

1a. And can I just confirm that you currently teach this subject to 4^th graders?

Yes 1 (SKIP TO #2)

No 2 (CONTINUE)

1b. Which of these subjects do you teach to 4^th graders? (READ RESPONSE OPTIONS)

Reading 1

Science 2

Other/Don’t teacher 4th graders 3 (Discontinue: Thank you for agreeing to participate, but at this time we are only conducting interviews with fourth-grade teachers who teach reading and language arts, math, or science.)

(IF CODES 1 or 3 in #1 OR CODES 1-2 IN 1b)

2. Do you have the questionnaire in front of you now?

Yes 1 (CONTINUE)

No, but can go retrieve it 2 (ASK R TO RETRIEVE IT)

No, cannot find it/did not receive it 3 (ASK R FOR EMAIL OR FAX AND SEND IT IMMEDIATELY. RESCHEDULE INTERVIEW IF NECESSARY)

3. So, with your permission I would like to audio-record this interview. Is that okay?

Yes 1 (TURN ON RECORDER, CONTINUE)

No 2 (SAY: “OK, WE WILL ONLY TAKE NOTES THEN” AND SKIP TEXT IMMEDIATELY BELOW)

[Respondent confirms]

Great!

I would like to get your feedback on some of the items in the questionnaire. Many of the questions ask you to think about a specific child by referencing “the child on the cover of the questionnaire.” So I’d like you to have a particular student from your 4^th grade math class in mind as we go through the questionnaire. You don’t need to tell me that student’s name, but please keep that particular child in mind as you answer the questions.

Do you have someone in mind? (IF YES: Great!)

(IF NO: Please think about the first student, alphabetically, in your first math class of the day.)

(NOTE TO COGNITIVE INTERVIEWER: RECORD ALL RESPONSES ON THE MATH TEACHER COGNITIVE INTERVIEW DATA COLLECTION FORM.)

Please turn to page 8 of your questionnaire and let’s first look at item B1.

Question B1

If respondent does not indicate a response, please probe: What would you mark on your questionnaire (if you were answering this question as part of the survey)?

Ask the following probes after each subitem a-i:

Were any of the examples in this particular item redundant or confusing?

Additional probes after all items have been asked about (ONLY ASK PROBES THAT HAVE NOT ALREADY BEEN COVERED IN DISCUSSION OF INDIVIDUAL ITEMS):

Are any of these the kinds of skills that are not taught to 4^th graders? Which ones?

How would you answer about a particular skill area if you teach some of the example activities more often than other example activities? For example, in item a, if you spend more time teaching about interpreting remainders than you spend on finding factor pairs.

How would you answer the question if you do not teach one of the example activities within a particular skill area at all? For example, in item f, if you explore the relative size of measurements, but do not ever discuss perimeter of geometric shapes?

Is it difficult to answer using the numeric categories that were provided? Would a different scale be better?

Are any of these too broad or too simplistic in how they have been written? Do we need to be more specific in the items to be able to find differences between classes or grades in what skills and concepts are being taught?

Now please turn to page 9 of your questionnaire, item B2.

Question B2

Please read this question out loud and then just focus on items i and m. Say what you are thinking about or what you would do to answer each item. Also, for each of these items, please tell me what answer you would mark. We will pause after each item so I can ask you some follow-up questions. (If needed: You don’t need to read the rest of the items.)

If respondent does not indicate a response, please probe: What would you mark on your questionnaire (if you were answering this question as part of the survey)?

Ask the following probes after each subitem:

(ASK ONLY FOR ITEM i:) What does it mean to “talk to the class” about math work?

(ASK ONLY FOR ITEM m:) What do you think this item is asking about? Does this include the interpretation of visual representations that the student has created him or herself? Does it include interpreting other students’ visual representations? What about interpreting those in a textbook?

Additional probes after all items have been asked about:

Are any of these activities used in other grades at your school, and NOT in 4^th grade? Which ones, and in what grades are they used?

Now please look at the supplemental pages that were sent with your questionnaire. Please read question 1 and tell me what you are thinking as you answer this question. Read all of the items before we talk about them.

Question 1

1. During this school year, how often has your school used the following ways to communicate with parents? MARK ONE RESPONSE ON EACH ROW.

	At least once a week	Once a week	Several times a month	Once a month	Less than once a month or never
a. Electronic communication to all parents, such as group emails, electronic newsletters, or other notices sent to all parents
b. Electronic communication with individual parents, such as individual emails or texts from the school
c. Non-electronic communication to all parents, such as letters, newsletters, or other notices sent to all parents
d. Non-electronic communication with individual parents, such as written notes, individual letters, or telephone calls from the school

Probes for Question 1:

Do you think your principal (or school administrator) would be able to answer these questions?

How easy or difficult is it to say how often each of these types of communication are happening at your school?

Would your principal be able to say how often communication is happening with individual parents?

Looking at item a, what does "electronic communication" mean to you? What kinds of communication are you thinking about? What are the possible ways a school could communicate electronically with parents?

(IF NOT ALREADY MENTIONED) If your school were to provide information in a format accessible to the general public, such as on a website, Facebook page, or in a Twitter feed, would you consider that to be the type of electronic communication being asked about in item a? Why or why not?

Looking at item c, what does "non-electronic communication" mean to you? Are there any other ways your school communicates non-electronically with all parents?

Now please read question 2.

Question 2

2. During this school year, has your school used an online tool or website for parents to log in to, to get information from the school? MARK ONE RESPONSE.

Yes (CONTINUE)

No (GO TO QUESTION 4)

NOTE TO COGNITIVE INTERVIEWER:

IF THE RESPONSE TO QUESTON 2 IS YES, CONTINUE WITH THE PROBES. IF IT IS NO, GO TO QUESTION 4.

Probes for Question 2:

What do you think this question is asking about?

Were you thinking about this tool or website when you answered Question 1?

Do the terms “online tool” and “website” capture this means of communication or is there a better term?

Now please read question 3.

Question 3

3. Has the information provided in the online tool or website included any of the following types of information? MARK ONE RESPONSE ON EACH ROW.

	YYes	NNo
a. School policies or practices
b. School events
c. Classroom-specific assignments, including homework
d. Child- or parent-specific information, such as progress reports between grading periods (with family privacy protected)

Probes for Question 3:

Are there other types of information that your school provides through an online tool or website to parents that are not on the list?

Would your principal be able to answer these questions, especially the last item? Is your principal involved with the website or online tool, or is it more a means for teachers to communicate with parents?

Next please look at question 4.

Question 4

I’d like you to read this question out loud and say what you are thinking about as you answer this question. Also, for each item, please tell me what answer you would mark. When it asks about the “child identified on the cover of this questionnaire,” please think about the student I asked you to keep in mind earlier. Please answer all of the items and then we’ll talk about them.

4. Please indicate how often each of these items applies to the child identified on the cover of this

questionnaire. MARK ONE RESPONSE ON EACH ROW.

	Doesn't apply ﴾seldom displays this behavior﴿	Sometimes applies ﴾occasionally displays this behavior﴿	Certainly applies ﴾often displays this behavior﴿
a. Likes to come to school
b. Dislikes school
c. Has fun at school
d. Likes being in school
e. Seems unhappy in school
f. Enjoys most classroom activities
g. Groans or complains about suggested activities

Probes for Question 4:

Do any of these items seem to have the same meaning or seem repetitive to you? Which ones?

(IF NOT MENTIONED) Are items a and d similar? Which one do you feel you could answer more accurately?

Are there any children in your class for whom you’d have difficulty answering these items? Why?

Now please look at question 5.

Question 5

5. Please evaluate the performance of the child identified on the cover of this questionnaire in the

following areas, using the categories below. MARK ONE RESPONSE ON EACH ROW.

	Very poor	Somewhat poor	Average	Good	Very good
a. Understands others’ feelings
b. Accurately interprets what a peer is trying to do
c. Generates good quality solutions to interpersonal solutions
d. Is aware of the effects of his/her behavior on others

Probes for Question 5:

Would there be any children in your class for whom you’d have difficulty answering these items? Why?

Are any of these items difficult to understand or rate? If yes, which ones and why?

Looking at item b, what is an example of a situation in which you might observe a child interpreting what a peer is trying to do?

Looking at item c, in your opinion, what does it mean to generate a “good quality” solution to an interpersonal problem?

And looking at item d, what is an example of a child being aware of the effects of his or her behavior on others?

CLOSING QUESTIONS

After having reviewed the questionnaire briefly before this call, what do you think about the overall length of this questionnaire?

Some teachers in the national study will be asked to complete the questions in part 1 for several children in their class. How would you feel about answering those questions for several children?

[TURN OFF THE RECORDING]

Did you have a chance to return the check information form?

If not, please fill out and return that form in the postage-paid envelope and we will mail a check out to you.

[If the form is lost say]: Please tell me how you would like your name to appear on the check and the best address to send the check to.

[Verify the spelling of the respondent’s name. Record information on the check information form and Return form]

Again, thank you for your time and input!

Case ID: T_

Date:

Time:

Interviewer:

Note taker:

Spring 2014

Science Teacher Questionnaire Protocol

Fourth-Grade Telephone-Based Science Teacher Cognitive Interview Protocol

First let me start by saying thank you for talking with me today. We would like your input on some questions that may be included in a questionnaire for fourth-grade science teachers participating in the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11. The study is a longitudinal study of elementary school students conducted by the National Center for Education Statistics of the U.S. Department of Education.

1. Before we go any further, I just need to identify which questionnaire you received in the mail to review. Does your questionnaire cover refer to reading, math or science? (Circle one)

Reading 1 (Skip to 1c)

Math 2 (Skip to 1c)

Science 3 (Continue)

1a. And can I just confirm that you currently teach this subject to 4^th graders?

Yes 1 (SKIP TO #2)

No 2 (CONTINUE)

1b. Which of these subjects do you teach to 4^th graders? (READ RESPONSE OPTIONS)

Reading 1

Math 2

Other/Don’t teacher 4th graders 4 (Discontinue: Thank you for agreeing to participate, but at this time we are only conducting interviews with fourth-grade teachers who teach reading and language arts, math, or science.)

(IF CODES 1 or 2 in #1 OR CODES 1-2 IN 1b)

2. Do you have the questionnaire in front of you now?

Yes 1 (CONTINUE)

No, but can go retrieve it 2 (ASK R TO RETRIEVE IT)

No, cannot find it/did not receive it 3 (ASK R FOR EMAIL OR FAX AND SEND IT IMMEDIATELY. RESCHEDULE INTERVIEW IF NECESSARY)

3. So, with your permission I would like to audio-record this interview. Is that okay?

Yes 1 (TURN ON RECORDER, CONTINUE)

No 2 (SAY: “OK, WE WILL ONLY TAKE NOTES THEN” AND SKIP TEXT BELOW)

[Respondent confirms]

Great!

I would like to get your feedback on some of the items in the questionnaire. Many of the questions ask you to think about a specific child by referencing “the child on the cover of the questionnaire.” So I’d like you to have a particular student from your 4^th grade science class in mind as we go through the questionnaire. You don’t need to tell me that student’s name, but please keep that particular child in mind as you answer the questions.

Do you have someone in mind? (IF YES: Great!)

(IF NO: Please think about the first student, alphabetically, in your first science class of the day.)

(NOTE TO COGNITIVE INTERVIEWER: RECORD ALL RESPONSES ON THE SCIENCE TEACHER COGNITIVE INTERVIEW DATA COLLECTION FORM.)

Please turn to page 7 of your questionnaire and look at item B1.

Question B1

If respondent does not indicate a response, please probe: What would you mark on your questionnaire (if you were answering this question as part of the survey)?

Probes:

Were any of the examples in this particular item redundant or confusing?

Additional probes after all items have been asked about (ONLY ASK PROBES THAT HAVE NOT ALREADY BEEN COVERED IN DISCUSSION OF INDIVIDUAL ITEMS):

Are any of these the kinds of skills that are not taught to 4^th graders? Which ones?

How would you answer about a particular skill area if you teach some of the example activities more often than other example activities? For example, in item c, if you teach about rocks and minerals more often than you teach about weather and erosion.

How would you answer the question if you do not teach one of the example activities within a particular skill area at all? For example, in item a, if you teach about electricity, but do not teach about magnetism?

Is it difficult to answer using the numeric categories that were provided? Would a different scale be better?

Now please turn to page 8 of your questionnaire, item B2.

Question B2

Please read this question out loud and then just focus on items g and L. Say what you are thinking about or what you would do to answer each item. Also, for each of these items, please tell me what answer you would mark. We will pause after each item so I can ask you some follow-up questions. (If needed: You don’t need to read the rest of the items.)

If respondent does not indicate a response, please probe: What would you mark on your questionnaire (if you were answering this question as part of the survey)?

Probes:

(ASK ONLY FOR ITEM G:) Looking at item g, is the word “children’s” helpful in this question? Does the meaning of the item change if we remove that word?

(ASK ONLY FOR ITEM G:) Is this activity grade-appropriate for 4^th graders, that is, is this something that fourth-graders would do in class?

(ASK ONLY FOR ITEM L:) Now looking at item L, is the phrase “about particular phenomena” helpful in this question? Does it change the meaning of the item if that phrase is removed?

Additional probes after all items have been asked about:

Are any of these activities used in other grades at your school, and NOT in 4^th grade? Which ones, and in what grades are they used?

Question 1

1. During this school year, how often has your school used the following ways to communicate with parents? MARK ONE RESPONSE ON EACH ROW.

	At least once a week	Once a week	Several times a month	Once a month	Less than once a month or never
a. Electronic communication to all parents, such as group emails, electronic newsletters, or other notices sent to all parents
b. Electronic communication with individual parents, such as individual emails or texts from the school
c. Non-electronic communication to all parents, such as letters, newsletters, or other notices sent to all parents
d. Non-electronic communication with individual parents, such as written notes, individual letters, or telephone calls from the school

Probes for Question 1:

Do you think your principal (or school administrator) would be able to answer these questions?

How easy or difficult is it to say how often each of these types of communication are happening at your school?

Would your principal be able to say how often communication is happening with individual parents?

Looking at item c, what does "non-electronic communication" mean to you? Are there any other ways your school communicates non-electronically with all parents?

Now please read question 2.

Question 2

2. During this school year, has your school used an online tool or website for parents to log in to, to get information from the school? MARK ONE RESPONSE.

Yes (CONTINUE)

No (GO TO QUESTION 4)

NOTE TO COGNITIVE INTERVIEWER:

IF THE RESPONSE TO QUESTON 2 IS YES, CONTINUE WITH THE PROBES. IF IT IS NO, GO TO QUESTION 4.

Probes for Question 2:

What do you think this question is asking about?

Were you thinking about this tool or website when you answered Question 1?

Do the terms “online tool” and “website” capture this means of communication or is there a better term

Now please read question 3.

Question 3

3. Has the information provided in the online tool or website included any of the following types of information? MARK ONE RESPONSE ON EACH ROW.

	YYes	NNo
a. School policies or practices
b. School events
c. Classroom-specific assignments, including homework
d. Child- or parent-specific information, such as progress reports between grading periods (with family privacy protected)

Probes for Question 3:

Are there other types of information that your school provides through the online tool or website to parents that are not on the list?

Next please look at question 4.

Question 4

4. Please indicate how often each of these items applies to the child identified on the cover of this

questionnaire. MARK ONE RESPONSE ON EACH ROW.

	Doesn't apply ﴾seldom displays this behavior﴿	Sometimes applies ﴾occasionally displays this behavior﴿	Certainly applies ﴾often displays this behavior﴿
a. Likes to come to school
b. Dislikes school
c. Has fun at school
d. Likes being in school
e. Seems unhappy in school
f. Enjoys most classroom activities
g. Groans or complains about suggested activities

Probes for Question 4:

Do any of these items seem to have the same meaning or seem repetitive to you? Which ones?

(IF NOT MENTIONED) Are items a and d similar? Which one do you feel you could answer more accurately?

Are there any children in your class for whom you’d have difficulty answering these items? Why?

Now please look at question 5.

5. For the items below, please think about the friends with whom the child identified on the cover of

this questionnaire has associated the most during the past month or two. Please indicate how

often you think each of these items applies to this group of children. MARK ONE RESPONSE ON

EACH ROW.

	Almost always untrue	Usually untrue	Sometimes true, sometimes untrue	Usually true	Almost always true
a. This is a good group of kids
b. I worry when this child is with this group of kids
c. Some of these kids are a bad influence on this child
d. These kids must be closely supervised by an adult
e. These kids are often in trouble
f. These kids are excellent students
g. These kids are hard workers
h. This child has a fun time with this group of children
i. Most of these kids are kind to other children

Probes for Question 5:

How easy or difficult would it be for you identify the group of friends that each child in your class associates with?

(IF NOT MENTIONED) If the students a particular child associates with are not students of yours, how easy or difficult would it be to answer this series of questions?

Do any of these items seem to have the same meaning or seem repetitive to you? Which ones?

How did the available response options work for this question? In particular, for item e, are you able to rate how often these kids are often in trouble using the scale?

CLOSING QUESTIONS

After having reviewed the questionnaire briefly before this call, what do you think about the overall length of this questionnaire?

Some teachers in the national study will be asked to complete the questions in part 1 for several children in their class. How would you feel about answering those questions for several children?

[TURN OFF THE RECORDING]

Did you have a chance to return the check information form?

If not, please fill out and return that form in the postage-paid envelope and we will mail a check out to you.

[If the form is lost say]: Please tell me how you would like your name to appear on the check and the best address to send the check to.

[Verify the spelling of the respondent’s name. Record information on the check information form and Return form]

Again, thank you for your time and input!

Case ID: T_

Date:

Time:

Interviewer:

Note taker:

Spring 2014

Teacher Timing Protocol

Fourth-Grade Telephone-Based Teacher Questionnaire Timing Protocol

First let me start by saying thank you for taking the time to fill out the questionnaires and talking with me today. As you know, the purpose of today’s call is to collect your timings from filling out the fourth-grade teacher questionnaires for in the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 and to get some general feedback on your experiences completing the questionnaires.

1. Did you have a chance to complete the questionnaires and fill out your Questionnaire Timings Form?

Yes 1 (CONTINUE)

No 2 (SET UP A TIME TO CALL BACK AFTER TEACHER HAS HAD A CHANCE TO FILL OUT QUESTIONNAIRES AND COMPLETE TIMINGS FORM)

No, did not receive them 3 (ASK R FOR EMAIL OR FAX AND SEND IT IMMEDIATELY. RESCHEDULE INTERVIEW IF NECESSARY)

2. Do you have the questionnaires and timing form in front of you now?

Yes 1 (CONTINUE)

No, but can go retrieve them 2 (ASK R TO RETRIEVE IT)

There are a few things you should know before we start. First, the interview today will take approximately 10 minutes. I’ll be taking notes, so I may pause momentarily or you may hear me writing (or typing). (IF APPROPRIATE: [NAME] is also here with me to take more detailed notes.) Even though I am taking notes, in a detailed interview like this, we often want to go back and verify our notes.

3. So, with your permission I would like to audio-record this interview. Is that okay?

Yes 1 (TURN ON RECORDER, CONTINUE)

No 2 (SAY: “OK, WE WILL ONLY TAKE NOTES THEN” AND SKIP TEXT IMMEDIATELY BELOW)

[Respondent confirms]

Let’s begin by having you look at your Questionnaire Timings Form and tell me how many minutes it took you to answer each section of the questionnaires. INTERVIEWER: RECORD TIMINGS ON THE RELEVANT QUESTIONNAIRE TIMINGS FORM.

Probes:

Thinking just about the Teacher Background Questionnaire (with the green color cover), were there any questions that were difficult for you to answer? (Which ones?)

(If respondent does not indicate a response:) You can refer to the survey to help remind you of what was asked.

Did any questions make you uncomfortable or did you feel any of them touched on a sensitive topic or issue? (Which ones?)

What do you think about the overall length of this questionnaire? Does it seem reasonable to ask a teacher to answer a survey of this length?

Thinking just about the Subject Area Questionnaire (Reading, Math or Science), what do you think about the overall length of this questionnaire? Does it seem reasonable to ask a teacher to answer a survey of this length?

Before we wrap up, do you have any other feedback you’d like to provide on these questionnaires?

Those are all of the questions that I have for you. Thank you for participating. Your feedback is very valuable and will help us to develop a questionnaire that is easier for other teachers to complete. As a thank you, we will mail you a $25 check.

[TURN OFF THE RECORDING]

Did you have a chance to return the check information form?

If not, please fill out and return that form in the postage paid envelope and we will mail a check out to you.

[If the form is lost say]: Please tell me how you would like your name to appear on the check and the best address to send the check to.

[Verify the spelling of the respondent’s name. Record information on the check information form and Return form]

Case ID: T_

Date:

Time:

Interviewer:

Note taker:

Spring 2014

Parent Questionnaire Protocol

Fourth-Grade Telephone-Based Parent Cognitive Interview Protocol

First let me start by saying thank you for talking with me today. We would like your input on some questions that may be included in an interview for parents of fourth-graders in the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11. The study is a longitudinal study of elementary school students conducted by the National Center for Education Statistics of the U.S. Department of Education.

1. Before we go any further, I just need to verify that you are a parent or guardian of a child in the fourth grade. Is that correct? (Circle one)

Yes, parent or guardian of fourth-grader....................... 1

No.................................................................................. 2 (STOP: Thank you for agreeing to participate, but at this time we are only conducting interviews with parents of fourth-graders.)

There are a few things you should know before we start. First, the interview today will take approximately 1 hour. I’ll be taking notes, so I may pause momentarily or you may hear me writing (or typing). In a detailed interview like this, I often like to go back and verify my notes.

2. So, with your permission I would like to audio-record this interview. Is that okay?

Yes 1 (TURN ON RECORDER, CONTINUE)

No 2 (SAY: “OK, WE WILL ONLY TAKE NOTES THEN” AND SKIP TEXT IMMEDIATELY BELOW)

[Respondent confirms]

Great! Let's get started.

What I would like to do is to get your feedback on some of the questions in the interview. I will ask you to read the questions out loud and talk about how you would go about answering them. In some cases, that may mean just talking about the mental steps you would take to arrive at the answer. After talking about the mental steps you would take to answer a question, if you are able to provide an answer, I would like to know the response you would give if you were participating in the interview. After each interview item that we talk about, I will ask some follow-up questions to make sure I understand how you are interpreting the question.

Here is the first question. Again, be sure to think out loud and tell me your thoughts as you come up with your answers to each question. The questions will focus on your fourth-grader.

RECORD ALL RESPONSES ON THE PARENT COGNITIVE INTERVIEW DATA COLLECTION FORM.

1. PIQ496d How often do you {or your spouse/partner} use a computer or any other electronic device, whether at home or at work or in another setting to find out how all students in {CHILD}’s school are doing as a group in areas such as standardized test scores or attendance rates?

[PROBE: Would you say never, once or twice a year, several times a year, at least once a month, or weekly?]

1 NEVER

2 ONCE OR TWICE A YEAR

3 SEVERAL TIMES A YEAR

4 AT LEAST ONCE A MONTH

5 WEEKLY

REFUSED

DON’T KNOW

IF RESPONDENT SAYS “NEVER”, “REFUSED” OR “DON’T KNOW”, SKIP PROBES AND GO TO #2.

PROBES FOR Q1:

What kinds of information does your school give to you?

Do you get this information through email, a website, through the regular mail, at an in-person meeting, or some other way?

2.PIQ585b How often would you say that {CHILD} seems to dread going to school? Would you say almost never, rarely, sometimes, a lot, or almost always?

[PROBE: Would you say almost never, rarely, sometimes, a lot, or almost always?]

1 ALMOST NEVER

2 RARELY

3 SOMETIMES

4 A LOT

5 ALMOST ALWAYS

REFUSED

DON’T KNOW

PROBES FOR Q2:

What does the word “dread” mean to you?

What do you think about the response choices? How well did they work for this question? (IF NECESSARY, READ:) Never, rarely, sometimes, a lot, almost always.

3. HEQ226 Are there family rules for {CHILD} about any of the following…

What {CHILD} looks at on the Internet?

1 YES

2 NO

REFUSED

DON’T KNOW

PROBES FOR Q3:

What do you think is meant by “family rules”?

If you use parental control settings on the Internet, were you thinking about that when you answered this question?

Here is some information that we may give people to explain what we mean by the Internet in this question. We would say: “Include everything the child may look at on the Internet, such as websites, videos, movies, television shows, online games, and picture or social media sites such as Instagram, Vine, Snapchat, Ask.fm, and Facebook.” Does hearing this statement make you change your answer?

Are you unfamiliar with any of these examples?

Is there anything important that we left off of this list of examples?

4. [HEQ227 Are there family rules for {CHILD} about any of the following…]

How many hours {CHILD} may spend on the Internet?

1 YES

2 NO

PROBES FOR Q4:

IF THERE ARE FAMILY RULES ABOUT HOURS: Are the rules for how many hours the child may spend on the Internet different during the week than on the weekend? If so, how would you answer this question?

5. HEQ227b Next, I am going to read a list of statements about the children whom {CHILD} has spent the most time with in the past month or two. Please tell me how true or untrue each statement is about that group of children.

These kids are often in trouble.

[PROBE: Would you say this is almost always untrue; usually untrue; sometimes true and sometimes untrue; usually true; or almost always true?]

1 ALMOST ALWAYS UNTRUE

2 USUALLY UNTRUE

3 SOMETIMES TRUE AND SOMETIMES UNTRUE

4 USUALLY TRUE

5 ALMOST ALWAYS TRUE

REFUSED

DON'T KNOW

PROBES FOR Q5:

When you were thinking about the children your child spent the most time with in the past month or two for this answer, was it easy or difficult to decide which children to think about?

Were you thinking only of your child’s friends? Did you include children your child hangs around with at school but who you don’t know very well? Would you be able to answer this question for children your child hangs around with at school but who you don’t know very well?

How did you answer if this was true for some of the kids but not all of them? For example, what if some kids in this group are always in trouble, but others only rarely?

6. HEQ227f [Please think about the children whom {CHILD} has spent the most time with in the past month or two.]

Some of these kids are a bad influence on {CHILD}.

[PROBE: Would you say this is almost always untrue; usually untrue; sometimes true and sometimes untrue; usually true; or almost always true?]

1 ALMOST ALWAYS UNTRUE

2 USUALLY UNTRUE

3 SOMETIMES TRUE AND SOMETIMES UNTRUE

4 USUALLY TRUE

5 ALMOST ALWAYS TRUE

REFUSED

DON'T KNOW

PROBE FOR Q6:

How would you answer if some kids in this group are sometimes a bad influence?

7. HEQ228 How many close friends does {CHILD} have?

PROBE: Your best guess is fine.

_______

PROBES FOR Q7:

What do you think is meant by “close friends”?

How easy or difficult is it for you to come up with an accurate answer to this question? That is, how easy or difficult it is to tell me a specific number?

8. HEQ229 What kind of influence is {CHILD}’s best friend? Would you say always a good influence, usually a good influence, neither a good nor a bad influence, usually a bad influence, or always a bad influence?

1 ALWAYS A GOOD INFLUENCE

2 USUALLY A GOOD INFLUENCE

3 NEITHER A GOOD NOR A BAD INFLUENCE

4 USUALLY A BAD INFLUENCE

5 ALWAYS A BAD INFLUENCE

6 DOES NOT APPLY/CHILD DOES NOT HAVE A BEST FRIEND

REFUSED

DON’T KNOW

PROBES FOR Q8:

If I had asked about {CHILD}’s closest friend, would you think of the same friend as {his/her} best friend? Is one wording easier to understand than the other?

9. HEQ229 What kind of influence are {CHILD}’s other friends?

Would you say always a good influence, usually a good influence, neither a good nor a bad influence, usually a bad influence, or always a bad influence?

1 ALWAYS A GOOD INFLUENCE

2 USUALLY A GOOD INFLUENCE

3 NEITHER A GOOD NOR A BAD INFLUENCE

4 USUALLY A BAD INFLUENCE

5 ALWAYS A BAD INFLUENCE

6 DOES NOT APPLY/CHILD DOES NOT HAVE OTHER FRIENDS

REFUSED

DON’T KNOW

PROBES FOR Q9:

Did you include your child’s best friend in your answer?

How easy or difficult is it to rate the influence of all of your child’s other friends?

IF THE PARENT REPORTED THE CHILD HAS AT LEAST ONE CLOSE FRIEND IN QUESTION 7: When you answered this question, did you think about all of your child’s friends or just the closest friends? Would this question be easier to answer if it just asked about your child’s close friends? Why or why not?

The next questions are about all of {CHILD}’s friends, not just close friends.

10. HEQ231 How often does {CHILD} hang out with friends who get in trouble? Would you say never, almost never, sometimes, often, or always?

1 NEVER

2 ALMOST NEVER

3 SOMETIMES

4 OFTEN

5 ALWAYS

REFUSED

DON’T KNOW

PROBES FOR Q10:

Were you thinking about time spent with friends at school, outside of school, or both?

Are you able to answer this question pretty well for all of your child’s friends, or does your child have friends you don’t know well enough to be able to accurately say how often they get in trouble?

11. HEQ232 How often does {CHILD} hang out with friends who fight a lot? Would you say never, almost never, sometimes, often, or always?

1 NEVER

2 ALMOST NEVER

3 SOMETIMES

4 OFTEN

5 ALWAYS

REFUSED

DON’T KNOW

PROBES FOR Q11:

What is meant by “fight”? Were you thinking of physical fighting, verbal fighting, or something else?

What is meant by “fight a lot”? How often is “a lot”?

12. HEQ233 How often does {CHILD} hang out with friends who take things that don’t belong to them? Would you say never, almost never, sometimes, often, or always?

1 NEVER

2 ALMOST NEVER

3 SOMETIMES

4 OFTEN

5 ALWAYS

REFUSED

DON’T KNOW

PROBES FOR Q12:

13. HEQ300 Outside of school hours in the past year, has {CHILD} participated in academic activities, like science, computers, math lab, or taking a class to learn a language other than English?

1 YES (GO TO QUESTION 14)

2 NO (GO TO QUESTION 15)

REFUSED

DON’T KNOW

PROBES FOR Q13:

How easy or difficult was this question to answer?

Were you thinking just about activities during the school year, or also about activities during the summer?

IF THE CHILD WAS IN ACADEMIC ACTIVITIES: What activities was {CHILD} in?

14. HEQ301 Did {he/she} specifically learn about math, science, or technology in any of those academic activities?

1 YES

2 NO

REFUSED

DON’T KNOW

PROBES FOR Q14:

Is this question asking the same thing or something different than the previous question? Why do you think that?

Here is the last question. 15. How often do you argue with {CHILD}? Would you say never, almost never, sometimes, often, or always?
1 NEVER 2 ALMOST NEVER
3 SOMETIMES
4 OFTEN
5 ALWAYS REFUSED
DON'T KNOW PROBES FOR Q15: What types of arguments were you thinking about when you answered this question? Did you only include larger or more significant arguments, or did you include all arguments, even minor ones like a child arguing about what to eat for dinner or what to wear to school?

CLOSING QUESTIONS

Now that we are finished with the regular questions, I just have some final questions. Did any of the questions make you uncomfortable or did you feel any of them touched on a sensitive topic or issue for you to answer about your child? (Which ones?)

Those are all of the questions that I have for you. Thank you for participating. Your feedback will help us to develop an interview that is easier for other parents to complete. As a thank you, we will mail you a $25 check.

[TURN OFF THE RECORDING]

Did you have a chance to return the check information form?

If not, please fill out and return that form in the postage paid envelope and we will mail a check out to you.

[If the form is lost say]: Please tell me how you would like your name to appear on the check and the best address to send the check to.

[Verify the spelling of the respondent’s name. Record information on the check information form and Return form]

Thank you again!

Case ID: T_

Date:

Time:

Interviewer:

Note taker:

Spring 2014

Parent Timing Questionnaire Protocol

Fourth-Grade Telephone-Based Parent Interview Timing Protocol

First let me start by saying thank you for talking with me today. We would like your input on an interview that may be used for parents of fourth-graders in the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11. The study is a longitudinal study of elementary school students conducted by the National Center for Education Statistics of the U.S. Department of Education.

1. Before we go any further, I just need to verify that you are a parent or guardian of a child in the fourth grade. Is that correct? (Circle one)

Yes, parent or guardian of fourth-grader....................... 1

2. So, with your permission I would like to audio-record this interview. Is that okay?

Yes 1 (TURN ON RECORDER, CONTINUE)

No 2 (SAY: “OK, WE WILL ONLY TAKE NOTES THEN” AND SKIP TEXT IMMEDIATELY BELOW)

[Respondent confirms]

Great! Now I’m going to ask you the questions in the parent interview. After you have answered the questions about your fourth-grader, I’ll ask for your overall feedback on the interview.

(INTERVIEWER: RECORD TIME AT BEGINNING OF PARENT INTERVIEW, AND AT END OF EACH SECTION OF THE INTERVIEW)

(AFTER FULL INTERVIEW HAS BEEN ADMINISTERED)

Now that we’re finished with the interview, I just have a few final questions for you to get your feedback on the interview experience.

Were there any questions that were difficult for you to answer? (Which ones?)

Did any questions make you uncomfortable or did you feel any of them touched on a sensitive topic or issue? (Which ones?)

What do you think about the overall length of this interview?

Those are all of the questions that I have for you. Thank you for participating. Your input will help us to develop an interview that is easier for other parents to complete. As a thank you, we will mail you a $25 check.

[TURN OFF THE RECORDING]

Please tell me how you would like your name to appear on the check and the best address to send the check to.

[Verify the spelling of the respondent’s name. Record information on the check information form and Return form]

Thanks again!

INTERVIEWER: TRANSFER TIMINGS TO THE PARENT INTERVIEW TIMINGS FORM

C-1

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Appendix C:
File Modified	0000-00-00
File Created	2021-01-27