High School and Beyond 2020 (HS&B:20) Base-Year Field Test Cognitive Testing Round 1

NCES Cognitive, Pilot, and Field Test Studies System

Attachments V-IX HS&B 2020 Cog Labs Round 1

High School and Beyond 2020 (HS&B:20) Base-Year Field Test Cognitive Testing Round 1

OMB: 1850-0803

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1850-0803 can be found here:

Document [docx]

Download: docx | pdf

National Center for Education Statistics (NCES)

Attachments V-IX

Survey Questions with Cognitive Interview Probes

High School and Beyond 2020 (HS&B:20)
Cognitive Testing Round 1

OMB# 1850-0803 v.240

September 2018

Attachment V: Cognitive Interview Protocol: 9^th Grade Students Survey 3

Attachment VI: Cognitive Interview Protocol: Parent Survey 25

Attachment VII: Cognitive Interview Protocol: Teacher Survey 46

Attachment VIII: Cognitive Interview Protocol: School Administrators Survey 69

Attachment IX: Cognitive Interview Protocol: School Counselors Survey 91

Attachment V:
Cognitive Interview Protocol: 9^th Grade Students Survey

For the next series of questions, please answer for the parent who brought you here today.

Question 1A.1

During the past week did your parent work at a job for pay?

 Yes

 No

Source: MGLS:2017 C020

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what this question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

ITEM SPECIFIC PROBES

Please explain what “past week” means to you?

Do not read answer options – select which applies:

 Last Sunday through Saturday

 The past 7 days

 Other: __________________________________________________________

Please explain what “job for pay” means to you. Please provide some examples.

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 1A.1 Handout

Question 1A.1

During the past week did your parent work at a job for pay?

 Yes

 No

Source: MGLS:2017 C020

INTERVIEWER NOTE: Respondent should respond to Q 1A.2 regardless of their response to Q 1A.1.

Question 1A.2

During the past week, was your parent on leave or vacation from a job?

 Yes

 No

Source: MGLS:2017 H001

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what this question is asking? Please try to explain without repeating the wording of the question.

_________________________________________________________________

__________________________________________________________________

_________________________________________________________________

__________________________________________________________________

ITEM SPECIFIC PROBES

Please explain what “on leave” means to you. Please provide some examples.

___________________________________________________________________

Please explain what “on vacation” means to you. Please provide some examples.

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

The next question is another way of asking about your parent.

Question 1B

During the past week, did your parent work for pay or income? (If your parent held a job but was not working because of temporary illness, vacation, strike, or jury duty answer "yes.")

 Yes

 No

Source: HSLS:09 Parent Survey C05

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what this question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

Did you find any words or parts of this question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

__________________________________________________________________

How did you come up with your answer?

__________________________________________________________________

ITEM SPECIFIC PROBES

In your own words, what does “income” mean to you?

Do you think the term “income” is helpful to include in this question?

___________________________________________________________________

Were the examples provided in the instructions helpful to you? (point to parentheses)

Why or why not?

___________________________________________________________________

Looking back at Question 1A.1, did you answer this question differently? Why or why not?

 Same

 Different

___________________________________________________________________

Which version (1A.1 or 1B) do you prefer? Why?

 1A.1

 1B

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 2

How far in school did your parent go? (Select highest level of education.)

a. Less than high school completion
b. Completed a high school diploma or equivalent (for example, GED, HiSET, TASC)
c. Completed a certificate or diploma from a school that provides occupational training
d. Completed an associate’s degree
e. Completed a bachelor’s degree
f. Completed a master’s degree
g. Completed a Ph.D., M.D., law degree, or other high level professional degree
h. Don’t Know

Source: ELS:02 83/HSLS:09 P1 C02

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what this question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

Did you find any words or parts of this question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Looking at sub-item c, in your own words, what does “completed a certificate or diploma” mean to you? Please try to explain without repeating the wording of the item.

___________________________________________________________________

Looking at sub-item c, in your own words, what does “school that provides occupational training” mean to you? Please try to explain without repeating the wording of the item.

Which of these three terms do you think would be easiest to understand:

___________________________________________________________________

 “school that provides occupational training”

 “technical institute”

 “trade school”

IF SELECTED “Don’t Know”: You selected “Don’t Know” as your answer, can you please explain why you chose this answer?

___________________________________________________________________

What are some differences, if any, between sub-items b, c, and d?

___________________________________________________________________

What are some differences, if any, between sub-items f and g?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 3

How many people live in your household, including yourself and your parent(s)?

______ people

Source: Modified from HSLS:09 (Parent) A08

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what this question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

ITEM SPECIFIC PROBES

Please explain what the phrase “live in the same household” means to you. Can you provide some examples?

___________________________________________________________________

Do you live in more than one household?

IF YES: In answering this question, which household did you choose? Why?

 Yes

 No

___________________________________________________________________

How did you come up with your answer? How did you decide who to include and who not to include?

___________________________________________________________________

Do you have any relatives or friends who sometimes live in your household?

IF YES: Did you include them in your answer?

 Yes

 No

 Yes

 No

Record observations and comments here

___________________________________________________________________

Question 4

How often do you…

Select one answer for each row.	Never	Rarely	Sometimes	Often
Go to class without your homework done?				
Go to class without a way to take notes?				
Go to class without books or required devices?				
Go to class late?				

Source: HSLS:09 E02

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what this question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

How sure are you about your answers to this item?

IF UNSURE OR VERY UNSURE: What do you need to be more sure of your answer?

 Very Unsure

 Unsure

 Sure

 Very Sure

___________________________________________________________________

ITEM SPECIFIC PROBES

Sub-item b

In your own words, what does “a way to take notes” mean to you? Please provide examples.

___________________________________________________________________

Sub-item c

Can you explain to me what “required devices” means to you? Please provide examples.

___________________________________________________________________

Do you think it would be helpful to include a “Does not apply” answer options? Why or why not?

___________________________________________________________________

Provide alternate version.

Looking at the wording on sub-items a and c, which version (original or alternate) do you prefer? Why?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 4 ALTERNATE

How often do you…

Select one answer for each row.	Never	Rarely	Sometimes	Often
Come to class without completing prior assignments or homework?				
Go to class without a way to take notes?				
Come to class without class materials (such as pencils, paper, tablet, books, or calculator)?				
Go to class late?				

Source: HSLS:09 E02

Question 5

Do you plan to enroll in an Advanced Placement (AP) course?

Yes
No
You haven't decided yet
You don't know what this is

Source: HSLS:09 F03

Question/Prompt	Response
GENERIC PROBES
Ask this probe AFTER the respondent has answered the question. Can you tell me, in your own words, what this question is asking? Please try to explain without repeating the wording of the question.	___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________
ITEM SPECIFIC PROBES
In your own words, what does “Advanced Placement (AP) course” mean to you? Please provide examples.	___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________

Question 6

How often do you have a problem with your internet at home when trying to do your homework or school assignments?

Do not have internet in the home

 Never

 Rarely

 Sometimes

 Often

 Very often

 Always

Source: MGLS:17 B.030

Question/Prompt

Response

GENERIC PROBE

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what this question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

How did you come up with your answer? (Note: Probe further on what “at home” means to kids with multiple households).

___________________________________________________________________

ITEM SPECIFIC PROBES

In your own words, what does “internet” mean to you? Please provide examples.

___________________________________________________________________

How do you or others access the internet? What devices do you use? What devices did you think about when answering?

___________________________________________________________________

Can you explain to me what “problem with your internet” means to you? What are some examples of problems with the internet? Do you think of problems with regards to doing homework or in general?

___________________________________________________________________

Record your observations and comments here

___________________________________________________________________

______________________________________________________________________________________________________________________________________

Question 7

How often do you…

Select one answer for each row.	Never	Less than once a week	A few times a week	About once a day	Many times a day
Play video games, computer games, or mobile games?	1	2	3	4	5
Make your own digital art or music (such as painting, graphics, videos, photos, music)?	1	2	3	4	5
Write computer programs (code) or develop apps?	1	2	3	4	5

Source: MGLS:17

Question/Prompt

Response

GENERIC PROBE

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what this question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

Did you find any words or parts of this question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

__________________________________________________________________

ITEM SPECIFIC PROBES

Sub-item a

In your own words, what does “video games, computer games, or mobile games” mean to you? Please provide examples.

___________________________________________________________________

Sub-item c

Can you explain to me what “write computer programs (code) or develop apps” means to you? Please provide examples.

___________________________________________________________________

Record your observations and comments here

___________________________________________________________________

______________________________________________________________________________________________________________________________________

Card Sort Activity

Never	Rarely	Sometimes
Often	Very Often	Always

Card Sort Activity

Instructions

Please look at the six cards in front of you. After you read the cards, try to order them from lowest to highest.

If you think that two or more cards mean the same thing, you can put the cards on top of each other.

[Place cards (printed out individually) on a pile in front of the student in random order.]

Response

Denote the following:

The order in which the student organized the cards from lowest to highest (1 to 6)

Any cards that the student places on top of each other using the same order number.

	Never		Rarely		Sometimes
	Often		Very Often		Always

Question/Prompt	Response
RESPONSE OPTION PROBES
Do you think all cards mean something different, or are there cards that mean the same to you?	___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________
Probe about cards that are placed on top of each other/mean the same. IF CARDS PLACED ON TOP/ MEAN THE SAME: Why did you put these cards on top of each other? How could we make these cards more different from each other? How would you describe the difference between [card] and [card]?	___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________

Record observations and comments here	___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________

General Debriefing and Thank You

Before we finish, I’d like to hear [any/other] thoughts you have about what you’ve been doing.

______________________________________________________________________________

Is there anything else you would like to tell me about working on the survey questions?

______________________________________________________________________________

Is there anything that you think could make [this/these] survey question(s) clearer?

______________________________________________________________________________

Thank participant for his/her time and provide gift card, as appropriate.

Attachment VI:
Cognitive Interview Protocol: Parent Survey

Question 1A.1

During the past week did you work at a job for pay?

 Yes

 No

Source: MGLS:2017 C020

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what the question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

ITEM SPECIFIC PROBES

Please explain what “past week” means to you?

Do not read answer options – select which applies:

 Last Sunday through Saturday

 The past 7 days

 Other: __________________________________________________________

Please explain what “job for pay” means to you. Please provide some examples.

___________________________________________________________________

If you were on vacation or leave in the past week, how would you answer this question?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

INTERVIEWER NOTE: Respondent should respond to Q 1A.2 regardless of their response to Q 1A.1.

Question 1A.2

During the past week, were you on leave or vacation from a job?

 Yes

 No

Source: MGLS:17 H001

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what the question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

ITEM SPECIFIC PROBES

Please explain what “on leave” means to you. Please provide some examples.

___________________________________________________________________

Please explain what “on vacation” means to you. Please provide some examples.

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

INTERVIEWER NOTE: Ask only if parent answered ‘no’ to the previous questions (Q1A.1 and Q1A.2)

Question 2

What were you doing most of last week? Would you say…

 a. Keeping house or caring for children

 b. Going to school

 c. Retired

 d. Unable to work

 e. Something else?

Source: MGLS:17 H020

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what the question is asking? Please try to explain without repeating the wording of the question.

_________________________________________________________________

__________________________________________________________________

_________________________________________________________________

__________________________________________________________________

Did you find any words or parts of this question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?


 Yes	 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Can you explain to me what part a “Keeping house or caring for children” means to you? Is there another word you would use instead?

_________________________________________________________________

IF SELECTED “Something Else”: You selected “Something else” as your answer to the question, can you please describe what else?

_________________________________________________________________

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

_________________________________________________________________

Record observations and comments here

___________________________________________________________________

The next questions are about your job. If you were not working during the past week, tell us about the job you had most recently.

Question 3.1

Please describe your job.

First, what is the job called? (For example: 4th grade teacher, entry-level plumber)

JOB_____________________________________________________________________

Source: NELS:88 4B

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Did you find any words or part of this question that were confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Please explain what “job” means to you.

___________________________________________________________________

Were the examples provided in the instructions helpful to you? (point to parentheses)

Why or why not?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 3.2

What do you actually do in that job? What are some of your main duties?

JOB DUTIES________________________________________________________________________

(For example, instruct and evaluate students and create lesson plans, assemble and install pipe sections and review building plans for work details)

Source: NELS:88 4C

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Did you find any words or part of this question that were confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

__________________________________________________________________

ITEM SPECIFIC PROBES

Please explain what the word “duties” means to you.

___________________________________________________________________

Were the examples provided in the instructions helpful to you? (point to parentheses)

Why or why not?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 4

What is the highest level of education you have completed?

a. Less than high school completion
b. Completed a high school diploma or equivalent (for example, GED, HiSET, TASC)
c. Completed a certificate or diploma from a school that provides occupational training
d. Completed an associate’s degree
e. Completed a bachelor’s degree
f. Completed a master’s degree
g. Completed a Ph.D., M.D., law degree, or other high level professional degree

Source: ELS:02 83/HSLS:09 P1 C02

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what the question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

Did you find any words or parts of this question that were confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Looking at part c, in your own words, what does “completed a certificate or diploma” mean to you? Please try to explain without repeating the wording of the item.

___________________________________________________________________

Looking at part c, in your own words, what does “school that provides occupational training” mean to you? Please try to explain without repeating the wording of the item.

Which of these three terms do you think would be easiest to understand:

___________________________________________________________________

 “school that provides occupational training”

 “technical institute”

 “trade school”

What are some differences, if any, between sub-items b, c, and d?

___________________________________________________________________

What are some differences, if any, between sub-items f and g?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 5

Since your 9^th grader started kindergarten, in how many different residences (houses, apartments, etc.) has [he/she] lived for four months or more?

Source: MGLS:17 I010

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what the question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

ITEM SPECIFIC PROBES

Were the examples provided in the question helpful to you? (point to parentheses)

Why or why not?

___________________________________________________________________

The question asks about residences lived in for “four months or more”. Does this make it easier to answer, harder to answer, or not make a difference? Why?

 Easier  No difference  Harder

___________________________________________________________________

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy  Difficult  Very Difficult

_________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 6

How many older siblings does your 9th grader have? Please include all older brothers and sisters including adopted siblings, half-siblings, step-siblings, and foster siblings.

Source: HSLS:09 A12

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what the question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

How did you come up with your answer?

___________________________________________________________________

__________________________________________________________________

ITEM SPECIFIC PROBES

Does your child have any siblings who are not currently in the home or live somewhere else?

IF YES: Did you include them in your answer?

 Yes

 No

___________________________________________________________________

__________________________________________________________________

Does your child have any step or half-siblings who were born in the same year?

IF YES: Do you consider them the same age or is one considered older (i.e., by birth date)?

 Yes

 No

___________________________________________________________________

______________________________________________________________________________________________________________________________________

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy  Difficult  Very Difficult

_________________________________________________________________

________________________________________________________________

_________________________________________________________________

Record your observations and comments here

___________________________________________________________________

______________________________________________________________________________________________________________________________________

Question 7 (Students with older sibling(s))

How many of your 9th grader’s siblings (including adopted, half-, step-, and foster brothers and sisters) have continued their education after high school?

___sibling(s)

Source: NELS:88 5B

Question/Prompt

Response

GENERIC PROBES

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what the question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

ITEM SPECIFIC PROBES

In your own words, what does the phrase “continued education after high school” mean to you?

___________________________________________________________________

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy  Difficult  Very Difficult

_________________________________________________________________

________________________________________________________________

_________________________________________________________________

Record your observations and comments here

___________________________________________________________________

______________________________________________________________________________________________________________________________________

Question 8

How many days of school has your 9th grader missed in the past 30 days due to health issues?

_____days

Source: New

Question/Prompt

Response

GENERIC PROBE

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what the question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

How did you come up with your answer?

___________________________________________________________________

__________________________________________________________________

ITEM SPECIFIC PROBES

In your own words, what does the term “health issues” mean to you? What would you consider or exclude when providing your response?

___________________________________________________________________

______________________________________________________________________________________________________________________________________

___________________________________________________________________

______________________________________________________________________________________________________________________________________

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy  Difficult  Very Difficult

_________________________________________________________________

________________________________________________________________

_________________________________________________________________

Record your observations and comments here

___________________________________________________________________

______________________________________________________________________________________________________________________________________

Question 9

If at any point in time your 9th grader continues his or her education after high school, how do you expect to pay for tuition, room, and board?

 (Check here if you are sure your 9th grader will never continue his or her education after high school.)

9th grader's own earnings and savings ___
9th grader's parents’ or relatives’ earnings and savings including a pre-paid tuition account or 529 plan ___
Scholarships or grants that do not have to be repaid ___
Federal or state loans in 9th grader’s name ___
Federal or state loans in 9th grader’s parent’s or relatives’ name ___
Private loan in 9th grader's name ___
Private loan in 9th grader’s parents’ or relatives’ names ___
Second mortgage ___
Withdrawal from retirement savings ___
Alimony or child support ___
Social Security or Veteran’s Administration benefits ___

1=Yes 2=No 3=Don't know

Source: HSLS 1st follow=up, plus NELS 82B and new subitems

Question/Prompt

Response

GENERIC PROBE

Ask this probe AFTER the respondent has answered the question.

Can you tell me, in your own words, what the question is asking? Please try to explain without repeating the wording of the question.

___________________________________________________________________

Did you find any words or parts of this question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?


 Yes	 No

__________________________________________________________________

___________________________________________________________________

ITEM SPECIFIC PROBES

Do you think providing some examples of any of these options [part a., b., c., etc.] would be helpful? If so, which sub item would be helpful to provide examples?

Mark all items that are recommended to include examples:

a. b. c. d. e. f. g. h. i. j. k.

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy  Difficult  Very Difficult

_________________________________________________________________

________________________________________________________________

_________________________________________________________________

In your own words, what does the phrase “continues [his/her] education after high school” mean to you? What would you include or exclude when providing your response?

___________________________________________________________________

______________________________________________________________________________________________________________________________________

___________________________________________________________________

Can you think of any other sources that should be on this list that are not included?

IF Yes: What sources would you add to this list?

 Yes

 No

___________________________________________________________________

Record your observations and comments here

___________________________________________________________________

______________________________________________________________________________________________________________________________________

General Debriefing and Thank You

Before we finish, I’d like to hear [any/other] thoughts you have about what you’ve been doing.

______________________________________________________________________________

Is there anything else you would like to tell me about working on the survey questions?

______________________________________________________________________________

Is there anything that you think could make [this/these] survey question(s) clearer?

______________________________________________________________________________

Thank participant for his/her time and provide gift card, as appropriate.

Attachment VII:
Cognitive Interview Protocol: Teacher Survey

Question 1

Did you enter teaching through an alternative certification program? An alternative certification program is a program that is designed to expedite the transition of non-teachers to a teaching career, for example, a state, district, or university alternative certification program.

Source: MGLS:17 C045

Question/Prompt

Response

GENERIC PROBES

Can you tell me, in your own words, what the question is asking?

___________________________________________________________________

Did you find any part of the question confusing?

IF YES: What did you find confusing? What could we do to make it less confusing?


 Yes	 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Do you think the definition for “alternative certification program” in this question is helpful? How can it be improved?

___________________________________________________________________

Are there other examples of alternative certification programs that are not listed here?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 2

How many hours of professional development did you receive during the previous school year (2017-2018) that was focused on math?

Not applicable, you are a first-year teacher in the 2018-2019 school year
None
Less than 6 hours
6-15 hours
16-35 hours
More than 35 hours

Source: MGLS:17 D001

Question/Prompt

Response

GENERIC PROBES

Did you find any part of the question confusing?

IF YES: What did you find confusing? What could we do to make it less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

What professional math development activities did you consider when answering the question? Can you please provide some examples?

___________________________________________________________________

Would you say answering this question for the previous school year was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

_________________________________________________________________

___________________________________________________________________

Did you find it useful to have a range of hours instead in the response options? Why or why not?

___________________________________________________________________

How would you answer this question if there were no response options provided?

Would you find it easier or more difficult to answer the question?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 3

Including this school year, how many years have you taught math at the following grade levels at any school?

(Please enter '0' if you have no math teaching experience at a level.)

any grade K-5 ____
any grade 6-8 ____
any grade 9–12 ____
any grade K–12 ____

Source: HSLS:09 M1 A19

Question/Prompt

Response

GENERIC PROBES

How did you come up with your answer?

___________________________________________________________________

Did you find any part of the question confusing?

IF YES: What did you find confusing? What could we do to make it less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

How did you calculate the number of years you have taught math for “grade K-12”?

___________________________________________________________________

Did you teach any math classes where you had students from several different grades in the same class? If so, how did you account for them in your answer?

___________________________________________________________________

Have you taught math at more than one school?

IF YES: What school(s) were you thinking about when answering this question? How did you decide what school(s) to consider?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Interviewer Directions: For this next set of questions, please think of one of your math class periods. You will be answering questions about this period. Remember, you are evaluating the survey questions, not the class.

Question 4

The next question asks for your opinion, so there are no right or wrong answers.

How much do you agree or disagree with the following statements?

	Strongly agree	Agree	Slightly agree	Slightly disagree	Disagree	Strongly disagree
You have a certain amount of intelligence and you really can’t do much to change it.						
Your intelligence is something about you that you can’t change very much.						
You can learn new things, but you can’t really change your basic intelligence.						

Source: MGLS Student A.035

Question/Prompt

Response

GENERIC PROBE

Did you find any part of the question confusing?

IF YES: What did you find confusing? What could we do to make it less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy  Difficult  Very Difficult

___________________________________________________________________

Would you be able to answer this question for another class, and for students in general? Why or why not?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 5

At this point in the school year, how would you rate the behavior of the students in your math class?

 Students misbehave very frequently and are almost always difficult to handle.

 Students misbehave frequently and are often difficult to handle.

 Students misbehave occasionally.

 Students behave well.

Students behave exceptionally well.

Source: MGLS:17 J040

Question/Prompt

Response

GENERIC PROBES

Did you find any part of the question confusing?

IF YES: What did you find confusing? What could we do to make it less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

___________________________________________________________________

About how long after the start of the school year can you assess students’ behavior in a class as a whole?

___________________________________________________________________

Are there other student behaviors that are not included here?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 6

How would you best describe the rigor of your math class?

 Remedial

 General

 Advanced/Honors

 Other (please specify) _______________

Source: NEW

Question/Prompt

Response

GENERIC PROBES

Can you tell me, in your own words, what the question is asking?

___________________________________________________________________

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

___________________________________________________________________

ITEM SPECIFIC PROBES

In your own words, what does “rigor” mean to you?

_________________________________________________________________

Are there other levels of rigor that are not listed here?

_________________________________________________________________

Record observations and comments here

___________________________________________________________________

Interviewer Directions: For this next set of questions, please think of one of your students. You will be answering questions about this student, but you do not need to tell us the student’s name. Remember, you are evaluating the survey questions, not the student.

Question 7

Please rate [student]'s skills in the following areas, as exhibited in your class.

Select one answer for each row.	Outstanding	Very good	Good	Fair	Poor	Not applicable or not observed
a. Ability to apply mathematical concepts to "real world" problems	1 	2 	3 	4 	5 	6 
b. Ability to complete or conduct proofs or demonstrations of [his/her] mathematical reasoning	1 	2 	3 	4 	5 	6 
c. Ability to talk about [his/her] reasoning or thinking in solving a problem	1 	2 	3 	4 	5 	6 
d. Ability to explain [his/her] reasoning in solving a problem in writing	1 	2 	3 	4 	5 	6 
e. Ability to use representations to model mathematical ideas	1 	2 	3 	4 	5 	6 
f. Ability to use a calculator to solve problems	1 	2 	3 	4 	5 	6 
g. Ability to fluently apply math facts and procedures	1 	2 	3 	4 	5 	6 

Source: MGLS:17 G005

Question/Prompt

Response

GENERIC PROBES

Did you find any words or part of [sub-item a, b, c, etc.] confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

Mark all items reported as Confusing

a b c d e f g

a._________________________________________________________________

b._________________________________________________________________

c._________________________________________________________________

d._________________________________________________________________

e._________________________________________________________________

f._________________________________________________________________

g._________________________________________________________________

ITEM SPECIFIC PROBES

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

___________________________________________________________________

Do you think your answers would vary depending on when during the school year you would evaluate the student(s)?

___________________________________________________________________

About how long after the start of the school year do you think you could answer this question about a student?

___________________________________________________________________

Ask this probe only for sub-items where the respondent selected “Not Applicable or Not Observed.”

I noticed you selected “Not Applicable or Not Observed” as your answer choice to [sub-item a, b, c etc.], can you please tell me more about why you selected this answer?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 8

For each item below, please think about [student]’s behavior during the past month.

Describe how often [student] demonstrated the behavior.

Select one answer for each row.	Never	Sometimes	Often	Very often	Always	No opportunity to observe this behavior
a. Organizes work	1 	2 	3 	4 	5 	6 
b. Appears motivated to learn new things	1 	2 	3 	4 	5 	6 
c. Works well independently	1 	2 	3 	4 	5 	6 
d. Adapts to changes in plans, requirements, or routines	1 	2 	3 	4 	5 	6 
e. Persists in completing tasks	1 	2 	3 	4 	5 	6 
f. Pays attention well	1 	2 	3 	4 	5 	6 

Source: MGLS:17 G025

Question/Prompt

Response

GENERIC PROBES

Did you find any words or part of [sub-item a, b, c, etc.] confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

Mark all items reported as Confusing

a b c d e f g

a._________________________________________________________________

b._________________________________________________________________

c._________________________________________________________________

d._________________________________________________________________

e._________________________________________________________________

f._________________________________________________________________

g._________________________________________________________________

ITEM SPECIFIC PROBES

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

___________________________________________________________________

Do you think your answers would vary depending on when during the school year you would evaluate the student(s)?

___________________________________________________________________

About how long after the start of the school year do you think you could answer this question about a student?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 9

Over the last month, how often did [student]...

Original:

Select one answer for each row.	0 times	1-2 times	3-6 times	7-9 times	10-12 times	13 or more times
a. Come to class without completing prior assignments or homework?	1 	2 	3 	4 	5 	6 
b. Come to class without class materials (such as pencils, paper, tablet, books, or calculator)?	1 	2 	3 	4 	5 	6 

Alternative 1:

Select one answer for each row.	Never	Less than once a week	Once a week	More than once a week	Every class
a. Come to class without completing prior assignments or homework?	1 	2 	3 	4 	5 
b. Come to class without class materials (such as pencils, paper, tablet, books, or calculator)?	1 	2 	3 	4 	5 

Alternative 2:

Select one answer for each row.	Never	Rarely	Sometimes	Often	Always
a. Come to class without completing prior assignments or homework?	1 	2 	3 	4 	5 
b. Come to class without class materials (such as pencils, paper, tablet, books, or calculator)?	1 	2 	3 	4 	5 

Source: MGLS:17 H010

Question/Prompt

Response

GENERIC PROBES

How did you come up with your answer?

___________________________________________________________________

Did you find any words or part of [sub-item a, b] confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

Mark all items reported as Confusing

a b

a._________________________________________________________________

b._________________________________________________________________

ITEM SPECIFIC PROBES

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

_________________________________________________________________

INTERVIEWER NOTE: Provide both handouts of alternative versions to participant and point out the frequency scales.

We are considering three different types of response scales. Out of the original and the two alternatives, which scale do you prefer? Why?

 Original  Alternative 1  Alternative 2

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 9 Participant Handout: Alternative 1

Select one answer for each row.	Never	Less than once a week	Once a week	More than once a week	Every class
a. Come to class without completing prior assignments or homework?	1 	2 	3 	4 	5 
b. Come to class without class materials (such as pencils, paper, tablet, books, or calculator)?	1 	2 	3 	4 	5 

Question 9 Participant Handout: Alternative 2

Select one answer for each row.	Never	Rarely	Sometimes	Often	Always
a. Come to class without completing prior assignments or homework?	1 	2 	3 	4 	5 
b. Come to class without class materials (such as pencils, paper, tablet, books, or calculator)?	1 	2 	3 	4 	5 

Question 10

Before being involved in this study, had you heard of the following?

	Yes	No
ED (Department of Education)	1 	2 
NCES (National Center for Education Statistics)	1 	2 
IES (Institute of Education Sciences)	1 	2 

Source: New

Question/Prompt

Response

ITEM SPECIFIC PROBES

(If yes to a) How did you hear of ED?

___________________________________________________________________

(If yes to a) Did you recognize it by its acronym, by its name, or both?

___________________________________________________________________

(If yes to a) When did you first hear of ED?

___________________________________________________________________

(If yes to b) How did you hear of NCES?

___________________________________________________________________

(If yes to b) Did you recognize it by its acronym, by its name, or both?

___________________________________________________________________

(If yes to b) When did you first hear of NCES?

___________________________________________________________________

(If yes to c) How did you hear of IES?

___________________________________________________________________

(If yes to c) Did you recognize it by its acronym, by its name, or both?

___________________________________________________________________

(If yes to c) When did you first hear of IES?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

General Debriefing and Thank You

Before we finish, I’d like to hear any other thoughts you have about the survey questions we worked on today.

________________________________________________________________________________________

Is there anything else you would like to tell me about working on the survey questions?

________________________________________________________________________________________

Is there anything that you think could make these survey questions clearer?

________________________________________________________________________________________

Thank participant for his/her time and provide gift card, as appropriate.

Attachment VIII:
Cognitive Interview Protocol: School Administrators Survey

Question 1

Which of the following describes your school?

(Check all that apply.)

 A comprehensive school

 A charter school (a school that in accordance with an enabling state statute, has been granted a charter exempting it from selected state or local rules and regulations)

 A special program school or magnet school such as a science or math school, performing arts school, talented or gifted school, or a foreign language immersion school

 A vocational or technical school

 An alternative school (a school that offers a curriculum designed to provide nontraditional education to students -- for example, to students at risk of school failure or dropout in a traditional setting)

 Schools-within-a-school

 9th-grade only school

 Online school

 Other________________________________

 None of the above

Source: HSLS:09 A06

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

In your own words, what does “comprehensive school” mean to you?

___________________________________________________________________

Can you think of any other types of schools that should be included here?

___________________________________________________________________

Are the descriptions of school types helpful? Why or why not?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 1 Handout

Question 1

Which of the following describes your school?

(Check all that apply.)

 A comprehensive school

 A charter school (a school that in accordance with an enabling state statute, has been granted a charter exempting it from selected state or local rules and regulations)

 A special program school or magnet school such as a science or math school, performing arts school, talented or gifted school, or a foreign language immersion school

 A vocational or technical school

 Schools-within-a-school

 9th-grade only school

 Online school

 Other________________________________

 None of the above

Source: HSLS:09 A06

Question 2

Does your school have a special focus?

(Check all that apply.)

STEM (science, technology, engineering and mathematics)
Early college
Performing or fine arts
Gifted and talented education
International baccalaureate or other special diploma program
Foreign language immersion
Newcomer and/or English language development program
Career academies
Freshman academies
Other________________________________
Your school does not have a special focus

Source: HSLS:09 A07 (modified)

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Are there other areas of school focus that are not listed here?

___________________________________________________________________

Looking back at question 1, [HAND COPY OF Q1 TO ADMINISTRATOR] do you think any of the options here might fit better in Question 1?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 3

Last school year (2017-2018), approximately what percentage of students had at least one parent or guardian participating in the following events?

Select one answer for each row.	0%-25%	26%-50%	51%-75%	76%-100%	Not applicable
Open house or back to school night					
All regularly scheduled schoolwide parent-teacher conferences					
Special subject-area events (e.g., science fair, concerts)					
Sporting events					
Parent education workshops or courses					
Signing of a school-parent compact (This is an agreement between school community members [e.g., parents, principals, teachers, and students] that acknowledges the shared responsibility for student learning and/or the school’s policies)					
Involvement in governance or fundraising (e.g., PTA or PTO meetings, school board, parent booster clubs)					
Involvement in budget decisions					
Career days					

Source: NTPS 3-2 2015-2016

Question/Prompt

Response

GENERIC PROBES

Did you find any words or part of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

_________________________________________________________________

ITEM SPECIFIC PROBES

Ask this probe only for sub-items where the respondent selected “Not Applicable.”

I noticed you selected “Not Applicable” as your answer choice to [sub-item a, b, c etc.], can you please tell me more about why you selected this answer?

___________________________________________________________________

Looking at sub-item g, do you think it makes sense to ask about governance and fundraising together? Why or why not?

___________________________________________________________________

Can you think of any events that should be on this list that are not included?

IF Yes: What events would you add to this list?

 Yes

 No

___________________________________________________________________

Provide alternate version.

Direct respondent’s attention specifically to the “response options” of the item.

Please go ahead and answer this question.

Would you say answering this question with these response options was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

_________________________________________________________________

___________________________________________________________________

Which version (original or alternate) do you prefer? Why?

 Original

 Alternate

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 3 ALTERNATE HANDOUT

Question 3

Last school year (2017-2018), about what percentage of students had a parent or guardian participate in each of the following activities?

Select one answer for each row.	All or almost all of the students	More than half of the students	About half of the students	Less than half of the students	None or almost none of the students
Open house or back to school night					
All regularly scheduled schoolwide parent-teacher conferences					
Special subject-area events (e.g., science fair, concerts)					
Sporting events					
Parent education workshops or courses					
Signing of a school-parent compact (This is an agreement between school community members [e.g., parents, principals, teachers, and students] that acknowledges the shared responsibility for student learning and/or the school’s policies)					
Involvement in governance or fundraising (e.g., PTA or PTO meetings, school board, parent booster clubs)					
Involvement in budget decisions					
Career days					

Source: NTPS 3-2 2015 - 2106

Question 4 (Public schools only)

Does your school participate in a public school choice program?

School choice programs are those aimed at giving families the opportunity to choose the school their children will attend, such as magnet schools, charter schools, and school vouchers. Do not include public school choice that is mandatory due to Adequate Yearly Progress requirements.

 Yes

 No

Source: HSLS:09 A08

Question/Prompt	Response
GENERIC PROBES
In your own words, what does “public school choice program” mean to you?	___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________
Record observations and comments here	___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________

Question 5 (Public schools that are part of a public school choice program)

In which of the following types of public school choice programs does your high school participate?

(Check all that apply.)

Students assigned to your school can choose to enroll in your school or another school within the district
Students can enroll in a public school in another district at no tuition cost to themselves or their families
Students from other districts can enroll in your school at no tuition cost to themselves or their families
Students assigned to your school can choose to enroll in a private school using state or district funds
Any other public school choice program___________________________________________

Source: HSLS:09 A09

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

_________________________________________________________________

ITEM SPECIFIC PROBES

Did you think any of the categories overlap? If so, which ones?

 Yes

 No

___________________________________________________________________

Can you think of any programs that should be on this list that are not included?

IF Yes: What programs would you add to this list?

 Yes

 No

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 6

In the 2018-2019 school year, does your school provide students with opportunities for competency-based advancement?

 Yes

 No

Source: High School Strategies Designed to Help At-Risk Students Graduate:15 104

Question/Prompt

Response

GENERIC PROBES

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

_________________________________________________________________

___________________________________________________________________

ITEM SPECIFIC PROBES

In your own words, what does “competency-based advancement” mean to you?

___________________________________________________________________

Do you think “competency-based learning” is the same or different from “personalized learning?”

Please explain.

 Same  Different

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 7

Are there enough slots available for the following courses so that all 9^th-grade students who have the necessary prerequisites and interest can enroll?

Select one answer for each row.	Yes	No	Not applicable
Algebra 1, part 1			
Algebra 1, part 2			
Algebra I			
Algebra II			
Geometry			

Source: Modified from MGLS:17 D10

Question/Prompt

Response

GENERIC PROBES

Did you find any words or part of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

How did you come up with your answer?

___________________________________________________________________

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

___________________________________________________________________

ITEM SPECIFIC PROBES

Ask this probe only for sub-items where the respondent selected “Not Applicable.”

I noticed you selected “Not Applicable” as your answer choice to [sub-item a, b, c etc.], can you please tell me more about why you selected this answer?

___________________________________________________________________

Can you think of any courses that should be on this list that are not included?

IF Yes: What courses would you add to this list?

 Yes

 No

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 8

What percentage of the total student body in your school….

(Please enter '0' if none.)

receives free or reduced-price lunch? ____%  Does not apply; this program is not offered at your school

are English language learners? _____%

Source: HSLS:09 B03

Question/Prompt

Response

GENERIC PROBES

How did you come up with your answer?

___________________________________________________________________

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

___________________________________________________________________

ITEM SPECIFIC PROBES

Can you think of any other terminology schools may use in place of “Free or Reduced-Price Lunch (FRPL)?”

___________________________________________________________________

Does your school offer FRPL to all students?

IF YES: Did this affect your response? Why or why not?

 Yes  No

___________________________________________________________________

What does the term “English Language Learners” mean to you?

___________________________________________________________________

How easy or difficult would you find it to answer this question, if it was asking ONLY about 9^th grade students?

IF Hard or Very Hard: What was difficult about answering this question? How could we make it easier?

 Very Easy  Easy  Hard  Very Hard

___________________________________________________________________

How easy or difficult would you find it to answer this question, if it was asking ONLY about 12^th grade students?

IF Hard or Very Hard: What was difficult about answering this question? How could we make it easier?

 Very Easy  Easy  Hard  Very Hard

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 9

What percentage of your school’s 2018-2019 9^th-grade class is repeating 9^th grade?

____%

Source: HSLS:09 B05

Question/Prompt

Response

GENERIC PROBES

How did you come up with your answer?

___________________________________________________________________

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

___________________________________________________________________

ITEM SPECIFIC PROBES

Provide alternate version.

Which version (original or alternate) do you prefer? Why?

 Original

 Alternate

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 9 ALTERNATE HANDOUT

Question 9

What percentage of your school’s 2018-2019 9^th-grade class was reclassified as a 9^th-grade student?

____%

Source: Modified from HSLS:09 B05

Question 10

When does the school day start for high school students in your school in the 2018-2019 school year?

__:____ AM

Source: New

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

How did you come up with your answer?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

General Debriefing and Thank You

Before we finish, I’d like to hear any other thoughts you have about the survey questions we worked on today.

_________________________________________________________________________________________

Is there anything else you would like to tell me about working on the survey questions?

_________________________________________________________________________________________

Is there anything that you think could make these survey questions clearer?

_________________________________________________________________________________________

Thank participant for his/her time and provide gift card, as appropriate.

Attachment IX:
Cognitive Interview Protocol: School Counselors Survey

Question 1

The following questions are about the high school counseling staff at your school as whole.

During the 2017-2018 school year, what percentage of work hours did your school’s counseling staff spend delivering the following services to high school students?

	5% or less	6%-10%	11%-20%	21%-50%	More than 50%
Choice and scheduling of high school courses					
Assisting students with college readiness, selection, and applications					
FAFSA completion					
Personal development					
Social development					
Academic development					
Academic testing					
Career assessment					
Occupational choice and career planning					
Job placement and employability skill development					
Students’ attendance, discipline, and other school and personal problems					
Non-counseling activities such as hall or lunch duty, substitute teaching, or bus duty					

Source: HSLS:09 11th grade Spring A05

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Would you say answering this question for all of your high school’s counseling staff was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

_________________________________________________________________

___________________________________________________________________

When you were answering this question, were you thinking about services provided one-on-one, services provided to a group or the whole school, or both? Did it depend on the item? Please explain.

 One-on-one  Group/ whole school

 Both

___________________________________________________________________

Did you think any of the categories overlap? If so, which ones?

 Yes

 No

___________________________________________________________________

Can you please provide me with a few examples of how you have assisted students with “college readiness, selection, and applications?”

___________________________________________________________________

Provide alternate version.

Looking at the wording on the response options, which version (original or alternate) do you prefer? Why?

 Original

 Alternate

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 1 ALTERNATE

During the 2017-2018 school year, how often did your school’s counseling staff deliver the following services to high school students?

	Never	Rarely	Sometimes	Often	Very Often
Choice and scheduling of high school courses					
Assisting students with college readiness, selection, and applications					
FAFSA completion					
Personal development					
Social development					
Academic development					
Academic testing					
Career assessment					
Occupational choice and career planning					
Job placement and employability skill development					
Students’ attendance, discipline, and other school and personal problems					
Non-counseling activities such as hall or lunch duty, substitute teaching, or bus duty					

Source: HSLS:09 11th grade Spring A05

Question 2

During the 2017–2018 school year, what percentage of work hours did your school’s counseling staff spend counseling high school students on the following topics?

	5% or less	6%-10%	11%-20%	21%-50%	More than 50%
Anxiety or depression					
Drug use					
Bullying					
Family relationships					
Peer relationships					
Romantic relationships					
Sexual behavior					
Sexual identity					
Sexual orientation					
Racial identity/ethnicity					

Source: New

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Would you say answering this question for all of your high school’s counseling staff was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

___________________________________________________________________

When you were answering this question, were you thinking about services provided one-on-one, services provided to a group or the whole school, or both? Did it depend on the item? Please explain.

 One-on-one  Group/ whole school

 Both

___________________________________________________________________

Looking back at Question 1,

[HAND COPY OF Q1 TO COUNSELOR] do you think that any of the categories overlap with Question 1?

IF YES: Which ones?

 Yes

 No

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 1 HANDOUT

During the 2017-2018 school year, what percentage of work hours did your school’s counseling staff spend delivering the following services to high school students?

	5% or less	6%-10%	11%-20%	21%-50%	More than 50%
Choice and scheduling of high school courses					
Assisting students with college readiness, selection, and applications					
FAFSA completion					
Personal development					
Social development					
Academic development					
Academic testing					
Career assessment					
Occupational choice and career planning					
Job placement and employability skill development					
Students’ attendance, discipline, and other school and personal problems					
Non-counseling activities such as hall or lunch duty, substitute teaching, or bus duty					

Source: HSLS:09 11th grade Spring A05

Question 3

Which of the following steps does your school take for high school students who need extra assistance?

	Yes	No
Tutoring during the regular school day by an adult		
Before- or after-school tutoring by an adult		
Peer tutoring		
School staff work with classroom teachers to provide extra assistance to the student		
Pull-out instruction during the regular school day		
Homework assistance program		
Special incentives or rewards tied to academic performance		
A school-arranged match with an adult mentor		
Positive behavior interventions and supports such as HS-BEP (high school behavior education program)		
Additional support outside the regular school day such as special programs, weekend programs, or summer school programs		
Summer program prior to entry into high school that provides supplemental instruction in reading and math		
Small learning communities or Achievement Academies for over-aged students who have not met high school entry criteria		
Small 9^th grade learning communities or academies separate from the rest of the school		
Catch-up courses or “double-dosing” of classes		
9th grade seminar or class(es) in study skills		
Specific professional development, coaches, or technical assistance for teachers working with struggling 9th graders		
Credit recovery program targeted at standards in which students were deficient. Programs may be offered by computer software, online instruction (such as a virtual school), or teacher-guided instruction		

Source: HSLS:09 11th grade Spring B16; HSLS:09 9th grade Fall A26

Question/Prompt

Response

GENERIC PROBES

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

___________________________________________________________________

ITEM SPECIFIC PROBES

Did you think any of the categories overlap? If so, which ones?

 Yes

 No

___________________________________________________________________

Does your school take any of these steps with students who are not in need of extra assistance? If yes, which steps?

 Yes

 No

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 4

In which of the following ways does your school support high-achieving students?

	Yes	No
Technology and software to support curriculum specifically to meet the needs of the high-achieving students
Gifted students receive pull-out instruction during the regular school day
Enrichment experiences such as Odyssey of the Mind, Science Olympiad, Academic Decathlon, math or science clubs, math or science teams
Scholarships for students to attend special events, programs, or classes
Special incentives or rewards tied to academic performance
A school-arranged match with an adult mentor
Special recognitions such as Honor Roll, Honor Society, or department awards

Source: HSLS:09 11th grade Spring B10

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Are there other ways your school supports high-achieving students? What are they?

___________________________________________________________________

What does the phrase “high-achieving students” mean to you?

___________________________________________________________________

Does your school support students who are not high achieving in any of these ways? If yes, which ways?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

These next three questions ask about Early Warning Systems. For the purposes of this survey, an early warning system is a system based on student-level data to detect students who exhibit behavior or academic performance issues that put them at risk of educational failure.

Question 5

In the 2018-2019 school year, does your school have an early warning system? (Please select only one)

 Yes

 No

Source: High School Strategies Deigned to Help At-Risk Students Graduate Q83

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

How did you come up with your answer?

___________________________________________________________________

Does your school have a formal dropout prevention program? This may be a whole-school restructuring program or a targeted program that operates on a smaller scale within the school or community organization(s) and enrolls students identified as at risk of dropping out.

IF YES AND HAS EARLY WARNING SYSTEM: Do the early warning system and dropout prevention program work together or are they independent? Please explain.

 Yes

 No

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 6 (Respondents whose school has an early warning system)

Which of the following data trigger an intervention in your early warning system?

(Check all that apply)

 State achievement test scores

 Attendance

 Truancy and/or chronic absenteeism

 Course grades

 Discipline incidents including suspensions or expulsions

 Involvement with criminal justice system

 Involvement with social services or foster care

 Pregnancy/teen parenthood

 Reports of substance abuse

 Homelessness or frequent address change

 Limited English proficiency

 Migrant status

 Students overage for their grade level

 Students who are behind in accumulating credits for their grade level

 Other (Please specify __________________)

Source: High School Strategies Deigned to Help At-Risk Students Graduate Q87

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Ask this probe only if the respondent selected “other.”

I noticed you selected “other” as your answer choice. Can you please tell me more about this?

___________________________________________________________________

Did you think any of the categories overlap? If so, which ones?

 Yes

 No

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 7 (Respondents whose school has an early warning system)

What action is taken if a student is flagged by an early warning system?

(Check all that apply)

 Meeting of teachers and administrators

 Meeting of parents

 Develop an intervention plan

 Assign a case manager

 Assign an adult mentor

 Arrange access to social services

 Assign to credit recovery courses

 Assign to double dose or academic support courses

 Assign to mandatory tutoring

 Refer to student support team

 Assigned to a dropout prevention program

 Other (Please specify _________)

Source: High School Strategies Designed to Help At-Risk Students Graduate:15 (Administrator) Q88

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Can you think of any other actions that should be on this list that are not included?

IF YES: What actions would you add to this list?

 Yes

 No

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 8

Does your school have a formal dropout prevention program for students in high school? This may be a whole-school restructuring program or a targeted program that operates on a smaller scale within the school or community organization(s) and enrolls students identified as at risk of dropping out.

 Yes

 No

Source: HSLS:09 9th grade Fall B24

Question/Prompt

Response

GENERIC PROBES

Did you find any part of the question confusing?

IF YES: What did you find confusing? What could we do to make it less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

What does the term “formal dropout prevention program” mean to you?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 9 (Respondents whose school has a dropout prevention program)

On what basis are students in high school recommended for your dropout prevention program?

(Check all that apply.)

 Absentee record

 Poor or failing grades

 Behind on credits

 Teacher’s referral

 Counselor’s referral

 Parental request

 Student request

 Disciplinary problems

 On another basis (Please specify_________ e.g., other early warning system criteria)

Source: HSLS:09 9th grade Fall B25

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Can you think of any other reasons that should be on this list that are not included?

IF YES: What reasons would you add to this list?

 Yes

 No

___________________________________________________________________

Did you think any of the categories overlap? If so, which ones?

 Yes

 No

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 10 (Respondents whose school has a dropout prevention program)

Which of the following services does your dropout prevention program offer?

(Check all that apply.)

 Occupationally focused courses

 Tutoring

 Incentives for better attendance or classroom performance

 Childcare for children of students

 Graduation counseling

 Job counseling

 Other (please specify ____________________________)

Source: HSLS:09 11th grade Spring B19

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Can you think of any other services that should be on this list that are not included?

IF YES: What services would you add to this list?

 Yes

 No

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 11

To what extent to you agree or disagree that your school makes sure that students from each of the following groups get the support they need to achieve success equal to other students?

	Strongly agree	Agree	Disagree	Strongly disagree
Low-income students				
English Language Learners (ELLs)				
Students with special needs				

Source: New

Question/Prompt

Response

GENERIC PROBES

In your own words, what does this question mean to you?

___________________________________________________________________

Did you find any part of the question confusing?

IF YES: What did you find confusing? What could we do to make it less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Do you offer different types of support or programs for low-income students, English Language Learners (ELLs), and students with special needs? What are they?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 12

Do high school students at this school keep the same counselor throughout their high school years?

 Yes

 No

Source: New

Question/Prompt

Response

GENERIC PROBES

In your own words, what does this question mean to you?

___________________________________________________________________

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 13

Are Career Clusters, Pathways, or Programs of Study (POS) offered to students in your school?

 Yes

 No

Source: HSLS:09 9th grade Fall (Counselor) B30

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Are there other words that you would use to describe career focused technical education?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 14 (Public schools only)

Which of the following describe the practices for enrollment of students in your school?

	Yes	No
All students in a particular geographic area (or district) enroll in this school		
Students in a particular geographic area (or district) are generally enrolled in this school but transfers are frequently allowed		
Students are enrolled from particular areas to achieve desired racial or ethnic composition in the school		
Students are enrolled from particular areas to achieve desired socio-economic status composition in the school		
Students are enrolled in this school based on an entrance test or other achievement criteria		
Students are enrolled in this school based on a lottery system		
Other (please specify)		

Source: NELS:88 (Administrator) 24

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Ask this probe only if the respondent selected “other.”

I noticed you selected “other” as your answer choice, can you please tell me more about your school’s approach?

___________________________________________________________________

Do students choose to go to your school rather than be assigned to it?

IF YES: How did you come up with your answer? What could we do to make it easier to answer the question?

 Yes

 No

___________________________________________________________________

In your own words, what does “particular geographic area (or district)” mean to you?

___________________________________________________________________

In your own words, what does “catchment area” mean to you? Which term is clearer to you: “particular geographic area (or district)” or “catchment area”? Why?

___________________________________________________________________

Are there other practices that are not listed here?

IF Yes: What practices would you add to this list?

 Yes

 No

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 15

How many Carnegie units in each of the following subjects is required by your school for high school graduation? The Carnegie unit is a standard of measurement that represents one credit for 120 hours of classroom time over the course of a year at the secondary-school level, equivalent to the completion of a 1-year academic course taken one period a day, 5 days a week.

 Check here if your school does not include 12th grade.

Select one answer for each row.	At least 4 Carnegie units	At least 3 Carnegie units	At least 2 Carnegie units	At least 1 Carnegie unit	No required amount
English/reading					
Mathematics					
Science					
Social Studies					
Computer Education					
Foreign Language					
Arts					
Music					
Physical Education/Health					

Source: NELS:88 (Administrator) 39

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

Are other subjects taught at your school that are required for graduation and are not included here? What are they?

___________________________________________________________________

Do you use something other than Carnegie units to describe minimum requirements?

IF YES: What do you use? How did you come up with your answer? What could we do to make it easier to answer the question?

 Yes

 No

___________________________________________________________________

How sure are you about your answers to this item?

 Very unsure  Unsure  Sure  Very sure

Record observations and comments here

___________________________________________________________________

Question 16

Does your school offer the following courses? (Check all that apply)

Online
Blended learning (e.g., online with an in-person facilitator)

Source: NEW

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

ITEM SPECIFIC PROBES

In your own words, what does the term “blended learning” mean to you?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

Question 17

What type of dual or concurrent enrollment program does your school offer?

	Yes	No
Students can earn college credits for college academic courses while enrolled in our school
Students can earn credits for college career and technical courses while enrolled in our school
Students can apply to a nearby “early college” high school on a partner college campus, where they will complete high school and earn college credits at the same time.
Students are automatically accepted into a partner college upon high school graduation
The credits students earn at a partner college are automatically accepted there once a student matriculates
The credits students earn at a partner college are automatically accepted by some or all other public two- and four-year colleges in our state once a student matriculates
Our school’s dual or concurrent enrollment program doesn’t specify details about credit acceptance by colleges

Source: HSLS:09 11th grade Spring (Counselor) B07

Question/Prompt

Response

GENERIC PROBES

Did you find any words or parts of the question confusing?

IF YES: What did you find confusing? What could we do to make the question less confusing?

 Yes

 No

___________________________________________________________________

Would you say answering this question was very easy, easy, difficult or very difficult?

IF DIFFICULT OR VERY DIFFICULT: How could we make it easier to answer this question?

 Very Easy  Easy

 Difficult  Very Difficult

___________________________________________________________________

ITEM SPECIFIC PROBES

In your own words, what does the term “dual or concurrent enrollment” mean to you?

___________________________________________________________________

In your own words, what does the term “early college” mean to you?

___________________________________________________________________

In your own words, what does the term “partner college” mean to you?

___________________________________________________________________

How sure are you about your answers to this item?

 Very unsure  Unsure  Sure  Very sure

Are there other types of dual or concurrent enrollment programs that are not listed here?

___________________________________________________________________

Record observations and comments here

___________________________________________________________________

General Debriefing and Thank You

Before we finish, I’d like to hear any other thoughts you have about the survey questions we worked on today.

_________________________________________________________________________________________

Is there anything else you would like to tell me about working on the survey questions?

_________________________________________________________________________________________

Is there anything that you think could make these survey questions clearer?

_________________________________________________________________________________________

Thank participant for his/her time and provide gift card, as appropriate.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-01-20

High School and Beyond 2020 (HS&B:20) Base-Year Field Test Cognitive Testing Round 1

NCES Cognitive, Pilot, and Field Test Studies System

Attachments V-IX HS&B 2020 Cog Labs Round 1