Middle Grades Longitudinal Study of 2016-2017 (MGLS:2017) Cognitive Interviews Reading and Accommodations

NCES Cognitive, Pilot, and Field Test Studies System

Volume 3 MGLS 2017 Cog Labs 2014 Reading & Disability - Cog Lab Protocols

Middle Grades Longitudinal Study of 2016-2017 (MGLS:2017) Cognitive Interviews Reading and Accommodations

OMB: 1850-0803

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1850-0803 can be found here:

Document [docx]

Download: docx | pdf

National Center for Education Statistics

Middle Grades Longitudinal Study of 2016-2017 (MGLS:2017) Cognitive Interviews

Set 2

Volume 3

Cognitive Laboratory Protocols

OMB# 1850-0803 v. 103

May 9, 2014

MGLS:2017 COGNITIVE INTERVIEW PROTOCOL

STUDENT MATHEMATICS ASSESSMENT FOR STUDENTS WITH DISABILITIES

MGLS: 2017 Cognitive Interview Protocol
Student Mathematics Assessment

I. Interviewer Welcome Script

The following welcome script should not be read verbatim. You, as the interviewer, need to be familiar enough with the scripts to ask the item probes in a conversational manner. Text written in the box is suggested content for you to be thoroughly familiar with in advance. You should project a warm and reassuring manner toward the participant to develop a friendly rapport and should use conversational language throughout. While the text in this introduction does not need to be read verbatim, all italicized text (notifying student of confidentiality and item-specific probes) is read verbatim to ensure we cover these important topics and ask students these questions in the same way.

Hello, my name is [NAME] and I am working with the U.S. Department of Education on a research study called the Middle Grades Longitudinal Study of 2016-17 (MGLS:2017).

Thank you for coming here today to help us. You may have already talked with your parents about what we are doing here, but just in case you still aren’t completely sure we want to take a minute or two to talk you through what’s going on.

You are here because we asked your [mom/dad] if you might be able to work with us as we create math questions for the MGLS:2017. These questions are going to be given to students all around the country, and what you are helping us with today is an important step in creating math questions that help us find out what children are learning in school. After we asked your [mom/dad] they agreed that you could participate and so you are here today.

We need your help selecting good math questions to ask students in grade [student’s grade level]. Today, we’re going to work through some math questions, and while we want you to do your best to answer the questions correctly, what we really want to learn is whether or not the questions make sense. That is, do you understand what the math questions are asking you to do, or is there something about the questions that make them confusing, tricky to understand, or especially hard to answer? Please go through the questions and try to answer them the best that you can. In addition to talking about things you may have found confusing, we would like to ask you about how you answered some of the questions and a little bit about what you were thinking when solving the questions and ways that we might improve some of the questions. It will take about an hour to go through all the questions and talk about them.

Your information and the answers you give today will be kept private. Only people working on the research study will know about your answers and ideas, but they won’t know that it was you who gave those answers and ideas. We will not talk about any of your answers or comments today with your parents, teachers, or school, but please feel free to tell your parents about anything we talk about today. Your help today will only be used for research purposes and may not be used for any other purpose except as required by law.

I will be taking notes, and I would also like to record what we are doing today so I can listen to it later when I write up my notes. No one outside our research team will hear the recording. Is it okay with you if I record our interview?

Just to remind you, your participation is completely voluntary, and if at any point you decide not to continue, that is okay.

Do you have any questions before we get started?

If, for any reason, the student is no longer interested in participating, thank the student for his or her time and end the interview. After answering questions and giving further explanation, begin the interview with the first item.

In addition to the notes you will take when talking with the student after they finish the assessment, make sure to observe the student while they take the assessment, noting signs of restlessness and frustration (such as fidgeting and time spent with eyes wandering around the room instead of on the task at hand). Note also the items for which the student uses a calculator or works on paper (and the location on the paper that has the work for each item). If a student returns to a previous item please note which item they were on, what item they returned to, and if they corrected or completed the item that they returned to.

II. Working with Students who have Disabilities and Middle Grade Students in General

Note that some of the students who participate in the cognitive laboratory will be students who have disabilities. The way you work with these students should not differ greatly from the way you work with middle grade students in general (that is, including those who do not have an identified disability). All cognitive lab participants are individuals with varying needs; you should observe each student carefully while he or she completes the assessment, adjusting your approach depending on the students’ level of frustration, comprehension, and comfort. Put yourself in the students’ shoes - if you faced the same challenges that they did, what would you find helpful, and how would you want to be treated?

Some students who participate in our cognitive laboratories will require accommodations. The general rule of thumb is that students can use accommodations that they would normally use when taking tests in the classroom. In other words, if they use a device or tool in their everyday learning, they can bring that device with them and use it while completing the assessments. The one exception is the use of calculators for the “no calculator” items. For those items, none of the students will use calculators. For all other items, students may use a calculator or a talking calculator or abacus if that is what they typically use.

You will be provided with any relevant details that were obtained from each students’ parent at the time of recruitment.

Here are some things to keep in mind when working with students who have disabilities, and students in general:

Use strong verbal communication skills. Speak clearly and distinctly to the student, pacing yourself based on his or her needs. You should pay attention to his or her cues and respond by adjusting your speed as needed. Some students will be able to follow your questions easily, while others may require some extra time to process what you have said.
Be patient with students. Allow time for them to think about what you’re asking and then answer you. Try counting to 10 before prompting a student if she doesn’t answer right away. On the other hand, if a student gets off topic, draw him or her gently but firmly back to the study question.

Be actively engaged. Pay close attention to how students respond to questions and watch for their cues. Your interaction should be natural and friendly but it’s also important to observe their behavior and responses carefully.
Consider whether their answers are matching the question. For example, if you ask the first math item “and the student says, “morning,” it may be time to try another approach. The student may be responding only to the last thing that you said (math class – and his math class is in the morning). Try rephrasing: “Here is a graph [point and watch for student to look at the graph]. Does your math book have graphs like this graph?”.
Watch for acquiescence. This happens when the respondent attempts to please the interviewer or give her the “desirable” response. You may notice, for example, that the participant provides the same response several times in a row or always says “yes.” In the example from the previous bullet (Does your math book have graphs like this graph?) you might rephrase “Where do you see graphs like this graph?”
Listen and watch for physical and verbal signs that the student is becoming uncomfortable, frustrated, or losing patience, perhaps indicating that it’s time for a break. You may hear a change in tone or a sigh, or the student may start interrupting part way through the discussion. Be aware of changes in body language that may mean the student is losing focus or is tired and uncomfortable. For example, students with physical disabilities may be uncomfortable sitting for long periods of time and may need help from someone in repositioning their body, or may be tired from the effort of using assistive technologies.
Observe whether the student seems to be experiencing intense emotions. Is he or she upset, embarrassed, or unusually happy? There could be a number of reasons for this, and it may happen at any time. A student who has difficulty communicating may have trouble telling you directly that he or she is upset. Because the student didn’t verbally communicate his or her feelings to you, it may seem that the student became frustrated out of the blue, even though there was a legitimate reason for his or her feelings. Make sure you’re respectful of any feelings participants may have, regardless of whether you understand why. Try taking a break and don’t force the issue if a problem is building.
Check your own body language. Be sure to keep your body language neutral but encouraging at all times during the interview so that students are answering what they really think and not looking for what pleases you. This means that you should sit up, face the student, and put on a friendly sort of “poker face”—with a positive affect that does not change based on the answers that they give. Keep in mind how your body language and positioning might be helpful, depending on the student’s particular disability; for example, if the student is hard of hearing, sit in a place where he or she can clearly see your face and mouth.
Be flexible. Follow the student’s lead, watch his or her cues, and be ready to adapt as needed. Our goal is to learn from the students and do our best to gain their full participation in the cognitive laboratories. Be responsive and adaptable. If necessary, you can enlist the help of that student’s parent, who will be nearby.

III. Example Items and Probes

Students will be instructed to complete a set of approximately 20 math questions as they normally would under naturalistic testing conditions (silently, without interruption). They will be asked to circle or highlight words that are difficult to understand and questions that they find confusing. They can skip any item that they find confusing. Then, after the task is completed, using a think-aloud protocol, they will be asked to talk about the questions that they skipped, found confusing, or found difficult to read or to understand. We then will ask them to try the problem again and talk about what they are thinking as they do the problem. In addition, we will ask questions about select items that have characteristics that we are interested in learning more about.

Some of these questions are for students in later grades and so may be very difficult. I would like you to circle or highlight anything that you find confusing or difficult to understand. You may skip any question that is too difficult to understand. You can use the calculator or extra paper to help you in solving the problems. You can ask me questions at any point.

Ask generic and item-specific probes, as appropriate, before going on to the next identified item.

When probing, think about the following kinds of details that we would like to know about how the student reads and understands each question:

Understanding of the Item: We are interested in learning if students have never seen the word before or if they forgot its meaning, and if they were still able to answer the question. Reassure them that it is okay if they did not know all the words and that their feedback is important – we want to make sure the questions are straight-forward for all students.
1. What questions and/or words were confusing or difficult to read?
  1. Have the student circle or highlight problem words and talk more about them – are they completely unfamiliar?
  2. Ask student to identify a word that may have been less confusing or easier. Ask how their teacher or textbook usually asks these questions, and what words they use.
2. Probe about word problems with more than two sentences in particular. Can they restate in their own words what the question is asking? If not, why?:
  1. Can you tell me, in your own words, what the question is asking?
  2. Have you answered questions like this in school?
  3. How would you [did you] find an answer to this question?
  4. Are there any words, phrases, or (if applicable) pictures/tables/figures/graphs that are difficult for you? If so, which ones?
  5. Is there anything about this question that confuses you? If yes, do you have a suggestion for how to make it easier to understand?
  6. (If applicable) If you tried these items on a computer, and it said to click on the answer, what would you do?
  7. (If applicable) How many answers are there for this question? How did you go about selecting those answers?

Presentation of the Item:
1. Certain items will have graphs or displays associated with the question. How does this affect the student’s understanding? Do they see questions like these in their math books or math class?
2. Probe the student about their familiarity with the way concepts are presented, especially with fractions or equations that can be represented in multiple ways.
  1. Are some conventions easier for the student to understand than others? Why?
3. Use Probes appropriate for item content:
  1. Ask the student to read the question to you (including the numbers). Note words that are difficult. Ask: Can you tell me, in your own words, what the question is asking?
  2. How did you answer this question?
  3. Are there any words, phrases, or (if applicable) pictures/tables/figures/graphs that are difficult for you? If so, which ones?
  4. Did you recognize the symbols in this question? Have you seen your math teacher or textbook use these symbols?[ point to symbols in the item]
  5. [If the student skipped the item] Did the symbols confuse you at all? Is there anything that would make it easier to understand the question?

Language:
1. If an item has a proper noun, such as a person or place’s name, does the student recognize that it is a name or are they spending time trying to decode it?
2. For sentences with item clauses (in other words, IF this, THEN that), does the student understand what the question is asking? Ask the student to put the question in their own words. Students can also show in writing what the question is asking, using the paper you provide.
3. Probes:
  1. Can you tell me, in your own words, what the question is asking?
  2. How did you answer this question?
  3. Can you please read the word in these parentheses out loud [point to difficult word(s)]?
  4. Are there any words, phrases, or [if applicable] pictures/tables/figures/graphs that are difficult for you? If so, which ones?
    1. Did it affect how you answered the question? How?
    2. What can you tell me about these words [Point to words mentioned above]? Did you need to read these words [proper names] in order to answer the question?

1. Understanding the Questions

With these questions, we are interested in probing whether the student understood what the question is asking.

Please take a moment to look at the following question.

CCSS 7.NS.1 – Applying

MGLS NS053

Starting at the point A marked on the number line below, subtract -3. Plot your result on the number line.

Item-Specific Probes:

Can you tell me, in your own words, what is the question asking?
How did you answer this question?
Are there any words, phrases, or [if applicable] pictures/tables/figures/graphs that are difficult for you? If so, which ones?
If this question was shown to you on a computer, and it asked you to click on the answer, what would you do?
Is there anything about this question that confuses you? If yes, do you have a suggestion for how to make it easier to understand?

Please take a moment to look at the following question.

CCSS 6.EE.1 – Knowing

2. MGLS EE010

Which of the following are equal to 2³? Choose all that apply.

6
8
9
12
3(2)
2 * 2 * 2
4(2)
2(2 * 2)
3 * 3
(2 * 2) * 3

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
How did you answer this question?
How many answers are there for this question?
Are there any words or phrases, or [if applicable] pictures/tables/figures/graphs that are difficult for you? If so, which ones?
Is there anything about this question that confuses you? If yes, do you have a suggestion for how to make it easier to understand?

3. Presentation

In this question, we want to observe whether the student recognizes the use of the “*” to represent multiplication and if the layout of this question is confusing.

Please take a moment to look at the following question.

CCSS 7.EE.3 – Applying

MGLS EE033

Evaluate this expression when P = 7.

Item-Specific Probes:

Ask the student to read the question to you (including the numbers). Note words that are difficult. Ask: Can you tell me, in your own words, what the question is asking?
If the student is unclear about the meaning of the *, point to * and ask: Have you seen your math teacher or textbook use this symbol or expressions like these?[repeat for ( ) ]
Are there any words or phrases that are difficult for you? If so, which ones?
Is there anything that would make it easier to understand the question?

4. Language Use (Example Item for Recognizing Proper Nouns / Names and Mathematical Terms)

In this question, we want to observe whether the student spends a lot of time trying to decode the name Pierre. We also will examine how students attempt the problem and understand the phrases “constant rate” and “represent the relationship,” particularly for students in grades 6 and 7 who may not been introduced to grade 8 functions standards (that is, have not had an Algebra class). We will probe about whether they read all the words or skipped over some (and whether they thought that they needed to read names), and whether the names Pierre or Louis made it difficult to understand.

In this item we will also specifically probe how they interpreted the sentence, “His friend Louis already painted some of the wall for Pierre.” Did they assume that Pierre would repaint that wall or did they understand that the area of the wall painted by Pierre was in addition to the wall that Louis painted?

CCSS 8.F.4a – Reasoning

MGLSF058

Pierre is painting a wall in his house. He paints at a constant rate. His friend Louis already painted some of the wall for Pierre. The table below shows the total area painted after Pierre paints for a certain number of minutes.

Number of minutes that Pierre spent painting (m)	Total area painted in square feet (a)
1	27.75
6	61.50

Pierre constructed the following function to represent the relationship between a, the total area painted in square feet, and m, the number of minutes that Pierre spent painting.

a = 6.75m

Is Pierre’s equation correct? Answer “yes” or “no.”
Explain your reasoning.

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
How did you answer this question?

What can you tell me about these words [Point to (Pierre) and (at a constant rate) and (represent the relationship)]. Did you know what this word/phrase means? Did you need to understand the word/phrase in order to answer the question?

Are there any other words or phrases that are difficult for you? If so, which ones?
1. Did it affect how you answered the question? How?
2. Did you need to read these words in order to answer the question?
Tell me about this part of the question [Point to (His friend Louis already painted some of the wall for Pierre.)]. Can you tell me, in your own words, what this sentence tells you? [Probe to also find out if they considered this part of the total painted area that Pierre had in his table or additional wall that was painted – did the student think Pierre repainted the wall that Louis had painted?]
What kinds of things did you think about in answering this question?

5. Collect the student’s assessment and review with them any items where the student circled a word they did not understand

At the end of the interview, ask the student again if there were any other words or phrases that they did not understand or things that they found confusing as they went through the items.

General Probes:

Have you seen this word before in your math book or does your teacher use this word?
Did you need to understand the word in order to answer the question?

How did not knowing this word affect how you answered the question?

How would you change this question to make it easier to understand?

IV. Debriefing and Thank You for Participation

Thank student for his or her time. Collect all hard copy forms and scratch paper the student used.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing this morning/afternoon.

Is there anything else you would like to tell me about working on the task?

Is there anything you would like to ask me about what we did today? [Answer student questions]

Thank you for helping us to improve our math questions.

MGLS:2017 COGNITIVE INTERVIEW PROTOCOL

STUDENT READING ASSESSMENT FOR TYPICALLY DEVELOPING STUDENTS AND FOR STUDENTS WITH DISABILITIES

MGLS:2017 Cognitive Interview Protocol

Student Reading Assessment for Typically Developing Students and Students with Disabilities

I. Interviewer Welcome Script

Hello, my name is [NAME] and I am working with the U.S. Department of Education on a research study called the Middle Grades Longitudinal Study of 2016-17 (MGLS:2017).

Thank you for coming here today to help us. You may have already talked with your parents about what we are doing here, but just in case you still aren’t completely sure, we want to take a second and talk to you about what’s going on.

You are here because we asked your [mom/dad] if you could work with us as we create new reading assessments for the MGLS:2017. (An assessment is another way of saying a test.) These assessments are going to be given to students all around the country, and what you are helping us with today is essential to creating reading assessments that help us find out what children are learning in school. After we asked your [mom/dad] they agreed that you could participate and so you are here today.

We need your help in trying out our new reading assessments for students in grade [student’s grade level]. Today, we are going to work through two sets of reading questions, and we would like you to do your best to answer the questions correctly. When you are done answering the reading questions, I will ask you for your feedback on what the questions were asking and how you selected your answers. For example, we would like to know whether you felt like you understood what you were supposed to do and if you had difficulty with specific questions. It will take about an hour to go through all the questions and talk about them.

Your information and the answers you give today will be kept private. Only people working on the research study will know about your answers and ideas, but they won’t know that it was you who gave those answers and ideas. We will not talk about any of your answers or comments with your parents, teachers, or school, but please feel free to tell your parents about anything we talk about today. Your help will only be used for research purposes and may not be used for any other purpose except as required by law.

Just to remind you, your participation is completely voluntary, and if at any point you decide not to continue, that is okay.

Do you have any questions before we get started?

Our main goals in this cognitive interview are to get timing information about each section of items the student will complete on the computer and how students interact with the computer interface, mouse, and keyboard during the assessments. We are also interested to know how students perceive the level of difficulty of the passages and questions, and how they understand the directions for each item type. The computer will automatically record the timing information while the student moves through the assessments. While the student completes the assessments, you will follow along, using the Reading Observation and Cognitive Interview Form provided, to record information for each section and some specific items. Information includes whether students appear to read the content on the screen or just skip it, scroll down to read the passage on items with a scroll bar, click on the tab to read the passage for items with a tab, etc. The form contains screenshots of the items. In most cases, you will be able to simply check Yes or No to state whether the student performed the navigational behaviors we would like to know about. There is room for additional comments if you notice anything you would like to comment on as a student works through a section or specific item. For example, if the student seems particularly distracted or confused. While observing the student, please note whether the student shows signs of restlessness and frustration (such as fidgeting and time spent with eyes wandering around the room instead of on the task at hand). During the interviews after each section, we will indirectly probe on these behaviors to determine if the restlessness or frustration is due to item difficulty, confusion, boredom, etc.

In addition to the notes you will take while the student is completing the assessments, you will take notes about your interview with the student after they finish the assessment. We will interview the student after the student has completed the first assessment (Block 1), then again after the second assessment (Block 2a or 2b). This will allow us to capture accurate timing data.

II. Working with Students who have Disabilities and Middle Grade Students in General

Note that some of the students who participate in the cognitive laboratory will be students who have disabilities. The way you work with these students should not differ greatly from the way you work with middle grade students in general (that is, including those who do not have an identified disability). All cognitive lab participants are individuals with varying needs; you should observe each student carefully while he or she completes the assessment, adjusting your approach depending on the students’ level of frustration, comprehension, and comfort. Put yourself in the students’ shoes–if you faced the same challenges that they did, what would you find helpful, and how would you want to be treated?

You will be provided with any relevant details that were obtained from each student’s parent at the time of recruitment.

Here are some things to keep in mind when working with students who have disabilities, and students in general:

Use strong verbal communication skills. Speak clearly and distinctly to the student, pacing yourself based on his or her needs. You should pay attention to his or her cues and respond by adjusting your speed as needed. Some students will be able to follow your questions easily, while others may require some extra time to process what you have said.
Be patient with students. Allow time for them to think about what you’re asking and then answer you. Try counting to 10 before prompting a student if she doesn’t answer right away. On the other hand, if a student gets off topic, draw him or her gently but firmly back to the study question.
Be actively engaged. Pay close attention to how students respond to questions and watch for their cues. Your interaction should be natural and friendly but it’s also important to observe their behavior and responses carefully.
Consider whether their answers are matching the question. For example, if you ask, “Can you tell me in your own words how to play this game?” and the student says, “I play card games at home,” it may be time to try another approach. Try rephrasing: “You just played the Hearts and Flowers game. If you had to tell a friend how to play that game, what would you tell them?”
Watch for acquiescence. This happens when the respondent attempts to please the interviewer or give her the “desirable” response. You may notice, for example, that the participant provides the same response several times in a row. For example, if you say, “Were the instructions difficult to understand or to read?” and the students says, “No” each time, you might rephrase, “What are some ways we can make the instructions easier to understand?”
Listen and watch for physical and verbal signs that the student is becoming uncomfortable, frustrated, or losing patience, perhaps indicating that it’s time for a break. You may hear a change in tone or a sigh, or the student may start interrupting part way through the discussion. Be aware of changes in body language that may mean the student is losing focus or is tired and uncomfortable. For example, students with physical disabilities may be uncomfortable sitting for long periods of time and may need help from someone in repositioning their body, or may be tired from the effort of using assistive technologies.
Observe whether the student seems to be experiencing intense emotions. Is he or she upset, embarrassed, or unusually happy? There could be a number of reasons for this, and it may happen at any time. A student who has difficulty communicating may have trouble telling you directly that he or she is upset. Because the student didn’t verbally communicate his or her feelings to you, it may seem that the student became frustrated out of the blue, even though there was a legitimate reason for his or her feelings. Make sure you’re respectful of any feelings participants may have, regardless of whether you understand why. Try taking a break and don’t force the issue if a problem is building.
Check your own body language. Be sure to keep your body language neutral but encouraging at all times during the interview so that students are answering what they really think and not looking for what pleases you. This means that you should sit up, face the student, and put on a friendly sort of “poker face”—with a positive affect that does not change based on the answers that they give. Keep in mind how your body language and positioning might be helpful, depending on the student’s particular disability; for example, if the student is hard of hearing, sit in a place where he or she can clearly see your face and mouth.
Be flexible. Follow the student’s lead and watch his or her cues, and be ready to adapt as needed. Our goal is to learn from the students and do our best to gain their full participation in the cognitive laboratories. Be responsive and adaptable. If necessary, you can enlist the help of that student’s parent, who will be nearby.

III. Observation Prompts and Probes for Reading Assessment Tasks

All students will first complete a block of questions that measure components of reading. Based on their performance on the first block, the computer program will present a message indicating which second block they should complete. The second block could contain additional reading components questions, or it could include questions embedded in a scenario-based comprehension assessment. The instructions have been organized according to the types of blocks the student may encounter.

There will be two sets of questions that you will work on today. Some of the questions you see today may be difficult while some may be easy. Please just try your best to work through all of them. The instructions for how to complete each question will be presented on the computer.

If students ask for help or indicate they don't understand how to complete a question, record that on the observation form. If a student is spending a long time on an item and not advancing, briefly probe for understanding. If the question is about directions and you know the answer, please explain to students and note this. If it is about the content of question (for example, “How do I answer this question?”), reassure the students and encourage them to do their best, and then have them move on to the next question. Be sure to identify whether the student has difficulty with the wording of the item or with its content. Take care not to make evaluative statements such as, “You’re doing a great job.” Use neutral language that thanks them for their hard work and encourages them to continue.

Block 1 - Reading Components

The first block contains three sections: vocabulary, morphological awareness and sentence comprehension. The online program will instruct the student on how to move through the assessment and how to answer the different item types. This section will contain no more than 46 items that can be done very quickly. We estimate this entire block will take the student approximately 10-15 minutes to complete.

You will be watching the student while they move through this block. In the Reading Observation and Cognitive Interview Form, there will be places to record your comments and answer specific questions about how the student navigates through various sections. Some example screens have been provided:

Observation questions for Block 1 - Reading Components: We are interested in whether students understand how to navigate through the first block and whether they seem comfortable switching between tasks. We are also interested in knowing whether students appear to read the directions and take time to understand the examples provided. Pictures of specific sections or question types have been included in the printed observation materials along with the specific observation questions associated with each question type. Below are the observation questions you will find on the form:

Does the student appear to read the instructions and the sample items (i.e., paused on the instruction screen long enough to read the content versus pressed the spacebar quickly; applied instructions and answered questions as directed; paused on the sample items long enough to read them versus randomly and/or rapidly hitting 1, 2 or 3)?
Does the student appear to use the spacebar easily to move on to the next question?
Does the student appear to respond with the 1, 2 or 3 key easily to give their answer?
Does the student generally seem engaged and to understand what they are supposed to do throughout the block (i.e., moved through the questions with little or no difficulty; did not exhibit signs of boredom, confusion or stress like excessive distractibility, eyes wandering around the room excessively, frequent scowling, spending long periods of time on single questions, clicking through rapidly)?

For Block 1, you will ask the following questions:

How easy or hard was this activity? Why? [If you noticed difficulty at any point during the observation, probe for what caused this difficulty. For example, I noticed you spent a long time on this question (point to question in print out). Can you tell me why?]
Did you read all the directions? [If no:] Why not?
Did you read the sample items? [If no:] Why not?
Did you have difficulty understanding any of the information? [If so:] What did you have difficulty understanding?
Did you understand what you were supposed to do throughout the assessment? What, if any, directions, questions, etc. did you have trouble with?
How comfortable were you navigating through the assessment? For example, using spacebar to move from page to page and answering questions using 1, 2 or 3?
Have you taken other reading tests like the ones you took today? Can you tell me about them? How were they similar? How were they different?
Is there anything else you would like to tell us about this activity?

Block 2a - Additional Reading Components

Some students will be assigned a second block that contains additional reading component questions. This block is designed to take the student approximately 20 minutes. As in the first block, the online program will instruct the student on how to move through the assessment and how to answer the different item types. Navigation in this block will be similar to the first block. Students will press the spacebar to go on to the next question, and they will use the 1, 2 or 3 keys to give their answer.

Two new tasks will be introduced in this block. The first is called a MAZE task. In this task, a passage will be presented one sentence at a time. Some of the sentences will have multiple ways to complete the sentence. The student's task is to select the word that best completes the sentence. An example has been provided.

The second new task is a passage comprehension section. In this task, the student will see the same passage that was presented in the MAZE task, but now the passage will appear complete (with all the correct completions). Some of these questions will include a scroll bar because the passage is too long to fit on the screen. An example has been provided.

Observation questions for Block 2a - Additional Reading Components: We are interested in whether students understand how to navigate through the MAZE and Passage Comprehension questions and whether they seem comfortable completing these tasks. We are also interested in knowing whether students appear to read the directions and take time to understand the examples provided. Pictures of specific sections or question types have been included in the form along with the specific observation questions associated with each question type. Below are some example observation questions:

Does the student appear to read the instructions and the sample items (i.e., paused on the instruction screen long enough to read the content versus pressed the spacebar quickly; applied instructions and answered questions as directed; paused on the sample items long enough to read them versus randomly and/or rapidly hitting 1, 2 or 3)?
Does the student appear to use the spacebar easily to move on to the next question?
Does the student appear to respond with the 1, 2 or 3 key easily to give their answer?
Does the student generally seem engaged and to understand what they are supposed to do throughout the block (i.e., moved through the questions with little or no difficulty; did not exhibit signs of boredom, confusion or stress like excessive distractibility, eyes wandering around the room excessively, frequent scowling, spending long periods of time on single questions, clicking through rapidly)?
For items with a scroll bar, did the student use the scroll bar to read the entire passage? Did the student appear to have any trouble with the scroll bar?

For Block 2a, you will ask the following questions:

How easy or hard was this activity? Why? [If you noticed difficulty at any point during the observation, probe for what caused this difficulty. For example, I noticed you spent a long time on this question (point to question in print out). Can you tell me why?]
Did you read all the directions? [If no:] Why not?
Did you read the sample items? [If no:] Why not?
Did you read the entire passages/texts (applies only to 2a)? [If no:] Why not?
Did you have difficulty understanding any of the information? [If so:] What did you have difficulty understanding?
Did you understand what you were supposed to do throughout the assessment? What, if any, directions, questions, etc. did you have trouble with?
How comfortable were you navigating through the assessment? For example, using spacebar to move from page to page and answering questions using 1, 2 or 3?
Have you taken other reading tests like the ones you took today? Can you tell me about them? How were they similar? How were they different?
Is there anything else you would like to tell us about this activity?

Block 2b–Scenario-Based Assessment of Comprehension

Some students will be assigned a second block of scenario-based comprehension questions. This block is designed to take the student approximately 20 - 25 minutes. In this block, students will be presented with a scenario (e.g., working with classmates to create a website) that will lead them through a variety of comprehension-related tasks on the same topic. In this block, students will encounter novel item types that require new ways of navigating on the screen. For example, some screens will have tabs that, if clicked, will reveal a previously-read passage. Other screens will have scrollbars because the passage is too long to present on the screen. Items may require a "click-and-click" response where the student first clicks on their answer and then clicks the location in a table where it belongs. Some examples have been provided:

Observation questions for Block 2b - Scenario-Based Assessment of Comprehension: We are interested in whether students understand how to navigate through the variety of questions and whether they seem comfortable completing the scenario-based task. We are also interested in knowing whether students appear to read the directions and take time to understand the examples provided. Pictures of specific questions have been included in the form along with the specific observation questions associated with each item. Below are some example observation questions:

Does the student appear to read the instructions or dialogue when there is not an item (i.e., paused on the instruction screen and examples long enough to read the content versus clicked the "Next" arrow quickly)?
Does the student generally seem engaged and to understand what they are supposed to do throughout the block (i.e., moved through the questions with little or no difficulty; did not exhibit signs of boredom, confusion or stress like excessive distractibility, eyes wandering around the room excessively, frequent scowling, spending long periods of time on single questions, clicking through rapidly)?
Does the student appear to easily use a mouse to make their response?
For items with a scroll bar, did the student use the scroll bar to read the entire passage? Did the student appear to have any trouble with the scroll bar?
For items with a tab, did the student click on the tab to re-read the passage? If so, what behaviors preceded clicking the tab? What did they do afterward?
For items that require click-and-click responses, did the student seem to understand the directions for completing the table (i.e., click-and-click, not drag and drop)?

For Block 2b, you will ask the following questions:

How easy or hard was this activity? Why? [If you noticed difficulty at any point during the observation, probe for what caused this difficulty. For example, I noticed you spent a long time on this question (point to question in print out). Can you tell me why?]
Did you read the entire passages and texts? [ If no:] Why not?
Did you have difficulty understanding any of the information? [Wait for answer.] [If so:] What did you have difficulty understanding?
Did you understand what you were supposed to do throughout the assessment? What, if any, directions, questions, etc. did you have trouble with?
How comfortable were you navigating through the assessment? For example, using the mouse, clicking the arrow to move from page to page and answering graphic organizer questions with “click and click” versus drag and drop?
Have you taken other reading tests like the ones you took today? Can you tell about each of them? How they were similar? How they were different?
What did you think of the topic? Was it interesting to you? Did you feel that there were clearly two sides to the issue?
Is there anything else you would like to tell us about this activity?

IV. Debriefing and Thank You for Participation

Thank the student for his or her time.

Collect all hard copy forms the student used.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing this morning/afternoon.

Is there anything else you would like to tell me about working on the tasks?

Is there anything you would like to ask me about what we did today?
[Answer student questions]

Thank you for helping us with our new reading assessments.

MGLS:2017 COGNITIVE INTERVIEW PROTOCOL

STUDENT EXECUTIVE FUNCTION ASSESSMENT FOR STUDENTS WITH DISABILITIES

MGLS:2017 Cognitive Interview Protocol

Student Executive Function Assessment

I. Interviewer Welcome Script

The following welcome script should not be read verbatim. You, as the interviewer, need to be familiar enough with the scripts to introduce the think-aloud process in a conversational manner. Text written in italics is suggested content for you to be thoroughly familiar with in advance. You should project a warm and reassuring manner toward the participant to develop a friendly rapport and should use conversational language throughout. The italicized text should be read verbatim, both in the introduction and the probes. This will ensure that we cover these important topics and ask students these questions in the same way.

Hello, my name is [NAME] and I am working with the U.S. Department of Education on a research study called the Middle Grades Longitudinal Study of 2016-17 (MGLS:2017).

You are here because we asked your [mom/dad] if you might be able to work with us as we develop activities to help us understand how students your age organize, remember, and use information. These activities are going to be given to students all around the country, and what you are helping us with today is essential to selecting our activities. After we asked your [mom/dad], they agreed that you could participate, and so you are here today.

Today, you will complete a few activities or games on the laptop. We need your help to make sure the activities work on the laptop and that they make sense. That is, although we want you to do your best to complete the activities as quickly and correctly as you can, what we really want to learn is whether the instructions are easy to understand and if you have any problems using the laptop to complete the activity. After you finish, I will ask you to tell me about any activities that were confusing or that did not make sense so that you can help us make sure these activities are clear for students.

What you tell me today will make these activities better. Your information and the answers you give today will be kept private. Only people working on the research study will know about your answers and ideas, but they won’t know that it was you who gave those answers and ideas. We will not talk about any of your answers or comments today with your parents, teachers, or school, but please feel free to tell your parents about anything we talk about today. Your help today will only be used for research purposes and may not be used for any other purpose except as required by law. I will be taking notes, and I would also like to video record what we are doing today so I can watch it later when I write up my notes about what we learn from you about these activities. No one outside our research team will see the video recording. Is it okay with you if I video record our interview?

Today you will help us with four activities. Each activity should take about five to ten minutes. Please do each carefully. Going through all the activities and my questions will take about 60 minutes.

Just to remind you, your participation is completely voluntary, and if at any point you decide not to continue, that is okay.

Do you have any questions before we get started?

If, for any reason, the student is no longer interested in participating, thank the student for his or her time and end the interview. If the student agrees to be video recorded, make sure to start the recording once you have obtained his or her permission. After answering any student questions, begin the interview with the first activity.

II. Working with Students who have Disabilities and Middle Grade Students in General

You will be provided with any relevant details that were obtained from each students’ parent at the time of recruitment.

Here are some things to keep in mind when working with students who have disabilities, and students in general:

Use strong verbal communication skills. Speak clearly and distinctly to the student, pacing yourself based on his or her needs. You should pay attention to his or her cues and respond by adjusting your speed as needed. Some students will be able to follow your questions easily, while others may require some extra time to process what you have said.
Be patient with students. Allow time for them to think about what you’re asking and then answer you. Try counting to 10 before prompting a student if she doesn’t answer right away. On the other hand, if a student gets off topic, draw him or her gently but firmly back to the study question.
Be actively engaged. Pay close attention to how students respond to questions and watch for their cues. Your interaction should be natural and friendly but it’s also important to observe their behavior and responses carefully.
Consider whether their answers are matching the question. For example, if you ask, “Can you tell me in your own words how to play this game?” and the student says, “I play card games at home,” it may be time to try another approach. Try rephrasing: “You just played the Hearts and Flowers game. If you had to tell a friend how to play that game, what would you tell them?”
Watch for acquiescence. This happens when the respondent attempts to please the interviewer or give her the “desirable” response. You may notice, for example, that the participant provides the same response several times in a row. For example, if you say, “Were the instructions difficult to understand or to read?” and the students says, “No” each time, you might rephrase, “What are some ways we can make the instructions easier to understand?”

Listen and watch for physical and verbal signs that the student is becoming uncomfortable, frustrated, or losing patience, perhaps indicating that it’s time for a break. You may hear a change in tone or a sigh, or the student may start interrupting part way through the discussion. Be aware of changes in body language that may mean the student is losing focus or is tired and uncomfortable. For example, students with physical disabilities may be uncomfortable sitting for long periods of time and may need help from someone in repositioning their body, or may be tired from the effort of using assistive technologies.
Observe whether the student seems to be experiencing intense emotions. Is he or she upset, embarrassed, or unusually happy? There could be a number of reasons for this, and it may happen at any time. A student who has difficulty communicating may have trouble telling you directly that he or she is upset. Because the student didn’t verbally communicate his or her feelings to you, it may seem that the student became frustrated out of the blue, even though there was a legitimate reason for his or her feelings. Make sure you’re respectful of any feelings participants may have, regardless of whether you understand why. Try taking a break and don’t force the issue if a problem is building.
Check your own body language. Be sure to keep your body language neutral but encouraging at all times during the interview so that students are answering what they really think and not looking for what pleases you. This means that you should sit up, face the student, and put on a friendly sort of “poker face”—with a positive affect that does not change based on the answers that they give. Keep in mind how your body language and positioning might be helpful, depending on the student’s particular disability; for example, if the student is hard of hearing, sit in a place where he or she can clearly see your face and mouth.
Be flexible. Follow the student’s lead and watch his or her cues, and be ready to adapt as needed. Our goal is to learn from the students and do our best to gain their full participation in the cognitive laboratories. Be responsive and adaptable. If necessary, you can enlist the help of that student’s parent, who will be nearby.

III. Observation Prompts and Probes for Executive Function Tasks

This section will include four measures: the Stop Signal, 3-Back with verbal stimulus, 2-Back with nonverbal stimulus, and the Hearts and Flowers task. Together, these measures will assess inhibitory control, working memory, and shifting in middle grade students. Students will be provided with a laptop and should be instructed to complete the tasks following the directions on the screen. They should complete the tasks as they normally would under naturalistic testing conditions (silently, without interruption). In general, we want to know if the instructions were clear and the student understood each task. Interviewers will observe students as they take the executive function assessment and interview the students about their experiences with the tasks. The interviewer will ask students about the instructions after each practice session and before the student attempts the actual task.

As they work through the tasks, the interviewer will observe students and note any evidence of engagement, understanding, boredom, or confusion in students’ facial expressions or body language. Interviewers should observe how students complete each executive function task, using the questions below as a guide.

Interviewer Observation Questions:

1. Does the student express signs of confusion, boredom, or excitement (for example, they fidget or seem to move their attention away from the task and let eyes wander the room)?

2. Does the student rapidly move through the Stop Signal items or take his or her time?

3. How often does the student change or switch their response to the items?

4. What is the start and stop time of each task? Write the information down on the timing form.

5. Does the student put on headphones when directed and check the sound? Does the student have any problems with continuing to wear headphones? Explain.

6. In the Hearts and Flowers and Stop Signal tasks, does the student use correct hand and finger placement on the arrow keys as directed? If not, explain.

7. In the N-back tasks, does the student use the space bar as directed? If not, explain.

8. How often do you need to redirect the student to the task? Explain.

9. Does the student appear to need additional directions or practice sessions? Explain.

After the student reads the instructions and completes the practice test for each section, the interviewer will ask them the following questions:

Can you tell me in your own words, how to play this game?
Were the instructions difficult to understand or to read?
How difficult do you think it will be for other students your age to understand the directions?

After students complete the tasks, the interviewer will debrief with the student about his or her experiences. Interviewers should probe students during the debriefing on any notable behaviors observed when students are completing the items in the executive function tasks. For example, if the interviewers see that a student is spending a lot of time on a particular item within a task, they should note that and probe on it later (for example, “I noticed you spent some time on this item. Can you tell me about…”).

We are particularly interested in learning if students understood what the task was asking them to do and whether they had difficulties with any of the tasks, including any problems with the laptop or program not working correctly.

The italicized text below should be read aloud to the student.

Were any of the instructions difficult to understand or hard to read?
Do you think other students your age would be able to understand the directions?
Was the laptop difficult to use for any parts of the activities?
Is there any other information that would have made these activities easier to understand or complete?

Was it clear how to do these activities?
1. Which activity did you like the most?
2. Which activity did you like the least? Why did you like it less than the others?
Do you have any other questions?

IV. Debriefing and Thank You for Participation

Thank the student for his or her time.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing this [morning/afternoon].

Is there anything else you would like to tell me about participating in these activities?

Is there anything you would like to ask me about what we did today? [Answer student questions]

Thank you for helping us.

MGLS:2017 COGNITIVE INTERVIEW PROTOCOL
STUDENT INTERVIEW FOR STUDENTS WITH DISABILITIES

MGLS:2017 Cognitive Interview Protocol

Student Interview

I. Interviewer Welcome Script

The following welcome script should not be read verbatim. You, as the interviewer, need to be familiar enough with the scripts below to introduce the MGLS:2017 study and the purpose of the cognitive interview in a conversational manner. Text in the box is suggested content for you to be thoroughly familiar with in advance. You should project a warm and reassuring manner toward the participant to develop a friendly rapport and should use conversational language throughout. Please note that any italicized text for item-specific probes should be read verbatim to ensure we cover these important topics and ask students these questions in the same way.

Hello, my name is [NAME] and I’m from [DIR/Mathematica], and we are working with the U.S. Department of Education on a research study called the Middle Grades Longitudinal Study of 2016-17 (MGLS:2017).

You are here because we asked your [mom/dad] if you might be able to work with us as we develop a student questionnaire for the MGLS:2017. This questionnaire is going to be given to children all around the country, and what you are helping us with today is essential to creating that questionnaire. After we asked your [mom/dad], they agreed that you could participate and so you are here today.

We need your help selecting good questions to ask students in grade [student’s grade level]. Today, I have some questions that middle-grade students like you in many different schools will be asked to answer about themselves, their education, and their families. Although we want you to do your best to answer the items, what we really want is your feedback on how easy questions are to understand. That is, do all of the questions make sense? Is there a difficult word that you don’t understand? Is there a better way for us to ask a question so that it makes more sense? There are no right or wrong answers to these questions, but please try your best, because the things you tell us today will be used to make improvements to these questions. This should take about one hour.

What you tell me today will make these questions better. Your information and the answers you give will be kept private. Only people working on the research study will know about your answers and ideas, but they won’t know that it was you who gave those answers and ideas. We will not talk about your answers or comments with anyone outside the research team, including your parents, teachers, or school, but please feel free to tell your parents about anything we talk about today. Your help today will only be used for research purposes and may not be used for any other purpose except as required by law. I will be taking notes, and I would also like to record what we are doing today so I can listen to it later when I write up my notes about what we learn from you about these questions. No one outside our research team will hear the recording. Is it okay with you if I record our interview?

Just to remind you, your participation is completely voluntary, and if at any point you decide not to continue, that is okay.

Do you have any questions before we get started?

If, for any reason, the student is no longer interested in participating, thank the student for his or her time and end the interview. After answering any student questions, begin the interview with the first item.

II. Working with Students who have Disabilities and Middle Grade Students in General

You will be provided with any relevant details that were obtained from each students’ parent at the time of recruitment.

Here are some things to keep in mind when working with students who have disabilities, and students in general:

Use strong verbal communication skills. Speak clearly and distinctly to the student, pacing yourself based on his or her needs. You should pay attention to his or her cues and respond by adjusting your speed as needed. Some students will be able to follow your questions easily, while others may require some extra time to process what you have said.
Be patient with students. Allow time for them to think about what you’re asking and then answer you. Try counting to 10 before prompting a student if she doesn’t answer right away. On the other hand, if a student gets off topic, draw him or her gently but firmly back to the study question.
Be actively engaged. Pay close attention to how students respond to questions and watch for their cues. Your interaction should be natural and friendly but it’s also important to observe their behavior and responses carefully.
Consider whether their answers are matching the question. For example, if working on item A1 and you ask “What grades of yours did you think about when answering each question? “and the student says, “7th grade,” it may be time to try another approach. Try rephrasing: “These questions were asking you about your grades or how well you do in your classes at school. Can you give me examples of things that you get grades on? Were you thinking about those things when you answered this?”
Watch for acquiescence. This happens when the respondent attempts to please the interviewer or give her the “desirable” response. You may notice, for example, that the participant provides the same response several times in a row or always says “yes.” For example at question A1, if a student responded “yes” for every answer, it would be inconsistent (“yes” at items j, k, or l are likely inconsistent with all “yes” responses to items a-h). You could gently ask the student about that discrepancy and remind them that there are no right or wrong answers; we are interested in learning about their experiences.
Listen and watch for physical and verbal signs that the student is becoming uncomfortable, frustrated, or losing patience, perhaps indicating that it’s time for a break. You may hear a change in tone or a sigh, or the student may start interrupting part way through the discussion. Be aware of changes in body language that may mean the student is losing focus or is tired and uncomfortable. For example, students with physical disabilities may be uncomfortable sitting for long periods of time and may need help from someone in repositioning their body, or may be tired from the effort of using assistive technologies.
Observe whether the student seems to be experiencing intense emotions. Is he or she upset, embarrassed, or unusually happy? There could be a number of reasons for this, and it may happen at any time. A student who has difficulty communicating may have trouble telling you directly that he or she is upset. Because the student didn’t verbally communicate his or her feelings to you, it may seem that the student became frustrated out of the blue, even though there was a legitimate reason for his or her feelings. Make sure you’re respectful of any feelings participants may have, regardless of whether you understand why. Try taking a break and don’t force the issue if a problem is building.
Check your own body language. Be sure to keep your body language neutral but encouraging at all times during the interview so that students are answering what they really think and not looking for what pleases you. This means that you should sit up, face the student, and put on a friendly sort of “poker face”—with a positive affect that does not change based on the answers that they give. Keep in mind how your body language and positioning might be helpful, depending on the student’s particular disability; for example, if the student is hard of hearing, sit in a place where he or she can clearly see your face and mouth.
Be flexible. Follow the student’s lead and watch his or her cues, and be ready to adapt as needed. Our goal is to learn from the students and do our best to gain their full participation in the cognitive laboratories. Be responsive and adaptable. If necessary, you can enlist the help of that student’s parent, who will be nearby.

III. Survey Items and Probes for Non-Assessment Student Questionnaire

The student interview consists of students answering eight items or item series that were newly created, modified, or need further testing with this age group. Because we are interested in understanding how students interpret and answer these items, this portion of the interview will be interactive: after the student is presented with and answers each item, you will ask item-specific probes. By asking probes right after each item or item series, students will be able to provide item-specific information while experiencing minimal recall difficulties.

Students will be instructed to complete each item or item series as they will do during the field test and national study (independently without interruption from the interviewer). Then, after the item is completed, they will be asked to describe what they were thinking. Ask item-specific probes before going on to the next item.

IV. Students’ Perceptions of Parents’ Behaviors

We are interested in understanding how students answer item series about various things that their parents may say or do in response to the good or bad grades they receive at school. We want to examine students’ comprehension of the items and whether their parents respond to their grades in ways that are not currently listed. We are also interested in how students may respond to items generally about “their parent(s)/guardian(s)” (that is, “do your parent(s)/guardian(s) encourage you…?”) versus asking questions pertaining to each individual parent (that is, “does your mother encourage you…?”; “does your father encourage you…?).

Please take a moment to read this question and the answers to the question. As you read through them, please circle any words that are difficult to understand or answers that you were not sure how to respond to.

SURVEY: Ginsburg & Bronstein, 1993

1. What do your parents/guardians do when you receive good grades in school?

When I get good grades my parents/guardians ….

	MARK ONE EACH ROW
	Yes	No
a. Give me a reward, like a present or gift	1 □	2 □
b. Give me money or increase my allowance ………	1 □	2 □
c. Take me to some place special, like out to dinner or to a movie	1 □	2 □
d. Give me more opportunities to make decisions for myself	1 □	2 □
e. Encourage me to try harder	1 □	2 □
f. Are less strict with me	1 □	2 □
g. Tell me I am a good student	1 □	2 □
h. Tell me they are proud of me	1 □	2 □
i. Say I should have done even better	1 □	2 □
j. Do not know about my good grades	1 □	2 □
k. Do not care about my good grades	1 □	2 □
l. Do not really do anything	1 □	2 □
m. Other (Please specify)	1 □	2 □

Student Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
Did you have trouble answering any of these statements?
1. Were there any words or statements that were confusing or unclear?
2. Discuss any of the statements or words the student has circled or noted when completing the item series, asking why they found it difficult and if they had suggestions for how to make it easier to understand.
What grades of yours did you think about when answering each question? Did you think about your recent test or homework grades, your most recent report card, this school year, or something else?
What does the phrase “Give me more opportunities to make decisions for myself” mean to you? Can you give me an example of this?
What does the phrase “Are less strict with me” mean to you? Can you give me an example of this?
Are there other things that your parents or guardians say or do when you get good grades that were not listed among the questions you answered?
1. [If so:] What other things do your parents or guardians say or do when you get good grades?
Who came to mind when you answered the question about how your parents or guardians would respond to the good grades you get?
1. Why did this person come to mind?

SURVEY: Ginsburg & Bronstein, 1993

2. What do your parents/guardians do when you receive bad grades in school?

When I get bad grades my parents/guardians ….

	MARK ONE EACH ROW
	Yes	No
a. Reduce my allowance	1 □	2 □
b. Give me fewer opportunities to make decisions for myself	1 □	2 □
c. Are more strict with me	1 □	2 □
d. Punish or ground me	1 □	2 □
e. Encourage me to try harder	1 □	2 □
f. Offer me a reward, such as money or a present, if I do better next time	1 □	2 □
g. Make me feel bad	1 □	2 □
h. Offer to help me with my school work	1 □	2 □
i. Offer to find me a tutor	1 □	2 □
j. Do not know about my bad grades	1 □	2 □
k. Do not care about my bad grades	1 □	2 □
l. Do not really do anything	1 □	2 □
m. Other (Please specify)	1 □	2 □

Student Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
1. How do you define “bad” grades?
Did you have trouble answering any of these statements?
1. Were there any words or statements that were confusing or unclear?
2. Discuss any of the statements or words the student have circled or noted when completing the item series, asking why they found it difficult and if they had suggestions for how to make it easier to understand.
What grades of yours did you think about when answering each question? Did you think about your recent test or homework grades, your most recent report card, this school year, or something else?
What does the phrase “give me fewer opportunities to make decisions for myself” mean to you? Can you give me an example of this?
What does the phrase “are more strict with me” mean to you? Can you give me an example of this?
As you read through the statements, did you think there were statements that were really asking the same thing?
1. [If so:] Which statements felt that way?
Are there other things that your parents or guardians say or do when you get bad grades that were not listed among the questions you answered?
1. If so: what other things do your parents or guardians say or do when you get bad grades?
Who came to mind when you answered these two questions about how your parents or guardians would respond to the grades you get? Why did this person come to mind?
1. Is there another adult who takes care of you who came to mind when answering this question?
  1. [If so:] Who else takes care of you?

If student names only one individual, proceed to question A3 in the interviewer protocol script. If student names two people or more (for example, mother, father, stepparent, relative, guardian), continue with the following set of probes below. Parent 1, Parent 2, Parent 3, etc. refer to the individuals the student mentions in probes 8 and 8a.

Thinking about the different individuals you named, how did you come up with your answer?
If the question had been specifically about “What does your [PARENT 1] say or do when you get good or bad grades?,” would you have responded (or answered) differently to any of these items?
If the question had been specifically about “What does your [PARENT 2] say or do when you get good or bad grades?,” would you have responded (or answered) differently to any of these items?
If applicable, continue to ask this question for all individuals mentioned in response to probes 8 and 8a.

Students’ Time Use

We are interested in understanding specific dimensions of students’ time use. With the first item in this section, we are interested in understanding how much time students spend with their friends and how the students differentiate spending time with their friends in person versus online. We would also like to know how students recall and estimate how much time they spend with their friends. With the second item in this section, we are interested in understanding what work or job experiences students may have. We are also interested in learning whether our lists of jobs adequately represent students’ experiences or if additional items need to be added.

All italicized text should be read aloud to the student.

Please take a moment to read this question and the answers to the question.

A3. During a typical weekday during the school year, how many hours do you spend hanging out or socializing with your friends?

SURVEY: High School Longitudinal Study of 2009

ARK ONE ONLY

1 □ Less than 1 hour

2 □ 1 to 2 hours

3 □ 2 to 3 hours

4 □ 3 to 4 hours

5 □ 4 to 5 hours

6 □ 5 or more hours

Student Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
How did you answer this question? Did you have trouble answering the question?
What does the phrase “hanging out or socializing with your friends” mean to you? Can you give me an example of this?
1. Is it something you do in person (for example, invite your friends over to your home after school, hang out at the mall together, go to the movies together, and so on)?
  1. [If so:] Can you give me an example of what you are doing when you are hanging out with or socializing with your friends in person?
2. Is it something you do online?
  1. [If so:] Can you give me an example of what you are doing when you are hanging out with or socializing with your friends online?
  2. What do you use to do this (in other words, do you use your phone, computer, or something else)?
  3. What specific websites or applications do you use to hang out or socialize with your friends online?
Who did you think about when answering this question?
1. Did you think of friends from school? From your neighborhood? Friends you know from somewhere else?
2. Did you think of friends who you know or met online?
  1. [If so:] How did you meet these online friends (for example, in a chat room, while playing games online, elsewhere)?
What does the phrase “typical weekday during the school year” mean to you?
How did you figure out the number of hours you spend hanging out or socializing with your friends during a typical day?
1. Did you think about what you typically did last week, last month, or even longer ago than that?
2. Was it difficult to separate out how much time you spend hanging out or socializing with your friends on the week days versus how much time you spend with them on the weekends?
  1. [If so:] How so?

This next question asks about getting paid for work you may do. This does not include things like doing chores at home or getting allowance. As you read through the question, please circle any words that are difficult to understand or answers that you were not sure how to respond to.

SURVEY: National Education Longitudinal Study, 1988

4. Which of the categories below describe the kind of work you do/did for pay? (Do not include chores or other work around the house, or an allowance you might receive.)

MARK ALL THAT APPLY

1 □ Lawn work

2 □ Waiter/waitress, dishwasher, or busser

3 □ Newspaper route

4 □ Babysitting or child care

5 □ Farm or agricultural work

6 □ Other manual labor

7 □ Store clerk, salesperson

8 □ Office or clerical

9 □ Odd jobs

10 □ Other (please specify)

11 □ I have not worked for pay

Student Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What kinds of things did you think about in answering this question?
What does the phrase “work you do/did for pay” mean to you?
Of the types of work that were listed, were there any that were difficult to understand?
1. [If so:] Which ones were difficult to understand?
Is there something you thought about, but were not sure which category it belonged to?
1. If so, what did you think about?
ASK ONLY IF STUDENT MARKED RESPONSE OPTION 10: Are there other types of work that you have done that were not listed above and that you did not write in yourself?
1. [If so:] What other type of work have you done?

Students’ Interests and/or Sparks

We are interested in understanding whether students have a “spark” in their lives. By “spark,” we mean any interests, hobbies, and talents the students are excited about. The list of talents, interests, and hobbies were primarily developed from open-ended responses reported in the MADICS survey and additional responses options were created to further capture students’ experiences. We are interested in learning about what talents, interests, or hobbies students engage in, and having students talk about what activities do or do not make this list and why.

All italicized text should be read aloud to the student.

Please take a moment to read the next few questions and the answers to the questions. As you read through them, please circle any words that are difficult to understand or answers that you were not sure how to respond to.

MODIFIED SURVEY: Maryland Adolescent Development In Context Study: Wave 1

5a. Below is a list of talents, interests, or hobbies that students your age are sometimes excited about. Please select the talents, interests, or hobbies that are very important to you. If you do not see yours listed, use the “Other” area to mark an “X” and write in that talent, interest, or hobby on the line provided.

What talents, interests, or hobbies are you excited about?

MARK ALL THAT APPLY

1 □ Math/science

2 □ Writing/reading

3 □ Computers/electronics

4 □ Team sports/athletics (Baseball, Basketball, Football, Gymnastics, Volleyball, etc.)

5 □ Physical activities (Biking, Running, Martial Arts, Skateboarding, Ice Skating etc.)

6 □ Dance

7 □ Music

8 □ Singing/choir

9 □ Art

10 □ Drama

11 □ Volunteering/service/activism

12 □ Being in nature, caring for animals, or participating in outdoor recreation

13 □ Doing construction, architecture, or other types of mechanics or engineering

14 □ Collecting (trading cards, stamps, models, etc.)

15 □ Cooking/baking

16 □ Other (Specify):

1 Shape1 7 □ I don’t have any talents, interests, or hobbies that I am excited about

(PUT DOWN YOUR PENCIL AND WAIT FOR FURTHER INSTRUCTIONS)

A5b. Among the talents, interests, or hobbies you selected above (question A4a), which are you the most excited about?

MARK ONE ONLY

MODIFIED SURVEY: Maryland Adolescent Development In Context Study: Wave 1

1 □ Math/science

2 □ Writing/reading

3 □ Computers/electronics

4 □ Team sports/athletics (Baseball, Basketball, Football, Gymnastics, Volleyball, etc.)

5 □ Physical activities (Biking, Running, Martial Arts, Skateboarding, Ice Skating etc.)

6 □ Dance

7 □ Music

8 □ Singing/choir

9 □ Art

10 □ Drama

11 □ Volunteering/service/activism

12 □ Being in nature, caring for animals, or participating in outdoor recreation

13 □ Doing construction, architecture, or other types of mechanics or engineering

14 □ Collecting (trading cards, stamps, models, etc.)

15 □ Cooking/baking

16 □ Other (Specify):

MODIFIED SURVEY: Maryland Adolescent Development In Context Study: Wave 1

6. Do you have a parent, mentor, friend, or another caring adult who has helped you pursue the talents, interests, or hobbies you are excited about?

1 □ Yes

0 □ No

Student Item-Specific Probes for A5a-b, and A6:

Can you tell me, in your own words, what the questions are asking?
1. What does “excited about” mean to you?
What does the phrase “talents, interests, or hobbies” mean to you?
Were there any talents, interests, or hobbies that you thought about but decided not to include? [If so:] What else came to mind? Was there a reason why you chose not to include it?
1. Were there any talents, interests, or hobbies that you have that you were not sure which category they went under? [If so:] Which one?
On a scale of 1 to 5, with 1 being easy and 5 being difficult, how hard would it be for kids your age to answer this question?

If student answered A6 ask:

When you said, [STUDENT’S RESPONSE TO A6], what kinds of things did you think about in answering this question?
Who did you think about in answering the question?
1. What [does that person/do those people] do to help you with your talents, interests, or hobbies? Can you give me an example?
What does the phrase “helped you pursue the talents, interests, or hobbies you are excited about” mean to you?

Students’ Educational Aspirations and Expectations

With this study question, we are interested in understanding students’ educational goals. Because these items were originally used with older students, we want to know how students in grades 6, 7, or 8 think about their educational goals and if they have any difficulties answering the questions.

All italicized text should be read aloud to the student.

A7. As things stand now, how far in school do you think you will get?

MARK ONE ONLY

MODIFIED SURVEY: High School Longitudinal Study of 2009

1 □ Won’t finish high school

2 □ Will graduate from high school, but won’t go any further

3 □ Will go to vocational, trade, or business school after high school

4 □ Will attend college

5 □ Will graduate from college

6 □ Will attend a higher level of school after graduating from college

7 □ Don’t know

Student Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
How did you answer this question?
What kinds of things did you think about in answering this question?
What does the phrase “vocational, trade, or business school” mean to you? Can you give me an example of what this means?
What does the phrase “higher level of school after graduating from college” mean to you? Can you give me an example of what this means?
Did you have a hard time picking your answer?
1. [If so:] Were you deciding between two answers?
2. [If so:] What helped you figure out which one to choose if you were deciding between these two answer choices?

If student responded 1, 2, or 3 to question A7, skip question A8 and go to the Debriefing and Thank You.

Let’s do another.

A8. How much do you agree with the following statement: I am definitely going to college.

MARK ONE ONLY

MODIFIED SURVEY: High School Longitudinal Study of 2009

1 □ Very strongly disagree

2 □ Strongly disagree

3 □ Disagree

4 □ Agree

5 □ Strongly agree

6 □ Very strongly agree

Student Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
What kinds of things did you think about in answering this question?
Did you have a hard time deciding between two of these answer choices?
1. [If so:] What helped you figure out which one to choose if you were deciding between these two answer choices?
On a scale of 1 to 5, with 1 being easy and 5 being difficult, how hard would it be for kids your age to answer this question?

V. Debriefing and Thank You for Participation

Thank the student for his or her time.

Before we finish, I’d like to hear any other thoughts you have about the questions you answered for us today.

Is there anything else you would like to tell me about working on the questions?

Is there anything you would like to ask me about what we did today? [Answer student questions]

Thank you for helping us to improve our questionnaire.

MGLS:2017 COGNITIVE INTERVIEW PROTOCOL
PARENT QUESTIONNAIRE FOR PARENTS OF STUDENTS WITH DISABILITIES

MGLS:2017 Cognitive Interview Protocol

Parent Questionnaire

I. Interviewer Welcome Script

The following script should not be read verbatim. Text in the box is suggested content for you to thoroughly review in advance and be familiar enough with to present in a conversational manner. You should project a warm and reassuring manner toward the participant to develop a friendly rapport and should use conversational language throughout. Please note that any italicized text for item-specific probes should be read verbatim to ensure we cover these important topics and ask parents these questions in the same way.

Hello, my name is <name>. I’m from Decision Information Resources, Inc., and we are working with the U.S. Department of Education.

Thank you for agreeing to help us today. Decision Information Resources and Mathematica Policy Research are working with the National Center for Education Statistics, part of the U.S. Department of Education, to design a new national longitudinal study of how children learn and progress during 6th, 7th, and 8th grade. This study, the Middle Grades Longitudinal Study of 2016-17 (or MGLS:2017), will collect information from students, parents, teachers, and school administrators.

You should have received a brief letter in the mail or e-mail that included a few questions for you to review. These are questions that parents of middle grade students across the country will be asked to answer about their children’s education, behavior, and family life. The questions also ask about their children’s IEP if they have one, and services that the child may receive outside of school. Today, I will ask you some follow-up questions about the topics in those questions. This should only take about 30 minutes. Your feedback will help us make these questions clearer for parents who will participate in the study in the future.

We would like to record this interview so the members of our research team can review the recordings later. No one else will hear the recording outside of the research team. What you say is very important to us, and we don’t want to miss anything. Your information will be kept strictly private and will not be shared with anyone outside the research team. Your feedback may be used only for research purposes and may not be used for any other purpose except as required by law. Is it okay if I record our interview?

Do you have any questions before we get started?

If, for any reason, the parent is no longer interested in participating, thank the parent for his or her time and end the interview. After answering questions and giving further explanation, begin the interview with the first item.

II. Newly added IEP and Disability Questions

With these questions, we are interested in learning more from parents about the child’s IEP and the parent’s role in contributing to their child’s IEP, services their child receives in and outside of school, and parental satisfaction with school services.

We will review each of the new disability-related items with participants in the parental cognitive laboratory, asking them to complete the series of questions and following up with a series of probes for each category.

III. Survey Topics, Items, and Probes

We will send parents survey questions in advance of the interview so they are aware of the topics we will cover. During the cognitive interview, the parent will read the questions and then you will follow-up with the probes.

All italicized text should be read verbatim to the parent.

Conversations About Future Planning

We are interested in the conversations that parents have with their children about school, future life events and decisions (careers, schooling after high school, life goals more generally). Here, we want to learn about the frequency and depth of conversations parents have with their children about the courses they take during and across the middle grades so we can develop and modify existing survey items as needed. The participant will read the question and answer the items, then the interviewer will ask the probes immediately after.

A1. Since the beginning of this school year, have you talked with your child about which math courses to take next school year?

MODIFIED SURVEY: High School Longitudinal Study of 2009

1 □ Yes

2 □ No

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
1. How did you answer this question?
Who initiated this conversation?
What kinds of things did you discuss?
Is your response based on discussions you had with your child only or did you consider discussions others in the household had with him or her as well?
1. [If “others”:] Who else did you include?
2. [If “others”:] What led you to base your response on the discussions of multiple people?

Would you have chosen a different answer if the question had asked “Have you talked with your child about which math courses to take next school year?”
Were there any words in the question that you found confusing? Was the way the question was worded confusing? What would have made the question clearer?

Social skills and positive behaviors

Here we are interested in how parents respond to questions about their child’s social skills and positive behaviors. The participant will read and answer statements about the child’s behaviors and the interviewer will follow with probes inquiring whether the questions were clear and easy to understand and to evaluate a new response scale (that is, a 5-point frequency scale).

Next we would like to ask you some questions about your child’s social skills and behaviors. Please read through the list of statements below and then I will ask a series of follow-up questions.

B1. Please read the following list of items that sometimes describe children. For each of the following characteristics please mark how often this occurred for your child over the last six months.

MARK ONE PER ROW	NEVER	RARELY	OCCASIONALLY	OFTEN	ALWAYS
a. He/she is considerate of other people’s feelings.	1 □	2 □	3 □	4 □	5 □
b MODIFIED SURVEY: ECLS-K:11 . He/she is helpful if someone is hurt, upset, or feeling ill	1 □	2 □	3 □	4 □	5 □
c. He/she is nervous in new situations, easily loses confidence	1 □	2 □	3 □	4 □	5 □
d. He/she often offers to help others (parents, teachers, children)	1 □	2 □	3 □	4 □	5 □
e. He/she gets along better with adults than with other youth	1 □	2 □	3 □	4 □	5 □

Item-Specific Probes:

Can you tell me, in your own words, what the question is asking?
1. How did you answer this question?
What behaviors or events came to mind when you heard the questions?
1. Why?
Were there any words in the question that you found confusing? Was the way the question was worded confusing? What would have made the question clearer?
Was there a particular statement about your child where you had a hard time deciding between two of these answer choices? [ If so:] What helped you figure out which one to choose if you were deciding between two answer choices?
Did the response categories make the question easy or difficult to understand?
1. Would making your selection have been easier with more answer choices? Less answer choices?
Can you tell me, in your own words, what the phrase “Rarely” means to you? Can you tell me, in your own words, what the phrase “Occasionally” means to you?

How did you decide between “Rarely” and “Occasionally”?

Parent involvement in school

With the following questions about parent involvement in school, we are interested in the relevance, understanding, and appropriateness of the response options and the questions for parents of middle grade students. Testing will focus on the most appropriate mix of items from those two sections. Additionally, across parenting topics we are interested in whether parent answers are based on his or her own personal behaviors alone or if multiple people’s behaviors (that is, both of the parents and/or other adults) in the home are taken into account.

Next, we would like to ask questions about your thoughts on your involvement in your child’s school. Read through and answer the list of statements. I will ask a series of follow-up questions once you are done.

C1. Indicate how much you AGREE or DISAGREE with each of the following statements. Please think about the current school year as you consider each statement.

	MARK ONE PER ROW
	DISAGREE VERY STRONGLY	DISAGREE	DISAGREE JUST A LITTLE	AGREE JUST A LITTLE	AGREE	AGREE VERY STRONGLY
a Hoover-Dempsey, K.V., & Sandler, H.M. (2005). Final Performance Report for OERI Grant # R305T010673: The Social Context of Parental Involvement: A Path to Enhanced Achievement. Presented to Project Monitor, Institute of Education Sciences, U.S. Department of Education, March 22, 2005. . It's my job to explain tough assignments to my child.	1 □	2 □	3 □	4□	5□	6 □
b. It's my job to make sure my child understands his or her assignments.	1 □	2 □	3 □	4□	5□	6 □
c. I make it my business to stay on top of things at school.	1 □	2 □	3 □	4□	5□	6 □
d. I assume my child is doing all right when I don't hear anything from the school.	1 □	2 □	3 □	4□	5□	6 □
e. The teacher has to let me know about a problem before I can do something about it.	1 □	2 □	3 □	4□	5□	6 □
f. I get most of my information about my child's progress from report cards.	1 □	2 □	3 □	4□	5□	6 □
g. My child's learning is mainly up to the teacher and my child.	1 □	2 □	3 □	4□	5□	6 □

Item-Specific Probes:

How did you answer this question?
Can you tell me, in your own words, what the phrase “I assume my child is doing all right when I don't hear anything from the school” is asking?
1. How did you answer this question?
What examples of your involvement in your child’s school came to mind when you heard the questions?
1. Why?
Do these type of things apply for your [6^th/7^th/8^th] grader or your school?
Did you have trouble picking a response from the six provided? (Follow-up probe if needed: What if you did not have a hard copy?)
For each item where you selected “Disagree very strongly” or “Disagree,” please tell me if the item simply did not apply to you and why.
Were your responses based on the way you think or act or did you consider the role of others in the household as well?
1. If “others,” who else did you include?
2. [If “others”:] What led you to base your response on the discussions of multiple people?
Were there any words in the question that you found confusing? Was the way the question was worded confusing? What would have made the question clearer?

Now we would like to ask questions about your involvement in your child’s learning and education. Read through and select a response to statements in the list below. I will ask a series of follow-up questions once you are done.

C2. Indicate HOW OFTEN you have done the following during this school year.

	MARK ONE EACH ROW
	NEVER	ONCE SO FAR THIS YEAR	ABOUT ONCE A MONTH	ONCE EVERY TWO WEEKS	ONCE A WEEK	DAILY
a. I kept an eye on my child's progress.	1 □	2 □	3 □	4 □	5 □	6 □
b. I made sure that my child's homework got done.	1 □	2 □	3 □	4 □	5 □	6 □
c. I helped my child study for tests or quizzes.	1 □	2 □	3 □	4 □	5 □	6 □
d. I talked to my child about what he or she is learning.	1 □	2 □	3 □	4 □	5 □	6 □
e. I took my child to the library, community events, or similar places.	1 □	2 □	3 □	4 □	5 □	6 □
f. I expected the school to notify me if my child had a problem.	1 □	2 □	3 □	4 □	5 □	6 □
g. I expected my child to do his or her homework at school.	1 □	2 □	3 □	4 □	5 □	6 □
h. I relied on the teacher to make sure my child understands his or her assignments.	1 □	2 □	3 □	4 □	5 □	6 □

Item-Specific Probes:

When considering your answer to the questions, how far back did you consider? One month? Six months?
1. Would the question have been easier to answer if it asked about a different timeframe?
Can you tell me, in your own words, what the phrase “I kept an eye on my child's progress” is asking?
1. How did you answer this question?
Can you tell me, in your own words, what the phrase “I talked to my child about what he or she is learning” is asking?
1. How did you answer this question?
Can you tell me, in your own words, what the phrase “I expected the school to notify me if my child had a problem” is asking?
1. How did you answer this question?
Can you tell me, in your own words, what the phrase “I expected my child to do his or her homework at school” is asking?
1. How did you answer this question?
Can you tell me, in your own words, what the phrase “I relied on the teacher to make sure my child understands his or her assignments” is asking?
1. How did you answer this question?
What examples of your involvement in your child’s learning and education came to mind when you heard the questions?
1. Why?
Do these type of things apply for your [6^th/7^th/8^th] grader or his or her school?
Did you have trouble picking a response from the six provided? (Follow-up probe if needed: What if you did not have a hard copy?)
For each item where you selected “Never,” please tell me if the item simply did not apply to you and why.
Were your responses based on your behavior only or did you consider the involvement of others in the household had with as well?
1. [If “others”:] Who else did you include?
2. [If “others”:] What led you to base your response on the discussions of multiple people?
Were there any words in the question that you found confusing? Was the way the question was worded confusing? What would have made the question clearer?

Thank you, your feedback is very helpful. I now want to transition to some questions that ask about your child’s Individualized Education Program or IEP, and also about services your child may receive. Read through the following questions and please select the appropriate responses. I will ask a series of follow-up questions once you are done. Please stop after question D12.

D1. Did [YOUTH] ever have an IEP (Individualized Education Program)?

Note: An IEP is a written statement for each student with a disability that sets goals for the student in school, says how progress will be measured, describes the special education and related services the school will provide, how much the student will be in the regular class with nondisabled students, and lists accommodations or modifications needed to measure what the student knows through tests.

 Yes 1

 No 0

Don’t know 2

IF D1 = 0 or 2, go to D13

D2. As of the beginning of the 2014-2015 school year, did [YOUTH] still have an IEP?

 Yes 1

 No 0

IF D2 = 1, go to D4

D3. Why does [he/she] no longer have an IEP?

Select all that apply

 No longer needs special services 1

 Met IEP goals 2

 [YOUTH] was declassified, school says no longer needs services 3

 No longer eligible, doesn’t qualify 4

 School doesn’t have the programs [YOUTH] needs 5

 Parent doesn’t want [YOUTH] in special education 6

 [YOUTH] did not want to be in special education 7

 [YOUTH] now has a 504 Plan 8

 [YOUTH] was never was in special education 9

 [YOUTH] home schooled by parent 10

 Other (specify) 99

D4. In the last 12 months, did you or another adult in the household go to a meeting about an IEP for [YOUTH’S] special education program or services?

 Yes 1

 No 0

Don’t know 2

D5. In the last 12 months, did [YOUTH] go to [that same/a] meeting about an IEP, for [his/her] special education program or services?

 Yes 1

 No 0

Don’t know 2

D6. In the last 12 months, has there been an IEP meeting about [YOUTH’S] special education program or services?

 Yes 1

 No 0

Don’t know 2

D7. Who came up with the goals on [YOUTH]'s IEP?

Select all that apply

 School 1

 Me or other parent/family member 2

 [YOUTH] 3

 Other (specify) 99

D8. Of the people who came up with the IEP goals, who came up with the most goals?

Select one only

 School 1

 Me or another parent/family member 2

 [YOUTH] 3

 Other (specify) 99

D9. How active was [YOUTH] in developing [his/her] IEP?

Select one only

 Very active; took a leadership role in IEP development 1

 Active; participated regularly in IEP development 2

 Somewhat active; participated occasionally in IEP development 3

 Not active; did not participate in IEP development 4

D10. Overall, how satisfied are you with the progress [YOUTH] has made towards [his/her] IEP

goals year? Are you...

Select one only

 Completely satisfied 1

 Very satisfied 2

 Fairly satisfied 3

 Somewhat dissatisfied 4

 Very dissatisfied 5

D11. Have you had any discussions with [YOUTH]'s IEP team about [his/her] transition to high school?

 Yes 1

 No 0

D12. Have you discussed the following topics with staff at [YOUTH]'s school?

Select one per row

	YES	NO
a. How my child can develop self-determination and self advocacy skills	1 	0 
b. How my child can expand social and community support networks	1 	0 
c. How my child can learn “soft” employment skills (such as appropriate dress, working well in a group, following instructions)	1 	0 
d. How my child can practice money management skills	1 	0 
e. How my child can connect with adult service providers	1 	0 
f. How my child can plan for health care needs	1 	0 
g. How my child can prepare for change	1 	0 

Probes about “Parent’s Role in Contributing to Their Child’s IEP” Items

How did you answer this question?
When you see the word “IEP,” what does it mean to you? Do you think this is a word that most parents would understand?
How easy or difficult was it to remember [INSERT ACTIVITY HERE for example, whether or not your child had an IEP during the specified time period, whether or not you attended an IEP meeting]? Please explain.
Did the possible responses make sense to you?
1. Was there anything not listed here that you think should be added? Please explain.
2. Was there anything in the list that you think should be removed? Please explain

Thank you. Now please answer question D13 and then stop.

D13. During the past 12 months, has [YOUTH] received any of the following accommodations or services? For each accommodation or service listed, please select whether [he/she] received it through school or through outside community sources.

Select one per row

	YES, RECEIVED THROUGH SCHOOL	YES, RECEIVED THROUGH OUTSIDE SOURCE	NO, DID NOT RECEIVE	DON'T KNOW
Academic Performance
a. Catch-up courses or double-dosing of classes during the regular school day	1 	2 	0 	d 
b. Supplemental instruction or tutoring in academic subjects before or after school	1 	2 	0 	d 
c. Supplemental instruction or tutoring in academic subjects on weekends	1 	2 	0 	d 
d. Help connecting students to outside academic transition services, supports, or activities (e.g., tutoring, mentoring)	1 	2 	0 	d 
e. Tutor	1 	2 	0 	d 
f. Braille instruction or other related services for visually impaired (e.g., Nemeth code, abacus)	1 	2 	0 	d 
Positive Skills and Behaviors
g. Help developing capability to dress, clean, care for self	1 	2 	0 	d 
h. Help connecting students to community skill- and behavior-related services, supports, or activities (e.g., assistive technology, networking)	1 	2 	0 	d 
i. Speech or language therapy, or communication services	1 	2 	0 	d 
j. Psychological or mental health services or counseling	1 	2 	0 	d 
k. Physical or occupational therapy	1 	2 	0 	d 
l. Orientation and mobility services (to help individuals navigate their environment)	1 	2 	0 	d 
Positive Life Functioning
m. Reader or interpreter, including sign language	1 	2 	0 	d 
n. Audiology services for hearing problems	1 	2 	0 	d 
o. Special transportation because of disability	1 	2 	0 	d 
Other
p. Other accommodations or services (specify)	1 	2 	0 	d 

Probes about “Services Received Outside of School” Items

How did you answer this question?
How far back did you consider, when answering this question? How easy or difficult was it to recall services received during this time period?
What do “services” mean to you? How easy or difficult was it to think about services received outside of school, as opposed to services received in school?
Did the list of services we provided make sense to you?
1. Was there anything not listed here that you think should be added? Please explain.
2. Was there anything in the list that you think should be removed? Please explain.
(If parent selected “Other/Specify” option) Please tell me more about how and why you selected the “Other/Specify” option.

For any QD13a-p=1, ask QD14a-p. Else, go to debrief script.

Thanks for that feedback. Please go ahead and answer the next question and when you finish it I have a few more questions for you.

D14. How helpful are or were the services your child or family is receiving or has received from the school during

the past 12 months?

Select one per row

	Very helpful	Helpful		Not helpful		Not at all helpful
Academic Performance
a. Catch-up courses or double-dosing of classes during the regular school day	1 		2 		0 	d 
b. Supplemental instruction or tutoring in academic subjects before or after school	1 		2 		0 	d 
c. Supplemental instruction or tutoring in academic subjects on weekends	1 		2 		0 	d 
d. Help connecting students to outside academic transition services, supports, or activities (e.g., tutoring, mentoring)	1 		2 		0 	d 
e. Tutor	1 		2 		0 	d 
f. Braille instruction or other related services for visually impaired (e.g., Nemeth code, abacus)	1 		2 		0 	d 
Positive Skills and Behaviors
g. Help developing capability to dress, clean, care for self	1 		2 		0 	d 
h. Help connecting students to community skill- and behavior-related services, supports, or activities (e.g., assistive technology, networking)	1 		2 		0 	d 
i. Speech or language therapy, or communication services	1 		2 		0 	d 
j. Psychological or mental health services or counseling	1 		2 		0 	d 
k. Physical or occupational therapy	1 		2 		0 	d 
l. Orientation and mobility services (to help individuals navigate their environment)	1 		2 		0 	d 
Positive Life Functioning
m. Reader or interpreter, including sign language	1 		2 		0 	d 
n. Audiology services for hearing problems	1 		2 		0 	d 
o. Special transportation because of disability	1 		2 		0 	d 
Other
p. Other accommodations or services (specify)	1 		2 		0 	d 

Probes about “Satisfaction with School Services” Items

How did you answer this question?
What kinds of services were you thinking about when answering these questions?
1. Did you think about services provided by the school only, or did you also think about services received outside of school? Please explain.
2. How easy or hard was it to differentiate between the two (services provided by the school or received outside of school)?Please explain.
Could you describe what would make you rate something very positively, and what would make you rate something as very poor? How easy or hard was it to decide?

IV. Debriefing and Thank You for Participation

Thank the parent for his or her time.

Before we finish, I’d like to hear any other thoughts you have about the topics we just discussed.

Is there anything else you would like to tell me about on these topics or any others?

Is there anything you would like to ask me about what we did today? [Answer parent questions]

Thank you for helping us to improve our parent questionnaire.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-01-28