National Assessment of Educational Progress (NAEP) Background Questions Cognitive Interviews for Students, Teachers, and School Administrators for NAEP 2013

NCES Cognitive, Pilot, and Field Test Studies System

Volume II NAEP Background Cog Labs for 2013

National Assessment of Educational Progress (NAEP) Background Questions Cognitive Interviews for Students, Teachers, and School Administrators for NAEP 2013

OMB: 1850-0803

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1850-0803 can be found here:

Document [docx]

Download: docx | pdf

National Assessment of Educational Progress

Volume II

Cognitive Interview Protocols

Request for Clearance for Cognitive Interview Study of Background Questions for Students, Teachers, and School Administrators

(Reading, Mathematics, Civics, Geography, U.S. History, and Technology and Engineering Literacy)

OMB# 1850-0803

Shape1

September 1, 2011

I. Paperwork Burden Statement 1

II. Welcome Script and Assent/Consent 2

III. Think Aloud Instructions and Practice 5

IV. Generic Probes and Think Aloud Hints for the Interviewer 6

V. Behavioral Coding Hints for Interviewer 7

VI. Data Collection Procedure 7

VII. Debriefing and Thank You 8

VIII. Existing Subject: Items and Item-Specific Probes 9

VIII. Technology and Engineering Literacy (TEL): Items and Item-Specific Probes 91

I. Paperwork Burden Statement

The appropriate Paperwork Reduction Act and OMB confidentiality texts are indicated below. Appropriate sections of this information are included in the consent forms and letters contained in the appendices for Volume I. The entire statements will be included on the front of the materials used in the interviews.

Existing Subject-Specific Questionnaire Items: Grade 4 and 8 Student Cognitive Interviews

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to take no more than 60 minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this research protocol, please write to: U.S. Department of Education, Washington, D.C. 20202-4537.

Your answers may be used only for statistical purposes and may not be disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].

Existing Subject-Specific Questionnaire Items: Grade 12 Student Cognitive Interviews

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to take no more than 30 minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this research protocol, please write to: U.S. Department of Education, Washington, D.C. 20202-4537.

TEL Questionnaire Items: Grade 8 Student Cognitive Interviews

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to take no more than 90 minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this research protocol, please write to: U.S. Department of Education, Washington, D.C. 20202-4537.

Existing Subject-Specific Questionnaire Items: Teacher or School Administrator Cognitive Interviews

TEL Questionnaire Items: School Administrator Cognitive Interviews

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to take no more than 90 minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this research protocol, please write to: U.S. Department of Education, Washington, D.C. 20202-4537.

II. Welcome Script and Assent/Consent

The following scripts should not be read verbatim. The interviewer needs to be familiar enough with the script to introduce the think aloud process in a conversational manner. Text written in italics is suggested content for the interviewer to be thoroughly familiar with in advance. The interviewer should project a warm and reassuring manner toward the participant to develop a friendly rapport and as such should use conversational language throughout.

Student (17 or younger) Participant Script

Hello, my name is <name >.

Thank you for coming here today to help us.

I have some questions that students like you in many different schools will be asked to answer about themselves, their education, and their families. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. Please try to answer the questions to the best of your ability, but there are no wrong answers. Your ideas will help the National Center for Education Statistics (NCES), part of the U.S. Department of Education, make these questions clearer for students like you. This interview is being recorded so researchers can review the recordings later. What you say may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573). The only people who will see anything related to your interview have signed confidentiality agreements.

Before continuing, ask the student if he/she has any questions. After answering questions and giving further explanation, continue with an explanation and demonstration of the think aloud process. If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

Student (18 or older) Participant Script

Hello, my name is <name >.

Thank you for coming here today to help us.

Before continuing, ask the student if he/she has any questions. After answering questions and giving further explanation, continue:

Before we begin, I’d like you to read the following consent form to confirm we have your permission to ask you the questions and write down and record your answers.

Give the participant a copy of the student consent form (see Volume I Appendix C or D). Ask the participant if he/she has any questions. After answering questions and giving further explanation, ask the participant to sign the consent form. Collect the signed consent form from the participant. Then continue with an explanation and demonstration of the think aloud process. If the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

Teacher Participant Script

Hello, my name is <name >.

Thank you for coming here today to help us.

I have some questions that teachers like you in many different schools will be asked to answer about themselves, their instructional practices, and their schools. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. Please try to answer the questions to the best of your ability, but there are no wrong answers. Your ideas will help the National Center for Education Statistics (NCES), part of the U.S. Department of Education, make these questions clearer for teachers like you. This interview is being recorded so researchers can review the recordings later. What you say may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573). The only people who will see anything related to your interview have signed confidentiality agreements.

Before continuing, ask the participant if he/she has any questions. After answering questions and giving further explanation, continue:

Before we begin, I’d like you to read the following consent form to confirm we have your permission to ask you the questions and write down and record your answers.

Give the participant a copy of the consent form (see Volume I Appendix E). Ask the participant if he/she has any questions. After answering questions and giving further explanation, ask the participant to sign the consent form. Collect the signed consent form from the participant. Then continue with an explanation and demonstration of the think aloud process. If the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

School Administrator Participant Script

Hello, my name is <name >.

Thank you for coming here today to help us.

I have some questions that school administrators like you in many different schools will be asked to answer about their school policies and characteristics. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. Please try to answer the questions to the best of your ability, but there are no wrong answers. Your ideas will help the National Center for Education Statistics (NCES), part of the U.S. Department of Education, make these questions clearer for school administrators like you. This interview is being recorded so researchers can review the recordings later. What you say may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573). The only people who will see anything related to your interview have signed confidentiality agreements.

Before continuing, ask the participant if he/she has any questions. After answering questions and giving further explanation, continue:

Before we begin, I’d like you to read the following consent form to confirm we have your permission to ask you the questions and write down and record your answers.

III. Think Aloud Instructions and Practice

The following scripts should not be read verbatim. You need to be familiar enough with the information to introduce the think aloud in a conversational manner.

We want to learn what people think about when answering questions. As you answer each question, I’d like you to think aloud. All that means is to just read the question aloud and then say what you’re thinking as you answer the question. Please tell me whatever is going through your mind as you answer the question.

Let me give you an example about making a piece of toast this morning. If I were thinking aloud while I got the toast ready, it would sound something like this.

“Bread drawer. Muffin or white bread? White bread. Where is the knife? Oh, the sink. Cut. Plug in the toaster oven, turn it on, close it. OK, Medium. Hmm (reading the white bread package). Serving size 1 piece of bread, Calories: 130; Calories from fat 10 – I guess that’s not so bad. Vitamin A 0%, Vitamin C 0%, Calcium 8%, Iron 8% So aren’t there B vitamins in bread? Hmm. (Look up at the corner of the ceiling) Wait! I forgot to put the bread in the toaster! OK, OK, put bread in toaster. Do I want butter or…(Ding) Open the toaster, don’t burn yourself! Use the knife to pull out the bread. Table, sit down, eat.”

The point of the think-aloud is not to describe what you are doing or to tell me what you plan to do. Just say aloud the words that are in your head.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I need to hear times when you are confused (and what is confusing you) and times when you understand what you are doing.

Sometimes it’s helpful to do a short “think aloud” practice. I’m going to ask you a simple question and ask you to think aloud as you decide on your answer. The question is:

“How many different kinds of fruit did you eat yesterday?”

Interviewer: Some participants will be silent after hearing the question. Immediately encourage the participant to say whatever he/she is thinking. You may need to remind the participant that the answer to the question has to be a number.

After you finish a question, or after you’ve gone through all the questions in the survey, I’d like to hear any other comments — good or bad— you have about the question and any changes that would make the question easier to understand or answer.

So, do you have any questions before we start?

Interviewer: After answering questions and giving further explanation, begin the interview with the first item.

IV. Generic Probes and Think Aloud Hints for the Interviewer

If the participant is silent for 5 or more seconds, use the following as a guideline for encouraging the participant to read the question aloud and say his/her thoughts, or to help him/her elaborate the responses.

We’re interested in capturing all the participant’s mental processes while answering these questions. Your goal is to have the participant speak aloud all his or her thoughts while answering the questions. By asking follow-up probes after each item, you will ensure that the data collected are as complete as possible.

If a participant is continually providing short responses or not answering, use “continuers” to encourage the participant to be more descriptive. The goal is to get participants to verbalize their thoughts without “putting words in their mouth.” Don’t ask questions that lead the participant’s response. You have to be as objective and unbiased as possible, but you may offer a verbal “nudge,” such as:

What are you thinking now?
Any other thoughts?
Tell me how you came to pick that answer.

Use your best judgment. If a participant exhibits difficulty reading or understanding an item, probe the participant without biasing the response.

Can you tell me in your own words what that question was asking? [comprehension and interpretation probe]
What does the word [term] mean to you as it’s used in this question? [comprehension and interpretation probe]

Use your best judgment. If a participant appears to have guessed or seems less than certain of the answer he/she has given, probe the participant.

How much would you say you know about [topic]? [confidence probe]
How sure are you of your answer? [confidence probe]
How did you come up with that answer? [recall strategy and bias probe]
Was this easy or hard to answer?” [comprehension and recall probe]
How easy or difficult is it to remember [topic]? [recall probe]
Did you find the list of answer choices easy or hard? [response probe]
How easy or hard was it to choose an answer from that list of choices? [response probe]

Use your best judgment. If a participant appears uncomfortable with a word or answering a question, probe the participant.

The question uses the word [term]. Does that sound OK to you, or would you choose something different? [sensitivity probe]
In general, how do you feel about this question? [sensitivity probe]

Use your best judgment. After testing an item, using the above generic probes to understand why the question as written is not clear, you may reword the question in a way that the participant will understand based on shared interviewer feedback or previous interview experiences. If this is done, be sure to document precisely the reworded question that was tested and test the item using relevant generic probes.

In addition to generic probes, interviewer protocols will include item-specific probes. Not all items have an item-specific probe. Item-specific probes are provided only if the researchers deemed it important to learn more about a cognitive process or issue that might not be raised naturally from the think aloud process or generic probing. If the issue to be inquired through the item-specific probe was addressed, or if time is running short, the item-specific probe does not need to be asked. Use your best judgment.

V. Behavioral Coding Hints for Interviewer

It is important to note when certain behaviors occur that could be evidence of problems with an item. When the following behaviors occur, note the occurrence and when it occurred:

Misreading or rereading a word or words in the instructions, question, or answer choices
Skipping a word or words in the instructions, question, or answer choices
Physical evidence of frustration (e.g., vocal exasperation, pained expression, long pauses)
Misinterpreting skip instruction (e.g., began reading an item that should have been skipped)

VI. Data Collection Procedure

Record your notes into the provided data entry spreadsheet or software program (e.g., Microsoft Excel or Morae). As described in your training session, you are asked to record information to identify problems with respondents’ survey response processes, which includes explicitly coding the occurrence of six different threats to obtaining valid and reliable survey responses.

1. Did the respondent skip over, misread, or reread any word(s)?

2. Did the respondent exhibit difficulty reading an item?

3. Did the respondent exhibit difficulty understanding the question?

4. Did the respondent exhibit difficulty understanding the meaning of particular words or concepts?

5. Did the respondent exhibit difficulty remembering the question or answer choices?

6. Did the respondent exhibit difficulty or discomfort answering the question?

In addition to the that explicit coding to identify problems, you should also record in the data entry sheets descriptions of identified problems, summarization of respondent verbal reports, and respondent responses to item-specific probes.

VII. Debriefing and Thank You

The following scripts should not be read verbatim. You need to be familiar enough with the information to introduce the think aloud in a conversational manner. In addition, only ask questions below that have not already been addressed by the participant.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

[Grade 12 Students, Teachers, and School Administrators Only] If you were not making comments and answering my extra questions, about how long do you think it would take you to fill out this questionnaire?

How did you feel about the questions you were asked?

Do you feel the questions were easy or difficult to understand?

Did you feel the questions were easy or difficult to answer?

[Teachers and School Administrators Only] Did the terminology used in the questionnaire items seem appropriate?

Thank participant for his/her time. Provide gift card (as appropriate) and get signature to acknowledge receipt.

VIII. Existing Subject: Items and Item-Specific Probes

Interviewer: Proceed in the following manner for each of the survey items in this section:

Direct the participant to the item and ask him/her to read the question (and any preceding or following instructions) out loud
Instruct the participant to answer the question and to think aloud while doing so
Record the participant’s comments
Ask generic and item-specific probes, as appropriate, before going on to the next item

Note: Some items are grade specific and will be asked only of students in that grade. The grade, subject, and respondent associated with the items are identified in brackets (e.g., [Grade 4 Reading–Student]) preceding the item.

Note: The introductions and questions being read by students are identified in the boxes on this and the remaining pages. Each item to be tested is presented on its own page.

Existing Subject: Student Questionnaire–Grade 4

[Grade 4 Reading–Student]

Item-Specific Probes:

What does “tutoring” mean to you? [Optional interviewer variant for similar questions throughout this questionnaire: “When you read/see/hear the word/words/phrases ‘xxxxxx,’ what does ‘xxxxxx,’ mean to you”]

When answering the question, what did you think of as examples of “receiving help or tutoring with reading”?

When answering the question, what did you think of as examples of “outside of school or after school”? Did you include help from friends or family members?

Have you ever received help or tutoring with reading after the regular school day, but in your school? [If “yes,”] Did you include that in your answer?

[If respondents select (B), (C), or (D)] Who has helped you with reading outside of your class? Did you ask that person [those people] for help? Did you participate in a regularly scheduled study [tutoring] [help] session? [If “yes,”] Were you the only participant or were you part of a group?

[Grade 4 Reading–Student]

Item-Specific Probes:

What do “Sometimes” and “Often” mean to you? Can you provide examples that would help someone understand how you would decide between those response choices?

What does “vocabulary” mean to you?

When answering the question, what did you think of as examples of discussing vocabulary?

When answering the question, did you include discussions that your teacher had with the whole class? Did you include discussions that your teacher had with you individually?

[Grade 4 Reading–Student]

Item-Specific Probes:

What does “for school this year” mean to you? Does “for school” include reading done in school as well as at home for school? Does “this year” mean school year or calendar year?

What does “computer” mean to you?

Are there any devices that you can hold in one hand that you would think of as a computer? [If yes,] What are the names of those devices? When answering the question, did you include times when you might have used those devices to read a story, article, or book?

[If respondents select (B), (C), or (D)] What types of things do you read on the computer? Do you read any of these types of things (story, article, or book) more often than the other types?

[Grade 4 U.S. History –Student]

Item-Specific Probes:

When answering the question, what does “memorizing material” mean to you? Is this something you do when you study?

How would you answer this question if you memorize material more than once or twice a month (option C) but not as much as once a week (option D)?

[Grade 4 Civics/Geography/U.S. History –Student]

Item-Specific Probes:

What does “social studies” mean to you? What subjects that are taught in your class would you call social studies subjects?

When answering the question, what did you think of as examples of “questions or assignments that involved social studies”?

How would you answer this question if you write short answers more than once or twice this year (option B) but not as much as once or twice a month (option C)? How would you respond if you write short answers more than once or twice a month (option C) but not as much as once a week (option D)?

[Grade 4 Civics/Geography–Student]

NOTE: The inclusion of this trend item in the cognitive lab is intended for comparison with the previous item (VE643503) and to determine whether a definition for “long answers” is needed.

Item-Specific Probes:

What does “long answers” mean to you? How many sentences would you think of as a “long answer”? Would you think a paragraph or more is a “long answer”? Did you include essays and reports in your answer? Did you include tests and quizzes where you are asked to write long answers?

How would you answer this question if you write long answers more than once or twice this year (option B) but not as much as once or twice a month (option C)? How would you respond if write long answers more than once or twice a month (option C) but not as much as once a week (option D)?

[Grade 4 Civics–Student]

Item-Specific Probes:

What do you think of as examples of “community service”?

What do you think of as examples of “volunteering?”

What does “community” mean to you? Do you consider your neighborhood your community? Do you consider the town you live in your community? Do you consider other nearby towns part of your community? Do you consider your school part of your community?

[For those who answer (A) “Yes,”] What types of community service or volunteer work have you done? Have you done volunteer work to help in other parts of the United States? Have you done volunteer work to help people in other countries? [If “yes,”], Did you include this in your answer to the question?

Existing Subject: Student Questionnaire–Grade 8

[Grade 8 Reading–Student]

VB345622 1. Please indicate how much you DISAGREE or AGREE with the following statements about reading and writing. Fill in one oval on each line.
	Strongly disagree	Disagree	Agree	Strongly agree
a. When I read books, I learn a lot.	(A)	(B)	(C)	(D)
b. Reading is one of my favorite activities.	(A)	(B)	(C)	(D)
c. Reading is enjoyable.	(A)	(B)	(C)	(D)

Item-Specific Probes:

What does “reading” mean to you? [Optional interviewer variant for similar questions throughout this questionnaire: “When you read/see/hear the word/words/phrases ‘xxxxxx,’ what does ‘xxxxxx,’ mean to you”]

What kinds of things do you usually read?

If instead of “Reading is enjoyable,” you were asked to indicate how much you disagree or agree with the statement “Reading is fun,” would you have given a different answer. [If “yes,”] Why?

When answering the questions, did you think about reading you do for school, or reading you do not for school, or reading regardless of the reason? [If respondent answers one of the first two] How would you answers change if I asked you to consider all reading regardless of the reason?

When answering the questions, did you also think about times where you might have used computers or computer-like devices for reading? [If “yes,”] How do you prefer to read: on paper, on computer, or some other way?

[Grade 8 Reading–Student]

Item-Specific Probes:

When answering the question, what did you think of as examples of “your regular English/language arts class”? Do you have a “Reading” class that you might not have included in your response?

What does “tutoring” mean to you?

When answering the question, what did you think of as examples of “receiving help or tutoring with reading”?

When answering the question, what did you think of as examples of “outside of your regular English/language arts class”? Did you include help from friends or family members?

Have you ever received help or tutoring with reading after the regular school day, but in your school? [If “yes,”] Did you include that in your answer?

[If respondents select (B), (C), or (D)] Who has helped you with reading outside of your class?

Did you ask that person [those people] for help? Did you participate in a regular study [tutoring] [help] session? [If “yes,”] Were you the only participant or were you part of a group?

[Grade 8 Reading–Student]

NOTE: The inclusion of this item in the cognitive lab is intended for comparison with sub-item (f) in the following item (VC504013).

Item-Specific Probes:

What does “long answers” mean to you? Would you think of a paragraph as a “long answer”?

When answering the question, what did you consider as “tests or assignments that involve reading”?

[Grade 8 Reading–Student]

VC504013 4. In your English/language arts class this year, how often does your class do each of the following? Fill in one oval on each line.
	Never or hardly ever	Once or twice a month	Once or twice a week	Every day or almost every day
a. Read aloud	(A)	(B)	(C)	(D)
b. Read silently	(A)	(B)	(C)	(D)
c. Discuss new or difficult vocabulary	(A)	(B)	(C)	(D)
d. Explain what we have read	(A)	(B)	(C)	(D)
e. Read books we have chosen ourselves	(A)	(B)	(C)	(D)
f. Write a paragraph or more about what we have read	(A)	(B)	(C)	(D)

NOTE: The inclusion of sub-item (f) in this item in the cognitive lab is intended for comparison with the previous item (VB345715).

Item-Specific Probes:

What does “a paragraph or more” mean to you? What do you think of as examples of a paragraph?

When answering item “d. Explain what we have read,” did you include times when you were asked to explain what you read out loud, or in writing, or both? [If respondent answers one of the first two], How would your answers change if I asked you to consider times you explained either out loud or in writing?

When answering item “f. Write a paragraph or more about what we have read,” what did you think of as examples of writing “about what we have read”? Did you consider times when the class took a test? Did you include assignments that you did inside the classroom, as well as any assignments that you were asked to do outside of school?

[Grade 8 Reading–Student]

VC504022 5. In your English/language arts class this year, when reading a story, article, or other passage, how often does your teacher ask you to do the following? Fill in one oval on each line.
	Never or hardly ever	Once or twice a month	Once or twice a week	Every day or almost every day
a. Summarize the passage	(A)	(B)	(C)	(D)
b. Interpret the meaning of the passage	(A)	(B)	(C)	(D)
c. Question the motives or feelings of the characters	(A)	(B)	(C)	(D)
d. Identify the main themes or main ideas of the passage	(A)	(B)	(C)	(D)
e. Evaluate the main arguments or evidence in a persuasive passage	(A)	(B)	(C)	(D)
f. Analyze the author’s organization of information in a passage	(A)	(B)	(C)	(D)
g. Critique the author’s craft or technique	(A)	(B)	(C)	(D)

Item-Specific Probes:

When answering item “e. Evaluate the main arguments or evidence in a persuasive passage,” what did “persuasive passage” mean to you? What did you think of as examples of a “persuasive passage”? How would you evaluate the main arguments or evidence in a persuasive passage?

When answering item “f. Analyze the author’s organization of information in a passage,” what did “author’s organization of information” mean to you? What did you think of as examples of author’s organization of information? How would you analyze an author’s organization of information?

When answering item “g. Critique the author’s craft or technique,” what did “author’s craft or technique” mean to you? What did you think of as examples of author’s craft or technique? How would you critique an author’s craft or technique?

[Grade 8 Reading–Student]

VC504027 6. In your English/language arts class this year, how often do you use a computer to do each of the following? Fill in one oval on each line.
	Never or hardly ever	Once or twice a month	Once or twice a week	Every day or almost every day
a. Learn and practice vocabulary	(A)	(B)	(C)	(D)
b. Practice spelling and grammar	(A)	(B)	(C)	(D)
c. Access reading-related websites (for example, websites with book reviews and lists of recommended books)	(A)	(B)	(C)	(D)
d. Conduct research for reading and writing projects	(A)	(B)	(C)	(D)
e. Read books or articles using a digital media device, such as an e-book reader or tablet computer	(A)	(B)	(C)	(D)

Item-Specific Probes:

What does “computer” mean to you?

Are there any devices that you can hold in one hand that you would consider a computer? [If yes,] What are the names of those devices?

When answering, did you include work you did for class that you did on a computer outside of school such as at home or in a library?

What types of reading-related websites do you go to? There were examples provided for item “c. Access reading-related websites,” but there could be other examples. What would you suggest as other examples?

When answering item “d. Conduct research for reading and writing projects” what does “conduct research” mean to you? What did you think of as examples of conducting research?

When answering “e. Read books or articles using a digital media device ...” what does “digital media device” mean to you? What does “e-book reader” mean to you? What does “tablet computer” mean to you?

[Grade 8 Civics –Student]

VE649049 7. If you found out about a problem in your community that you wanted to do something about, how well do you think you would be able to do each of the following?
	I definitely can’t	I probably can’t	Maybe	I probably can	I definitely can
a. Create a plan to address the problem	(A)	(B)	(C)	(D)	(E)
b. Get other people to care about the problem	(A)	(B)	(C)	(D)	(E)
c. Organize and run a meeting	(A)	(B)	(C)	(D)	(E)
d. Express your views in front of a group of people	(A)	(B)	(C)	(D)	(E)
e. Identify individuals or groups who could help you with the problem	(A)	(B)	(C)	(D)	(E)
f. Write an opinion letter to a local newspaper	(A)	(B)	(C)	(D)	(E)
g. Contact an elected official about the problem	(A)	(B)	(C)	(D)	(E)
h. Organize a petition	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

What does “community” mean to you? Do you consider your neighborhood your community? Do you consider the town you live in your community? Do you consider neighboring towns part of your community? Do you consider your school part of your community?

Can you give examples of problems a community may experience? Do you think including examples in the question would be helpful?

What does item “a. Create a plan to address the problem” mean to you?

What does item “b. Get other people to care about the problem” mean to you?

What does item “e. Identify individuals or groups who could help you with the problem” mean to you?

Would your responses about what you would be able to do be different if you were required or assigned to do tasks as part of your schoolwork versus doing them voluntarily?

For each activity listed “a.”-“h.”, have you ever done this activity?

[For those who respond B, C, D, or E to item (g)], can you give examples of elected officials who you might contact about a community problem?

What does “petition” mean to you?

Existing Subject: Student Questionnaire–Grade 12

[Grade 12 Civics –Student]

VE649049 1. If you found out about a problem in your community that you wanted to do something about, how well do you think you would be able to do each of the following?
	I definitely can’t	I probably can’t	Maybe	I probably can	I definitely can
a. Create a plan to address the problem	(A)	(B)	(C)	(D)	(E)
b. Get other people to care about the problem	(A)	(B)	(C)	(D)	(E)
c. Organize and run a meeting	(A)	(B)	(C)	(D)	(E)
d. Express your views in front of a group of people	(A)	(B)	(C)	(D)	(E)
e. Identify individuals or groups who could help you with the problem	(A)	(B)	(C)	(D)	(E)
f. Write an opinion letter to a local newspaper	(A)	(B)	(C)	(D)	(E)
g. Contact an elected official about the problem	(A)	(B)	(C)	(D)	(E)
h. Organize a petition	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

What does “community” mean to you? [Optional interviewer variant for similar questions throughout this questionnaire: “When you read/see/hear the word/words/phrases ‘xxxxxx,’ what does ‘xxxxxx,’ mean to you”]

Do you consider your neighborhood your community? Do you consider the town you live in your community? Do you consider neighboring towns part of your community? Do you consider your school part of your community?

Can you give examples of problems a community may experience? Do you think including examples in the question would be helpful?

What does item “a. Create a plan to address the problem” mean to you?

What does item “b. Get other people to care about the problem” mean to you?

What does item “e. Identify individuals or groups who could help you with the problem” mean to you?

Would your responses about what you would be able to do be different if you were required or assigned to do tasks as part of your schoolwork versus doing them voluntarily?

For each activity listed “a.”-“h.”, have you ever done this activity?

[For those who respond B, C, D, E to item (g)], can you give examples of elected officials who you might contact about a community problem?

What does “petition” mean to you?

[Grade 12 U.S. History–Student]

Item-Specific Probes:

[For those who answer (A) “yes”], When or in what grade did you take the IB course?

[For those who answer (B) “No” or (C) “I don’t know”], Are you familiar with the International Baccalaureate program?

[For those who answer (C) “I don’t know”], Why did you choose “I don’t know”? [Interviewer note: Determine whether reason is lack of familiarity with the program, uncertainty about whether a course you are taking is IB, do not understand the question, or something else.]

Existing Subject: Teacher Questionnaire–Grade 4

[Grade 4 Reading–Teacher]

Item-Specific Probes:

What does “computer” mean to you? [Optional interviewer variant for similar questions throughout this questionnaire: “When you read/see/hear the word/words/phrases ‘xxxxxx,’ what does ‘xxxxxx,’ mean to you”]

Are there any devices that you can hold in one hand that you would consider a computer? [If “yes,”] What are the names of those devices?

When answering the question, what do you think of as having “reading instruction” or doing “reading activities”?

When answering the question, did you consider time that either you or the students in your class used a computer?

When answering the question, what types of activities did you think of as examples of “using a computer” for reading instruction and/or activities? Can you describe the different ways that you use the computer for reading instruction and/or reading activities?

When answering the question, did you include classroom preparation time in your response?

When answering the question, did you include time spent on these activities in the classroom and somewhere else in the school (e.g., computer lab)?

[Grade 4 Reading–Teacher]

VE658916 2. When you have reading instruction and/or do reading activities, how often do you use the following strategies? Fill in one oval on each line.
	Never or hardly ever	Once or twice a month	Once or twice a week	Every day or almost every day
a. I teach reading as a whole-class activity.	(A)	(B)	(C)	(D)
b. I create same-ability groups.	(A)	(B)	(C)	(D)
c. I create mixed-ability groups.	(A)	(B)	(C)	(D)
d. I use individualized instruction for reading.	(A)	(B)	(C)	(D)
e. Students work independently on an assigned plan or goal.	(A)	(B)	(C)	(D)
f. Students work independently on a goal they choose themselves.	(A)	(B)	(C)	(D)
g. Other (specify): _____________________	(A)	(B)	(C)	(D)

Item-Specific Probes:

Are there any common approaches used by teachers to group students for reading instruction and/or activities that we should include in the list? [If “yes,”] Please describe those approaches and how frequently you expect they occur relative to the other choices on the list.

[Grade 4 Civics/Geography/U.S. History–Teacher]

Item-Specific Probes:

Are there any common approaches used by teachers to group students for social studies instruction and/or activities that we should include in the list? [If “yes,”] Please describe those approaches and how frequently you expect they occur relative to the other choices on the list.

What did you consider for response choice (B) “Ability groups”? Would you group students of equal abilities, a range of different abilities, leadership, or academic ability?

How would you define response choice (D) “Diversity”? What do you think of as “diversity” in instructional groups?

What does response choice (E) “Seating assignments” mean to you?

[Grade 4 Civics/Geography/U.S. History–Teacher]

VB608036 4. How often do you use the following resources to teach social studies in this class? Fill in one oval on each line.
	Never or hardly ever	Once or twice a month	Once or twice a week	Almost every day
a. A social studies textbook	(A)	(B)	(C)	(D)
b. Books, newspapers, magazines, or other periodicals	(A)	(B)	(C)	(D)
c. Primary documents	(A)	(B)	(C)	(D)
d. Quantitative data (such as that on maps, charts, or graphs)	(A)	(B)	(C)	(D)
e. Computer software	(A)	(B)	(C)	(D)
f. Films, videos, or filmstrips	(A)	(B)	(C)	(D)
g. Materials from other subject areas	(A)	(B)	(C)	(D)
h. Online textbooks	(A)	(B)	(C)	(D)
i. Other online resources (e.g., websites that provide primary source materials, podcasts, or exhibits)	(A)	(B)	(C)	(D)

Item-Specific Probes:

Are there any resources that you use for social studies instruction that are not included among the items listed “a” through “i”? [If “yes,”] What other resources do you use?

When answering (e), what did you think of as examples of “Computer software”?

When answering (h), what did you think of as examples of “Online textbooks”? Did you include electronic textbooks that are not necessarily available on the Internet?

When answering (i), what do you consider to be “Other online resources”? Did you find the examples provided to be helpful?

[Grade 4 Civics/Geography/U.S. History–Teacher]

VE229585 5. How often do you use each of the following to assess student progress in social studies? Fill in one oval on each line.
	Never or hardly ever	Once or twice a month	Once or twice a week	Almost every day
a. Tests with multiple-choice, true/false, or matching type questions	(A)	(B)	(C)	(D)
b. Tests with fill-in-the-blank questions	(A)	(B)	(C)	(D)
c. Tests with essays	(A)	(B)	(C)	(D)
d. Paragraph-length written responses about what students have read	(A)	(B)	(C)	(D)
e. Extended essays/papers on assigned topics	(A)	(B)	(C)	(D)
f. Individual projects	(A)	(B)	(C)	(D)
g. Individual presentations	(A)	(B)	(C)	(D)
h. Group projects	(A)	(B)	(C)	(D)
i. Group presentations	(A)	(B)	(C)	(D)

Item-Specific Probes:

Are there any common assessment approaches used by teachers for social studies instruction that are not included in the list?

For items (a) through (c), what did you consider as “tests”? Did you include both teacher-made and standardized tests?

For item (d) “Paragraph-length written responses about what students have read,” did you include such responses given on tests (item [c])?

For item (e) “Extended essays/papers on assigned topics,” did you include such responses given on tests (item [c])?

[Grade 4 Civics/Geography/U.S. History–Teacher]

VC776876 6. To what extent have you emphasized each of the following topics in your social studies class? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. Change in U.S. democracy	(A)	(B)	(C)	(D)
b. People from various cultures	(A)	(B)	(C)	(D)
c. Technological changes	(A)	(B)	(C)	(D)
d. Role of U.S. in the world	(A)	(B)	(C)	(D)
e. Politics and government	(A)	(B)	(C)	(D)
f. Foundations of U.S. democracy	(A)	(B)	(C)	(D)
g. The U.S. Constitution	(A)	(B)	(C)	(D)
h. World affairs	(A)	(B)	(C)	(D)
i. Roles of citizens in U.S. democracy	(A)	(B)	(C)	(D)
j. Space and place (i.e., basic concepts of physical and human geography)	(A)	(B)	(C)	(D)
k. Environment and society (i.e., how people adapt to, depend on, and are affected by the natural environment)	(A)	(B)	(C)	(D)
l. Spatial dynamics and connections (i.e., variation among regions and how people interact across space via communication, transportation, and trade)	(A)	(B)	(C)	(D)

Item-Specific Probes:

What does item “a. Change in U.S. democracy” mean to you?

How helpful were the examples provided for items j, k, and l?

[For those who respond (A) “Not at all” to any of the sub-items], Did you respond “Not at all” because these topics are not required in your fourth-grade curriculum?

For this question, what do “small extent,” “moderate extent,” and “large extent” mean to you? Can you provide the cut-points or examples that would help someone understand how you would distinguish between the response choices? [Interviewer note for similar questions throughout this questionnaire: If respondent doesn’t understand cut-point, explain to them it is the potentially overlapping space between two categories (or in this case response choices such as “small extent” and “moderate extent”). The goal is to understand how the respondent distinguishes between these response choices at the points where they might intersect.]

[Grade 4 Civics/Geography/U.S. History–Teacher]

VB336276 7. When your students work on social studies, to what extent do they use computers to do the following? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. Locate and retrieve social studies information through the Internet	(A)	(B)	(C)	(D)
b. Look up social studies information in electronic reference works (for example, atlases, almanacs, encyclopedias)	(A)	(B)	(C)	(D)
c. Use exploration or simulation software to “experience” history or geography	(A)	(B)	(C)	(D)
d. Write social studies reports using word processing	(A)	(B)	(C)	(D)
e. Create social studies presentations or projects using multimedia software	(A)	(B)	(C)	(D)
f. Organize social studies information using spreadsheets or databases	(A)	(B)	(C)	(D)
g. Create or use maps using a computer or the Internet	(A)	(B)	(C)	(D)

Item-Specific Probes:

What does “computer” mean to you?

Are there any devices that you can hold in one hand that you would consider a computer? [If “yes,”] What are the names of those devices?

When answering the question, did you include time spent on these activities in the classroom and somewhere else in the school (e.g., computer lab)? Did you include time you expect them to have spent on these activities outside of the school (e.g., homework assignments)?

Do students perform the activities listed in the question on computers because they are required to do so or because they choose to use computers, software, and the Internet to complete their assignments?

For item (e), what types of programs or applications did you think about as examples of “multimedia software”?

For item (g), what types of programs or applications did you think about as examples of “using a computer or the Internet” to “create” maps? What types of activities did you think about in reference to using maps?

For this question, what do “small extent,” “moderate extent,” and “large extent” mean to you? Can you provide the cut-points or examples that would help someone understand how you would distinguish between the response choices?

[Grade 4 Civics/Geography/U.S. History–Teacher]

Item-Specific Probes:

For this question, did you include only teaching or instructional time in your response, or did you include lesson preparation time as well?

When considering access to the Internet, did you include Internet access only in the classroom or somewhere else in the school (e.g., a computer lab or the library)?

[For respondents who answer B “Yes, sometimes,”] How did you interpret “sometimes”? Is the access intermittent, or is it available in your school but not your classroom?

[For respondents who answer C “No,”] Do you not have access or is access available but you do not use it for social studies instruction?

[Grade 4 Civics/Geography/U.S. History–Teacher]

Item-Specific Probes:

For this question, what does “computer or other digital technologies” mean to you?

Can you provide some examples of what you consider to be “other digital technologies”? Is there another term that can be used instead of “digital technologies” that might make this question clearer or easier to understand?

For this question, did you include only teaching or instructional time in your response, or did you include lesson preparation time as well?

Existing Subject: Teacher Questionnaire–Grade 8

[Grade 8 Reading–Teacher]

VC976477 1. How often do you ask your students to do the following things? Fill in one oval on each line.
	Never or hardly ever	Once or twice a month	Once or twice a week	Every day or almost every day
a. Summarize what they have read	(A)	(B)	(C)	(D)
b. Interpret the meaning of what they have read	(A)	(B)	(C)	(D)
c. Question the motives or feelings of the characters in what they have read	(A)	(B)	(C)	(D)
d. Identify the main ideas of what they have read	(A)	(B)	(C)	(D)
e. Relate what they have read to their own lives	(A)	(B)	(C)	(D)
f. Evaluate the main arguments or evidence in a persuasive passage they have read	(A)	(B)	(C)	(D)
g. Analyze the author’s organization of information in what they have read	(A)	(B)	(C)	(D)
h. Critique the author’s craft or technique in what they have read	(A)	(B)	(C)	(D)

Item-Specific Probes:

When answering item “f. Evaluate the main arguments or evidence in a persuasive passage they have read,” what did “persuasive passage” mean to you? [Optional interviewer variant for similar questions throughout this questionnaire: “When you read/see/hear the word/words/phrases ‘xxxxxx,’ what does ‘xxxxxx,’ mean to you”]

What did you think of as examples of a “persuasive passage”? What do you expect students to do to evaluate the main arguments or evidence in a persuasive passage?

When answering item “g. Analyze the author’s organization of information in what they have read,” what did “author’s organization of information” mean to you? What did you think of as examples of author’s organization of information? What do you expect students to do to analyze an author’s organization of information?

When answering item “h. Critique the author’s craft or technique in what they have read,” what did “author’s craft or technique” mean to you? What did you think of as examples of author’s craft or technique? What do you expect students to do to critique an author’s craft or technique?

[Grade 8 Reading–Teacher]

Item-Specific Probes:

What does “computer” mean to you?

Are there any devices that you can hold in one hand that you would consider a computer? [If “yes,”] What are the names of those devices?

When answering the question, what do you think of as having “reading instruction” or doing “reading activities”?

When answering the question, what types of activities did you think of as examples of “use a computer” for reading instruction and/or activities? Did you include classroom preparation time in your response?

When answering the question, did you include time spent on these activities in the classroom and somewhere else in the school (e.g., computer lab)?

[Grade 8 Reading–Teacher]

VE658916 3. When you have reading instruction and/or do reading activities, how often do you use the following strategies? Fill in one oval on each line.
	Never or hardly ever	Once or twice a month	Once or twice a week	Every day or almost every day
a. I teach reading as a whole-class activity.	(A)	(B)	(C)	(D)
b. I create same-ability groups.	(A)	(B)	(C)	(D)
c. I create mixed-ability groups.	(A)	(B)	(C)	(D)
d. I use individualized instruction for reading.	(A)	(B)	(C)	(D)
e. Students work independently on an assigned plan or goal.	(A)	(B)	(C)	(D)
f. Students work independently on a goal they choose themselves.	(A)	(B)	(C)	(D)
g. Other (specify): _____________________	(A)	(B)	(C)	(D)

Item-Specific Probes:

[Grade 8 Civics/Geography/U.S. History–Teacher]

Item-Specific Probes:

What did you consider for response choice (B) “Ability groups”? Would you group students of equal abilities, a range of different abilities, leadership, or academic ability?

How would you define response choice (D) “Diversity”? What do you think of as “diversity” in instructional groups?

What does response choice E “Seating assignments” mean to you?

[Grade 8 Civics/Geography/U.S. History–Teacher]

VB608036 5. How often do you use the following resources to teach social studies in this class? Fill in one oval on each line.
	Never or hardly ever	Once or twice a month	Once or twice a week	Almost every day
a. A social studies textbook	(A)	(B)	(C)	(D)
b. Books, newspapers, magazines, or other periodicals	(A)	(B)	(C)	(D)
c. Primary documents	(A)	(B)	(C)	(D)
d. Quantitative data (such as that on maps, charts, or graphs)	(A)	(B)	(C)	(D)
e. Computer software	(A)	(B)	(C)	(D)
f. Films, videos, or filmstrips	(A)	(B)	(C)	(D)
g. Materials from other subject areas	(A)	(B)	(C)	(D)
h. Online textbooks	(A)	(B)	(C)	(D)
i. Other online resources (e.g., websites that provide primary source materials, podcasts, or exhibits)	(A)	(B)	(C)	(D)

Item-Specific Probes:

Are there any resources that you use for social studies instruction that are not included among the items listed “a” through “i”? [If “yes,”] What other resources do you use?

When answering (e), what did you consider as examples of “Computer software”?

When answering (h), what did you consider as examples of “Online textbooks”? Did you include electronic textbooks that are not necessarily available on the Internet?

When answering (i), what do you consider to be “Other online resources”? Did you find the examples provided to be helpful?

[Grade 8 Civics/Geography/U.S. History–Teacher]

VE229585 6. How often do you use each of the following to assess student progress in social studies? Fill in one oval on each line.
	Never or hardly ever	Once or twice a month	Once or twice a week	Almost every day
a. Tests with multiple-choice, true/false, or matching type questions	(A)	(B)	(C)	(D)
b. Tests with fill-in-the-blank questions	(A)	(B)	(C)	(D)
c. Tests with essays	(A)	(B)	(C)	(D)
d. Paragraph-length written responses about what students have read	(A)	(B)	(C)	(D)
e. Extended essays/papers on assigned topics	(A)	(B)	(C)	(D)
f. Individual projects	(A)	(B)	(C)	(D)
g. Individual presentations	(A)	(B)	(C)	(D)
h. Group projects	(A)	(B)	(C)	(D)
i. Group presentations	(A)	(B)	(C)	(D)

Item-Specific Probes:

Are there any common assessment approaches used by teachers for social studies instruction that are not included in the list?

For items (a) through (c), what did you consider as “tests”? Did you include both teacher-made and standardized tests?

For item (d) “Paragraph-length written responses about what students have read,” did you include such responses given on tests (item [c])?

For item (e) “Extended essays/papers on assigned topics,” did you include such responses given on tests (item [c])?

[Grade 8 Civics/Geography/U.S. History–Teacher]

VC776876 7. To what extent have you emphasized each of the following topics in your social studies class? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. Change in U.S. democracy	(A)	(B)	(C)	(D)
b. People from various cultures	(A)	(B)	(C)	(D)
c. Technological changes	(A)	(B)	(C)	(D)
d. Role of U.S. in the world	(A)	(B)	(C)	(D)
e. Politics and government	(A)	(B)	(C)	(D)
f. Foundations of U.S. democracy	(A)	(B)	(C)	(D)
g. The U.S. Constitution	(A)	(B)	(C)	(D)
h. World affairs	(A)	(B)	(C)	(D)
i. Roles of citizens in U.S. democracy	(A)	(B)	(C)	(D)
j. Space and place (i.e., basic concepts of physical and human geography)	(A)	(B)	(C)	(D)
k. Environment and society (i.e., how people adapt to, depend on, and are affected by the natural environment)	(A)	(B)	(C)	(D)
l. Spatial dynamics and connections (i.e., variation among regions and how people interact across space via communication, transportation, and trade)	(A)	(B)	(C)	(D)

Item-Specific Probes:

What does item “a. Change in U.S. democracy” mean to you?

How helpful were the examples provided for items j, k, and l?

[For those who respond (A) “Not at all” to any of the sub-items], Did you respond “Not at all” because these topics are not required in your eighth-grade curriculum?

[Grade 8 Civics/Geography/U.S. History–Teacher]

VB336276 8. When your students work on social studies, to what extent do they use computers to do the following? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. Locate and retrieve social studies information through the Internet	(A)	(B)	(C)	(D)
b. Look up social studies information in electronic reference works (for example, atlases, almanacs, encyclopedias)	(A)	(B)	(C)	(D)
c. Use exploration or simulation software to “experience” history or geography	(A)	(B)	(C)	(D)
d. Write social studies reports using word processing	(A)	(B)	(C)	(D)
e. Create social studies presentations or projects using multimedia software	(A)	(B)	(C)	(D)
f. Organize social studies information using spreadsheets or databases	(A)	(B)	(C)	(D)
g. Create or use maps using a computer or the Internet	(A)	(B)	(C)	(D)

Item-Specific Probes:

What does “computer” mean to you?

Are there any devices that you can hold in one hand that you would consider a computer? [If “yes,”] What are the names of those devices?

For item (e), what types of programs or applications did you think about as examples of “multimedia software”?

[Grade 8 Civics/Geography/U.S. History–Teacher]

Item-Specific Probes:

For this question, did you include only teaching or instructional time in your response, or did you include lesson preparation time as well?

When considering access to the Internet, did you include Internet access only in the classroom or somewhere else in the school (e.g., a computer lab or the library)?

[For respondents who answer B “Yes, sometimes,”] How did you interpret “sometimes”? Is the access intermittent, or is it available in your school but not your classroom?

[For respondents who answer C “No,”] Do you not have access or is access available but you do not use it for social studies instruction?

Existing Subject: School Questionnaire–Grade 4

[Grade 4 Reading–School]

VB380370 1. During the last two years, to what extent have professional development activities offered to teachers in your school focused on the following? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. Use of language arts across the curriculum	(A)	(B)	(C)	(D)
b. Interpreting and analyzing literature	(A)	(B)	(C)	(D)
c. Interpreting and analyzing informational texts	(A)	(B)	(C)	(D)
d. Understanding the process of reading or writing	(A)	(B)	(C)	(D)
e. Use of rubrics to evaluate student work	(A)	(B)	(C)	(D)
f. Instructional strategies for teaching language arts	(A)	(B)	(C)	(D)
g. Effective use of technology in reading instruction	(A)	(B)	(C)	(D)

Item-Specific Probes:

Are these activities or programs occurring more often at your school or at a location outside of your school?

[Grade 4 Reading–School]

Item-Specific Probes:

What does “reading specialist” mean to you? [Optional interviewer variant for similar questions throughout this questionnaire: “When you read/see/hear the word/words/phrases ‘xxxxxx,’ what does ‘xxxxxx,’ mean to you”]

Did you include “literacy coach” as a “reading specialist” when thinking about your response?

[Grade 4 Reading–School]

VE659100 3. To what extent are each of the following a responsibility of the reading specialist(s) available to fourth-grade students at your school? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. Provide reading instruction to students on various topics	(A)	(B)	(C)	(D)
b. Provide reading instruction to students at various ability levels	(A)	(B)	(C)	(D)

Item-Specific Probes:

What does “provide reading instruction” mean to you? Would you consider tutoring and supplemental educational services or only in-class instruction?

[Grade 4 Reading–School]

Item-Specific Probes:

What does “literacy coach” mean to you? Did you include “reading specialist” as a “literacy coach” when thinking about your response?

When answering the question, did you consider [include in your decision] only fourth-grade English/language arts teachers or all fourth-grade teachers at the school?

[Grade 4 Reading–School]

VE659050 5. To what extent are each of the following a responsibility of the literacy coach(es) available to fourth-grade teachers at your school? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. Provide assistance/support to individual teachers about reading content or the teaching of reading	(A)	(B)	(C)	(D)
b. Conduct professional development for groups of teachers about reading content or the teaching of reading	(A)	(B)	(C)	(D)

Item-Specific Probes:

What are some examples of the type of assistance/support to individual teachers that a literacy coach would provide? What proportion of the assistance/support is directed toward content? What proportion is directed toward the teaching of reading?

What are some examples of the type of professional development for groups of teachers that a literacy coach would provide? What proportion of the assistance/support is directed toward content? What proportion is directed toward the teaching of reading?

[Grade 4 Reading–School]

VC190984 6. To what extent is your school’s reading program structured according to the following resources? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. State curriculum standards or frameworks	(A)	(B)	(C)	(D)
b. District curriculum standards or curriculum guides	(A)	(B)	(C)	(D)
c. Results from state/district assessments	(A)	(B)	(C)	(D)
d. In-school curriculum frameworks and standards for learning	(A)	(B)	(C)	(D)
e. Results from school assessments	(A)	(B)	(C)	(D)
f. Recommendations from school reading/language arts department	(A)	(B)	(C)	(D)
g. Discretion of individual teachers	(A)	(B)	(C)	(D)
h. Commercially designed programs	(A)	(B)	(C)	(D)
i. Internet resources	(A)	(B)	(C)	(D)

Item-Specific Probes:

For item “f. Recommendations from school reading/language arts department,” do you consider “school reading/language arts department” to include district departments or only your school’s reading/language arts department?

What does item “h. Commercially designed programs” mean to you? When answering item h, what did you think of as examples of “Commercially designed programs”?

What does item “i. Internet resources” mean to you? When answering item i, what did you think of as examples of “Internet resources”?

[Grade 4 Reading–School]

VE659159 7. In addition to reading teachers, does your school have the following personnel to assist with reading class instruction for fourth-grade students with disabilities (SD)? Fill in one oval on each line.
	Yes	No
a. Special Education teachers (and related service providers)	(A)	(B)
b. Reading specialists or literacy coaches	(A)	(B)
c. Speech pathologists	(A)	(B)
d. Paraprofessionals or teacher aides who are trained to work with students with disabilities	(A)	(B)
e. Paraprofessionals or teacher aides who are not trained to work with students with disabilities	(A)	(B)
f. Other staff not trained to work with students with disabilities	(A)	(B)
g. Parent volunteers	(A)	(B)

Item-Specific Probes:

What are some examples of the type of assistance that would be provided by paraprofessionals or teacher aides who are not trained to work with students with disabilities?

When answering item “f. Other staff not trained to work with students with disabilities,”, what did you think of as examples of “other staff...”? What are some examples of the type of assistance that would be provided by these other staff members?

Is there anyone else who should be included in this list of personnel who provide assistance with reading class instruction for fourth-grade students with disabilities?

[Grade 4 Reading–School]

VE659190 8. In addition to reading teachers, does your school have the following personnel to assist with reading class instruction for fourth-grade English language learners (ELL)? Fill in one oval on each line.
	Yes	No
a. Reading specialists or literacy coaches	(A)	(B)
b. Speech pathologists	(A)	(B)
c. Certified ELL/bilingual education teachers	(A)	(B)
d. Paraprofessionals or teacher aides who are trained to work with students who are ELL	(A)	(B)
e. Paraprofessionals or teacher aides who are not trained to work with students who are ELL	(A)	(B)
f. Other staff not trained to work with students who are ELL	(A)	(B)
g. Parent volunteers	(A)	(B)

Item-Specific Probes:

What are some examples of the type of assistance that would be provided by paraprofessionals or teacher aides who are not trained to work with English language learners?

When answering item “f. Other staff not trained to work with students who are ELL,” what did you think of as examples of “other staff...”? What are some examples of the type of assistance that would be provided by these other staff members?

Is there anyone else who should be included in this list of personnel who provide assistance with reading class instruction for fourth-grade students who are English language learners?

[Grade 4 Mathematics - School]

VE659399

9. In addition to their regular classroom teacher, is there a mathematics resource teacher available (full- or part-time) to fourth-grade students at your school?

(A) Yes, available full-time to fourth-grade students → Go to Question X.

(B) Yes, available part-time to fourth-grade students → Go to Question X.

Item-Specific Probes:

What does “mathematics resource teacher” mean to you? Did you include “mathematics coach” as a “mathematics resource teacher” when thinking about your response?

Would you consider teacher aids or Title 1 teachers to be mathematics resource teachers?

Did you include staff who work in afterschool programs, or only staff who work during regular school hours?

[Grade 4 Mathematics - School]

VE659419

10. To what extent are each of the following a responsibility of the mathematics resource teacher(s) available to fourth-grade students at your school? Fill in one oval on each line.

	Not at all	Small extent	Moderate extent	Large extent
a. Provide mathematics remediation/intervention to individual students	(A)	(B)	(C)	(D)
b. Provide mathematics remediation/intervention to groups of students	(A)	(B)	(C)	(D)
c. Provide mathematics enrichment to individual students	(A)	(B)	(C)	(D)
d. Provide mathematics enrichment to groups of students	(A)	(B)	(C)	(D)

Item-Specific Probes:

What do the terms “remediation/intervention” and “enrichment” mean to you; and, how would you say they were or were not different from each other?

For items a and b, can you identify examples of “providing remediation/intervention”?

For items c and d, can you identify examples of “providing enrichment”?

When answering this question, did you include tutoring and supplemental educational services or only in-class instruction?

[Grade 4 Mathematics - School]

VE659450

11. Is there a mathematics coach available (full- or part-time) to fourth-grade teachers at your school?

(A) Yes, available full-time to fourth-grade teachers → Go to Question X.

(B) Yes, available part-time to fourth-grade teachers → Go to Question X.

Item-Specific Probes:

What does “mathematics coach” mean to you?

Did you include “mathematics resource teacher” as a “mathematics coach” when thinking about your response?

Did you include district-based personnel (such as mathematics specialists, curriculum specialists, lead teachers, or mathematics consultants) or only school-based personnel in your response?

Did you include staff who work in afterschool programs, or only staff who work during regular school hours?

When answering the question, did you consider [include in your decision] only fourth-grade mathematics teachers or all fourth-grade teachers at the school?

[Grade 4 Mathematics - School]

VE659461

12. To what extent are each of the following a responsibility of the mathematics coach(es) available to fourth-grade teachers at your school? Fill in one oval on each line.

	Not at all	Small extent	Moderate extent	Large extent
a. Provide support or assistance about mathematics content	(A)	(B)	(C)	(D)
b. Provide support or assistance about the teaching of mathematics to individual teachers	(A)	(B)	(C)	(D)
c. Conduct professional development about mathematics or the teaching of mathematics for groups of teachers	(A)	(B)	(C)	(D)

Item-Specific Probes:

What does “provide support or assistance” mean to you? When answering “a. Provide support or assistance about mathematics content,” what did you think of as examples of “providing support or assistance”?

When answering “b. Provide support or assistance about the teaching of mathematics to individual teachers,” what did you think of as examples of “providing support or assistance”?

When answering “c. Conduct professional development about mathematics or the teaching of mathematics for groups of teachers,” what did you think of as examples of “conducting professional development for groups of teachers”?

[Grade 4 Mathematics - School]

VC311202

13. To what extent is your school’s mathematics program structured according to the following resources? Fill in one oval on each line.

	Not at all	Small extent	Moderate extent	Large extent
a. State curriculum standards or frameworks	(A)	(B)	(C)	(D)
b. District curriculum standards or curriculum guides	(A)	(B)	(C)	(D)
c. Results from state/district assessments	(A)	(B)	(C)	(D)
d. In-school curriculum frameworks and standards for learning	(A)	(B)	(C)	(D)
e. Results from school assessments	(A)	(B)	(C)	(D)
f. Recommendations from school mathematics department	(A)	(B)	(C)	(D)
g. Discretion of individual teachers	(A)	(B)	(C)	(D)
h. Commercially designed programs	(A)	(B)	(C)	(D)
i. Internet resources	(A)	(B)	(C)	(D)

Item-Specific Probes:

For item “f. Recommendations from school mathematics department,” do you consider “school mathematics department” to include district departments or only your school’s mathematics department?

What does item “h. Commercially designed programs” mean to you? When answering item h, what did you think of as examples of “Commercially designed programs”?

What does “Internet resources” mean to you? When answering item “i. Internet resources,” what did you think of as examples of “Internet resources”?

[Grade 4 Civics/Geography/U.S. History–School]

VB337329 14. In what grade do students in your school typically receive instruction with a primary focus on the following subjects? Fill in all ovals that apply.
	Prior to the 4th grade	In the 4th grade	After the 4th grade	Students do not receive instruction with a primary focus on this subject.
a. State and local history	(A)	(B)	(C)	(D)
b. United States history	(A)	(B)	(C)	(D)
c. Civics/government	(A)	(B)	(C)	(D)
d. Geography	(A)	(B)	(C)	(D)
e. Economics	(A)	(B)	(C)	(D)

Item-Specific Probes:

What does “primary focus” mean to you? Do you consider a standalone or specific subject course to provide a primary focus (e.g., a course focused on state history or a course focused on geography)?

Does your curriculum distinguish between state and local history? How would you respond to “a. State and local history” if students are taught state history or local history, but not both?

[Grade 4 Civics/Geography/U.S. History–School]

VE013622 15. Does your school offer any of the following services for fourth-grade students who fall behind or need extra help in social studies? Fill in one oval on each line.
	Yes	No
a. Tutoring	(A)	(B)
b. Placement of instructional aides in classrooms to provide assistance	(A)	(B)
c. Use of instructional aides for pull-out instruction	(A)	(B)
d. Extra work or homework	(A)	(B)
e. Before- or after-school programs	(A)	(B)
f. Saturday classes	(A)	(B)
g. Summer school program	(A)	(B)
h. Extra help for English language learners	(A)	(B)
i. Remediation using computers or other digital technologies (e.g., online tutorials or support)	(A)	(B)

Item-Specific Probes:

Were there any items among the “a” through “i” list that you did not understand? [If “yes,”] Please describe the problem you had with the item and what you interpreted the item to mean.

For all items, would you include programs that occurred on school grounds as well as those that occurred off school grounds?

How helpful were the examples provided for item (i) about remediation?

[Grade 4 Civics/Geography/U.S. History–School]

VE013637 16. To what extent is your school’s social studies program for fourth-graders structured according to the following resources? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. District curriculum standards or frameworks	(A)	(B)	(C)	(D)
b. State curriculum standards or frameworks	(A)	(B)	(C)	(D)
c. National curriculum standards or frameworks	(A)	(B)	(C)	(D)
d. School-based curriculum standards or frameworks	(A)	(B)	(C)	(D)
e. Results from district or school assessments	(A)	(B)	(C)	(D)
f. Results from state assessments	(A)	(B)	(C)	(D)
g. Results from national assessments	(A)	(B)	(C)	(D)
h. Recommendations from district social studies department	(A)	(B)	(C)	(D)
i. Discretion of individual teachers	(A)	(B)	(C)	(D)
j. Commercially designed programs	(A)	(B)	(C)	(D)
k. Textbooks	(A)	(B)	(C)	(D)

Item-Specific Probes:

What does “Commercially designed programs” mean to you? When answering (j), what did you think of as examples of “Commercially designed programs”? Would you consider a textbook to be a “commercially designed program”?

Existing Subject: School Questionnaire–Grade 8

[Grade 8 Reading–School]

VB380370 1. During the last two years, to what extent have professional development activities offered to teachers in your school focused on the following? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. Use of language arts across the curriculum	(A)	(B)	(C)	(D)
b. Interpreting and analyzing literature	(A)	(B)	(C)	(D)
c. Interpreting and analyzing informational texts	(A)	(B)	(C)	(D)
d. Understanding the process of reading or writing	(A)	(B)	(C)	(D)
e. Use of rubrics to evaluate student work	(A)	(B)	(C)	(D)
f. Instructional strategies for teaching language arts	(A)	(B)	(C)	(D)
g. Effective use of technology in English/language arts instruction	(A)	(B)	(C)	(D)

Item-Specific Probes:

Are these activities or programs occurring more often at your school or at a location outside of your school?

Are there any other activities that should be included in this list of professional development activities or programs related to reading and English/language arts?

[Grade 8 Reading–School]

Item-Specific Probes:

Did you include “literacy coach” as a “reading specialist” when thinking about your response?

[Grade 8 Reading–School]

VE659230 3. To what extent are each of the following a responsibility of the reading specialist(s) available to eighth-grade students at your school? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. Provide reading instruction to students on various topics	(A)	(B)	(C)	(D)
b. Provide reading instruction to students at various ability levels	(A)	(B)	(C)	(D)

Item-Specific Probes:

What does “provide reading instruction” mean to you? Would you consider tutoring and supplemental educational services or only in-class instruction?

[Grade 8 Reading–School]

Item-Specific Probes:

What does “literacy coach” mean to you?

Did you include “reading specialist” as a “literacy coach” when thinking about your response?

[Grade 8 Reading–School]

VE659066 5. To what extent are each of the following a responsibility of the literacy coach(es) available to eighth-grade teachers at your school? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. Provide assistance/support to individual teachers about English/language arts content or the teaching of English/language arts	(A)	(B)	(C)	(D)
b. Conduct professional development for groups of teachers about English/language arts content or the teaching of English/language arts	(A)	(B)	(C)	(D)

Item-Specific Probes:

[Grade 8 Reading–School]

VC191175 6. To what extent is your school’s English/language arts program structured according to the following resources? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. State curriculum standards or frameworks	(A)	(B)	(C)	(D)
b. District curriculum standards or curriculum guides	(A)	(B)	(C)	(D)
c. Results from state/district assessments	(A)	(B)	(C)	(D)
d. In-school curriculum frameworks and standards for learning	(A)	(B)	(C)	(D)
e. Results from school assessments	(A)	(B)	(C)	(D)
f. Recommendations from school reading/language arts department	(A)	(B)	(C)	(D)
g. Discretion of individual teachers	(A)	(B)	(C)	(D)
h. Commercially designed programs	(A)	(B)	(C)	(D)
i. Internet resources	(A)	(B)	(C)	(D)

Item-Specific Probes:

What does item “h. Commercially designed programs” mean to you? When answering item h, what did you think of as examples of “Commercially designed programs”?

What does item “i. Internet resources” mean to you? When answering item i, what did you think of as examples of “Internet resources”?

[Grade 8 Reading–School]

VE659259 7. In addition to reading teachers, does your school have the following personnel to assist with reading class instruction for eighth-grade students with disabilities (SD)? Fill in one oval on each line.
	Yes	No
a. Special Education teachers (and related service providers)	(A)	(B)
b. Reading specialists or literacy coaches	(A)	(B)
c. Speech pathologists	(A)	(B)
d. Paraprofessionals or teacher aides who are trained to work with students with disabilities	(A)	(B)
e. Paraprofessionals or teacher aides who are not trained to work with students with disabilities	(A)	(B)
f. Other staff not trained to work with students with disabilities	(A)	(B)
g. Parent volunteers	(A)	(B)

Item-Specific Probes:

What are some examples of the type of assistance that would be provided by paraprofessionals or teacher aides who are not trained to work with eighth-grade students with disabilities?

Is there anyone else who should be included in this list of personnel who provide assistance with reading class instruction for eighth-grade students with disabilities?

[Grade 8 Reading–School]

VE659284 8. In addition to reading teachers, does your school have the following personnel to assist with reading class instruction for eighth-grade English language learners (ELL)? Fill in one oval on each line.
	Yes	No
a. Reading specialists or literacy coaches	(A)	(B)
b. Speech pathologists	(A)	(B)
c. Certified ELL/bilingual education teachers	(A)	(B)
d. Paraprofessionals or teacher aides who are trained to work with students who are ELL	(A)	(B)
e. Paraprofessionals or teacher aides who are not trained to work with students who are ELL	(A)	(B)
f. Other staff not trained to work with students who are ELL	(A)	(B)
g. Parent volunteers	(A)	(B)

Item-Specific Probes:

What are some examples of the type of assistance that would be provided by paraprofessionals or teacher aides who are not trained to work with eighth-grade English language learners?

Is there anyone else who should be included in this list of personnel who provide assistance with reading class instruction for eighth-grade students who are English language learners?

[Grade 8 Mathematics - School]

VE659418

9. In addition to their regular classroom teacher, is there a mathematics resource teacher available (full- or part-time) to eighth-grade students at your school?

(A) Yes, available full-time to eighth-grade students → Go to Question X.

(B) Yes, available part-time to eighth-grade students → Go to Question X.

Item-Specific Probes:

What does “mathematics resource teacher” mean to you?

Did you include “mathematics coach” as a “mathematics resource teacher” when thinking about your response?

Would you consider teacher aids or Title 1 teachers to be mathematics resource teachers?

Did you include staff who work in afterschool programs, or only staff who work during regular school hours.

[Grade 8 Mathematics - School]

VE659430

10. To what extent are each of the following a responsibility of the mathematics resource teacher(s) available to eighth-grade students at your school? Fill in one oval on each line.

	Not at all	Small extent	Moderate extent	Large extent
a. Provide mathematics remediation/intervention to individual students	(A)	(B)	(C)	(D)
b. Provide mathematics remediation/intervention to groups of students	(A)	(B)	(C)	(D)
c. Provide mathematics enrichment to individual students	(A)	(B)	(C)	(D)
d. Provide mathematics enrichment to groups of students	(A)	(B)	(C)	(D)

Item-Specific Probes:

What do the terms “remediation/intervention” and “enrichment” mean to you; and, how would you say they were or were not different from each other?

When answering this question, would you consider tutoring and supplemental educational services or only in-class instruction?

For items a and b, can you identify examples of “providing remediation/intervention”?

For items c and d, can you identify examples of “providing enrichment”?

[Grade 8 Mathematics - School]

VE659454

11. Is there a mathematics coach available (full- or part-time) to eighth-grade teachers at your school?

(A) Yes, available full-time to eighth-grade teachers → Go to Question X.

(B) Yes, available part-time to eighth-grade teachers → Go to Question X.

Item-Specific Probes:

What does “mathematics coach” mean to you?

Did you include “mathematics resource teacher” as a “mathematics coach” when thinking about your response?

Did you include district-based personnel (such as mathematics specialists, curriculum specialists, lead teachers, or mathematics consultants) or only school-based personnel in your response?

Did you include staff who work in afterschool programs, or only staff who work during regular school hours.

[Grade 8 Mathematics - School]

VE659468

12. To what extent are each of the following a responsibility of the mathematics coach(es) available to eighth-grade teachers at your school? Fill in one oval on each line.

	Not at all	Small extent	Moderate extent	Large extent
a. Provide support or assistance about mathematics content	(A)	(B)	(C)	(D)
b. Provide support or assistance about the teaching of mathematics to individual teachers	(A)	(B)	(C)	(D)
c. Conduct professional development about mathematics or the teaching of mathematics for groups of teachers	(A)	(B)	(C)	(D)

Item-Specific Probes:

When answering “b. Provide support or assistance about the teaching of mathematics to individual teachers,” what did you think of as examples of “providing support or assistance”?

[Grade 8 Mathematics - School]

VC311202

13.To what extent is your school’s mathematics program structured according to the following resources? Fill in one oval on each line.

	Not at all	Small extent	Moderate extent	Large extent
a. State curriculum standards or frameworks	(A)	(B)	(C)	(D)
b. District curriculum standards or curriculum guides	(A)	(B)	(C)	(D)
c. Results from state/district assessments	(A)	(B)	(C)	(D)
d. In-school curriculum frameworks and standards for learning	(A)	(B)	(C)	(D)
e. Results from school assessments	(A)	(B)	(C)	(D)
f. Recommendations from school mathematics department	(A)	(B)	(C)	(D)
g. Discretion of individual teachers	(A)	(B)	(C)	(D)
h. Commercially designed programs	(A)	(B)	(C)	(D)
i. Internet resources	(A)	(B)	(C)	(D)

Item-Specific Probes:

What does item “h. Commercially designed programs” mean to you? When answering item h, what did you think of as examples of “Commercially designed programs”?

What does “Internet resources” mean to you? When answering item “i. Internet resources,” what did you think of as examples of “Internet resources”?

[Grade 8 Civics/Geography/U.S. History–School]

VB338391 14. At what grade do students in your school typically take the following classes? Fill in all ovals that apply.
	6th grade	7th grade	8th grade	9th grade	This class is not offered in my school.
a. A class primarily focused on U.S. history	(A)	(B)	(C)	(D)	(E)
b. A class primarily focused on geography	(A)	(B)	(C)	(D)	(E)
c. A class primarily focused on civics or government	(A)	(B)	(C)	(D)	(E)
d. A class primarily focused on economics	(A)	(B)	(C)	(D)	(E)
e. A class primarily focused on world history or world cultures	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

What does “primarily focused” mean to you? Do you consider a standalone or specific subject course to provide a primary focus (e.g., a course on world history or a course on geography?

[Grade 8 Civics/Geography/U.S. History–School]

VE015686 15. Does your school offer any of the following services for eighth-grade students who fall behind or need extra help in social studies (e.g., U.S. history, civics, or geography)? Fill in one oval on each line.
	Yes	No
a. Tutoring	(A)	(B)
b. Placement of instructional aides in classrooms to provide assistance	(A)	(B)
c. Use of instructional aides for pull-out instruction	(A)	(B)
d. Extra work or homework	(A)	(B)
e. Before- or after-school programs	(A)	(B)
f. Saturday classes	(A)	(B)
g. Summer school program	(A)	(B)
h. Extra help for English language learners	(A)	(B)
i. Remediation using computers or other digital technologies (e.g., online tutorials or support)	(A)	(B)

Item-Specific Probes:

For all items, would you include programs that occurred on school grounds as well as those that occurred off school grounds?

How helpful were the examples provided for item (i) about remediation?

[Grade 8 Civics/Geography/U.S. History–School]

VE015700 16. To what extent is your school’s social studies program for eighth-graders (e.g., U.S. history, civics, or geography) structured according to the following resources? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. District curriculum standards or frameworks	(A)	(B)	(C)	(D)
b. State curriculum standards or frameworks	(A)	(B)	(C)	(D)
c. National curriculum standards or frameworks	(A)	(B)	(C)	(D)
d. School-based curriculum standards or frameworks	(A)	(B)	(C)	(D)
e. Results from district or school assessments	(A)	(B)	(C)	(D)
f. Results from state assessments	(A)	(B)	(C)	(D)
g. Results from national assessments	(A)	(B)	(C)	(D)
h. Recommendations from district social studies department	(A)	(B)	(C)	(D)
i. Discretion of individual teachers	(A)	(B)	(C)	(D)
j. Commercially designed programs	(A)	(B)	(C)	(D)
k. Textbooks	(A)	(B)	(C)	(D)

Item-Specific Probes:

Existing Subject: School Questionnaire–Grade 12

[Grade 12 Civics/Geography/U.S. History–School]

VB338399 1. At what grade do students in your school typically take the following courses? Fill in all ovals that apply.
	9th grade	10th grade	11th grade	12th grade	This course is not offered in my school.
a. A course primarily focused on U.S. history	(A)	(B)	(C)	(D)	(E)
b. A course primarily focused on geography	(A)	(B)	(C)	(D)	(E)
c. A course primarily focused on civics or government	(A)	(B)	(C)	(D)	(E)
d. A course primarily focused on economics	(A)	(B)	(C)	(D)	(E)
e. A course primarily focused on world history or world cultures	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

What does “primarily focused” mean to you? [Optional interviewer variant for similar questions throughout this questionnaire: “When you read/see/hear the word/words/phrases ‘xxxxxx,’ what does ‘xxxxxx,’ mean to you”]

Do you consider a standalone or specific subject course to provide a primary focus (e.g., a course on world history or a course on geography)?

[Grade 12 Civics/Geography/U.S. History–School]

VE015741 2. Does your school offer any of the following services for twelfth-grade students who fall behind or need extra help in social studies (e.g., U.S. history, civics, or geography)? Fill in one oval on each line.
	Yes	No
a. Tutoring	(A)	(B)
b. Placement of instructional aides in classrooms to provide assistance	(A)	(B)
c. Use of instructional aides for pull-out instruction	(A)	(B)
d. Extra work or homework	(A)	(B)
e. Before- or after-school programs	(A)	(B)
f. Saturday classes	(A)	(B)
g. Summer school program	(A)	(B)
h. Extra help for English language learners	(A)	(B)
i. Remediation using computers or other digital technologies (e.g., online tutorials or support)	(A)	(B)

Item-Specific Probes:

For all items would you include programs that occurred on school grounds as well as those that occurred off school grounds?

How helpful were the examples provided for item (i) about remediation?

[Grade 12 Civics/Geography/U.S. History–School]

VE015755 3. To what extent is your school’s social studies program for twelfth-graders (e.g., U.S. history, civics, or geography) structured according to the following resources? Fill in one oval on each line.
	Not at all	Small extent	Moderate extent	Large extent
a. District curriculum standards or frameworks	(A)	(B)	(C)	(D)
b. State curriculum standards or frameworks	(A)	(B)	(C)	(D)
c. National curriculum standards or frameworks	(A)	(B)	(C)	(D)
d. School-based curriculum standards or frameworks	(A)	(B)	(C)	(D)
e. Results from district or school assessments	(A)	(B)	(C)	(D)
f. Results from state assessments	(A)	(B)	(C)	(D)
g. Results from national assessments	(A)	(B)	(C)	(D)
h. Recommendations from district social studies department	(A)	(B)	(C)	(D)
i. Discretion of individual teachers	(A)	(B)	(C)	(D)
j. Commercially designed programs	(A)	(B)	(C)	(D)
k. Textbooks	(A)	(B)	(C)	(D)

Item-Specific Probes:

[Grade 12 Civics/Geography/U.S. History–School]

VE230108 4. Of the years required for social studies, how many years (or Carnegie-unit equivalents) of coursework does your state, district, or school require in each of the following subjects? Fill in one oval on each line.
	None	One-half year	One year	Two years	Three years	Four years
a. A course primarily focused on U.S. history	(A)	(B)	(C)	(D)	(E)	(F)
b. A course primarily focused on geography	(A)	(B)	(C)	(D)	(E)	(F)
c. A course primarily focused on civics or government	(A)	(B)	(C)	(D)	(E)	(F)
d. A course primarily focused on economics	(A)	(B)	(C)	(D)	(E)	(F)
e. A course primarily focused on world history or world cultures	(A)	(B)	(C)	(D)	(E)	(F)

Item-Specific Probes:

What does “primarily focused” mean to you?

Would it be easier to answer this question if it asked for number of credits hours or units instead of years?

Is it possible that a school’s requirements would be greater than a district/state, or that a district’s requirements would be greater than the state? If so, how would you answer this question?

[Grade 12 Civics/Geography/U.S. History–School]

VB607901 5. How many students in your school are currently enrolled in Advanced Placement courses in each of the following subjects? Fill in one oval on each line.
	None	1–5	6–10	11–25	26–50	More than 50
a. Human Geography	(A)	(B)	(C)	(D)	(E)	(F)
b. U. S. Government and Politics	(A)	(B)	(C)	(D)	(E)	(F)
c. Comparative Government and Politics	(A)	(B)	(C)	(D)	(E)	(F)
d. U. S. History	(A)	(B)	(C)	(D)	(E)	(F)
e. World History	(A)	(B)	(C)	(D)	(E)	(F)
f. European History	(A)	(B)	(C)	(D)	(E)	(F)
g. Macroeconomics	(A)	(B)	(C)	(D)	(E)	(F)
h. Microeconomics	(A)	(B)	(C)	(D)	(E)	(F)

Item-Specific Probes:

Did you include all students in your school enrolled in AP courses or only twelfth-grade students enrolled in AP courses.

Did you include students who may have taken a semester course during the year in your calculation of students who are currently enrolled in these classes or only those who are currently enrolled during this current academic term?

How sure are you of your answer?

[For those who answered (A) “None” to any of the items], Does your school offer the course? [If yes], Was student enrollment too low for the course to currently be offered?

[Grade 12 Civics/Geography/U.S. History–School]

VE015764 6. Does your school offer any of the following courses for twelfth-grade students? Fill in one oval on each line.
	Yes	No
a. Online U.S. history courses for credit	(A)	(B)
b. Online civics/government courses for credit	(A)	(B)
c. Online geography courses for credit	(A)	(B)
d. Online economics courses for credit	(A)	(B)
e. Online world history or world cultures courses for credit	(A)	(B)

Item-Specific Probes:

[For those who respond (A) “Yes” to any of the items], Is (are) the courses available to all interested twelfth-grade students? [If no], For which students are the courses available (e.g., top-track students, those have completed specific prerequisites, etc.)?

What does “online course” mean to you?

Would you consider a “distance learning course” to mean the same thing as an “online course”? [If “no”], How do they differ from one another?

[If respondent selects (A) “Yes,” for any of the items in the list “a” – “e”,] Are these courses for high school credit or for college credit hours or units?

[Grade 12 Civics/Geography/U.S. History–School]

VE664139

7. Approximately what percentage of students in this year’s graduating class has enrolled in a history course for dual credit taught at the following locations?

(Dual credit is defined for this survey as a course or program where high school students can earn both high school and postsecondary credits for the same course. Postsecondary institutions include public and private 2- or 4-year colleges or universities, community colleges, and technical or vocational schools. The dual credit options must either be legislated by your state or have an articulated or formal written agreement between your school and a postsecondary institution. Do NOT include Advanced Placement and International Baccalaureate^® courses). Fill in one oval on each line.

	None	1-5%	6-10%	11-25%	26-50%	51-75%	Over 75%
a. On your high school campus	(A)	(B)	(C)	(D)	(E)	(F)	(G)
b. On a postsecondary campus	(A)	(B)	(C)	(D)	(E)	(F)	(G)
c. Through distance learning (Internet)	(A)	(B)	(C)	(D)	(E)	(F)	(G)
d. Other (specify): ___________	(A)	(B)	(C)	(D)	(E)	(F)	(G)

Item-Specific Probes:

[For those who answer (B), (C), (D), (E), (F), or (G) to any of the items,] What types of history courses are students taking for dual credit (e.g., U.S. history. world history, European history)?

How are you arriving at this percentage estimation? If you are calculating a ratio, what numbers are you using in the numerator and in the denominator?

How sure are you of your answers?

Did you find the parenthetical explanation for dual credit helpful or not?

Approximately, what percentage of this year’s graduating class is on a college preparatory or honors track? What percentage is on a technical or vocational track?

How is dual credit given in these courses? Which institutions (your high school, two-year colleges, four-year colleges, etc.) are granting the credit?

VIII. Technology and Engineering Literacy (TEL): Items and Item-Specific Probes

Interviewer: Proceed in the following manner for each of the survey items in this section:

Direct the participant to the item and ask him/her to read the question (and any preceding or following instructions) out loud
Instruct the participant to answer the question and to think aloud while doing so
Record the participant’s comments
Ask generic and item-specific probes, as appropriate, before going on to the next item

Note: The introductions and questions being read by students are identified in the boxes on this and the remaining pages. Each item to be tested is presented on its own page.

TEL: Student Questionnaire–Grade 8

[Grade 8 TEL–Student]

Item-Specific Probes:

Before we get to the first question, please tell me in your own words what this introduction is telling you.

Before today, have you ever heard of technology described in this way? [If “yes,”] Where have you heard technology described in this way?”

Does the way technology is described change how you think about what technology means? [If “yes,”] How did it change how you think about or define technology?

Before today, have you ever heard of engineering described in this way? [If “yes,”] Where have you heard technology described in this way?”

Does the way engineering is described change how you think about what engineering means? [If “yes,”] How did it change how you think about or define engineering?

What does “natural environment” mean to you? [Optional interviewer variant for similar questions throughout this questionnaire: “When you read/see/hear the word/words/phrases ‘xxxxxx’ what does it mean to you”]

What does “solve problems” mean to you?

[Grade 8 TEL–Student]

Item-Specific Probes:

What does “courses” mean to you?

Do you think the words “courses in school” means something different than “classes in school” or “subjects in school” [If “yes,”] How would you describe the different meanings? How would you answer the question differently if “classes” or “subjects” had been used in the question instead of “courses”?

Would you be more likely to use the word “courses,” “classes,” or “subjects” to describe the topics or areas that you study in school?

What does “industrial technology” mean to you? When answering the question, what do you think are examples of “industrial technology” courses?

[For those who answer (A) “Industrial technology,”] What was the name of the industrial technology course or courses that you took? What did you learn about in the course(s)?

When answering the question, what do you think are examples of “engineering” courses?

[For those who answer (C) “Any course that involves learning to use, program, or build computers,”] What was the name of the computer course or courses that you took? What did you learn about in the course(s)?

What does “technology-related” mean to you? When answering the question, what do you think are examples of “technology-related” courses?

[For those who answer (D) “Any other technology-related course,”] What was the name of the technology-related course or courses that you took? What did you learn about in those courses?

[Grade 8 TEL–Student]

Item-Specific Probes:

When you read the word technology in this question, did you think about technology topics using the definition in the introduction or your own definition? [Interviewer: confirm whether the student remembers the definition of technology that was provided in the introduction.]

When you saw the word engineering in this question, did you think about engineering topics using the definition in the introduction or your own definition? [Interviewer: confirm whether the student remembers the definition of engineering that was provided in the introduction.]

What do you think are examples of “technology or engineering topics”?

[For those who answer (A), (B), or (C),], What kind of technology or engineering topics are examples of what you learned?

“Have you ever studied technology or engineering topics in any course in school that was not listed in the question? [If “yes,”] In what courses did you study these topics and what are some examples of the technology or engineering topics that you studied?” [Interviewer: If student mentioned any courses in the preceding question (Question 1) related to technology and engineering courses, then ask the student not to include those when responding to this probe. Also, please note in record to refer also to previous question notes for more information for analysis.]

[Grade 8 TEL–Student]

VE639151 3. How often do you do the following? Select one circle in each row.
	Never or almost never	A few times a year	Once or twice a month	Once or twice a week	Every day or almost every day
a. E-mail other students about schoolwork	(A)	(B)	(C)	(D)	(E)
b. Send text messages to other students about schoolwork	(A)	(B)	(C)	(D)	(E)
c. E-mail teachers about schoolwork or to submit homework	(A)	(B)	(C)	(D)	(E)
d. Download, upload, or browse material from your school’s website	(A)	(B)	(C)	(D)	(E)
e. Check your school’s website for announcements	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

[For those who answer (B), (C), (D), or (E) to “a. E-mail students…”, “b. Send text…”, or “c. E-mail teachers”] When you are e-mailing or texting other students or teachers about schoolwork, what are you discussing?

[For those who answer (B), (C), (D), or (E) to any activity in the list] Are you doing it because you want to or are you required to do that activity [any of those activities]? [If “yes,”] Is it required by your teacher or the school or someone else?

Do you ever send instant messages to other students about school work? [If “yes,”] Did you include that in your response to “a. E-mail other students about schoolwork,” or “b. Send text messages to other students about schoolwork,” or not at all because you think instant messages are different from e-mail or text messages?

Do you ever use social networks such as Facebook or Twitter to discuss schoolwork with other students? [If “yes,”] Did you include that in your response to “a. E-mail other students about schoolwork,” or “b. Send text messages to other students about schoolwork,” or not at all because you think communicating via social networks is different from e-mail or text messages?

Aside from downloading, uploading, or browsing material from your school’s website,” are there other reasons to go to your school’s website? [If “yes,”] What were they and did you include those reasons when giving your answer to “d. Download, upload, or browse material from your school’s website”?

[Grade 8 TEL–Student]

VE639172 4. How often do you do the following? Select one circle in each row.
	Never or almost never	A few times a year	Once or twice a month	Several times a month	At least once a week
a. Participate in clubs, camps, or competitions about technology or engineering	(A)	(B)	(C)	(D)	(E)
b. Go to museums or events to learn about technology or engineering	(A)	(B)	(C)	(D)	(E)
c. Edit digital photographs or other graphic images	(A)	(B)	(C)	(D)	(E)
d. Work with technological hobby kits or resources (for example, robotics, remote controlled vehicles, model cars)	(A)	(B)	(C)	(D)	(E)
e. Work in a shop or garage with industrial technologies (for example, auto mechanics, machining, metalworking, construction, woodworking)	(A)	(B)	(C)	(D)	(E)
f. Work with drafting or design tools (for example, computer aided design [CAD], systems analysis)	(A)	(B)	(C)	(D)	(E)
g. Take online courses to learn more about technology or engineering	(A)	(B)	(C)	(D)	(E)
h. Watch television shows or videos to learn more about technology or engineering	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

[Interviewer: Due to length of set member list, after student responds to last item, confirm that they recall the question stem (i.e., what the main part of the question is asking) based on their think-aloud response or through the following direct probe:] Without looking back at the question, do you remember what the main question was?

This question asked you about some activities you might have done in the past. What did you think about the number of activities that were asked about? What did you think about the length of the sentences used to describe the activities?

What does “several times a month” mean to you? When answering this question, how many times is “several”?

When you saw the words “technology or engineering” in this question, how were you defining technology? And, how were you defining engineering? [This probe is slightly different, yet very similar, to some earlier and later probes. The goal of these probes is to examine throughout the course of the survey whether students maintain their own definition, use the definition provided, or use a hybrid; also, whether the definition used changes or remains consistent throughout the survey.]

Were any of the descriptions of activities difficult to understand? Examples were provided for some activities. What did you think of the examples? Did the examples help you better understand the meaning of the activity? Did your understanding of any activity change after seeing the examples? [If “yes,”] How? Are there any other examples that you would include to better describe the listed activities?

What does “f. Work with drafting or design tools” mean to you?

What does “online course” mean to you?

Do you ever watch television shows or videos about technology or engineering simply because they are entertaining or interesting? [If “yes,”] Did you include the times you did that when giving your answer? [If “no,”] Why not? [Interviewer: Note whether respondent is clearly distinguishing “to learn about” when thinking through their answer.

[For those who answer (B), (C), (D), or (E) to “h. Watch television shows or videos…”] What are some examples of the kind of shows that you watch?

[Grade 8 TEL–Student]

VE639065 5. Are the following items available for you to use at home? Select one circle in each row.
	Yes, and I use it.	Yes, but I do not use it.	No
a. Desktop computer	(A)	(B)	(C)
b. Portable laptop or notebook computer	(A)	(B)	(C)
c. Tablet computer	(A)	(B)	(C)
d. E-book reader or e-reader	(A)	(B)	(C)
e. Printer	(A)	(B)	(C)
f. Dial-up Internet connection through a modem and telephone line	(A)	(B)	(C)
g. Broadband or high-speed Internet connection	(A)	(B)	(C)
h. Home video game console	(A)	(B)	(C)
i. Handheld video game console	(A)	(B)	(C)
j. Portable music or video player	(A)	(B)	(C)
k. USB flash drive	(A)	(B)	(C)

Item-Specific Probes:

How often do you have to use an item before you would answer “Yes, and I use it”? Did you or would you answer “Yes, and I use it” if you used an item only once ever?

Are there any other things that you can think of that are related to computers, that you use at home, but are not included on this list? [If “yes,”] Can you tell me what they are? Did you think about any of them when answered any of the questions?

How do you know when someone has a dial-up Internet connection or a broadband Internet connection?

Are there any items in the list that you do not know what they were? [If “yes,”] Which one(s)?

For those who answered (C) “No” to “g. Broadband or high-speed Internet connection,”] Do you have wireless Internet in your home?

For those who answered (A) or (B) to “j. Portable music or video player,”] What kind of portable music or video player do you have?

What does “USB flash drive” mean to you? Would you use any other name to describe it? What does “thumb drive” mean to you?

[Grade 8 TEL–Student]

VE681474 6. Are the following items available for you to use outside of your home? Select one circle in each row.
	Yes, and I use it.	Yes, but I do not use it.	No
a. Desktop computer	(A)	(B)	(C)
b. Portable laptop or notebook computer	(A)	(B)	(C)
c. Tablet computer	(A)	(B)	(C)
d. E-book reader or e-reader	(A)	(B)	(C)
e. Printer	(A)	(B)	(C)
f. Dial-up Internet connection through a modem and telephone line	(A)	(B)	(C)
g. Broadband or high-speed Internet connection	(A)	(B)	(C)
h. Home video game console	(A)	(B)	(C)
i. Handheld video game console	(A)	(B)	(C)
j. Portable music or video player	(A)	(B)	(C)
k. USB flash drive	(A)	(B)	(C)

Item-Specific Probes:

Without looking back at the question, can you tell me any ways this question is different from the last question? [Interviewer: Note whether student recalls that this question is asking about “outside of your home”]

In answering the question, what did you think of as “outside of your home”?

In answering the question, were you including things that you can use or borrow that other people own? Or, were you including only things that belong to you or your family that you can take or use outside of your home?

In answering the question, did you consider the availability of these items to use or borrow when you are at another relative’s house? What about at a friend’s house? What about at the library? What about at school?

[Grade 8 TEL–Student]

Item-Specific Probes:

[Grade 8 TEL–Student]

VE681533 8. Different cell phones have different features. Can you use your cell phone to do the following? Select one circle in each row.
	Yes, and I use it.	Yes, but I do not use it.	No
a. Send or receive text messages	(A)	(B)	(C)
b. Send or receive e-mail	(A)	(B)	(C)
c. Take a picture	(A)	(B)	(C)
d. Send or receive pictures	(A)	(B)	(C)
e. Play music	(A)	(B)	(C)
f. Send or receive an instant message (or IM)	(A)	(B)	(C)
g. Record a video	(A)	(B)	(C)
h. Send or receive a video	(A)	(B)	(C)
i. Play a game	(A)	(B)	(C)
j. Use a social networking site	(A)	(B)	(C)
k. Use an application (or app)	(A)	(B)	(C)

Item-Specific Probes:

What does “use an application (or app)” mean to you?

[For those who answer (B) “Yes, but I do not use it.” to any activity in the list,] Why do you not use this feature?

[For those who answer (C) “No” to any activity in the list,] Does your phone not have this feature or are you unsure whether your phone has the feature?

[For those who answer (A) “Yes, and I use it.” or (B) “Yes, but I do not use it.” to item “k. Use and application (or app),”] Do you know how to install an app on your phone or do you use only the apps that are preinstalled or someone else installed on your phone?

[Grade 8 TEL–Student]

VE639123 9. How often do you use a computer or other digital technology for the following activities? Select one circle in each row.
	Never or almost never	A few times a year	Once or twice a month	Once or twice a week	Every day or almost every day
a. Play one-player games	(A)	(B)	(C)	(D)	(E)
b. Play multi-player games	(A)	(B)	(C)	(D)	(E)
c. Read and write e-mail	(A)	(B)	(C)	(D)	(E)
d. Chat online	(A)	(B)	(C)	(D)	(E)
e. Participate in online forums, social networking sites, or virtual communities	(A)	(B)	(C)	(D)	(E)
f. Maintain a personal website or blog	(A)	(B)	(C)	(D)	(E)
g. Browse the Internet for fun (for example, watching videos)	(A)	(B)	(C)	(D)	(E)
h. Compare products or shop online	(A)	(B)	(C)	(D)	(E)
i. Download music, films, games or software from the Internet	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

What does “computer or other digital technology” mean to you? What examples do you think of as “computer or other digital technology”?

What does “chatting online” mean to you? What are some of the different ways that you “chat online”?

Do you think there is a difference between maintaining a “personal website” and maintaining a profile on a social networking website such as Facebook or MySpace? [If “yes,”] How would you describe the difference? Would you include updating your profile on your social networking site in your response to “f. Maintain a personal website or blog”?

[For those who answer (B) “A few times a year,” (C) “Once or twice a month,” (D) “Once or twice a week,” or (E) “Every day or almost every day” for “h. Compare products or shop online,”] How do you usually compare products online? In other words, what things do you do to help you choose between different products?

Do you use the computer regularly for any other activities not included among the activities listed in the question?[If “yes,”] Please describe the activities.

[Grade 8 TEL–Student]

VE639025 10. How much do you disagree or agree with these statements? Select one circle in each row.
	Strongly Disagree	Disagree	Agree	Strongly agree
a. Technology and engineering are important to society.	(A)	(B)	(C)	(D)
b. Technology and engineering are important to my daily life.	(A)	(B)	(C)	(D)
c. Learning about technology and engineering will help me get the job I want.	(A)	(B)	(C)	(D)
d. Learning about technology and engineering will help me in the future.	(A)	(B)	(C)	(D)
e. I enjoy learning about technology and engineering.	(A)	(B)	(C)	(D)

Item-Specific Probes:

When you saw the words “technology and engineering” in this question, how were you defining technology? What did you think are examples of technology? And, how were you defining engineering? What did you think are examples of engineering?

How would you respond to these items “a” – “e” about “technology and engineering” if you felt one way about technology, but a different way about engineering?

When answering item “a. Technology and engineering are important to society,” did you feel any pressure to answer in a particular way? [If “yes,”] Can you describe how you were feeling and why you answered the way you did?

When answering item “d. Learning about technology and engineering will help me in the future,” what does “in the future” mean to you?

[Grade 8 TEL–Student]

Item-Specific Probes:

[For those who answer (A) “Yes, in school,” (B) “Yes, outside of school,” or (C) “Yes, both in school and outside of school,”], Can you give some examples of different ways you would like to learn more about technology or engineering?

[For those who answer (A) “Yes, in school,” (B) “Yes, outside of school,” or (C) “Yes, both in school and outside of school,”], Can you give some examples of what type of things you want to learn more about related to technology or engineering?

[Grade 8 TEL–Student]

VE639166 12. How interested are you in learning about the following areas of technologies? Select one circle in each row.
	Not at all interested	Not too interested	Somewhat interested	Very interested
a. Information and communication	(A)	(B)	(C)	(D)
b. Transportation	(A)	(B)	(C)	(D)
c. Manufacturing	(A)	(B)	(C)	(D)
d. Construction	(A)	(B)	(C)	(D)
e. Power and energy	(A)	(B)	(C)	(D)
f. Environmental and green technologies	(A)	(B)	(C)	(D)
g. Agricultural and related biotechnologies	(A)	(B)	(C)	(D)
h. Medical technologies	(A)	(B)	(C)	(D)

Item-Specific Probes:

What do “Not too interested,” “Somewhat interested,” and “Very interested” mean to you? Can you provide examples that would help someone understand how you would distinguish between those response choices?

What do you think are examples of “information and communication”?

What do you think are examples of “transportation”?

What do you think are examples of “manufacturing”?

What do you think are examples of “construction”?

What do you think are examples of “power and energy”?

In item “f. Environmental and green technologies,” what does “environmental and green technologies” mean to you? When answering the question, what do you think are examples of “environmental and green technologies”?

In item “g. Agricultural and related biotechnologies,” what does “agricultural and related biotechnologies” mean to you? When answering the question, what do you think are examples of “agricultural and related biotechnologies”?

What do you think are examples of “medical technologies”?

[Grade 8 TEL–Student]

VE681624 13. In school, how often do you learn about or discuss the following? Select one circle in each row.
	Never	Rarely	Sometimes	Often
a. How to judge or evaluate the accuracy of information sources	(A)	(B)	(C)	(D)
b. How to credit others for their ideas	(A)	(B)	(C)	(D)

Item-Specific Probes:

What do “Rarely,” “Sometimes,” and “Often” mean to you? Can you provide examples that would help someone understand how you would distinguish between those response choices?

What time period were you thinking about when answering this question—for example were you thinking about the last week, the last month, the last year, or longer? If you learned about one of these listed items “often” last month in school, but not this month, how would you have answered this question? If you learned about one of these listed items “often” last year in school, but not this year, how would you have answered this question?

Have you learned or talked about the topics on this list in an afterschool program or club? [If “yes,”] Does that afterschool program or club meet in your school on your school’s grounds? [If “yes,”] Did you include the time learning or talking about those topics in that after school program or club in your answer?

How would you describe “a. How to judge or evaluate the accuracy of information sources” in your own words? When thinking about “information sources,” did you include information sources on the Internet in your answer?

[Note: The following probes are designed to both identify problems with interpretation of set members (or sub-items in this case 13a and 13b). However, these probes will also be used to determine the viability and wording of a future self-efficacy item related to one or both of these constructs.]

Do you ever question things you read or hear, or do you usually accept what you’ve read or heard as being truthful and accurate?

How good do you think you would be at figuring out whether an information source, such as a person on television or on a website, is telling the truth or hiding some of the facts?

When you write a report or an essay for school, do you ever include information you get from the Internet? How about things you’ve heard or seen on TV? [If “yes,”] When you include information you get from the Internet or TV in a report do you credit or describe the information source (for example, in an endnote or footnote)? How often do you do that?

[Grade 8 TEL–Student]

VE639119 14. In school, how often do you use a computer or other digital technology to do the following? Select one circle in each row.
	Never or almost never	A few times a year	Once or twice a month	Once or twice a week	Every day or almost every day
a. Search for information	(A)	(B)	(C)	(D)	(E)
b. Communicate with other people	(A)	(B)	(C)	(D)	(E)
c. Create a presentation	(A)	(B)	(C)	(D)	(E)
d. Work with others to solve a problem	(A)	(B)	(C)	(D)	(E)
e. Write	(A)	(B)	(C)	(D)	(E)
f. Draw	(A)	(B)	(C)	(D)	(E)
g. Find data to solve a problem	(A)	(B)	(C)	(D)	(E)
h. Share information with others	(A)	(B)	(C)	(D)	(E)
i. Get information from experts	(A)	(B)	(C)	(D)	(E)
j. Run simulations	(A)	(B)	(C)	(D)	(E)
k. Create a spreadsheet	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

What does “computer or other digital technology” mean to you? What examples do you think of as “computer or other digital technology”?

Do you do any of the listed activities in an afterschool program or club? [If “yes,”] Does that afterschool program or club occur in your school or on your school’s grounds? [If “yes,”] Did you include the time spent doing those activities in that afterschool program or club in your answer?

What does item “i. Get information from experts” mean to you? Would you consider reading a book or article by an expert as getting information from an expert?

What does item “j. Run simulations” mean to you? What do you think are examples of “simulations”?

What does item “k. Create a spreadsheet” mean to you? [For those who responded (B)-(E) on “k”], What kinds of things do you do when you create a spreadsheet?

[Grade 8 TEL–Student]

VE639143 15. Outside of school, how often do you use a computer or other digital technology to do the following? Select one circle in each row.
	Never or almost never	A few times a year	Once or twice a month	Once or twice a week	Every day or almost every day
a. Search for information	(A)	(B)	(C)	(D)	(E)
b. Communicate with other people	(A)	(B)	(C)	(D)	(E)
c. Create a presentation	(A)	(B)	(C)	(D)	(E)
d. Work with others to solve a problem	(A)	(B)	(C)	(D)	(E)
e. Write	(A)	(B)	(C)	(D)	(E)
f. Draw	(A)	(B)	(C)	(D)	(E)
g. Find data to solve a problem	(A)	(B)	(C)	(D)	(E)
h. Share information with others	(A)	(B)	(C)	(D)	(E)
i. Get information from experts	(A)	(B)	(C)	(D)	(E)
j. Run simulations	(A)	(B)	(C)	(D)	(E)
k. Create a spreadsheet	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

In answering the question, what did you think of as “outside of school”?

Would you think of an afterschool program or club in your school or on your school’s grounds as “outside of school”? Would you think of a field trip as “outside of school”?

[For those who answer (B), (C), (D), or (E) for “b. Communicate with other people”], What type of computer or other digital technology do you use to communicate? Who are you communicating with? How does the computer or the digital technology that you use to communicate vary (or differ) depending on who you are communicating with?

[Grade 8 TEL–Student]

Item-Specific Probes:

What does “using computers or other digital technology for collecting or sharing information” mean to you? What examples do you think of as “using computers or other digital technology for collecting or sharing information”?

In this question you were asked to choose from which sources you learned the most. What did you think about when you were comparing each of the sources on the list? Was it difficult to choose from among the choices of where you learned the most? [If “yes,”] What made it difficult choose?

Do you ever read articles on websites or on online forums? [If “yes,”] Would you consider articles on a website or online forum as best fitting the answer choice (B) “Books or articles in print form” or the answer choice (D) “Websites or online forums”?

What does “online forums” mean to you?

[Grade 8 TEL–Student]

Item-Specific Probes:

In this question you were asked to choose who taught you the most. If you had to, how would you rank the choices? Was it difficult to choose from among the choices of who taught you the most? [If “yes,”] What made it difficult choose?

[Grade 8 TEL–Student]

VE682217 18. Do you think that you would be able to do each of the following? Select one circle in each row.
	I definitely can’t	I probably can’t	Maybe	I probably can	I definitely can
a. Publish or maintain a personal website or blog	(A)	(B)	(C)	(D)	(E)
b. Create presentations with sound, pictures, or video	(A)	(B)	(C)	(D)	(E)
c. Organize information into a chart, graph, or spreadsheet	(A)	(B)	(C)	(D)	(E)
d. Compare products using the Internet	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

In your own words, tell me what you think this question is asking you to do?

Do you have a social network website such as a profile on Facebook or My Space? [If “yes,”] Did you include that in your response to “a. Publish or maintain a personal website or blog”?

For each activity listed “a”-“d,” have you ever done this activity?

What does “compare products using the Internet” mean to you?

[Grade 8 TEL–Student]

VE682225 19. In school, how often do you learn about or discuss the following? Select one circle in each row.
	Never	Rarely	Sometimes	Often
a. The use and purpose of tools, machines, or devices	(A)	(B)	(C)	(D)
b. The care or maintenance of tools, machines, or devices	(A)	(B)	(C)	(D)
c. Designing or creating something to solve a problem	(A)	(B)	(C)	(D)
d. Designing something when there is limited time, money, or materials	(A)	(B)	(C)	(D)
e. Figuring out how to fix something	(A)	(B)	(C)	(D)
f. Finding the right people to work with or get help from to fix something	(A)	(B)	(C)	(D)

Item-Specific Probes:

What do “Rarely,” “Sometimes,” and “Often” mean to you? Can you provide examples that would help someone understand how you would distinguish between those response choices?

What do “tools,” “machines,” and “devices” mean to you? Do you think there is a difference between “tools,” “machines,” and “devices”? [If “no,”] What do you think are some examples of “tools,” “machines,” or “devices”? [If “yes,”] What do you think are some examples of “tools”? What do you think are some examples of “machines”? What do you think are some examples of “devices”?

What does “a. The use and purpose of tools, machines, or devices” mean to you?

What does “b. The care or maintenance of tools, machines, or devices” mean to you?

Do you think there is a difference between “a. The use and purpose of tools, machines, or devices” and “b. The care or maintenance of tools, machines, or devices”? [If “yes,”] How would you describe the difference?

What does “c. Designing or creating something to solve a problem” mean to you? What examples do you think of as “designing or creating something to solve a problem”?

What does “d. Designing something when there is limited time, money, or materials” mean to you? What examples do you think of as “designing something when there is limited time, money, or materials”?

What does “e. Figuring out how to fix something” mean to you? What do you think are some examples of ways to figure out how to fix something?

[Grade 8 TEL–Student]

VE682232 20. Outside of school, how often do you learn about or discuss the following? Select one circle in each row.
	Never	Rarely	Sometimes	Often
a. The use and purpose of tools, machines, or devices	(A)	(B)	(C)	(D)
b. The care or maintenance of tools, machines, or devices	(A)	(B)	(C)	(D)
c. Designing or creating something to solve a problem	(A)	(B)	(C)	(D)
d. Designing something when there is limited time, money, or materials	(A)	(B)	(C)	(D)
e. Figuring out how to fix something	(A)	(B)	(C)	(D)
f. Finding the right people to work with or get help from to fix something	(A)	(B)	(C)	(D)

Item-Specific Probes:

In answering the question, what did you think of as “outside of school”? Would you think of an afterschool program or club in your school or on your school’s grounds as “outside of school”? Would you think of a field trip as “outside of school?”

What do “Rarely,” “Sometimes,” and “Often” mean to you? Can you provide examples that would help someone understand how you would distinguish between those response choices?

What time period were you thinking about when answering this question—for example were you thinking about the last week, the last month, the last year, or longer? If you learned about one of these listed items “often” last month outside of school, but not this month, how would you have answered this question? If you learned about one of these listed items “often” last year outside of school, but not this year, how would you have answered this question?

[Grade 8 TEL–Student]

VE638956 21. In school, how often have you ever done the following activities? Select one circle in each row.
	Never	Once or twice	Three to five times	More than five times
a. Used tools or materials to fix or build something	(A)	(B)	(C)	(D)
b. Used different tools, materials, or machines to see which are best for a given purpose	(A)	(B)	(C)	(D)
c. Built or tested a model to see if it solves a problem	(A)	(B)	(C)	(D)
d. Figured out why something is not working in order to fix it	(A)	(B)	(C)	(D)
e. Taken something apart in order to fix it or see how it works	(A)	(B)	(C)	(D)
f. Designed a computer program	(A)	(B)	(C)	(D)

Item-Specific Probes:

What time period were you thinking about when answering this question—for example were you thinking about the last week, the last month, the last year, or longer? If you did one of these activities “once or twice” last month in school, but not this month, how would you have answered this question? If you did one these activities “once or twice” last year in school, but not this year, how would you have answered this question?

For those who answer (A “Never”) to item “c. Built or tested a model to see if it solves a problem”], What does “c. Built or tested a model to see if it solves a problem” mean to you?

[For those who answer (B “Once or twice”), (C “Three to five times”), or (D “More than five times”) to item “c. Built or tested a model to see if it solves a problem”], What types of models have you worked with? What types of problem(s) were you trying to solve?

Do you think there is a difference between “d. Figured out why something is not working in order to fix it” and “e. Taken something apart in order to fix it or see how it works”? [If “yes,”] How would you describe the difference?

[For those who answer (B “Once or twice”), (C “Three to five times”), or (D “More than five times”) to item “f. Designed a computer program”], What kind of computer program did you design? How did you design it? What kind of programming software or programming language did you use?

[Grade 8 TEL–Student]

VE638983 22. Outside of school, how often have you ever done the following activities? Select one circle in each row.
	Never	Once or twice	Three to five times	More than five times
a. Used tools or materials to fix or build something	(A)	(B)	(C)	(D)
b. Used different tools, materials, or machines to see which are best for a given purpose	(A)	(B)	(C)	(D)
c. Built or tested a model to see if it solves a problem	(A)	(B)	(C)	(D)
d. Figured out why something is not working in order to fix it	(A)	(B)	(C)	(D)
e. Taken something apart in order to fix it or see how it works	(A)	(B)	(C)	(D)
f. Designed a computer program	(A)	(B)	(C)	(D)

Item-Specific Probes:

What time period were you thinking about when answering this question—for example were you thinking about the last week, the last month, the last year, or longer? If you did one of these activities “once or twice” last month outside of school, but not this month, how would you have answered this question? If you did one these activities “once or twice” last year outside of school, but not this year, how would you have answered this question?

[Grade 8 TEL–Student]

Item-Specific Probes:

In this question you were asked to choose from which sources you learned the most. If you had to, how would you rank the choices? Was it difficult to choose from among the choices of where you learned the most? [If “yes,”] What made it difficult choose?

In this question, you were asked about “building things, fixing things, and how things work”? If you learned more about “building things” from one source and “how things work” from another source, how would you answer this question?

What does “building things” mean to you? What examples do you think of as “building things”?

What does “fixing things” mean to you? What examples do you think of as “fixing things”?

What does “how things work” mean to you? What examples do you think of as “how things work”?

What does “online forums” mean to you?

[Grade 8 TEL–Student]

Item-Specific Probes:

In this question, you were asked about “building things, fixing things, and how things work”? If you learned more about “building things” from one person and “how things work” from another person, how would you answer this question?

[Grade 8 TEL–Student]

VE682276 25. Do you think that you would be able to do each of the following? Select one circle in each row.
	I definitely can’t	I probably can’t	Maybe	I probably can	I definitely can
a. Build a model using a kit	(A)	(B)	(C)	(D)	(E)
b. Build a model without using a kit	(A)	(B)	(C)	(D)	(E)
c. Use tools or materials to fix something	(A)	(B)	(C)	(D)	(E)
d. Take something apart in order to fix it or see how it works	(A)	(B)	(C)	(D)	(E)
e. Design a computer program	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

In your own words, tell me what you think this question is asking you to do?

For items “a. Build a model using a kit” and “b. Build a model without using a kit,” what do you consider to be “models”? What do you think are examples of a model?

What does “a kit” mean to you?

For each activity listed “a”-“e”, have you ever done this activity?

[Grade 8 TEL–Student]

VE638999 26. In school, how often do you learn about or discuss the following? Select one circle in each row.
	Never	Rarely	Sometimes	Often
a. Inventions that change the way people live	(A)	(B)	(C)	(D)
b. Choices people make that affect the environment	(A)	(B)	(C)	(D)
c. Conditions that influence the use or availability of machines or devices	(A)	(B)	(C)	(D)
d. The ways people work together to solve problems in their community or the world	(A)	(B)	(C)	(D)

Item-Specific Probes:

What do “Rarely,” “Sometimes,” and “Often” mean to you? Can you provide examples that would help someone understand how you would distinguish between those response choices?

What do you think are examples of “a. Inventions that change the way people live”?

What do you think are examples of “b. Choices people make that affect the environment”?

What do you think are examples of “c. Conditions that influence the use or availability of machines or devices”?

What do you think are examples of “d. The ways people work together to solve problems in their community or the world”?

[Grade 8 TEL–Student]

VE639008 27. Outside of school, how often do you learn about or discuss the following? Select one circle in each row.
	Never	Rarely	Sometimes	Often
a. Inventions that change the way people live	(A)	(B)	(C)	(D)
b. Choices people make that affect the environment	(A)	(B)	(C)	(D)
c. Conditions that influence the use or availability of machines or devices	(A)	(B)	(C)	(D)
d. The ways people work together to solve problems in their community or the world	(A)	(B)	(C)	(D)

Item-Specific Probes:

In answering the question, what did you think of as “outside of school”?

Would you think of an afterschool program or club in your school or on your school’s grounds as “outside of school”? Would you think of a field trip as “outside of school?”

In answering the question, when thinking about “outside of school,” did you include watching a television documentary at home related to any of the listed topics in your answer? For example, if you watched a documentary on the invention of something, would you have considered that when giving a response to “a. Inventions that change the way people live”?

[Grade 8 TEL–Student]

Item-Specific Probes:

What does “the relationship between technology, people, and the environment” mean to you? What are some examples of “the relationship between technology, people, and the environment”?

What does “online forums” mean to you?

[Grade 8 TEL–Student]

Item-Specific Probes:

What are some examples of what you were taught about the relationship between technology, people, and the environment?

[Grade 8 TEL–Student]

VE682317 30. Do you think that you would be able to do each of the following? Select one circle in each row.
	I definitely can’t	I probably can’t	Maybe	I probably can	I definitely can
a. Describe how a society might change with the introduction of a new invention	(A)	(B)	(C)	(D)	(E)
b. Compare the effects that different activities might have on the environment	(A)	(B)	(C)	(D)	(E)
c. Explain why people in different parts of the world might develop or use different tools, machines, or devices	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

In your own words, tell me what you think this question is asking you to do?

What do you think are some examples of how a society might change with the introduction of a new invention?

What do you think are some examples of activities that might affect the environment?

What do you think are some examples of why people in different parts of the world might develop or use different tools, machines, or devices?

For each activity listed “a”-“c,” have you ever done this activity?

[Grade 8 TEL–Student]

VE638978 31. Before today, have you ever done any of the following?
	Yes	No
a. Taken an interactive computer test in school	(A)	(B)
b. Taken an interactive computer test outside of school	(A)	(B)

Note: Student will not be taking a test prior to cognitive interview. Therefore, this question will be presented, but student will be informed that they are being shown this question and asked some follow-up questions simply to gauge their understanding.

Item-Specific Probes:

What does “interactive computer test” mean to you?

How would you describe an interactive computer test to a friend? What makes it different from a regular test on a computer? How do you think an interactive computer test might differ from a regular test on a computer?

[For those who answer (A) “Yes” to either item], What was that interactive computer test like? What were you asked to do on that test?

[For those who answer (B) “No” to both items], Have you ever taken any kind of test on a computer?

Have you ever used any kind of computer where you received a response or feedback based on what you entered or the computer? [If “yes,”] What are some examples of what you did?

Have you ever played an education or quiz game on a computer or any computer-like device including a video game system? [If “yes,”] What are some examples of what you did in those games? What were the names of the games? Did you include tests as part of those games when giving your answer?

TEL: School Questionnaire–Grade 8

Item-Specific Probes:

Before we get to the first question, please tell me in your own words what this introduction is telling you.

Before today, have you ever heard of technology described the way it was in the introduction?

Does the way technology is described in the introduction change how you think about or define technology? [If “yes,”] How did you think about or define technology before seeing the definition provided in the introduction?

Before today, have you ever heard of engineering described the way it was in the introduction?

Does the way engineering is described in the introduction change how you think about or define engineering? [If “yes,”] How did you think about or define engineering before seeing the definition provided in the introduction?

VE638313 1. In your school, prior to or in eighth grade, are the following topics required in the curriculum? Fill in one oval in each row.
	Required in the curriculum	Not required in the curriculum
a. How objects are designed to solve problems and meet people’s needs	(A)	(B)
b. The interactions among technology, society, and the environment	(A)	(B)
c. The role of technological systems (e.g., energy usage, healthcare, communications)	(A)	(B)
d. How computers, the Internet, and other digital technologies affect society	(A)	(B)
e. Careers in technical fields (e.g., engineer, medical technician, computer programmer)	(A)	(B)
f. Concepts related to specific technologies (e.g., electronics, biotechnology, agriculture)	(A)	(B)

Item-Specific Probes:

What does “required in the curriculum” mean to you? [Optional interviewer variant for similar questions throughout this questionnaire: “When you read/see/hear the word/words/phrases ‘xxxxxx’ what does it mean to you”]

What grade levels prior to and including eighth grade are included in your school?

Regardless of whether they are required in your curriculum, are there any other topics that you would add to this list? [If “yes”] What else should be added?

How did you interpret the meaning of items “c. The role of technological systems,” “e. Careers in technical fields,” and “f. Concepts related to specific technologies”? Do you think the examples contained in parentheses in the items helped you to better understand the particular topics being referred to in this question?

[For those who answered (A) “Required in the curriculum” for each activity listed “a”-“f,”] How is this topic required in the curriculum (e.g., part of what course or activity)? What are some examples of how this topic is addressed in your school’s curriculum?

VE638378 2. In your school, what proportion of eighth-grade students should have received instruction in each of the following topics prior to or in eighth grade? Fill in one oval in each row.
	All or almost all students	Some students	Not included in the curriculum
a. How objects are designed to solve problems and meet people’s needs	(A)	(B)	(C)
b. The interactions among technology, society, and the environment	(A)	(B)	(C)
c. The role of technological systems (e.g., energy usage, healthcare, communications)	(A)	(B)	(C)
d. How computers, the Internet, and other digital technologies affect society	(A)	(B)	(C)
e. Careers in technical fields (e.g., engineer, medical technician, computer programmer)	(A)	(B)	(C)
f. Concepts related to specific technologies (e.g., electronics, biotechnology, agriculture)	(A)	(B)	(C)

Item-Specific Probes:

What does “should have received instruction” mean to you? How would your response have been different if the question had instead asked “have received” or “estimated to have received”?

When responding to this question, approximately what percentage of eighth-grade students did you consider as “some students”?

[For those who answered (B “Some students” to any sub-item], Is there any particular pattern to the types of students receiving instruction? For example, are they students who are taking an elective, or are they students in a particular track or program within the school?

Are there any additional topics that you would include in this list? [If “yes”] What else should be included?

Do you think the examples in sub-items “c. The role of technological systems,” “e. Careers in technical fields,” and “f. Concepts related to specific technologies” help you better understand the particular topics being referred to in this question?

VE638432 3. How are each of the following areas addressed in your school’s curriculum prior to or in eighth grade? Fill in one or more ovals in each row.
	Required	Elective	After school	Not offered
a. Technology and Society (i.e., the effects that technology has on society and on the natural world and the ethical questions that arise from those effects)	(A)	(B)	(C)	(D)
b. Design and Systems (i.e., the nature of technology, the engineering design process by which technologies are developed, and basic approaches to dealing with everyday technologies, including maintenance and troubleshooting)	(A)	(B)	(C)	(D)
c. Information and Communication Technology (e.g., computers and software learning tools; networking systems and protocols; handheld digital devices; and other technologies for accessing, creating, and communicating information for facilitating creative expression)	(A)	(B)	(C)	(D)

Item-Specific Probes:

If an area was “Required” in any grade prior to or in grade 8, did you select “Required” for that area?

For each of the sub-items are you able to answer one and only one response option? Are there any areas that are addressed both as an elective and after school? Are there any areas that are “Required” at some grades but “Elective” in other grades?

How did you interpret the meaning of each of the three areas? Do you think the examples for each area help you better understand the particular topics being referred to in this question?

[For those who answered (A) “Required” for each area listed “a”-“c”] How is this topic required in the curriculum (e.g., part of what course or activity)? What are some examples of how this topic is addressed in your school’s curriculum?

VE638446

4. Does your school require technology or engineering instruction prior to or in eighth grade?

(A) Yes → Continue to Question 5 [VE638450].

(B) No → Skip to Question 6 [VE638334].

Item-Specific Probes:

When answering the question, what qualifies as “instruction”? Did you consider “instruction” as either integrated within a general course or as a separate course?

[For those who answer (B) “No” and indicate they interpreted the question to ask about a “separate course,”], Would your response to the question be different if you included technology or engineering instruction that is integrated into other courses?

VE638450 5. To what extent is your school’s technology or engineering curriculum based on the following? Fill in one oval in each row.
	Not at all	Small extent	Moderate extent	Large extent
a. State curriculum standards or frameworks	(A)	(B)	(C)	(D)
b. District curriculum standards or curriculum guides	(A)	(B)	(C)	(D)
c. Results from state/district assessments	(A)	(B)	(C)	(D)
d. In-school curriculum frameworks or standards for learning	(A)	(B)	(C)	(D)
e. Results from school assessments	(A)	(B)	(C)	(D)
f. Recommendations from your teachers or content specialists	(A)	(B)	(C)	(D)
g. Commercially available programs or curricula	(A)	(B)	(C)	(D)
h. Textbooks	(A)	(B)	(C)	(D)
i. Other (specify):____________	(A)	(B)	(C)	(D)

Item-Specific Probes:

[For those who answered (A “Not at all” to any sub-item], Did you respond “Not at all” to any sub-item because they were not a factor, or because they were a factor, but were not the basis of decisions about the curriculum?

When answering the question, did you include curriculum for technology and engineering courses only or did you also include curriculum for other courses that include technology or engineering concepts?

For item “e. Results from school assessments,” what types of “school assessments” did you consider? What subject areas did you consider when answering the sub-item?

For item “f. Recommendations from your teachers or content specialists,” can you provide me some insight on what the phrase “content specialists” mean to you?

For item “h. Textbooks,” did you consider textbooks that are prescribed by a commercially available program or curricula? Did you consider any textbooks that are currently prescribed in your school’s curriculum, as well as other textbooks not currently prescribed?

As they are used in this question, can you define what a “small extent,” “moderate extent,” and “large extent” mean to you? Can you provide the cut-points or examples that would help someone understand how you would distinguish between the response choices? [Interviewer note for similar questions throughout this questionnaire: If respondent doesn’t understand cut-point, explain to them it is the potentially overlapping space between two categories (or in this case response choices such as “small extent” and “moderate extent”). The goal is to understand how the respondent distinguishes between these response choices at the points where they might intersect.]

VE638334

6. Prior to or in eighth grade, does your school offer any courses or afterschool programs that cover technology or engineering concepts?

(A) Yes → Continue to Question 7 [VE681573].

(B) No → Skip to Question 8 [VE638483].

Item-Specific Probes:

[For those who answered (A “Yes”], Are the courses mainly offered as part of the regular school day or after school?

VE681573

7. In the rows below, please identify the most relevant courses that cover technology or engineering concepts (up to five) that your school offers to students prior to or in eighth grade.

a. Course 1:___________________________________________________________

b. Course 2:___________________________________________________________

c. Course 3:___________________________________________________________

d. Course 4:___________________________________________________________

e. Course 5:___________________________________________________________

Item-Specific Probes:

[Interviewer note: If respondent has difficulty naming any courses, provide some example courses to spark memory by using the following probe:] Perhaps some courses in your school might have titles like “Introduction to Design, Engineering and Technology,” “Invention and Innovation,” “Computer Programming,” “Systems: Design and Modeling,” or “Robotics.”

What criteria did you use to select and include as the “most relevant courses”?

Are the courses that you selected offered mainly at one grade, or across a range of grades prior to and including eighth grade? Are the courses that you selected required, electives, or afterschool programs?

Please describe briefly the content in the courses that you listed?

VE638483 8. In a typical school year, how often are your eighth-grade students assessed in technology or engineering knowledge using the following methods? Fill in one oval in each row.
	Never	Once or twice a year	Once or twice a month	Once or twice a week	Every day or almost every day	I don’t know.
a. Standardized or performance tests	(A)	(B)	(C)	(D)	(E)	(F)
b. Teacher-developed tests and quizzes	(A)	(B)	(C)	(D)	(E)	(F)
c. Student assignments, projects, or homework	(A)	(B)	(C)	(D)	(E)	(F)

Item-Specific Probes:

How certain are you regarding the frequency that each assessment method is used? Is there any sub-item for which you are least sure about your answer?

[For those who answered (B) – (E) to items “a. Standardized or performance tests” or “b. Teacher-developed tests and quizzes,”] Were any of these interactive computer tests?

What does “Teacher-developed tests and quizzes” mean to you? Can you provide examples of what you considered as “teacher-developed tests and quizzes”? Did you include in your response tests that were provided to a school as a curriculum package developed by teachers?

[For those who answered (F) to any item,] For the item(s) where you responded “I don’t know,” do you have access to a reliable source or personnel (e.g., teacher) who might have access to this information?

VE638475 9. The goal of the following set of questions is to gather information about the student-computer ratio for eighth-grade students at your school.
	Number
a. At your school, what is the total number of students in the eighth grade?	_, _ _ _
b. Approximately how many computers in the school are available to eighth-grade students for educational purposes?	_, _ _ _
c. Approximately how many of these computers are connected to the Internet?	_, _ _ _
d. Approximately how many of all computers available to eighth-grade students are portable (e.g., laptops, tablets)?	_, _ _ _

If you answered 1 or any number greater than 1 to 9d, continue to Question 10. If you answered 0 to 9d, skip to Question 11.

Item-Specific Probes:

How did you determine the numbers for each sub-item? Did you include specific or estimated values for each sub-item?

For sub-item “b. Approximately how many computers in the school...,” what does “educational purposes” mean to you?

VE675587

10. Are your students allowed to take portable computers home with them?

(A) Yes, and they are allowed to keep them permanently.

(B) Yes, but they must return them (e.g., at the end of the week or end of the school year).

Item-Specific Probes:

What does “portable computers” mean to you? Can you provide examples of what you considered as “portable computers”?

What does “permanently” mean to you?

[For those who answered (B “Yes, but they must return them”)], Can you elaborate on your school’s policy concerning portable computers? How long may students borrow the computers?

VE638517 11. In your school, what percentage of students prior to or in eighth grade has taken advantage of the following school-sponsored resources during or after school? Fill in one oval in each row.
	School does not provide this resource to students.	0–5%	6–20%	21–50%	Over 50%
a. Clubs, competitions, exhibits, etc. related to some aspect of technology and engineering	(A)	(B)	(C)	(D)	(E)
b. Technology resources such as robotics or digital photography	(A)	(B)	(C)	(D)	(E)
c. Workshop for industrial technologies (e.g., auto mechanics, machining, metalworking, construction, woodworking)	(A)	(B)	(C)	(D)	(E)
d. Workshop or laboratory for drafting or design tools (e.g., computer-aided design [CAD], drafting, systems analysis)	(A)	(B)	(C)	(D)	(E)
e. Online courses in any subject	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

How did you calculate the percentage for each sub-item? Is the percentage for each sub-item a cumulative percentage across the grades?

Do you think the examples in sub-items “c. Workshop for industrial technologies” and “d. Workshop or laboratory for drafting or design tools” are helpful in better understanding what type of resources the questions refer to?

[For those who respond (B “0–5%”), (C “6–20%”), (D “21–50%”), or (E “Over 50%”) to any of the sub-items], Are these resources available at your school or through a partnering institution?

How sure are you of your answer? In what ways do you communicate with teachers and students to keep abreast of what and how students take advantage of the listed school-sponsored resources?

VE638436 12. In your school, are the following resources available to teachers for teaching or professional development? Fill in one oval in each row.
	Yes	No
a. Reimbursed expenses for professional development related to technology and engineering	(A)	(B)
b. Reimbursed expenses for association memberships related to technology and engineering	(A)	(B)
c. Reimbursed expenses for university or online courses related to technology and engineering	(A)	(B)
d. Technology resources such as robotics or digital photography	(A)	(B)
e. Workshop for industrial technologies (e.g., auto mechanics, machining, metalworking, construction, woodworking)	(A)	(B)
f. Workshop or laboratory for drafting or design tools (e.g., computer-aided design [CAD], drafting, systems analysis)	(A)	(B)
g. Supplies or equipment for technology demonstrations	(A)	(B)
h. Textbooks or digital tutorials related to technology and engineering	(A)	(B)

Item-Specific Probes:

[For those who answer (A) “Yes” to any sub-items], Are these resources available to all teachers or only to those who teach courses considered to have technology or engineering content?

[For those who answer (A) “Yes” to sub-items “a. Reimbursed expenses for professional development...,” “b. Reimbursed expenses for association memberships...,” or “c. Reimbursed expenses for university or online courses...”], Are there any special requirements teachers must meet to be reimbursed for expenses?

[For those who answer (A) “Yes” to sub-item “d. Technology resources such as robotics or digital photography”], Can you elaborate on the kinds of technology resources that are available at your school?

Do you think the examples in items “e. Workshop for industrial technologies” and “f. Workshop or laboratory for drafting or design tools” are helpful in better understanding what type of resources the questions refer to?

For sub-item “h. Textbooks or digital tutorials related to technology and engineering,” did you include both print and electronic textbooks? What does the phrase “digital tutorials” mean to you?

Are there any other resources that your school has available to teachers that were not included in the categories on this list of sub-items? [If “yes”], What other types of resources are available?

Would you consider “a. Reimbursed expenses for professional development...” and “c. Reimbursed expenses for university or online courses...” mutually exclusive?

Would you consider “d. Technology resources such as robotics or digital photograph” and “g. Supplies or equipment for technology demonstrations” mutually exclusive?

VE675659 13. Does your school have equipment available for instruction with the following capabilities?
	Not available	Available for some teachers	Available for all teachers
a. Viewing movies and television programs	(A)	(B)	(C)
b. Recording video	(A)	(B)	(C)
c. Taking digital images	(A)	(B)	(C)
d. Converting non-digital images or content (e.g., scanner)	(A)	(B)	(C)
e. Projecting digital images	(A)	(B)	(C)
f. Projecting interactive data (e.g., interactive whiteboard that responds to user control via stylus, finger, or other device)	(A)	(B)	(C)
g. Collecting data (e.g., tools such as sensors or probes that detect or collect information such as motion, pH, temperature, light)	(A)	(B)	(C)
h. Mobile computing (e.g., handheld or portable computer devices)	(A)	(B)	(C)

Item-Specific Probes:

[For those who answer (B “Available for some teachers”) to any sub-items], Which teachers have access to these resources?

[For those who answer (B “Available for some teachers”) and (C “Available for all teachers”) to any sub-items], Is this equipment available in all classrooms or in specific rooms? Do teachers need to reserve equipment in advance?

Do you think the examples in sub-items “d. Converting non-digital images or content,” “f. Projecting interactive data,” “g. Collecting data,” and “h. Mobile computing” are helpful in better understanding what type of resources the questions refer to?

VE638523 14. To what extent is your school’s capability to provide instruction in technology or engineering hindered by any of the following? Fill in one oval in each row.
	Not at all	Small extent	Moderate extent	Large extent
a. Lack of qualified teachers trained in technological or engineering content	(A)	(B)	(C)	(D)
b. Lack of technical support personnel	(A)	(B)	(C)	(D)
c. Lack or inadequacy of instructional materials (e.g., textbooks, computers, software, etc.)	(A)	(B)	(C)	(D)
d. Lack or inadequacy of Internet connectivity	(A)	(B)	(C)	(D)
e. Lack or inadequacy of laboratory or workshop equipment	(A)	(B)	(C)	(D)
f. Lack or inadequacy of audio-visual resources	(A)	(B)	(C)	(D)
g. Lack of curriculum development expertise or standards specificity	(A)	(B)	(C)	(D)
h. Lack of time because of demands for other curriculum content	(A)	(B)	(C)	(D)

Item-Specific Probes:

Are there any other instructional capabilities that you would include in this list? [If “yes”] What else should be included?

How did you interpret the meaning of item “c. Lack or inadequacy of instructional materials”? Do you think the examples contained in parentheses in the items helped you to better understand the particular topics being referred to in this question?

[For those who answer (B) “Small extent”, (C) “Moderate extent”, or (D) “Large extent” to sub-item “d. Lack or inadequacy of Internet connectivity”], Please describe what your school’s Internet connectivity is like? Is it available in specific classrooms or the school library? Is it reliable?

[For those who answer (B “Small extent”), (C “Moderate extent”), or (D “Large extent”) to sub-item “g. Lack of current development expertise...”], Please elaborate on your response to item (g)? Is there a lack of standards specificity, curriculum development expertise, or both?

For this question, what do “small extent,” “moderate extent,” and “large extent” mean to you? Can you provide the cut-points or examples that would help someone understand how you would distinguish between the response choices?

VE638496 15. In the past two years, what percentage of teachers who teach technology or engineering concepts in your school has participated in professional development in any of the following? Fill in one oval in each row.
	Not applicable	0%	1–25%	26–50%	51–75%	Over 75%	I don’t know.
a. Content, curriculum, or pedagogy related to engineering design	(A)	(B)	(C)	(D)	(E)	(F)	(G)
b. Content, curriculum, or pedagogy related to technology or technological literacy	(A)	(B)	(C)	(D)	(E)	(F)	(G)
c. Integrating information and communications technology into instruction	(A)	(B)	(C)	(D)	(E)	(F)	(G)

Item-Specific Probes:

How did you calculate the percentage? Did you include all teachers who teach any courses that include technology or engineering concepts, or did you include only teachers who specifically teach courses dedicated to technology or engineering?

Did you come up with a specific or estimated value for the numerator and denominator separately when arriving at your answer?

How certain are you about the specific proportions of teachers you indicated in your answer? If you had to guess, to what extent might your estimates vary from the actual percentage of teachers who participated in these professional development activities?

Do you think that the distinctions between sub-items a, b, and c are clear? [If “no”] What do you think would help to clarify the differences?

[For those who answered (G) to any item,] For the item(s) where you responded “I don’t know.”, do you have access to a reliable source or personnel (e.g., teacher) who might have access to this information?

When answering the question for item “b. Content, curriculum, or pedagogy related to technology or technological literacy,”, what does “technological literacy” mean to you?

VE638333 16. In your school, prior to or in eighth grade, how much emphasis is placed on teaching students the following? Fill in one oval in each row.
	None	A little	Some	A lot	I don’t know.
a. Inventions change the way people live	(A)	(B)	(C)	(D)	(E)
b. Choices people make affect the environment	(A)	(B)	(C)	(D)	(E)
c. Circumstances influence the use or availability of machines or devices	(A)	(B)	(C)	(D)	(E)
d. How people work together to solve problems in their community or the world	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

[For those who respond (A) “None”, (B) “A little”, (C) “Some”, or (D) “A lot” to any of the items], How sure are you of your answer? In what ways do you communicate with teachers and students to keep abreast of what and how these topics are emphasized in the classroom?

How would you answer this question in cases where some teachers or grade levels emphasize these ideas a lot or to a great extent and others only a little or not at all? Emphasis can vary by subject, are you basing your response emphasis on a particular course, a set of courses, or all courses in the school?

For item “c. Circumstances influence the use or availability...,” how did you interpret “circumstances”?

For this question, what do “a little,” “some,” and “a lot” mean to you? Can you provide the cut-points or examples that would help someone understand how you would distinguish between the response choices?

[For those who answered (E) to any item,] For the item(s) where you responded “I don’t know.”, do you have access to a reliable source or personnel (e.g., teacher) who might have access to this information?

VE638350 17. In your school, prior to or in eighth grade, to what extent do students do the following activities? Fill in one oval in each row.
	Not at all	Small extent	Moderate extent	Large extent	I don’t know.
a. Describe how a society might change with the introduction of a new invention	(A)	(B)	(C)	(D)	(E)
b. Compare the effects that different activities might have on the environment	(A)	(B)	(C)	(D)	(E)
c. Explain why people in different parts of the world might develop or use different tools, machines, or devices	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

[For those who respond (A) “Not at all”, (B) “Small extent”, (C) “Moderate extent”, or (D) “Large extent” to any of the items], How sure are you of your answer? In what ways do you communicate with teachers and students to keep abreast of the activities students do?

Did you include activities that students may do in and outside of school?

How would you answer this question in cases where some teachers or grade levels do these activities often and others only a little or not at all?

For item “a. Describe how a society might change...,” what types of “activities” did you consider?

For item “b. Compare the effects...,” what types of “activities” did you consider?

For item “c. Explain why people in different parts of the world...,” what types of “activities” did you consider?

For item “c. Explain why people in different parts of the world...,” what do “tools,” “machines,” and “devices” mean to you?

VE638372 18. In your school, prior to or in eighth grade, how much emphasis is placed on teaching students the following? Fill in one oval in each row.
	None	A little	Some	A lot	I don’t know.
a. The use and purpose of tools, machines, or devices	(A)	(B)	(C)	(D)	(E)
b. The maintenance and care of tools, machines, or devices	(A)	(B)	(C)	(D)	(E)
c. Designing or creating something to solve a problem	(A)	(B)	(C)	(D)	(E)
d. Designing something when there is limited amount of time, money, or materials	(A)	(B)	(C)	(D)	(E)
e. Figuring out why something is not working in order to fix it	(A)	(B)	(C)	(D)	(E)
f. The appropriate people to work with or get help from to fix something	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

[For those who respond (A “None”), (B “A little”), (C “Some”), or (D “A lot”) to any of the items], How sure are you of your answer? In what ways do you communicate with teachers and students to keep abreast of the topics emphasized?

How would you answer this question in cases where some teachers or grade levels emphasize these ideas a lot and others only a little or not at all? Emphasis can vary by subject, are you basing your response emphasis on a particular course, a set of courses, or all courses in the school?

For items “a. The use and purpose of tools, machines, or devices” and “b. The maintenance and care of tools machines, or devices,” what do “tools,” “machines,” and “devices” mean to you?

For items “c. Designing or creating something to solve a problem” and “d. Designing something where there is limited...,” what does “designing” mean to you? Can you provide an example?

[For those who respond (A) “None”, (B) “A little”, (C) “Some”, or (D) “A lot” to “d. Designing something where there is limited...”], In teaching these topics in your school, is the concept of design criteria and modeling addressed? How about the concept of constraints? How about the concept of trade-offs?

Would you consider items “e. Figuring out why something is not working...” and “f. The appropriate people to work with...” to be mutually exclusive? Do you think the distinctions between the two are clear?

For this question, what do “a little,” “some,” and “a lot” mean to you? Can you provide the cut-points or examples that would help someone understand how you would distinguish between the response choices?

VE638380 19. In your school, prior to or in eighth grade, to what extent do students do the following activities? Fill in one oval in each row.
	Not at all	Small extent	Moderate extent	Large extent	I don’t know.
a. Use tools and materials to fix something	(A)	(B)	(C)	(D)	(E)
b. Use different tools, materials, or machines to see which are best for a given purpose	(A)	(B)	(C)	(D)	(E)
c. Build or test a model to see if it solves a problem	(A)	(B)	(C)	(D)	(E)
d. Figure out why something is not working in order to fix it	(A)	(B)	(C)	(D)	(E)
e. Take something apart in order to fix it or see how it works	(A)	(B)	(C)	(D)	(E)
f. Design a computer program	(A)	(B)	(C)	(D)	(E)
g. Examine how parts, processes, or people work together in a system	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

Did you include activities that students may do in and outside of school?

How would you answer this question in cases where some teachers or grade levels do these activities often and others only a little or not at all?

Would you consider items “a. Use tools and materials to fix something” and “b. Use different tools materials or machines...” to be mutually exclusive? Do you think the distinctions between the two are clear?

Would you consider items “d. Figure out why something is not working...” and “e. Take something apart in order to fix it...” to be mutually exclusive? Do you think the distinctions between the two are clear?

For item “c. Build or test a model...,” what does “build or test a model” mean to you? Can you provide examples of what you consider to be “models”?

For item “f. Design a computer program,” what does “design a computer program” mean to you?

For item “g. Examine how parts, processes, or people...,” what does “parts, processes, or people work together in a system” mean to you? This item is trying to ascertain the extent to which students do activities that reflect an understanding of technological principles and that collaboration is part of this process. Do you have suggestions that could make this intent clearer?

VE638391 20. In your school, prior to or in eighth grade, how much emphasis is placed on teaching students the following? Fill in one oval in each row.
	None	A little	Some	A lot	I don’t know.
a. How to judge or evaluate the accuracy of information sources	(A)	(B)	(C)	(D)	(E)
b. How to credit others for their ideas	(A)	(B)	(C)	(D)	(E)
c. How to collaborate or share information with others	(A)	(B)	(C)	(D)	(E)
d. Using simulations	(A)	(B)	(C)	(D)	(E)
e. How to consult with experts to get help	(A)	(B)	(C)	(D)	(E)
f. People can work together in different ways to solve problems.	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

For item “d. Using simulations,” what does “using simulations” mean to you? Do you think examples would be helpful?

For item “e. How to consult with experts...,” what does “consult with experts to get help” mean to you? Who did you consider as examples of “experts”? Did you include consultations that students have with teachers or others in the school?

VE638410 21. In your school, prior to or in eighth grade, to what extent do students do the following activities? Fill in one oval in each row.
	Not at all	Small extent	Moderate extent	Large extent	I don’t know.
a. Use digital tools to gather and display information in order to test a hypothesis	(A)	(B)	(C)	(D)	(E)
b. Select and use appropriate digital technologies to create a presentation	(A)	(B)	(C)	(D)	(E)
c. Use a computer or other digital technology to simulate a system and explain different outcomes	(A)	(B)	(C)	(D)	(E)
d. Give feedback to others when working together	(A)	(B)	(C)	(D)	(E)

Item-Specific Probes:

[For those who respond (A “Not at all”), (B “Small extent”), (C “Moderate extent”), or (D “Large extent”) to any of the items], How sure are you of your answer? In what ways do you communicate with teachers and students to keep abreast of the activities students do?

Did you include activities that students may do in and outside of school?

How would you answer this question in cases where some teachers or grade levels do these activities often and others only a little or not at all?

For item “a. Use digital tools to gather and display information...,” what does “digital tools” mean to you? Can you provide an example? Do your students do activities that include the collection, analysis, and presentation of data? [If “yes,”] Did you include those activities in your response? [If “no,”] Can you provide an example of these activities that were not included?

For item “b. Select and use appropriate digital technologies...,” what does “digital technologies” mean to you? Do you think of digital tools in item (a) differently from “digital technologies” in item (b)?

For item “c. Use a computer or other digital technology...,” what does “simulate a system” mean to you?

Would you consider items “a. Use digital tools to gather and display information...,” “b. Select and use appropriate digital technologies...,” and “c. Use a computer or other digital technology...” to be mutually exclusive? Do you think the distinctions between the three are clear?

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS
Author	BlairJ
File Modified	0000-00-00
File Created	2021-01-31