National Assessment of Educational Progress (NAEP) Science Scenario-Based Tasks Cognitive Interviews

National Center for Education Statistics

National Assessment of Educational Progress

Appendices AA – CC

Cognitive Interview Study of the Effects of Visual Representations on Student Performance on NAEP Science Scenario-Based Tasks

OMB# 1850-0803 v. 156

April 15, 2016

Table of Contents

Appendix AA: AIR Recruitment Phone Script for Parents/Guardians of Student Participants 1

Appendix BB: Consent Form for Parent/Guardian of Student Participants 7

Appendix CC: Elliot Benson/Nichols Research Recruitment Phone Script for Parents/Guardians of Student Participants 9

Appendix AA: AIR Recruitment Phone Script for Parents/Guardians of Student Participants

Script when returning calls from potential participants, or calling parents/guardian from a previously compiled list:

Introduction when calling parents/guardians from a previously compiled list: Good morning/afternoon/evening. Is this the <NAME FROM CALL LIST> household? My name is <CALLER NAME>, and I am calling from the American Institutes for Research (AIR). We are administering a study on behalf of the U.S. Department of Education’s National Center for Education Statistics. This study is strictly for research purposes. Your participation is voluntary, and you will receive no sales pitch or follow-up calls based on your participation. I’d like to tell you a little bit about the study first; is now a good time? (YES or NO and call back later)

We are recruiting students currently in grades 8 to help us evaluate new digitally based science tasks, delivered on tablet computers. These tasks are part of the National Assessment of Educational Progress, which is delivered to students nationwide. [If parent/guardians asks what NAEP is: This national assessment is administered to students in elementary, middle, and high schools throughout the United States.]

1. Are there any 8th grade students in your household?

	SELECT ONE
Yes	1	(CONTINUE)
No	2	(THANK AND TERMINATE)

We are conducting research interviews to explore how students work through science digitally based tasks. Our goal is to ensure that the tasks are clear and understandable to a broad range of students. If selected to participate, your child will receive $40 to thank <him OR her> for <his OR her> time. In addition, if transportation to the site is involved, you will also receive $25 as appreciation for bringing your child to and from the interview site.

The research interview will take place at <NAME OF INTERVIEW SITE AND LOCATION>. It can be scheduled at your convenience and will last about two hours. During the interview, your child will be asked to respond to one 15 or 30-minute science task on a tablet computer and then talk to the interviewer about his or her experience completing the task.

CONTINUE TO Q. 2

Introduction when calling a parent/guardian who responded to a flyer:

Good morning/afternoon/evening. My name is <NAME>. I’m calling from the American Institutes for Research on behalf of the National Center for Education Statistics, part of the U.S. Department of Education. Thank you for contacting us about the possibility of your child participating in a research interview to help evaluating a new digitally based science test of the National Assessment of Educational Progress, or NAEP.

We are conducting research interviews to explore how students work through science digital-based tasks. Our goal is to ensure that the tasks are clear and understandable to a broad range of students. If selected to participate, your child will receive $40 as thanks for his or her time. In addition, if transportation to the site is involved, you will also receive $25 as appreciation for bringing your child to and from the interview site.

The research interview will take place at <NAME OF INTERVIEW SITE AND LOCATION>. It can be scheduled at your convenience and will last about two hours. During the interview, your child will be asked to respond to one 15 or 30-minute science task on a tablet computer and then talk to the interviewer about his or her experience completing the task.

1. In what grade is your child?

	SELECT ONE
Grade 8	1	(CONTINUE TO Q. 2)
Not
Not Grade 8	2	(THANK AND TERMINATE)

CONTINUE TO Q. 2

2. If your child qualifies, do you give your permission for us to invite him/her to participate?

	SELECT ONE
Yes	1	(CONTINUE)
No	2	(THANK AND TERMINATE)

3. What is your child’s sex? (READ THE QUESTION AND RESPONSE OPTIONS)

	SELECT ONE
Male	1	(ATTEMPT TO RECRUIT A MIX)
Female	2

4. From your perspective, on a scale from 1 to 4 where 1 means “not at all comfortable,” and 4 means “very comfortable,” how comfortable would your child be expressing his/her thoughts to a researcher he/she has never met before? (READ LIST)

	SELECT ONE
Very Comfortable	4	(CONTINUE)
Somewhat Comfortable	3
Not Too Comfortable	2	(THANK AND TERMINATE)
Not At All Comfortable	1

If your child participates in the research study, you must sign and return a parental consent form. Your child can either bring the signed form to the interview or you can scan it and email it back. Without a signed form, your child cannot participate in the interview and receive the 40 dollars. If your child chooses to participate, I will send you a consent form.

5. Would you like me to send the consent form by e-mail or by FedEx mail?

SELECT ONE
e-mail	(CONTINUE TO 5A)
postal mail	(CONTINUE TO 5B)

5A. What is your e-mail address?

RECORD PARENT’S/GUARDIANS EMAIL ADDRESS:

____________________________________________________________________________________

5B. For sending the consent form via FedEx, we need your mailing address and a telephone number. What mailing address and phone number should we use?

[IF THE PARENT/GUARDIAN PROVIDES A PO BOX, NOTE THAT FEDEX DOESN’T DELIVER TO PO BOXES SO WE NEED A STREET ADDRESS]

RECORD PARENT’S/GUARDIAN’S MAILING ADDRESS AND PHONE NUMBER:

____________________________________________________________________________________

The next few questions are for classification purposes only.

6. Does your child participate in the National School Lunch program?

   	SELECT ONE
   Yes	1	→(ATTEMPT TO RECRUIT A MIX)
   No	2
   Prefer not to answer	3

7. What is the main language spoken in your home?

→ (ATTEMPT TO RECRUIT A MIX OF ENGLISH SPEAKING AND OTHER)

8. We are conducting interviews between <DATE> and <DATE>. If your child chooses to participate, what day and time would work best for you to take [HIM/HER] to the interview?

[CHECK SCHEDULE FOR RESPONDENT’S LOCATION FOR AVAILABLE TIMES. IF NECESSARY, SUGGEST A DIFFERENT PLACE OR TIME.]

RECORD DATE AND TIME:

____________________________________________________________________________________

9. Now that you have agreed for your child to participate in the study if he/she is selected, I would like to be able to ask your child a few questions, during which you are welcome to stay on the line. Is he/she available to speak with me at this time? (If not, get a call back time and call back) What is your student’s name?

RECORD NAME:

_________________________________________________________________________________

Again, you are welcome to stay on the line, but please allow your student to speak for him/herself. Thank you.

Script for eighth grade student:

RECRUITER: Please determine how talkative and articulate the student is. It is critical that we speak with students who are comfortable expressing themselves with someone they are meeting for the first time. Please use your best judgment and recruit only students that are thoughtful and freely express themselves in a manner that is relevant to the topic being discussed.

All recruits MUST be articulate students who can answer open-ended questions, and can provide support or explanation for their responses (e.g., “I like it because...,” “I don’t like it because...”)

DO NOT recruit any who have difficulty answering the questions, whose parent/guardian answers FOR them, who seem hesitant or shy speaking with you (as an adult they do not know), who answer “I don’t know,” or who CANNOT provide an explanation supporting their responses.

Hi, <STUDENT’S NAME>. My name is and I have been speaking with your parent/guardian for a few minutes.

As I explained to your parent/guardian, I work for a research company and we are conducting an important study to help us evaluate science tasks for a nationwide student assessment, the National Assessment of Educational Progress, or NAEP. For the research interview, you will be asked to visit <LOCATION> and respond to a sample science task. Your response will not be graded. The National Center for Education Statistics simply wants to know how you would do the task if it were to be included in the National Assessment of Educational Progress.

The task focuses on science and is interesting and fun to do. You will be asked to do the task on a tablet computer. To thank you for participating, you will receive $40. The interview will last about two hours. Your participation is totally voluntary. If you participate, no one but the study’s researchers will see your interview responses and all other interview materials, and we will not use your name to identify you.

10. Do you think you would be interested in participating if selected?

	SELECT ONE
Yes	1	(CONTINUE)
No	2	(READ THE FOLLOWING)

Thank you for your time. I will need to let your parent/guardian know that you would not like to participate. Is he or she available?

TO PARENT/GUARDIAN: Your child has decided not to participate, but I would like to thank you very much for taking the time to speak with us about this opportunity.

11. What grade are you in?

RECORD GRADE

(MUST BE IN 8^TH GRADE, OTHERWISE THANK AND TERMINATE)

12. What type of school do you currently attend? (READ LIST)

	SELECT ONE
Public school	1	(RECRUIT A MIX, BUT OK IF EVERYONE IS FROM PUBLIC SCHOOL)
Private school	2
Don’t know	3	(THANK AND TERMINATE)

13. How comfortable do you think you would feel talking with a researcher you haven’t met before about how clear and understandable the survey questions are? For example, questions about how much you use computers. Would you be: (READ LIST)

	SELECT ONE
Very Comfortable	4	(CONTINUE)
Somewhat Comfortable	3
Not Too Comfortable	2	(THANK AND TERMINATE)
Not At All Comfortable	1

14. Please tell me what your favorite subject is in school and why. (WRITE VERBATIM)

NOTE TO INTERVIEWERS: IT IS EXTREMELY IMPORTANT THAT STUDENTS IN THIS STUDY BE ARTICULATE. IF STUDENT CANNOT OR WILL NOT EASILY GIVE A ONE TO TWO SENTENCE UNPROMPTED ANSWER IN WELL UNDERSTOOD ENGLISH, PLEASE TERMINATE.

Invitation:

You qualify to participate in an interview session with a researcher in < LOCATION> to respond to a sample science task. Your parent/guardian is available to bring you to the interview on <DAY/DATE AND TIME>. The session will last two hours. You will receive $40 for participating. Your parent/guardian gave permission for you to attend if you are available. Will you be able to attend the interview?

	SELECT ONE
Yes	1	(CONTINUE)
No	2	(THANK AND TERMINATE)

To attend the session and receive your payment we must receive the parental consent form that we will [e-mail OR send by FedEx] to your parent/guardian. Your parent/guardian can scan and e-mail the consent form to us or you can bring it with you and give it to the interviewer. Do you understand that you must have a completed and signed parental consent form to participate?

	SELECT ONE
Yes	1
No	2

Appendix BB: Consent Form for Parent/Guardian of Student Participants

PARENTAL CONSENT FORM

FOR PARTICIPATION IN RESEARCH

American Institutes for Research (AIR)

Title of Research Study: Cognitive Laboratory Interviews for NAEP Science Scenerio Based Tasks

Thank you for your interest in participating in the research study described below. This consent form explains the research study. Please read it carefully, and feel free to ask questions about anything you do not understand. If you do not have questions now, you may ask them later as they occur to you.

1. What is this study about?

The National Center for Education Statistics (NCES), part of the U.S. Department of Education, is conducting research interviews with students currently in grade 8 about new digitally based science tasks, delivered on tablet computers, for the National Assessment of Educational Progress (NAEP). Students’ input will help ensure that the tasks are clear and understandable to a broad range of students.

2. Where will the research interviews take place?

The interviews will take place at <NAME OF INTERVIEW LOCATION>, located at <LOCATION>. There will be more than one adult present.

3. What will happen during the research interview?

At the start of a session, your child will be given a brief introduction to taking NAEP on a tablet computer. Following this introduction, your child will complete one 15 or 30-minute science task on a tablet computer. Computer software will record how your child responded to the task shown on the screen. After the task is completed, the administrator will play back a recording of your child’s performance and ask her/him to explain what they were thinking as they worked through the task. Students will also complete a short questionnaire about their experiences learning science and working with computers.

4. How long will the research interview last?

The interview will last up to 2 hours. The interview requires only one visit to the interview site.

5. Will you keep information private and confidential?

Your child will be assigned a unique student identifier (ID), and at no time will his/her name be linked to any of his/her answers. Your child’s responses will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573]. Audio, video, and screen activity will be recorded for analysis, but no personal identifying information will be recorded or retained.

6. What are the possible risks of being in the study?

There are no known risks to participating in this study.

7. Whom can I contact with questions or for further information?

<CONTACT NAME> is coordinating this session at AIR on behalf of NCES. If you have any questions, please call <CONTACT PHONE> or e-mail <CONTACT E-MAIL>.

8. Is participation required?

Participation in this study is completely voluntary. Your child has the right to refuse to answer particular questions. Your child may elect to withdraw from this study at any time and can end early if he/she wishes.

By signing below, you agree that your child, _______________________________, may participate in this study. Your child will receive $40 at the conclusion of the interview.

In addition, if transportation to the site is involved, you will receive $25 to thank you for bringing your child to and from the interview site.

Your signature below will indicate that your questions have been answered satisfactorily, and that you have read and understood the information provided above.

Print Name: ___________________________________________________________________________

Signature: ______________________________________________________Date: __________________

E-mail address: _________________________________________________________________________

Appendix CC: Elliot Benson/Nichols Research Recruitment Phone Script for Parents/Guardians of Student Participants

Script when calling parents/guardians from a previously compiled list:

Good morning/afternoon/evening. Is this the <NAME FROM CALL LIST> household? My name is <CALLER NAME>, and I am calling from <Elliot Benson/Nichols Research>. We are helping the American Institutes for Research (AIR) recruit students for a study being conducted on behalf of the U.S. Department of Education’s National Center for Education Statistics. This study is strictly for research purposes. Your participation is voluntary, and you will receive no sales pitch or follow-up calls based on your participation. I’d like to tell you a little bit about the study first; is now a good time? (YES or NO and call back later)

We are recruiting students currently in grades 8 to help us evaluate new digitally based science tasks, delivered on tablet computers. These tasks are part of the National Assessment of Educational Progress, which is delivered to students nationwide. [If parent /guardian asks what NAEP is: This national assessment is administered to students in elementary, middle, and high schools throughout the United States.]

1. Are there any 8th grade students in your household?

	SELECT ONE
Yes	1	(CONTINUE)
No	2	(THANK AND TERMINATE)

We are conducting research interviews to explore how students work through science digitally based tasks. Our goal is to ensure that the tasks are clear and understandable to a broad range of students. If selected to participate, your child will receive $40 to thank <him OR her> for <his OR her> time. In addition, if transportation to the site is involved, you will also receive $25 to thank you for bringing your child to and from the interview site.

The research interview will take place at <NAME OF INTERVIEW SITE AND LOCATION>. It can be scheduled at your convenience and will last about two hours. During the interview, your child will be asked to respond to one 15 or 30-minute science task on a tablet computer and then talk to the interviewer about his or her experience completing the task.

2. If your child qualifies, do you give your permission for us to invite him/her to participate?

	SELECT ONE
Yes	1	(CONTINUE)
No	2	(THANK AND TERMINATE)

3. What is your child’s sex?  (READ THE QUESTION AND RESPONSE OPTIONS)

	SELECT ONE
Male	1	(ATTEMPT TO RECRUIT A MIX)
Female	2

4. From your perspective, on a scale from 1 to 4 where 1 means “not at all comfortable,” and 4 means “very comfortable,” how comfortable would your child be expressing his/her thoughts to a researcher he/she has never met before? (READ LIST)

	SELECT ONE
Very Comfortable	4	(CONTINUE)
Somewhat Comfortable	3
Not Too Comfortable	2	(THANK AND TERMINATE)
Not At All Comfortable	1

If your child participates in the research study, you must sign and return a parental consent form. Your child can either bring the signed form to the interview or you can scan it and email it back. Without a signed form your child cannot participate in the interview and receive the $40 payment. If your child chooses to participate, I will send you a consent form.

5. Would you like me to send the consent form by e-mail or by FedEx mail?

SELECT ONE
e-mail	(CONTINUE TO 5A)
postal mail	(CONTINUE TO 5B)

5A. What is your e-mail address?

RECORD PARENT’S/GUARDIAN’S EMAIL ADDRESS:

____________________________________________________________________________________

5B. For sending the consent form via FedEx, we need your mailing address and a telephone number. What mailing address and phone number should we use?

[IF PARENT/GUARDIAN PROVIDES A PO BOX, NOTE THAT FEDEX DOESN’T DELIVER TO PO BOXES SO WE NEED A STREET ADDRESS]

RECORD PARENT’S/GUARDIAN’S MAILING ADDRESS AND PHONE NUMBER:

____________________________________________________________________________________

The next few questions are for classification purposes only.

6. Does your child participate in the National School Lunch program?

   	SELECT ONE
   Yes	1	→(ATTEMPT TO RECRUIT A MIX)
   No	2
   Prefer not to answer	3

7. What is the main language spoken in your home?

→ (ATTEMPT TO RECRUIT A MIX OF ENGLISH SPEAKING AND OTHER)

8. We are conducting interviews between <DATE> and <DATE>. If your child chooses to participate, what day and time would work best for you to take [HIM/HER] to the interview?

[CHECK SCHEDULE FOR RESPONDENT’S LOCATION FOR AVAILABLE TIMES. IF NECESSARY, SUGGEST A DIFFERENT PLACE OR TIME.]

RECORD DATE AND TIME:

____________________________________________________________________________________

9. Now that you have agreed for your child to participate in the study if he/she is selected, I would like to be able to ask your child a few questions, during which you are welcome to stay on the line. Is he/she available to speak with me at this time? (If not, get a call back time and call back) What is your student’s name?

RECORD NAME:

_________________________________________________________________________________

Again, you are welcome to stay on the line, but please allow your student to speak for him/herself. Thank you.

Script for eighth grade student:

RECRUITER: Please determine how talkative and articulate the student is. It is critical that we speak with students who are comfortable expressing themselves with someone they are meeting for the first time. Please use your best judgment and recruit only students that are thoughtful and freely express themselves in a manner that is relevant to the topic being discussed.

All recruits MUST be articulate students who can answer open ended questions, and can provide support or explanation for their responses (e.g., “I like it because...,” “I don’t like it because...”)

DO NOT recruit any who have difficulty answering these questions, whose parent/guardian answers FOR them, who seem hesitant or shy speaking with you (as an adult they do not know), who answer “I don’t know,” or who CANNOT provide an explanation supporting their responses.

Hi, <STUDENT’S NAME>. My name is and I have been speaking with your parent/guardian for a few minutes.

As I explained to your parent/guardian, I work for a research company and we are conducting an important study to help us evaluate science tasks for a nationwide student assessment, the National Assessment of Educational Progress, or NAEP. For the research interview, you will be asked to visit <LOCATION> and respond to a sample science task. Your response will not be graded. The National Center for Education Statistics simply wants to know how you would do the task if it were to be included in the National Assessment of Educational Progress.

The task focuses on science and is interesting and fun to do. You will be asked to do the task on a tablet computer. To thank you for participating, you will receive $40. The interview will last about two hours. Your participation is totally voluntary. If you participate, no one but the study’s researchers will see your interview responses and all other interview materials, and we will not use your name to identify you.

10. Do you think you would be interested in participating if selected?

	SELECT ONE
Yes	1	(CONTINUE)
No	2	(READ THE FOLLOWING)

Thank you for your time. I will need to let your parent/guardian know that you would not like to participate. Is he or she available?

TO PARENT/GUARDIAN: Your child has decided not to participate, but I would like to thank you very much for taking the time to speak with us about this opportunity.

11. What grade are you in?

RECORD GRADE

(MUST BE IN 8^TH GRADE, OTHERWISE THANK AND TERMINATE)

12. What type of school do you currently attend? (READ LIST)

	SELECT ONE
Public school	1	(RECRUIT A MIX, BUT OK IF EVERYONE IS FROM PUBLIC SCHOOL)
Private school	2
Don’t know	3	(THANK AND TERMINATE)

13. How comfortable do you think you would feel talking with a researcher you haven’t met before about how clear and understandable the survey questions are? For example, questions about how much you use computers. Would you be: (READ LIST)

	SELECT ONE
Very Comfortable	4	(CONTINUE)
Somewhat Comfortable	3
Not Too Comfortable	2	(THANK AND TERMINATE)
Not At All Comfortable	1

14. Please tell me what your favorite subject is in school and why. (WRITE VERBATIM)

NOTE TO INTERVIEWERS: IT IS EXTREMELY IMPORTANT THAT STUDENTS IN THIS STUDY BE ARTICULATE. IF STUDENT CANNOT OR WILL NOT EASILY GIVE A ONE TO TWO SENTENCE UNPROMPTED ANSWER IN WELL UNDERSTOOD ENGLISH, PLEASE TERMINATE.

Invitation:

You qualify to participate in an interview session with a researcher in < LOCATION> to respond to a sample science task. Your mother/father is available to bring you to the interview on <DAY/DATE AND TIME>. The session will last two hours. You will receive $40 for participating. Your parent/guardian gave permission for you to attend if you are available. Will you be able to attend the interview?

	SELECT ONE
Yes	1	(CONTINUE)
No	2	(THANK AND TERMINATE)

To attend the session and receive your payment we must receive the parental consent form that we will [e-mail OR send by FedEx] to your parent/guardian. Your parent/guardian can scan and e-mail the consent form to us, or you can bring it with you and give it to the interviewer. Do you understand that you must have a completed and signed parental consent form to participate?

	SELECT ONE
Yes	1
No	2