National Assessment of Educational Progress (NAEP) Writing Tasks Cognitive Interviews

Good morning/afternoon/evening. My name is <CONTACT NAME> and I am contacting you from Educational Testing Service (ETS). We are conducting an important study for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. We are recruiting students who are currently in grades 8 or 12 to participate in research interviews supporting the development of new writing tasks for the National Assessment of Educational Progress (NAEP). The research interviews will explore how students work through tasks on a tablet computer to ensure that the tasks are clear and understandable to a broad range of students. NCES simply wants to know how students would respond to the tasks if they were on an actual test, and would like to hear students’ thoughts to improve the tasks and make them easier to understand.

The voluntary research interviews will last no more than 90 minutes for 8^th or 12^th graders, respectively. Students will be asked to respond to several writing tasks and will be asked about them. Research staff will take notes of students’ reactions to these tasks and will subsequently use students’ feedback to revise and refine the tasks. We will observe and ask about the students’ level of interest in, and understanding of, the tasks. Students will also be asked whether they have any suggestions about how to enhance the tasks.

Student responses will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by law. [Recruiter note: If asked what law is being referenced, please provide its name: Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573.

We would like your help in distributing and posting flyers. We will follow up directly with those who contact us. We will hold some sessions at <LOCATION>; however, if the parents/guardians of students cannot provide transportation to <LOCATION>, we would like to set up sessions at your school.

For each participating student, a $25 gift card (from a major credit card company) will be issued in appreciation for participants’ time. In addition, if transportation to the site is involved, a parent or legal guardian of each student will receive a $25 gift card as appreciation for bringing the child to and from the interview site.

These interviews are an important stage of the NAEP test development process. The comments of students attending your school will help NCES improve test questions.

Would your school be interested in participating in or helping us to recruit students for this research study?

If NO: Thank you for your time. Have a wonderful day/evening.

If YES: Good, thank you very much. To ensure that we interview a broad mix of students and determine the best interview location, I have a few brief screening questions to ask you. It should take only a few minutes.

Screener

[Recruiter: Determine recruitment needs in advance. Look in public documents and school website to find what grades a school includes, the size of each grade, the SES, and racial/ethnic makeup of the student body. Record all responses to Screener questions. End Screener at whatever point the school staff member gives a response that confirms that their group has already been sampled adequately. End screener script: “Based on the requirements of this study, we are not able to include you in the interviews at this time.”]

Do you have <##> [specify recruitment needs] 8th graders who might be available to participate in the study?
Do you have <##> [specify recruitment needs] 12th graders who might be available to participate in the study?

Thank you for taking the time to answer the screening questions. Your school is generally eligible to participate in the research interviews.

Would you be able to distribute and post flyers regarding this study? After receiving a contact of interest, an ETS staff member would follow up with the parent/guardian by phone and ask them some screening questions to ensure that a diverse sample of students is included in this study.

If NO: Thank you for your time. Have a good day/evening. [Interviewer: End call.]

If YES: Would you be able to make a room at your school available where ETS could conduct some 90-minute research sessions?

If YES: That’s great, we will contact you to schedule some sessions at your school after we have some students signed up. (Interviewer: go to Closing]

If NO:

(Closing)

Thank you for agreeing to disseminate information about this important research study. We will send you e-mail with informational flyers.

We may need to reach you by telephone. Is the number I called the best number to use?

Do you have any questions at this time?

[Recruiter: Answer any questions.]

If you do have any questions, please call <PHONE NUMBER>.

Thank you again for agreeing to be part of this important research project.

Appendix G: ETS Recruitment Phone Script for Parents/Guardians of Student Participants

Script when returning calls from potential participants, or calling parents from a previously compiled list:

Good morning/afternoon/evening. Is this the <NAME FROM CALL LIST> household? My name is <CALLER NAME>, and I am calling from Educational Testing Service (ETS). We are conducting a study on behalf of the U.S. Department of Education’s National Center for Education Statistics. We are recruiting students currently in grades 8 or 12 to sign up to participate in a research study supporting the development of new writing tasks for the National Assessment of Educational Progress or NAEP. [If parent asks what this is: This national assessment is administered to students in elementary, middle, and high schools throughout the United States.]

Are there any 8th or 12th grade students in your household?

Yes (Great! Let me give you some additional information about the study.)

No (Thank you, have a good day/evening.)

We are conducting research interviews to explore how students work through sample writing tasks to ensure that they are clear and understandable to a broad range of students. If selected to participate, your child will receive a $25 gift card (from a major credit card company) as thanks for his or her time. In addition, if transportation to the site is involved, you will also receive a $25 gift card as appreciation for bringing your child to and from the interview site.

The research interview will take place at <NAME OF INTERVIEW SITE AND LOCATION>. It can be scheduled at your convenience and will last about 90 minutes. During the interview, your child will be asked to respond to sample writing tasks on a tablet computer. NCES simply wants to know: if these tasks were to be included in an actual NAEP assessment, how would students respond to them?

These research interviews are an important stage of the NAEP test development process. By sharing their feedback with us, students can directly guide the future of this nationwide assessment by helping NCES evaluate the tasks to create the best test possible.

Do you believe that your child would be interested in participating?

No (Thank you anyway, have a good day/evening.) (END CALL)

Yes…

Great! To ensure that we interview a broad mix of students, I have a few brief questions to ask about you and your child. It should take only a few minutes. First, I want to collect some additional information about your child to find out whether he or she meets the participation requirements.

(Skip to Screener, on following page.)

Script when calling a parent/guardian who responded to email or letter

Good morning/afternoon/evening. My name is <NAME>. I’m calling from Educational Testing Service (ETS) on behalf of the National Center for Education Statistics), part of the U.S. Department of Education. Thank you for contacting us about the possibility of your child participating in a research interview to help develop writing tasks for the National Assessment of Educational Progress, or NAEP. To ensure that we interview a broad mix of students, I have a few brief screening questions to ask about you and your child. It should take only a few minutes. First, I want to collect some additional information about your child to find out whether he or she meets the participation requirements.

Screener

(INTERVIEWER: Determine recruitment needs in advance. Record all responses. End Screener at whatever point the parent or legal guardian gives a response that confirms that their group has already been sampled adequately. End Screener script: “Thank you for answering my questions. Based on the requirements of this study, we are not able to include your child in the interviews at this time.”)

What is your child’s name?

What grade is your child currently in? (RECRUITER: DO NOT READ LIST)

 K – 7^th  (Unfortunately, we are interviewing only students currently in grade 8 or grade 12 for this study. Thank you for your time. Have a good day/evening.)

 8^th – Good.

 9^th – 11^th (Unfortunately, we are interviewing only students currently in grade 8 or grade 12 for this study. Thank you for your time. Have a good day/evening.)

 12^th – Good.

What school does your child attend? (INTERVIEWER: Note if homeschooled)

____________________________________________________________

What school district does he or she attend? (INTERVIEWER: Note if parochial, charter, homeschooled) __________________________________________________

What is your child’s race or ethnicity? (INTERVIEWER: Read list only if parent doesn’t immediately reply. “Are you…?”)

Ethnicity:  Hispanic or Latino  Not Hispanic or Latino

Race (one or more of the categories below):

 American Indian or Alaska Native

 Asian

 Black or African American

 Hispanic or Latino

 Native Hawaiian or other Pacific Islander

 White or Caucasian

Just to be sure we're interviewing a broad range of students for this study, does your child participate in the National School Lunch program?

[If the respondent indicates they prefer not to disclose this information, the interviewer should say “That’s fine” and move on to the next question.]

Would you consider the area you live in to be …

Urban

Suburban

Rural

Ok. Thank you and what is the zip code of the area that you live in?

________________________

Thank you. We would like to speak with your child to confirm if (he/she) is interested in participating in the research interview. It will take only a couple of minutes and then I will ask to speak with you again for another moment. Is your child available to speak with us now or should we call (him/her) back at another time?

 Yes, the child is available for screening questions  (SKIP TO SCRIPT FOR 8th or 12th GRADE STUDENTS)

 No, please call back.

Is the phone number you gave me earlier the best number to call?

Other phone numbers: _________________________________

Is there a good time for me to call? When would be best for you?

INTERVIEWER: IF CHILD NEEDS TO RECEIVE A CALL BACK, RECORD NAME, PHONE NUMBER, DATE/TIME

Parent/Guardian Name ______________________________________________

Child Name ________________________________________________________

Parent Phone ______________________________________________________

Date ______________________________________________________________

Time ______________________________________________________________

SCRIPT FOR 8th or 12th GRADE STUDENTS

Good morning/afternoon/evening. My name is <NAME> and I’m calling from Educational Testing Service (ETS). We are doing an important study for the National Center for Education Statistics, part of the U.S. Department of Education.

We have spoken to your parent and we are pleased that you may be interested in helping us develop writing tasks for a nationwide student assessment, the National Assessment of Educational Progress, or NAEP. Before I tell you more about what you would be asked to do, I need to ask you one question:

What grade are you currently in? (RECRUITER: DO NOT READ LIST)

 K – 7^th  (Unfortunately, we are interviewing only students currently in grade 8 or grade 12 for this study. Thank you for your time. Have a good day/evening.)

 8^th – Good.

 9^th – 11^th (Unfortunately, we are interviewing only students currently in grade 8 or grade 12 for this study. Thank you for your time. Have a good day/evening.)

 12^th – Good.

Thank you. For the research interview, you will be asked to visit <LOCATION> and respond to a sample writing task. Your response will not be graded. The National Center for Education Statistics simply wants to know how you would do the task if it were to be included in the National Assessment of Educational Progress. The task focuses on writing and is interesting and fun to do. You will be asked to do the task on a tablet computer. Your input will help evaluate the test that will be given to students throughout the nation. To thank you for participating, you will receive a $25 gift card (from a major credit card company). The interview will last about 90 minutes. Your participation is totally voluntary. If you participate, no one but the study’s researchers will see your interview responses and all other interview materials, and we will not use your name to identify you.

Would you like to participate?

 Yes

 No  (THANK AND END CALL)

If NO: Thank you for your time. I will need to let your parent know that you would not like to participate. Is he or she available?

TO PARENT: Your child has decided not to participate, but I would like to thank you very much for taking the time to speak with us about this opportunity.

If YES: Thank you for agreeing to participate in this important research study. I will need to speak to your parent to talk about the date/time of the interview. Is your parent available?

 Yes

 No

INTERVIEWER: IF CALLING PARENT BACK, RECORD NAME, PHONE NUMBER, DATE/TIME

Parent/Guardian Name _________________________________

Child Name ____________________________________________

Parent Phone ___________________________________________

Date __________________________________________________

Time __________________________________________________

IF PARENT AVAILABLE AND CURRENTLY SCHEDULING INTERVIEWS

Thank you again for allowing your child to participate in the NAEP research interviews. We will be holding our current interviews from <DATE> to <DATE>. What would be the best date/time to schedule the interview?

Date ____________________

Time ____________________

Thank you. We will follow up with a confirmation email/letter and a consent form. What is your email or postal address for us to send these to?

____________________________________________________________________________________

IF PARENT AVAILABLE AND NOT CURRENTLY SCHEDULING INTERVIEWS

Thank you again for allowing your child to participate in the NAEP research interviews. We are in between interview sessions, and will need to call you back when we are actively scheduling the interviews to determine the date and time that will work best. What are your scheduling preferences? (SELECT ALL THAT APPLY)

 Morning

 Afternoon

 After school

 Weekends

Thank you so much for talking with us today. We look forward to communicating with you soon.

(END CALL)

Appendix H: ETS Recruitment Phone Script for Students Age 18 or Above Who Responded to Email, Advertisement, or Invitation

Good morning/afternoon/evening. My name is <NAME>. I’m calling from Educational Testing Service (ETS) on behalf of the National Center for Education Statistics, part of the U.S. Department of Education. Thank you for contacting us about the possibility of participating in a research interview to help develop writing tasks for the National Assessment of Educational Progress, or NAEP. [If student asks what this is: This national assessment is administered to students in elementary, middle, and high schools throughout the United States.]

To ensure that we interview a broad mix of students, I have a few brief screening questions to ask you. It should take only a few minutes. First, I want to collect some additional information about you to find out whether you meet the participation requirements.

Screener

(INTERVIEWER: Determine recruitment needs in advance. Record all responses. End Screener at whatever point the response confirms that this group has already been sampled adequately. End Screener script: “Thank you for answering my questions. Based on the requirements of this study, we are not able to include you in the interviews at this time.”)

What grade are you currently in at school? (INTERVIEWER: DO NOT READ LIST)

 K – 7^th  (Unfortunately, we are interviewing only students currently in grade 8 or grade 12 for this study. Thank you for your time. Have a good day/evening.)

 8^th – Good.

 9^th – 11^th (Unfortunately, we are interviewing only students currently in grade 8 or grade 12 for this study. Thank you for your time. Have a good day/evening.)

 12^th – Good.

What school do you attend? (INTERVIEWER: Note if homeschooled)

____________________________________________________________

What school district do you attend? (INTERVIEWER: Note if parochial, charter, homeschooled) __________________________________________________

What is your race or ethnicity? (INTERVIEWER: Read list only if student doesn’t immediately reply. “Are you…?”)

Ethnicity:  Hispanic or Latino  Not Hispanic or Latino

Race (one or more of the categories below):

 American Indian or Alaska Native

 Asian

 Black or African American

 Hispanic or Latino

 Native Hawaiian or other Pacific Islander

 White or Caucasian

Just to be sure we're interviewing a broad range of students for this study, do you participate in the National School Lunch program?

[If the respondent indicates they prefer not to disclose this information, the interviewer should say “That’s fine” and move on to the next question.]

Would you consider the area you live in to be …

Urban

Suburban

Rural

Ok. Thank you and what is the zip code of the area that you live in?

________________________

Thank you. For the research interview, you will be asked to visit <LOCATION> and respond to a sample writing task. Your response will not be graded. The National Center for Education Statistics simply wants to know how you would do the task if it were to be included in the National Assessment of Educational Progress. The task focuses on writing and is interesting and fun to do. You will be asked to do the task on a tablet computer. Your input will help evaluate the test that will be given to students throughout the nation. To thank you for participating, you will receive a $25 gift card (from a major credit card company). The interview will last about 90 minutes. Your participation is totally voluntary. If you participate, no one but the study’s researchers will see your interview responses and all other interview materials, and we will not use your name to identify you.

Would you like to participate?

 Yes

 No  (THANK AND END CALL)

IF CURRENTLY SCHEDULING INTERVIEWS

Thank you again for agreeing to participate in the NAEP research interviews. We will be holding our current interviews from <DATE> to <DATE>. What would be the best date/time to schedule the interview?

Date ____________________

Time ____________________

IF NOT CURRENTLY SCHEDULING INTERVIEWS

Thank you again for agreeing to participate in the NAEP research interviews. We are in between interview sessions, and will need to call you back when we are actively scheduling the interviews to determine the date and time that will work best. What are your scheduling preferences? (SELECT ALL THAT APPLY)

 Morning

 Afternoon

 After school

 Weekends

Thank you so much for talking with us today. We look forward to communicating with you soon.

(END CALL)

Appendix L: ETS Consent Form for Parent/Guardian of Student Participants (Below Age 18)

PARENTAL CONSENT FORM

FOR PARTICIPATION IN RESEARCH

Educational Testing Service (ETS)

Title of Research Study: Cognitive Laboratory Interviews for NAEP Writing

Thank you for your interest in participating in the research study described below. This consent form explains the research study. Please read it carefully, and feel free to ask questions about anything you do not understand. If you do not have questions now, you may ask them later as they occur to you.

What is this study about?

The National Center for Education Statistics (NCES), part of the U.S. Department of Education, is conducting research interviews with students currently in grades 8 and 12 about new test questions for the National Assessment of Educational Progress (NAEP). The questions we are testing focus on writing and will be part of a computer-based test. In these interviews, students will see early versions of computer-based tasks. We will ask questions that focus on students’ ability to understand the questions presented to them. Students will also be asked whether they have any suggestions about how we might improve the tasks.

Where will the research interviews take place?

The interviews will take place at <NAME OF INTERVIEW LOCATION>, located at <LOCATION>. There will be more than one adult present.

What will happen during the research interview?

Each participant will take an early version of a new NAEP writing task on a tablet computer. An interviewer will ask what your child was thinking as he or she completed the task, and whether he or she believes the task is clear and understandable. Your child’s responses will not be graded. ETS staff will use computer software to record what students do on screen, as well as microphones to record students’ reactions to tasks, descriptions of their work, and responses to interviewer questions. Interviewers will also take notes about students’ reactions to the writing tasks so that tasks can be revised and refined.

How long will the research interview last?

The interview will last up to 90 minutes. The interview requires only one visit to the interview site.

Will you keep information private and confidential?

Your child will be assigned a unique student identifier (ID), and at no time will his/her name be linked to any of his/her answers. Your child’s responses will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573]. Audio and screen activity will be recorded for analysis, but no personal identifying information will be recorded or retained.

Will the results go to my child’s school?

Your child’s answers will not be disclosed to his/her school, and will not be linked to any personally identifiable information such as the school’s name.

What are the possible risks of being in the study?

There are no known risks to participating in this study.

Whom can I contact with questions or for further information?

<CONTACT NAME> is coordinating this session at ETS on behalf of NCES. If you have any questions, please call <CONTACT PHONE> or e-mail <CONTACT E-MAIL>.

Can participation end early?

Participation in this study is completely voluntary. Your child has the right to refuse to answer particular questions. Your child may elect to withdraw from this study at any time and can end early if he/she wishes.

By signing below, you agree that your child, _______________________________, may participate in this study. Your child will receive a $25 gift card (from a major credit card company) at the conclusion of the interview.

In addition, if transportation to the site is involved, you will receive a $25 gift card as appreciation for the time bringing your child to and from the interview site.

Your signature below will indicate that your questions have been answered satisfactorily, and that you have read and understood the information provided above.

Print Name: ___________________________________________________________________________

Signature:______________________________________________________Date:__________________

E-mail address:_________________________________________________________________________

School that student attends:___________________________________________________________

Appendix M: ETS Consent Form for Student Participants Age 18 or Above

CONSENT FORM

FOR PARTICIPATION IN RESEARCH

Educational Testing Service (ETS)

Title of Research Study: Cognitive Laboratory Interviews for NAEP Writing

What is this study about?

Where will the research interviews take place?

The interviews will take place at <NAME OF INTERVIEW LOCATION>, located at <LOCATION>.

What will happen during the research interview?

Each participant will take an early version of a new NAEP writing task on a tablet computer. An interviewer will ask what you were thinking as you completed the task, and whether you believe that the task is clear and understandable. Your responses will not be graded. ETS staff will use computer software to record what students do on screen, as well as microphones to record students’ reactions to tasks, descriptions of their work, and responses to interviewer questions. Interviewers will also take notes about students’ reactions to the writing tasks so that tasks can be revised and refined.

How long will the research interview last?

The interview will last up to 90 minutes. The interview requires only one visit to the interview site.

Will you keep information private and confidential?

You will be assigned a unique student identifier (ID), and at no time will your name be linked to any of your answers. All of your responses will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573]. Audio and screen activity will be recorded for analysis, but no personal identifying information will be recorded or retained.

Will the results go to my school?

Your answers will not be disclosed to your school, and will not be linked to any personally identifiable information such as the school’s name.

What are the possible risks of being in the study?

There are no known risks to participating in this study.

Whom can I contact with questions or for further information?

<CONTACT NAME> is coordinating this session at ETS on behalf of NCES. If you have any questions, please call <CONTACT PHONE> or e-mail <CONTACT E-MAIL>.

Can my participation end early?

Your participation in this study is completely voluntary and you have the right to refuse to answer particular questions. You may also elect to withdraw from this study at any time, and you can end early if you wish.

By signing below, you agree that you will participate in this research study. You will receive a $25 gift card (from a major credit card company) at the conclusion of each session. In addition, if transportation to the site is involved, your parent or legal guardian will receive a $25 gift card in appreciation for the time bringing you to and from the interview site. Your signature below will indicate that your questions have been answered satisfactorily, and that you have read and understood the information provided above.

Print Name: ___________________________________________________________________________

Signature: ______________________________________________________Date:___________________

E-mail address:_________________________________________________________________________

School that student attends:______________________________________________________________

Appendix S: EurekaFacts Recruitment Phone Script for Schools of Potential Student Participants

Hello. May I please speak to <NAME of contact at school>?

Good morning/afternoon/evening. My name is <CONTACT NAME> and I am contacting you from EurekaFacts, a local research company. We are conducting an important study for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. We are recruiting students who are currently in grades 8 or 12 to participate in research sessions supporting the development of new writing tasks for the National Assessment of Educational Progress (NAEP). The research sessions will explore how students work through the tasks on tablet computer to ensure that the tasks are clear and understandable to a broad range of students. NCES simply wants to know how students would do the tasks if they were on an actual test, and would like to hear students’ thoughts to improve the tasks and make them easier to understand.

The voluntary research sessions will last no more than 90 minutes. Students will be asked to respond to writing tasks and will be asked about them. EurekaFacts staff will take notes of students’ reactions to the tasks, and students’ feedback will be used to revise and refine the tasks. We will observe and ask about the students’ understanding of the tasks. Students may also be asked whether they have any suggestions about how to enhance the tasks.

Student responses will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by law.

[INTERVIEWER note: If asked what law is being referenced, please provide its name: Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573.]

We would like your help in distributing flyers to students, and we will follow up directly with any who contact us. We will hold some sessions at <LOCATION>; however, if the parents/guardians of students cannot provide transportation to <LOCATION>, we would like to set up sessions at your school.

These sessions are an important stage of the NAEP test development process. The comments of students attending your school will help NCES improve test questions.

Would your school be interested in participating in or helping us to recruit students for this research study?

If NO: Thank you for your time. Have a good day/evening.

If YES: Great, thank you very much. To ensure that we work with a broad mix of students and determine the best session location, I have a few brief screening questions to ask you. It should take only a few minutes.

Screener

[INTERVIEWER: Determine recruitment needs in advance. Look in public documents and school website to find what grades a school includes, the size of each grade, the SES, and racial/ethnic makeup of the student body. Record all responses to screener questions. End screener at whatever point the school staff member gives a response that confirms that their group has already been sampled adequately. End screener script: “Based on the requirements of this study, we are not able to include you in the study at this time.”]

Are you at a school with a population that includes:

8th-grade students
12th-grade students

How many students are enrolled in your school?
1. 299 students or fewer
2. 300-599 students
3. 600 students or more

Eligible Participant Script

Thank you for taking the time to answer the screening questions. Students at your school are generally eligible to participate in the research sessions. After receiving a contact of interest, a EurekaFacts staff member will follow up with the parent/guardian by phone and ask them some screening questions to ensure that a diverse sample of students is included in this study.

Would you be able to send flyers to parents of potential student participants?

If NO: Thank you for your time. Have a good day/evening. [Interviewer: End call.]

If YES: Would you be able to make a room at your school available where EurekaFacts could conduct some 90-minute research sessions?

If YES: That’s great, we will contact you to schedule some sessions at your school after we have some students signed up. (Interviewer: go to Closing]

If NO:

(Closing)

Thank you for agreeing to disseminate information about this important research study. We will send you e-mail with informational flyers.

Is [INSERT EMAIL ADDRESS] the best email to reach you? OR What is the best email address to reach you?

We may need to reach you by telephone. Is the number I called the best number to use?

Do you have any questions at this time?

[INTERVIEWER: Answer any questions.]

If you do have any questions, please call <PHONE NUMBER>.

Thank you again for agreeing to be part of this important research project. Have a wonderful day/evening.

Appendix X: EurekaFacts Recruitment Phone Script for Parents/Guardians of Student Participants (Below Age 18)

Script when calling parents from previously compiled list:

Good morning/afternoon/evening. Is this the <NAME FROM CALL LIST> household? My name is <CALLER NAME>, and I am calling from EurekaFacts, a local research company. We are conducting a study on behalf of the U.S. Department of Education’s National Center for Education Statistics.

We are recruiting students currently in grades 8 or 12 to participate in a research study supporting the development of new questions for the National Assessment of Educational Progress or NAEP. [If parent asks what this is: This national assessment is administered to students in elementary, middle, and high schools throughout the United States.]

Are there any 8th or 12th grade students in your household?

No: (Thank you, have a good day/evening.)

Yes: (Great! Let me give you some additional information about the study.)

We are looking for students in grade 8 and grade 12 to help us test new computer-delivered writing tasks. These tasks will be part of the National Assessment of Educational Progress (NAEP) delivered to students nationwide.

The research session will take place at <NAME OF SESSION SITE AND LOCATION>. It can be scheduled at your convenience and will last about 90 minutes. During the session, your child will be asked to respond to sample writing tasks on a tablet computer. NCES simply wants to know how students would do the tasks if they were on an actual test, and would like to hear students’ thoughts to improve the tasks and make them easier to understand.

If selected to participate, your child will receive a $25 gift card (from a major credit card company) as thanks for his or her time. In addition, if transportation to the site is involved, you will also receive a $25 gift card as appreciation for bringing your child to and from the research site.

These research sessions are an important stage of the NAEP test development process. By sharing their feedback with us, students can directly guide the future of this nationwide assessment by helping NCES evaluate the test questions and instructions to create the best test possible.

Do you believe that your child would be interested in participating?

No: (Thank you anyway, have a good day/evening.)

Yes: (Great! Thank you so much. To ensure that we work with a broad mix of students, I have a few brief questions to ask about you and your child. It should take only a few minutes. First, I want to collect some additional information about your child to find out whether he or she meets the participation requirements.)

(Skip to Screener, on following page.)

Script when calling a parent/guardian who responded to email or letter

Good morning/afternoon/evening. My name is <NAME>. I’m calling from EurekaFacts on behalf of the National Center for Education Statistics, part of the U.S. Department of Education. Thank you for contacting us about the possibility of your child participating in a research session to help assess test questions for the National Assessment of Educational Progress, or NAEP. To ensure that we work with a broad mix of students, I have a few brief screening questions to ask about you and your child. It should only take a few minutes. First, I want to collect some additional information about your child to find out whether he or she meets the participation requirements.

Screener

[INTERVIEWER: Determine recruitment needs in advance. Record all responses. End screener at whatever point the parent or legal guardian gives a response that confirms that their group has already been sampled adequately. End Screener script: “Thank you for answering my questions. Based on the requirements of this study, we are not able to include your child in the sessions at this time.”)

What is your child’s name?

What grade is your child currently in? (INTERVIEWER: DO NOT READ LIST)

 K – 7^th  (Unfortunately, we need only students currently in grade 8 or grade 12 for this study. Thank you for your time. Have a good day/evening.)

 8^th – Good.

 9^th – 11^th (Unfortunately, we need only students currently in grade 8 or grade 12 for this study. Thank you for your time. Have a good day/evening.)

 12^th – Good.

What school does your child attend? (INTERVIEWER: Note if homeschooled)

____________________________________________________________

What school district does he or she attend? (INTERVIEWER: Note if parochial, charter, homeschooled) __________________________________________________

What is your child’s race or ethnicity? (INTERVIEWER: Read list only if parent doesn’t immediately reply. “Are you…?”)

Ethnicity:  Hispanic or Latino  Not Hispanic or Latino

Race (one or more of the categories below):

 American Indian or Alaska Native

 Asian

 Black or African American

 Hispanic or Latino

 Native Hawaiian or other Pacific Islander

 White or Caucasian

Just to be sure we're working with a broad range of students for this study, does your child participate in the National School Lunch program?

 Yes

 No

 Refused to disclose

[If the respondent indicates they prefer not to disclose this information, the interviewer should say “That’s fine” and move on to the next question.]

Would you consider the area you live in to be …

Urban

Suburban

Rural

Ok. Thank you and what is the zip code of the area that you live in?

______________

Thank you. We would like to speak with your child to confirm if (he/she) is interested in participating in the research session. It will take only a couple of minutes and then I will ask to speak with you again for another moment.

Is your child available to speak with us now or should we call (him/her) back at another time?

 Yes, the child is available for screening questions  (SKIP TO SCRIPT FOR 8th GRADE OR 12th GRADE STUDENTS)

 No, please call back.

Is the phone number you gave me earlier the best number to call?

Other phone numbers: ____________________________

Is there a good time for me to call? When would be best for you?

INTERVIEWER: IF CHILD NEEDS TO RECEIVE A CALL BACK, RECORD NAME, PHONE NUMBER, DATE/TIME

Parent/Guardian Name ___________________________________________________________

Child Name ____________________________________________________________________

Child Phone ____________________________________________________________________

Date ____________________

Time ____________________

SCRIPT FOR 8th or 12th GRADE STUDENTS

Good morning/afternoon/evening. My name is <NAME> and I’m calling from EurekaFacts, a local research company. We are doing an important study for the National Center for Education Statistics, part of the U.S. Department of Education.

We have spoken to your parent and we are pleased that you may be interested in helping us develop writing test questions for a nationwide student assessment, the National Assessment of Educational Progress, or NAEP. Before I tell you more about what you would be asked to do, I need to ask you one question:

What grade are you currently in? (INTERVIEWER: DO NOT READ LIST)

 K – 7^th  (Unfortunately, we need only students currently in grade 8 or grade 12 for this study. Thank you for your time. Have a good day/evening.)

 8^th – Good.

 9^th – 11^th (Unfortunately, we need only students currently in grade 8 or grade 12 for this study. Thank you for your time. Have a good day/evening.)

 12^th – Good.

Thank you. For the research session, you will be asked to visit <LOCATION> and respond to a sample of test questions. Your responses will not be graded. The questions focus on writing and are interesting and fun to answer. You will be asked to answer the questions on a tablet computer.

Your input will help evaluate the test that will be given to students throughout the nation. To thank you for participating, you will receive a $25 gift card (from a major credit card company). The session will last about 90 minutes. Your participation is totally voluntary. If you participate, no one but the study’s researchers will see your responses and all other session materials, and we will not use your name to identify you.

Would you like to participate?

 Yes

 No  (THANK AND END CALL)

If NO: Thank you for your time. I will need to let your parent know that you would not like to participate. Is he or she available?

TO PARENT: Your child has decided not to participate, but I would like to thank you very much for taking the time to speak with us about this opportunity.

If YES: Thank you for agreeing to participate in this important research study. I will need to speak to your parent to talk about the date/time of the session. Is your parent available?

 Yes

 No

INTERVIEWER: IF CALLING PARENT BACK, RECORD NAME, PHONE NUMBER, DATE/TIME

Parent/Guardian Name ______________________________________________

Child Name ________________________________________________________

Parent Phone ______________________________________________________

Date ______________________________________________________________

Time ______________________________________________________________

IF PARENT AVAILABLE AND CURRENTLY SCHEDULING INTERVIEWS

Thank you again for allowing your child to participate in the NAEP research sessions. We will be holding our current sessions from <DATE> to <DATE>. What would be the best date/time to schedule the session?

Date ____________________

Time ____________________

Thank you. We will follow up with a confirmation email/letter and a consent form. What is your email or postal address for us to send these to?

____________________________________________________________________________________

IF PARENT AVAILABLE AND NOT CURRENTLY SCHEDULING SESSIONS

Thank you again for allowing your child to participate in the NAEP research sessions. We are in between research sessions, and will need to call you back when we are actively scheduling the sessions to determine the date and time that will work best. What are your scheduling preferences? (SELECT ALL THAT APPLY)

 Morning

 Afternoon

 After school

 Weekends

Thank you so much for talking with us today. We look forward to communicating with you soon.

Appendix Y: EurekaFacts Recruitment Phone Script for Students Age 18 or Above Who Responded to Email, Advertisement, or Invitation

Good <MORNING/AFTERNOON/EVENING>. My name is <NAME>. I’m calling from EurekaFacts on behalf of the U.S. Department of Education’s National Center for Educational Statistics, also known as NCES. Thank you for contacting us about the possibility of participating in a research session to help test new test questions for the National Assessment of Educational Progress, also known as the NAEP assessment. To ensure that we interview a broad mix of students, I have a few brief screening questions to ask you about yourself. It should take only a few minutes of your time. First, I want to collect some additional information about you to identify if you meet the participation requirements.

Before proceeding, please confirm that you are still interested in the possibility of participating in this research interview?

Yes: (INTERVIEWER: Proceed to “Screener.”)

No: (Thank you for your time. Have a good <DAY/EVENING>.)

Screener:

What is your name?

What grade are you currently in? (If the student indicates that he/she is in any grade other than 12th, thank and end call: Unfortunately we are interviewing only students currently in grades 8, or 12 for this study. Thank you for your time. Have a good <DAY/EVENING>.)

What school do you attend? (INTERVIEWER: Note if homeschooled)

____________________________________________________________

What school district do you attend? (INTERVIEWER: Note if parochial, charter, homeschooled) __________________________________________________

What is your race or ethnicity? (INTERVIEWER: Read list only if respondent doesn’t immediately reply. “Are you…?”)

Ethnicity:  Hispanic or Latino  Not Hispanic or Latino

Race (one or more of the categories below):

American Indian or Alaska Native
Asian
Black or African American
Hispanic or Latino
Native Hawaiian or other Pacific Islander
White or Caucasian

Just to be sure we're working with a broad range of students for this study, do you participate in the National School Lunch program?

Yes
No
Refused to disclose

[If the respondent indicates they prefer not to disclose this information, the interviewer should say “That’s fine” and move on to the next question.]

Would you consider the area you live in to be …

Urban
Suburban
Rural

Ok. Thank you and what is the zip code of the area that you live in?

________________________

Would you like to participate?

Yes
No  (THANK AND END CALL)

If NO: Thank you for your time, and have a wonderful <DAY/EVENING.>

If YES:

Thank you for agreeing to participate in this important research study. For the research session, you will be asked to visit EurekaFacts headquarters in Rockville, MD and respond to a sample of test questions. Your responses will not be graded. The questions focus on writing and are interesting and fun to answer. You will be asked to answer the questions on a tablet computer.

Your input will help evaluate the test that will be given to students throughout the nation. To thank you for participating, you will receive a $25 gift card (from a major credit card company). If your parent or legal guardian brings you to and from our office, he or she will also receive a $25 gift card. The session will last about 90 minutes. Your participation is totally voluntary. If you participate, no one but the study’s researchers will see your responses and all other session materials, and we will not use your name to identify you.

Please note that before your interview starts, you will be asked to sign a consent form to confirm that you have agreed to participate in the interview. You will not be able to participate until the form has been signed.

IF CURRENTLY SCHEDULING INTERVIEWS

Thank you again for agreeing to participate in the NAEP research sessions. We will be holding our current sessions from <DATE> to <DATE>. What would be the best date/time to schedule the session?

Date ____________________

Time ____________________

IF NOT CURRENTLY SCHEDULING SESSIONS

Thank you again for agreeing to participate in the NAEP research sessions. We are in between research sessions, and will need to call you back when we are actively scheduling the sessions to determine the date and time that will work best. What are your scheduling preferences? (SELECT ALL THAT APPLY)

 Morning

 Afternoon

 After school

 Weekends

Thank you so much for talking with us today. We look forward to communicating with you soon.

Appendix GG: EurekaFacts Consent Form for Parents of Student Participants (Below Age 18): Writing Cog Labs

PARENTAL CONSENT DOCUMENT

FOR PARTICIPATION IN RESEARCH

EurekaFacts, LLC

Title of Research Study: Cognitive Laboratory Interviews for NAEP Writing

Thank you for your interest in participating in the research project described below. This consent form explains the research study. Please read it carefully, and feel free to ask questions about anything you do not understand. If you do not have questions now, you may ask them if they occur to you later.

What is this study about?

The National Center for Education Statistics (NCES), part of the U.S. Department of Education, is conducting research interviews with students currently in grades 8 and 12 about new test questions for the National Assessment of Educational Progress (NAEP). The questions we are testing focus on writing and will be part of a computer-based test. In these interviews, students will see early versions of computer-based questions. EurekaFacts staff will ask questions that focus on students’ level of interest and their ability to understand the questions presented to them. Students will also be asked if they have any suggestions about how we might improve the questions.

Where will the research interviews take place?

The interviews will take place at <NAME OF INTERVIEW LOCATION>, located at <LOCATION>, near <NEARBY LANDMARK OR METRO STATION>.

What will happen during the research interview?

Students will be asked to complete new NAEP writing tasks on a tablet computer. The research sessions will explore how students work through the tasks on a tablet computer to ensure that the tasks are clear and understandable to a broad range of students. NCES simply wants to know how students would do the tasks if they were on an actual test, and would like to hear students’ thoughts to improve the tasks and make them easier to understand.

Students will be asked to respond to writing tasks and will be asked about them. EurekaFacts staff will use computer software to record what students do on screen, as well as microphones to record students’ reactions to tasks, descriptions of their work, and responses to interviewer questions. EurekaFacts staff will also take notes about students’ reactions to the writing tasks so that tasks can be revised and refined. We will observe and ask about the students’ understanding of the tasks. Students may also be asked whether they have any suggestions about how to enhance the tasks.

How long will the research interview last and when it will take place?

The interview will last up to 90 minutes and will require only one visit to our office.

Will you keep information private and confidential?

Will the results go to my child’s school?

Your child’s answers will not be disclosed to his/her school, and will not be linked to any personally identifiable information such as the school’s name.

What are the possible risks of being in the study?

There are no known risks to participating in this study.

Who can I contact with questions or for further information?

<CONTACT NAME> is conducting this study at EurekaFacts on behalf of NCES. If you have any questions, please call him/her at <CONTACT PHONE> or by e-mail <CONTACT E-MAIL>.

Can participation end early?

Your participation in this study is completely voluntary. Your child has the right to refuse to answer particular questions. You may elect to withdraw from this study at any time.

In addition, if transportation to the site is involved, you will receive a $25 gift card as appreciation for the time bringing your child to and from the research site.

Your signature below will indicate that your questions have been answered satisfactorily, and that you have read and understood the information provided above.

Print Name: _________________________________________________________________________

Signature: ___________________________________________________________________________

Date:_______________________________________________________________________________

School that student attends: ____________________________________________________________

Appendix HH: EurekaFacts Consent Form for Adult Student Participants (Age 18 and Above): Writing Cog Labs

PARTICIPATION CONSENT DOCUMENT

FOR PARTICIPATION IN RESEARCH

EurekaFacts, LLC

Title of Research Study: Cognitive Laboratory Interviews for NAEP Writing

What is this study about?

The National Center for Education Statistics (NCES), part of the U.S. Department of Education, is conducting research interviews with students currently in grades 8 and 12 about new test questions for the National Assessment of Educational Progress (NAEP). The questions we are testing focus on writing and will be part of a computer-based test. In these interviews, students will see early versions of computer-based questions. EurekaFacts staff will take notes about students’ reactions to these questions to revise and refine test content. We will ask questions that focus on students’ level of interest and their ability to understand the questions presented to them. Students will also be asked if they have any suggestions about how we might improve the questions.

Where will the research interviews take place?

The interviews will take place at <NAME OF INTERVIEW LOCATION>, located at <LOCATION>, near <NEARBY LANDMARK OR METRO STATION>.

What will happen during the research interview?

Students will be asked to complete new NAEP writing tasks on a tablet computer. The research sessions will explore how you work through the tasks on a tablet computer to ensure that the tasks are clear and understandable to a broad range of students. NCES simply wants to know how you would do the tasks if they were on an actual test, and would like to hear your thoughts to improve the tasks and make them easier to understand.

How long will the research interview last and when it will take place?

The interview will last up to 90 minutes and will require only one visit to our office.

Will you keep information private and confidential?

You will be assigned a unique student identifier (ID), and at no time will your name be linked to any of your answers. Your responses will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573]. Audio and screen activity will be recorded for analysis, but no personal identifying information will be recorded or retained.

Will the results go to my school?

Your answers will not be disclosed to your school, and will not be linked to any personally identifiable information such as the school’s name.

What are the possible risks of being in the study?

There are no known risks to participating in this study.

Who can I contact with questions or for further information?

<CONTACT NAME> is conducting this study at EurekaFacts on behalf of NCES. If you have any questions, please call him/her at <CONTACT PHONE> or by e-mail <CONTACT E-MAIL>.

Can participation end early?

Your participation in this study is completely voluntary. You have the right to refuse to answer particular questions. You may elect to withdraw from this study at any time.

By signing below, you agree that you will participate in this study. You will receive a $25 gift card (from a major credit card company) at the conclusion of the session.

In addition, if transportation to the site is involved, your parent/guardian providing transportation to and from the research site will also receive a $25 gift card as appreciation for bringing you to and from the research site.

Your signature below will indicate that your questions have been answered satisfactorily, and that you have read and understood the information provided above.

Print Name: _________________________________________________________________________

Signature: ___________________________________________________________________________

Date:_______________________________________________________________________________

School that you attend: ________________________________________________________________

File Type	application/msword
File Title	11-14-13_version_updated appdx. Q and X
Author	joc
Last Modified By	Kashka Kubzdela
File Modified	2015-01-22
File Created	2015-01-22

National Assessment of Educational Progress (NAEP) Writing Tasks Cognitive Interviews

NCES Cognitive, Pilot, and Field Test Studies System

Appendices NAEP Writing Tasks Cog Labs - Screeners and Consent Forms