Appendices NAEP SAIL Science Development 2014-15

You have been very helpful in the past in working with Educational Testing Service (ETS) to recruit students to help us test our tests. The students you helped to recruit provided rich information, and we are requesting your help again in recruiting more students to help us evaluate and improve test questions.

We are looking for students in grades 8 and 9 to help us test new computer-delivered questions, 3D models, and a virtual science lab for the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) Study. NAEP serves as the central means by which the United States measures student proficiency in core academic domains. It is vital for us to get student feedback while developing test questions.

These reviews are sessions where we show students versions of computer-delivered questions, 3D models, and a virtual science lab. Students will be informed that their participation is completely voluntary, and that they will not be evaluated. They will also be told they have a key role in helping us improve our tests and they can make a difference in the experience of students who may use these tests in the future.

ETS staff will take notes of students’ reactions to these questions and interactions with the virtual models and virtual lab and use students’ feedback to revise and refine the questions, virtual models, and virtual lab. We will record students as we observe and ask questions about the students’ level of interest and ability to understand the questions and use the simulations. Students will also be asked if they have any suggestions on how to improve them.

The sessions will be scheduled for up to 90 minutes, and students will be given $25 Visa gift cards for each session as a thank you for their time and as an incentive to help us. In addition, if transportation to the site is involved, a parent or legal guardian of each student will receive a $25 Visa gift card as a thank you for the time in bringing the child to and from the <play testing/cognitive interview> site.

We would like your help in distributing flyers. We will hold sessions at ETS offices located in Princeton, New Jersey. If you are interested and willing to help us recruit students for this effort, or would like additional information about the effort, please contact <staff member at ETS organizing the sessions>. We are attaching a flyer that can be shared with your group, and will be happy to provide you with printed copies if you prefer.

Thank you for your time and interest. If you have any questions, please contact <CONTACT INFORMATION>.

Sincerely,

Educational Testing Service

660 Rosedale Rd

Princeton, NJ 08541

Appendix B: Informational Flyer

Can You Help Us Test Our Test?

Educational Testing Service (ETS) is looking for students in grades 8 and 9 to help us test new computer-delivered assessments in virtual environments for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. These questions are part of NAEP SAIL (Survey Assessments Innovations Lab) research projects exploring new test approaches. You will be given early drafts of test questions and we will be recording as you respond to them and give your opinion about them. We call this <Play Testing/Cognitive Interviews>.

The information we collect from you as you explore and react to our test will only be used to help improve the test. The information will not be shared with or used by any school or district.

<Play testing/cognitive interview> sessions will be held at ETS at 660 Rosedale Road in Princeton, NJ on <note days of the week when sessions are scheduled and the window of time when these sessions will be held>. These sessions will last no more than 90 minutes.

You will get a $25 gift card for your time in helping us to improve our tests. In addition, if transportation to the site is involved, your parent or legal guardian will receive a $25 Visa gift card to thank him or her for the time in bringing you to and from the <play testing/cognitive interview> site.

If you are interested, please ask your parent or guardian to fill out an intake form at [WEBSURVEYOR LINK], or contact [CONTACT PERSON] at [PHONE] or [EMAIL] for more information or to give the necessary information over the phone.

Appendix C: Websurveyor Intake Form

NAEP SAIL Study

The study will require that students be video recorded as they spend approximately 90 minutes completing the following tasks either individually or in small groups:

Use virtual models or virtual science lab to solve a science problem
Think aloud as they participate in the activity
Answer some questions about their experience with the tasks

Students will receive a $25 Visa gift card for participation in the study. In addition, if transportation to the site is involved, the parent or legal guardian will receive a $25 Visa gift card to thank him or her for the time in bringing participant(s) to and from the <play testing/cognitive interview> site. The responses to questions and taking part in this study may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573]. Participation is voluntary and the student may withdraw from the study at any time.

We would like to have a balanced group of students take part in the study. To that end, we’d like you to give us some information about your student by filling out the form on the following pages. We will let you know within <TIME> whether your student has been selected to take part in this <play testing/cognitive interview> study. If your child is not selected for this study, we hope you will consider letting him or her be a part of other studies in the future.

If you have any questions, please contact (CONTACT INFORMATION).

Thanks for your interest in this study!

(NAME)

Student Name (Last, First):

Parent Name (Last, First):

Parent Phone Number:

Parent Email Address:

What grade is your child in? ____8th ___9th

Student’s school:

*Student’s Age:

*Student’s Gender:

*Student’s Race/Ethnicity:

Ethnicity:  Hispanic or Latino  Not Hispanic or Latino

Race (one or more of the categories below):

American Indian or Alaska Native
Asian
Black or African American
Native Hawaiian or other Pacific Islander
White or Caucasian

(* Indicates that the question is voluntary.)

Possible time for 90 minute session: Please mark all the sessions that would work for your child(ren)

(Not all these times will be used. We are trying to schedule students in a few convenient sessions)

(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)

If some other time during these weeks would be better, what 90 minute time frame would that be?

To complete submission of this form, please click the “Submit” button. You will be contacted within <TIME> to let you know whether or not your child has been selected to participate in this study.

(SUBMIT BUTTON)

Appendix D: Recruitment Phone Script for Parents or Legal Guardians of Student Participants

(To be used if direct outreach to parents is necessary)

Script when calling parents from previously compiled list:

Good morning/afternoon/evening. Is this the <NAME FROM CALL LIST> household? My name is <NAME>, and I am calling from Educational Testing Service (ETS). We are conducting a study on behalf of the U.S. Department of Education’s National Center for Education Statistics, or NCES. We are recruiting students currently in grades 8 and 9 to sign up to participate in a research study supporting the development of new science assessments and <interactive virtual models / lab> for a project being sponsored by the National Assessment of Educational Progress or NAEP. [If parent asks what this is: This national assessment is administered to students in elementary, middle, and high schools throughout the United States.]

Are there any 8th or 9th grade students in your household?

Yes (Great! Let me give you some additional information about the study.)

No (Thank you, have a good day/evening.)

We are conducting research interviews to explore how students work through sample science test questions and interact with virtual models / a virtual science lab to ensure that they are clear and understandable to a broad range of students. If selected to participate, your child will receive a $25 Visa gift card. In addition, if transportation to the site is involved, you will receive a $25 Visa gift card to thank you for the time in bringing your child to and from the interview site.

The research interview will take place at Educational Testing Service in Princeton, NJ. It can be scheduled at your convenience and will last about 90 minutes. During the interview, your child will be asked to respond to sample science test questions on a computer, and think aloud about the process they use for answering those questions. NCES simply wants to know how students would use the virtual models / virtual science lab and answer the questions if they were included on an actual NAEP assessment.

These research interviews are an important stage of the NAEP test development process. By sharing their feedback with us, students can directly impact the future of this nationwide assessment by helping NCES evaluate the new types of tests which are being explored.

Do you believe your child would be interested in participating?

Yes (Proceed to protocol script below)

No (Thank you, have a good day/evening.)

Great! To ensure that we interview a broad mix of students, I have a few brief questions to ask about you and your child. It should only take a few minutes of your time. First, I want to collect some additional information about your child to identify if <HE/SHE> meets the participation requirements. The information I collect from you may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

(Skip to screener.)

Script when calling a parent who responded to email or letter

Good morning/afternoon/evening. My name is <NAME>. I’m calling from Educational Testing Service (ETS) on behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education. I’d like to speak to <NAME>. Thank you for contacting us about the possibility of your child participating in a research interview to help develop innovative science test questions for the National Assessment of Educational Progress, or NAEP. To ensure that we interview a broad mix of students, I have a few brief screening questions to ask about you and your child. It should only take a few minutes of your time. First, I want to collect some additional information about your child to identify if <HE/SHE> meets the participation requirements. The information I collect from you may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

Screener

(INTERVIEWER: Determine recruitment needs in advance. Record all responses. End screener at whatever point the parent or legal guardian gives a response that confirms that their group has already been sampled adequately. End screener script: Thank you for answering my questions. Based on the requirements of this study, we are not able to include your child in the interviews at this time.)

What is your child’s name?

What grade is your child currently in? (INTERVIEWER: DO NOT READ LIST)

 8th

 9th

 Other (Unfortunately, we are only interviewing students currently in grade 8 or grade 9 for this study. Thank you for your time. Have a good day/evening.)

What school does your child attend? (INTERVIEWER: Note if homeschooled)

____________________________________________________________

How old is your child?

What is your child’s gender?

What is your child’s race or ethnicity? (INTERVIEWER: Is she/he …..?)

Student’s Race/Ethnicity:

Ethnicity:  Hispanic or Latino  Not Hispanic or Latino

Race (one or more of the categories below):

American Indian or Alaska Native
Asian
Black or African American
Native Hawaiian or other Pacific Islander
White or Caucasian

[If the respondent indicates they prefer not to disclose this information, the interviewer should say “That’s fine” and move on to the next question.]

Thank you. We would like to speak with your child to confirm if (he/she) is interested in participating in the research interview. It will only take a couple of minutes and then I will ask to speak with you again for another moment. Is your child available to speak with us now or should we call (him/her) back at another time?

 Yes, the child is available for screening questions  (SKIP TO SCRIPT FOR STUDENTS)

 No, please call back.

Is the phone number you gave me earlier the best number to call?

Other phone numbers: ____________________________

Is there a good time for me to call? When would be best for you?

INTERVIEWER: IF CHILD NEEDS TO RECEIVE A CALL BACK, RECORD NAME, PHONE NUMBER, DATE/TIME

Parent Name ____________________

Child Name ____________________

Child Phone ____________________

Date ____________________

Time ____________________

SCRIPT FOR STUDENTS

Good morning/afternoon/evening. My name is <NAME> and I’m calling from Educational Testing Service (ETS). We are doing an important study for the National Center for Education Statistics (NCES), part of the U.S. Department of Education.

We have spoken to your parent/guardian and we are pleased that you may be interested in helping us develop innovative science test questions for a new test that is being developed by the National Assessment of Educational Progress, or NAEP. Before I tell you more about what you would be asked to do, I’d like to ask you one question first:

What grade are you currently in? (INTERVIEWER: DO NOT READ LIST)

 8th

 9th

 Other (Unfortunately, we are only interviewing students currently in grade 8 or grade 9 for this study. Thank you for your time. Have a good day/evening.)

Thank you. For the research interview, you will be asked to visit Educational Testing Service in Princeton and respond to a sample of science test questions. Your responses will not be graded. The National Center for Education Statistics simply wants to know how you would interact with a virtual model / virtual science lab and answer the questions if they were to be included on an actual test. The questions focus on science and are really interesting and fun to answer. You will be asked to answer the questions on a computer. Your input will help evaluate and improve the test questions and virtual environment. To thank you for participating, you will receive a $25 Visa gift card. The interview will last about 90 minutes and will be recorded. Your interactions with the virtual model / virtual science lab will be videotaped as well but the data will only be used for research purposes. Your participation is totally voluntary. If you participate, your interview responses and all other interview materials may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

Would you like to participate?

 Yes

 No  (THANK AND TERMINATE)

If NO - Thank you for your time. I will need to speak to your parent/guardian to thank (him/her) for letting you take the time to speak with us.

TO PARENT: Your child has decided not to participate, but I would like to thank you very much for taking the time to speak with us today/this evening.

If YES - Thank you for agreeing to participate in this important research study. I will need to speak to your parent again to make arrangements for your participation. Is he/she available?

 Yes

 No

TO PARENT: Your child has indicated that he/she would be interested in participating in this study. We will be in touch with you within <TIME> to let you know whether or not your child has been selected to participate in this study, and to discuss scheduling and send you a consent form to fill out if he/she will be participating.

INTERVIEWER: IF CALLING PARENT BACK, RECORD NAME, PHONE NUMBER, DATE/TIME

Parent Name ____________________

Child Name ____________________

Parent Phone ____________________

Date ____________________

Time ____________________

Appendix E: Consent Form for Parents/Legal Guardians of Student Participants

PARTICIPATION CONSENT DOCUMENT

FOR PARTICIPATION IN RESEARCH

Dear Parent/Guardian:

Staff members at Educational Testing Service (ETS) and I are currently working on the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) Study, a federally funded project which is studying new types of computer delivered assessments. We are inviting 8th and 9th graders to participate in our study. For this project, a staff member from ETS will be meeting with students individually or in small groups for a study that lasts around 90 minutes at the ETS location on Rosedale Road in Princeton, NJ. Parents will not be in the room with their child, and should plan to return to pick up their child at the end of the session. Students will receive a $25 Visa gift card for their participation. In addition, if transportation to the site is involved, your parent or legal guardian will receive a $25 Visa gift card to thank him or her for the time in bringing you to and from the <play testing/cognitive interview> site.

The project team will video record students as they take part in problem-solving activities related to science. They will be asked to manipulate virtual objects on a screen / interact with a virtual science lab while answering questions about science. Students will be asked to think aloud as they engage in these activities. Afterwards, they will answer a questionnaire about their experience with the activity.

Your child’s performance is not an indication of his or her abilities; it will inform us how the tasks work and help us to develop and refine our ideas for future tasks. All information obtained from your child may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

We hope that you will give your consent for your child to participate in this important activity by signing the accompanying permission form. Students should bring the signed permission form with them when they come to ETS.

Please Note: Parents are responsible for providing transportation for their children

to and from ETS’s Princeton campus, where the sessions will be held.

To check or confirm your appointment, or if you have further questions, please call (CONTACT) of ETS at (PHONE NUMBER) or email (EMAIL).

Sincerely,

Parent Consent Form

Student’s name: ____________________________________________________

Please print student’s name

I am the parent/guardian of the child named above, and I hereby give my permission for my child to take part in the NAEP SAIL Study, including having the session video recorded for use by the project team. I understand that my child’s information may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

Having read the accompanying letter, I voluntarily consent to my child’s participation in this study. I understand that my child’s participation is voluntary, my child may withdraw from the study at any time without prejudice, and that my child has the right to refuse to answer any questions they don’t feel comfortable responding to.

I understand that if I have any questions about any part of my child’s participation, I may call the Project Director, <SCIENTIST CONTACT INFORMATION>.

_____________________________________________ __________________________

Parent/Guardian Signature Date

Rectangle 2 Yes, I consent to my child participating in this research project, including the video capture of my child completing the assigned task and answering interview questions.

Rectangle 3 No, I do not consent to my child participating in this research project.

________________________________________________________________________________________

In the future, we may have additional opportunities for students to participate in studies. If you are interested in being contacted about your child participating in future studies, please check the box below.

Rectangle 6 I consent to being contacted about opportunities for my child to participate in future NAEP SAIL studies.

________________________________________________________________________________________

The data collected in this project will be valuable information for the larger cognitive learning field, and as such, your child’s responses or clips from these videos may be selected for use in research reports, presentations and/or conferences. Your child’s name will not be identified.

Rectangle 4 Yes, I consent to my child’s data (video and responses) being included in external publications and/or research conferences.

Rectangle 5 No, I do not consent to my child’s data (video and responses) being included in external publications and/or research conferences.

Please bring this completed form with you at the time of the

child’s scheduled appointment.

Appendix F: Email/Letter Interview Confirmation for Parents/Legal Guardians of Student Participants

IF SENT AS EMAIL - Subject: Confirmation of <Play Testing/Cognitive Interview> Session

<DATE>

Dear <NAME>,

Thank you for agreeing to allow your child to participate in a study to give us feedback on the questions and simulation being developed for the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) project. During the session, your child will be asked to respond to sample test activities and interact with a virtual model / virtual lab which is being developed to assess knowledge of science. The National Center for Education Statistics (NCES), part of the U.S. Department of Education, simply wants to know what students such as your child think about the tests and how they might answer questions on that test. Your child will work individually or in a small group of several students who will also be reviewing and reacting to the potential test questions. Your child’s feedback and responses will be noted and audio and video recorded to help us revise and improve our tests.

All of your child’s responses may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

Your child has been selected to attend a session at <START TO END TIME> on <DATE> located at the Princeton campus of ETS. To thank your child for <his/her> time, <he/she> will receive a $25 Visa gift card as soon as he/she has finished the session. In addition, if transportation to the site is involved, you will receive a $25 Visa gift card to thank you for the time in bringing your child to and from the <play testing/cognitive interview> site.

Please confirm that this time slot still works for your child by <DATE>. We are attaching a consent form that must be signed and returned for your child to participate. Please fill it out and send it back to us, or have your child bring it with them to the <play testing/cognitive interview> site.

We look forward to meeting with your child and getting your child’s feedback. If you have any questions or if your child is unable to attend this session, please contact me using the information below.

Sincerely,

Educational Testing Service

Appendix G: Acknowledgement Email to Parents/Legal Guardians of Students Not Selected to Participate

IF SENT AS EMAIL - Subject: NAEP SAIL Study

<DATE>

Dear <NAME>,

Thank you for your interest in having your son or daughter be a part of the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) <SUBJECT> Study. Unfortunately, due to the number of students who were interested in participating, we are not able to ask your child to be a part of the study at this time. However, we anticipate that we will be conducting additional studies related to this project in the future. If you are interested in possibly being contacted about having your son or daughter participate at a later time if another opportunity becomes available, please let us know (see contact information below).

Thank you again for taking the time to respond to our participation request.

Sincerely,

Educational Testing Service

Appendix H: Thank You Email/Letter to Parent or Legal Guardian of Student Participants

IF SENT AS EMAIL - Subject: Thank you for your Child’s Participation in a Research Study

<DATE>

Dear <NAME>,

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, Educational Testing Service (ETS) would like to thank you for allowing your child to participate in the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) research study. Thanks to your child’s participation, we are better able to refine, revise, and develop the simulation and questions that are part of this assessment.

Your child’s assistance helped us better understand how students approach these test questions and use the virtual environment. <HIS/HER> open, candid, and insightful comments were very helpful.

Again, thank you for your child’s assistance.

Sincerely,

Educational Testing Service

Appendix I: Thank You Email/Letter to School/ Organization/ Association of Student Participants

IF SENT AS EMAIL - Subject: Thank you for your Participation in a Research Study

<DATE>

Dear <NAME>,

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, Educational Testing Service (ETS) would like to thank you for your <school’s/ organization’s/ association’s> participation in the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) research study. Thanks to the students at your <school/ organization/ association> we are better able to refine, revise, and develop the virtual environment and questions which are part of this assessment.

Your <school’s/ organization’s/ association’s> assistance helped us better understand how students approach the <SUBJECT> test questions. Students’ open, candid, and insightful comments were very helpful.

Again, thank you for your assistance.

Sincerely,

< COMPANY REPRESENTATIVE >

Educational Testing Service

Appendices – Tryouts

Appendix J: Site Recruitment Email/Letter (To be Sent to School Administrators/Staff Members)

IF SENT AS EMAIL - Subject: Research Study Opportunity

<DATE>

My name is <CONTACT NAME> and I am contacting you from Educational Testing Service (ETS). We are conducting an important study for the National Center for Education Statistics (NCES), part of the U.S. Department of Education.

We are looking for students in grades 8 and 9 to help us test new computer-delivered questions and simulations in <SUBJECT>. These questions are part of a new assessment being developed by National Assessment of Educational Progress (NAEP) and may someday be delivered to students across the country. NAEP serves as the central means by which the United States measures student proficiency in core academic domains. It is vital for us to get student feedback while developing test questions.

These voluntary tryouts will last no more than 90 minutes. Students will be video recorded while responding to several <SUBJECT> test questions and use a simulation to help answer the questions. ETS staff will take notes of students’ reactions to these questions and how they use the simulation, and use this information to revise and refine the questions.

Students will be informed that their participation is completely voluntary, and that they will not be evaluated. They will also be told they have a key role in helping us improve our tests and they can make a difference in the experience of thousands of students who may someday take these tests.

We would like your help in distributing flyers and consent forms to students and in collecting consent forms. We will hold some sessions at <LOCATION>; however, if the parents of students cannot provide transportation to <LOCATION>, we would like to set up sessions at your <ORGANIZATION> located at <address>.

For each participating student, a $25 Visa gift card will be issued as a thank you for participants’ time. In addition, if transportation to the site is involved, a parent or legal guardian of each student will receive a $25 Visa gift card as a thank you for the time in bringing the child to and from the interview site. If the interviews take place at a school during school hours, the $25 gift card for each student will be issued to your school.

If you are interested and willing to help us recruit students for this study, or would like additional information about the study, please contact <staff member at ETS organizing the sessions> and we will follow up with flyers for you to distribute.

Thank you for your time and interest. If you have any questions, please contact <CONTACT INFORMATION>.

Sincerely,

<NAME>

Educational Testing Service

Appendix K: Site Recruitment Phone Script (When Calling School Administrators/Staff Members)

Good morning/afternoon/evening. May I please speak to <NAME of contact at school/organization>?

Good morning/afternoon/evening. My name is <CONTACT NAME> and I am contacting you from Educational Testing Service (ETS). We are conducting an important study for the National Center for Education Statistics (NCES), part of the U.S. Department of Education.

The research interviews will explore how students work through <SUBJECT> test questions on computer to ensure that the questions are clear and understandable to a broad range of students. NCES simply wants to know how students would answer the questions if they were on an actual test and would like to hear students’ thoughts to improve the <SUBJECT> test questions and make the questions easier to understand.

Student responses may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002), 20 U.S.C §9573].

[Recruiter note: this reference is for informational purposes only. If asked what law is being referenced, please provide the information contained within this bracketed text.]

These interviews are an important stage of the NAEP test development process. The comments of students attending your <ORGANIZATION> will help NCES improve test questions.

Would your school be interested in participating in or helping us to recruit students for this research study?

If NO: Thank you for your time. Have a wonderful day/evening.

If YES: Great, thank you very much. To ensure that we interview a broad mix of students and determine the best interview location, I have a few brief screening questions to ask you. It should only take a few minutes of your time.

Screener

[Recruiter: Determine recruitment needs in advance. Look in public documents and school/organization website to find what grades a school/organization includes, the size of each grade, the SES, and racial/ethnic makeup of the student body. Record all responses to screener questions. End screener at whatever point the staff member gives a response that confirms that their group has already been sampled adequately. End screener script: Based on the requirements of this study, we are not able to include you in the interviews at this time.]

Do you have XX number [specify recruitment needs] of <GRADE> graders who might be available to participate in the study?
Would you be able to give ETS a room in which to conduct the study at your <ORGANIZATION>?

Would you be able to send flyers and consent forms to parents of potential student participants?

Thank you for taking the time to answer the screening questions. Your school has students who are eligible to participate in the tryouts. Again, each session will last up to 90 minutes. Would you like to schedule the interview sessions at your <ORGANIZATION> or help us recruit student participants who can attend interview sessions at <LOCATION>?

If NO: Thank you for your time. Have a wonderful day/evening.

If YES: Schedule the interview sessions or make arrangements to send recruiting materials to the school.

We may need to reach you by telephone. Is the number I called the best number to use?

Do you have any questions at this time?

[Recruiter: Answer any questions.]

If you do have any questions, please call <PHONE NUMBER>.

Thank you again for agreeing to be part of this very important research project.

Appendix L: Recruitment Email/Letter (To be Sent to Teachers/Staff Members)

IF SENT AS EMAIL - Subject: Research Study Opportunity

<DATE>

Dear <Teacher or Staff Member at After School Program>,

We are looking for students in grades 8 and 9 to help us test new computer-delivered questions for the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) research study. NAEP serves as the central means by which the United States measures student proficiency in core academic domains. It is vital for us to get student feedback while developing test questions.

In these sessions, students will interact with a virtual model / virtual lab and answer questions that have been developed for the science project. Students will be informed that their participation is completely voluntary, and that they will not be evaluated. They will also be told they have a key role in helping us improve our tests and they can make a difference in the experience of students who may someday take these or similar tests. ETS staff will record students as they observe the way in which students interact with the simulation and questions to evaluate what has been developed.

Student responses may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA 2002, 20 U.S.C §9573].

The sessions will be scheduled for 90 minutes, and students will be given $25 Visa gift cards for each session as a thank you for their time and as an incentive to help us. In addition, if transportation to the site is involved, a parent or legal guardian of each student will receive a $25 Visa gift card as a thank you for the time in bringing the child to and from the study site.

We would like your help in distributing flyers. We will hold sessions at ETS offices located in Princeton, New Jersey or at <LOCATION>. If you are interested and willing to help us recruit students for this effort, or would like additional information about the effort, please contact <staff member at ETS organizing the sessions>. We are attaching a flyer, which can be shared with your group, and will be happy to provide you with printed copies if you prefer.

Thank you for your time and interest. If you have any questions, please contact <CONTACT INFORMATION>.

Sincerely,

Educational Testing Service

660 Rosedale Rd

Princeton, NJ 08541

Appendix M: Informational Flyer

Can You Help Us Test Our Test?

Educational Testing Service (ETS) is looking for students in grades 8 and 9 to help us test new computer-delivered science test questions for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. These questions are part of a new test being developed that deals with science. You will be interacting with a virtual model / virtual lab and recorded as you answer test questions as part of a tryout of materials which have been developed.

The information we collect from you as you explore and react to our test will only be used to help improve the test. The information will not be shared with or used by any school or district.

These sessions will be held at <location > on <note days of the week when sessions are scheduled and the window of time when these sessions will be held>. These sessions will last no more than 90 minutes.

You will get a $25 Visa gift card for your time in helping us to improve our tests. In addition, if transportation to the site is involved, your parent or legal guardian will receive a $25 Visa gift card to thank him or her for the time in bringing you to and from the tryout site.

Appendix N: Websurveyor Intake Form

NAEP SAIL (SUBJECT) Study

Staff members at Educational Testing Service (ETS) and I are currently working on the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) research study, a federally funded project which is studying the effects of interactive computer delivery and content of NAEP SAIL assessments. We are inviting 8th and 9th graders to participate in our study. This will inform us how the tasks work and help us to develop and refine our ideas for future tasks.

The study will require that students be video recorded as they spend approximately 90 minutes using a computer simulation to answer questions about science, and then answer some questions about their experience with the tasks

We would like to have a balanced group of students take part in the study. To that end, we’d like you to give us some information about your student by filling out the form on the following pages. We will let you know within <TIME> whether your student has been selected to take part in this study. If your student is not selected for this study, I hope you will consider letting him or her be a part of other studies in the future.

If you have any questions, please contact (CONTACT INFORMATION).

Thanks for your interest in this study!

(NAME)

Student Name (Last, First):

Parent Name (Last, First):

Parent Phone Number:

Parent Email Address:

What grade is your child in? ____8th ___9th

Student’s school:

*Student’s Age:

*Student’s Gender:

*Student’s Race/Ethnicity:

Ethnicity:  Hispanic or Latino  Not Hispanic or Latino

Race (one or more of the categories below):

American Indian or Alaska Native
Asian
Black or African American
Native Hawaiian or other Pacific Islander
White or Caucasian

(* Indicates that the question is voluntary.)

Possible time for 90 minutes session: Please mark all the sessions that would work for your child(ren)

(It is possible that not all these times will be used.)

(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)

If some other time during these weeks would be better, what 90 minutes time frame would that be?

To complete submission of this form, please click the “Submit” button. You will be contacted within <TIME> to let you know whether or not your student has been selected to participate in this study.

(SUBMIT BUTTON)

Appendix O: Recruitment Phone Script for Parents or Legal Guardians of Student Participants

(To be used if direct outreach to parents is necessary)

Script when calling parents from previously compiled list:

Good morning/afternoon/evening. Is this the <NAME FROM CALL LIST> household? My name is <NAME>, and I am calling from Educational Testing Service (ETS). We are conducting a study on behalf of the U.S. Department of Education’s National Center for Education Statistics, or NCES. We are recruiting students currently in grade(s) 8 AND 9 to sign up to participate in a research study supporting the development of new science test questions and interactive virtual environments for a project being sponsored by the National Assessment of Educational Progress or NAEP. [If parent asks what this is: This national assessment is administered to students in elementary, middle, and high schools throughout the United States.]

Are there any 8th or 9th grade students in your household?

Yes (Great! Let me give you some additional information about the study.)

No (Thank you, have a good day/evening.)

We are conducting tryouts to explore how students work through sample science test questions and interact with a virtual model / virtual lab to ensure that they are clear and understandable to a broad range of students. If selected to participate, your child will receive a $25 Visa gift card. In addition, if transportation to the site is involved, you will receive a $25 Visa gift card to thank you for the time in bringing your child to and from the interview site.

The tryouts will take place at Educational Testing Service or at <LOCATION>. It can be scheduled at your convenience and will last about 90 minutes. During the interview, your child will be asked to respond to sample science test questions on a computer. NCES simply wants to know how students would use the virtual model / virtual lab and answer the questions if they were included on an actual NAEP assessment.

These tryouts are an important stage of the NAEP test development process. By sharing their feedback with us, students can directly impact the future of this nationwide assessment by helping NCES evaluate the new types of tests that are being explored.

Do you believe your child would be interested in participating?

Yes (Proceed to protocol script below)

No (Thank you, have a good day/evening.)

(Skip to screener.)

Script when calling a parent who responded to email or letter

Good morning/afternoon/evening. My name is <NAME>. I’m calling from Educational Testing Service (ETS) on behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education. Thank you for contacting us about the possibility of your child participating in a research interview to help develop innovative science test questions for the National Assessment of Educational Progress, or NAEP. To ensure that we interview a broad mix of students, I have a few brief screening questions to ask about you and your child. It should only take a few minutes of your time. First, I want to collect some additional information about your child to identify if <HE/SHE> meets the participation requirements. The information I collect from you may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

Screener

What is your child’s name?

What grade is your child currently in? (RECRUITER: DO NOT READ LIST)

 8th

 9th

 Other (Unfortunately, we are only interviewing students currently in grade 8 or grade 9 for this study. Thank you for your time. Have a good day/evening.)

What school does your child attend? (INTERVIEWER: Note if homeschooled)

____________________________________________________________

How old is your child?

What is your child’s gender?

What is your child’s race or ethnicity? (INTERVIEWER:. Is <he/she> …?)

*Student’s Race/Ethnicity:

Ethnicity:  Hispanic or Latino  Not Hispanic or Latino

Race (one or more of the categories below):

American Indian or Alaska Native
Asian
Black or African American
Native Hawaiian or other Pacific Islander
White or Caucasian

[If the respondent indicates they prefer not to disclose this information, the interviewer should say “That’s fine” and move on to the next question.]

 Yes, the child is available for screening questions  (SKIP TO SCRIPT)

 No, please call back.

Is the phone number you gave me earlier the best number to call?

Other phone numbers: ____________________________

Is there a good time for me to call? When would be best for you?

INTERVIEWER: IF CHILD NEEDS TO RECEIVE A CALL BACK, RECORD NAME, PHONE NUMBER, DATE/TIME

Parent Name ____________________

Child Name ____________________

Child Phone ____________________

Date ____________________

Time ____________________

SCRIPT FOR STUDENTS

We have spoken to your parent and we are pleased that you may be interested in helping us develop innovative science test questions for a new test that is being developed by the National Assessment of Educational Progress, or NAEP. Before I tell you more about what you would be asked to do, I’d like to ask you one question first:

What grade are you currently in? (RECRUITER: DO NOT READ LIST)

 8th

 9th

 Other (Unfortunately, we are only interviewing students currently in grade 8 or grade 9 for this study. Thank you for your time. Have a good day/evening.)

Thank you. For the study, you will be asked to visit Educational Testing Service in Princeton or meet with someone at <LOCATION> and respond to a sample of science test questions. Your responses will not be graded. The National Center for Education Statistics simply wants to know how you would interact with a virtual model / virtual lab and answer the questions if they were to be included on an actual test. The questions focus on science and are really interesting and fun to answer. You will be video recorded as you answer the questions on a computer but the data will only be used for research purposes. Your input will help evaluate and improve the test questions and simulation. To thank you for participating, you will receive a $25 Visa gift card. In addition, if transportation to the site is involved, your parent or legal guardian will receive a $25 Visa gift card to thank him or her for the time in bringing you to and from the study site. The study will last about 90 minutes and will be recorded. Your interaction with the virtual environment will be videotaped as well. Your participation is totally voluntary. If you participate, your responses and all other materials may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

Would you like to participate?

 Yes

 No  (THANK AND TERMINATE)

If NO - Thank you for your time. I will need to speak to your parent to thank (him/her) for letting you take the time to speak with us.

TO PARENT: Your child has decided not to participate, but I would like to thank you very much for taking the time to speak with us today/this evening.

If YES - Thank you for agreeing to participate in this important research study. I will need to speak to your parent again to make arrangements for your participation. Is he/she available?

 Yes

 No

INTERVIEWER: IF CALLING PARENT BACK, RECORD NAME, PHONE NUMBER, DATE/TIME

Parent Name ____________________

Child Name ____________________

Parent Phone ____________________

Date ____________________

Time ____________________

Appendix P: Consent Form for Parents/Legal Guardians of Student Participants

PARTICIPATION CONSENT DOCUMENT

FOR PARTICIPATION IN RESEARCH

Dear Parent/Legal Guardian:

Staff members at Educational Testing Service (ETS) and I are currently working on the National Assessment of Educational Progress (NAEP) SAIL (Survey Assessments Innovations Lab) research study, a federally funded project which is studying new types of computer delivered assessments. We are inviting 8th and 9th graders to participate in our study. For this project, a staff member from ETS will be meeting with students for a study that lasts around 90 minutes at the ETS location on Rosedale Road in Princeton, NJ, or at <LOCATION>. Parents will not be in the room with their child, and should plan to return to pick up their child at the end of the session. Students will be given $25 in gift cards in exchange for their participation. In addition, if transportation to the site is involved, the parent or legal guardian will receive a $25 Visa gift card to thank him or her for the time in bringing their child to and from the study site.

The project team will video record students as they take part in problem-solving activities related to science in a virtual environment. They will be asked to manipulate virtual objects / interact with a virtual lab on a screen while answering questions about science. At the end of the test, they will be asked questions about their experience taking it.

Please Note: Parents are responsible for providing transportation for their children

to and from ETS’s Princeton campus, where the sessions will be held.

To check or confirm your appointment, or if you have further questions, please call (CONTACT) of ETS at (PHONE NUMBER) or email (EMAIL).

Sincerely,

Parent Consent Form

Student’s name: ____________________________________________________

Please print student’s name

I am the parent/guardian of the child named above, and I hereby give my permission for my child to take part in the NAEP SAIL research study, including having the session video recorded for use by the project team. I understand that my child will not be identified by name, and that my child’s information may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

I understand that if I have any questions about any part of my child’s participation, I may call the Project Director, <SCIENTIST CONTACT INFORMATION>.

_____________________________________________ __________________________

Parent/Guardian Signature Date

Rectangle 12 Yes, I consent to my child participating in this research project, including the video capture of my child completing the assigned task.

Rectangle 13 No, I do not consent to my child participating in this research project.

________________________________________________________________________________________

Rectangle 16 I consent to being contacted about opportunities for my child to participate in future NAEP SAIL studies.

________________________________________________________________________________________

Rectangle 14 Yes, I consent to my child’s data (video and responses) being included in external publications and/or research conferences.

Rectangle 15 No, I do not consent to my child’s data (video and responses) being included in external publications and/or research conferences.

Please bring this completed form with you at the time of the

child’s scheduled appointment.

Appendix Q: Email/Letter Interview Confirmation for Parents/Legal Guardians of Student Participants

IF SENT AS EMAIL - Subject: Confirmation of Tryout Session

<DATE>

Dear <NAME>,

Thank you for agreeing to allow your child to participate in a study to give us feedback on the questions and simulation being developed for the National Assessment of Educational Progress (NAEP) SAIL (Survey Assessments Innovations Lab) research project. During the session, your child will be asked to respond to sample test activities and interact with a virtual model / virtual lab which is being developed to assess knowledge of science. The National Center for Education Statistics (NCES), part of the U.S. Department of Education, simply wants to know how a student such as your child might answer questions on that test. Your child’s feedback and responses will be noted and audio and video recorded to help us revise and improve our tests.

Your child has been selected to attend a session at <START TO END TIME> on <DATE> located at <LOCATION>. To thank your child for <his/her> time, <he/she> will receive a $25 Visa gift card as soon as he/she has finished the session. In addition, if transportation to the site is involved, you will receive a $25 Visa gift card to thank you for the time in bringing your child to and from the tryout site.

Sincerely,

Educational Testing Service

Appendix R: Acknowledgement Email to Parents/Legal Guardians of Students Not Selected to Participate

IF SENT AS EMAIL - Subject: NAEP SAIL Study

<DATE>

Dear <NAME>,

Thank you for your interest in having your son or daughter be a part of the National Assessment of Educational Progress (NAEP) SAIL (Survey Assessments Innovations Lab) <SUBJECT> Study. Unfortunately, due to the number of students who were interested in participating, we are not able to ask your child to be a part of the study at this time. However, we anticipate that we will be conducting additional studies related to this project in the future. If you are interested in possibly being contacted about having your son or daughter participate at a later time if another opportunity becomes available, please let us know (see contact information below).

Thank you again for taking the time to respond to our participation request.

Sincerely,

Educational Testing Service

Appendix S: Thank You Email/Letter to Parent or Legal Guardian of Student Participants

IF SENT AS EMAIL - Subject: Thank you for your Child’s Participation in a Research Study

<DATE>

Dear <NAME>,

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, Educational Testing Service (ETS) would like to thank you for allowing your child to participate in our National Assessment of Educational Progress (NAEP) SAIL (Survey Assessments Innovations Lab)research study. Thanks to your child’s participation, we are better able to refine, revise, and develop the simulation and questions that are part of this assessment.

Your child’s assistance helped us better understand how students approach these test questions and use the simulation. <HIS/HER> help was very much appreciated.

Again, thank you for your child’s assistance.

Sincerely,

Educational Testing Service

Appendix T: Thank You Email/Letter to School/ Organization/ Association of Student Participants

IF SENT AS EMAIL - Subject: Thank you for your Participation in a Research Study

<DATE>

Dear <NAME>,

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, Educational Testing Service (ETS) would like to thank you for your <school’s/ organization’s/ association’s> participation in the National Assessment of Educational Progress (NAEP) SAIL (Survey Assessments Innovations Lab)research study. Thanks to the students at your <school/ organization/ association> we are better able to refine, revise, and develop the virtual environment and questions which are part of this assessment.

Your <school’s/ organization’s/ association’s> assistance helped us better understand how students approach the <SUBJECT> test questions. Students’ help was very much appreciated.

Again, thank you for your assistance.

Sincerely,

< COMPANY REPRESENTATIVE >

Educational Testing Service

File Type	application/msword
File Title	NATIONAL CENTER FOR EDUCATION STATISTICS
Author	joc
Last Modified By	Kashka Kubzdela
File Modified	2014-04-25
File Created	2014-04-17