National Assessment of Educational Progress (NAEP) Survey Assessments Innovations Lab (SAIL) Pretesting Activities: Virtual World for English Language Arts Assessment - Updated

You have been very helpful in the past in working with Educational Testing Service (ETS) to recruit students to help us test our tests. The students you helped to recruit provided rich information, and we are requesting your help again in recruiting more students to help us evaluate and improve test questions.

We are looking for students in grades 8 and 9 to help us test new computer-delivered English language arts questions in a virtual environment for the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) Study. NAEP serves as the central means by which the United States measures student proficiency in core academic domains. It is vital for us to get student feedback while developing test questions.

These reviews are sessions where we show students versions of computer-delivered questions in a virtual environment. Students will be informed that their participation is completely voluntary, and that they will not be evaluated. They will also be told they have a key role in helping us improve our tests and they can make a difference in the experience of students who may use these tests in the future.

ETS staff will take notes of students’ reactions to these questions and interactions with the virtual models environment and use students’ feedback to revise and refine the questions and virtual environment. We will record students as we observe and ask questions about the students’ level of interest and ability to understand the questions and use the simulations. Students will also be asked if they have any suggestions on how to improve them.

The sessions will be scheduled for up to <90 minutes for playtesting, 120 minutes for cognitive interviews>, and students will be given <$25 for playtesting, $30 for cognitive interviews> Visa gift cards for each session as a thank you for their time and as an incentive to help us. In addition, if transportation to the site is involved, a parent or legal guardian of each student will receive a $25 Visa gift card as a thank you for the time in bringing the child to and from the <playtesting/cognitive interview> site.

We would like your help in distributing flyers. We will hold sessions at ETS offices located in Princeton, New Jersey. If you are interested and willing to help us recruit students for this effort, or would like additional information about the effort, please contact <staff member at ETS organizing the sessions>. We are attaching a flyer that can be shared with your group, and will be happy to provide you with printed copies if you prefer.

Thank you for your time and interest. If you have any questions, please contact <CONTACT INFORMATION>.

Sincerely,

Educational Testing Service

660 Rosedale Rd

Princeton, NJ 08541

Appendix B: Informational Flyer

Can You Help Us Test Our Test?

Educational Testing Service (ETS) is looking for students in grades 8 and 9 to help us test new computer-delivered assessments in virtual environments for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. These questions are part of NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) research projects exploring new test approaches. You will be given early drafts of test questions and we will be recording as you respond to them and give your opinion about them. We call this <Play Testing/Cognitive Interviews>.

The information we collect from you as you explore and react to our test will only be used to help improve the test. The information will not be shared with or used by any school or district.

<Play testing/cognitive interview> sessions will be held at ETS at 660 Rosedale Road in Princeton, NJ on <note days of the week when sessions are scheduled and the window of time when these sessions will be held>. These sessions will last no more than <90 minutes (playtesting)/120 minutes (cognitive interviews)>.

You will get a <$25 Visa gift card (playtesting)/$30 Visa gift card for cognitive interviews> for your time in helping us to improve our tests. In addition, if transportation to the site is involved, your parent or legal guardian will receive a $25 Visa gift card to thank him or her for the time in bringing you to and from the <play testing/cognitive interview> site.

If you are interested, please ask your parent or guardian to fill out an intake form at [WEBSURVEYOR LINK], or contact [CONTACT PERSON] at [PHONE] or [E-MAIL] for more information or to give the necessary information over the phone.

Appendix C: Websurveyor Intake Form

NAEP SAIL Study

Staff members at Educational Testing Service (ETS) and I are currently working on the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) Study, a federally funded project that is studying the effects of interactive computer delivery and content of NAEP SAIL assessments. We are inviting 8th and 9th graders to participate in our study. This will inform us how the tasks work and help us to develop and refine our ideas for future tasks.

The study will require that students be video recorded as they spend approximately <90 minutes (playtesting)/120 minutes (cognitive interviews)> completing the following tasks either individually or in small groups:

Use a virtual environment to answer English language arts questions
Think aloud as they participate in the activity
Answer some written and oral questions about their experience with the tasks.

Students will receive a <$25 Visa gift card (playtesting)/$30 Visa gift card (cognitive interviews)> for participation in the study. In addition, if transportation to the site is involved, the parent or legal guardian will receive a $25 Visa gift card to thank him or her for the time in bringing participants to and from the <play testing/cognitive interview> site. Unless you give specific permission, your child’s responses to questions may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573]. Participation is voluntary and a student may withdraw from the study at any time.

We would like to have a balanced group of students take part in the study. To that end, we’d like you to give us some information about your student by filling out the form on the following pages. We will let you know within <TIME> whether your student has been selected to take part in this <play testing/cognitive interview> study. If your child is not selected for this study, we hope you will consider letting him or her be a part of other studies in the future.

If you have any questions, please contact (CONTACT INFORMATION).

Thanks for your interest in this study!

(NAME)

Student Name (Last, First):

Parent Name (Last, First):

Parent Phone Number:

Parent E-mail Address:

What grade is your child in? ____8th ___9th

Student’s school:

*Student’s Age:

*Student’s Gender:

*Student’s Race/Ethnicity:

Ethnicity:  Hispanic or Latino  Not Hispanic or Latino

Race (one or more of the categories below):

American Indian or Alaska Native
Asian
Black or African American
Native Hawaiian or other Pacific Islander
White or Caucasian

(* Indicates that the question is voluntary.)

Possible times for <90-minute/120-minute> session: Please mark all the sessions that would work for your child(ren)

(Not all these times will be used. We are trying to schedule students in a few convenient sessions)

(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)

If some other time during these weeks would be better, what <90-minute/120-minute> time frame would that be?

To complete submission of this form, please click the “Submit” button. You will be contacted within <TIME> to let you know whether or not your child has been selected to participate in this study.

(SUBMIT BUTTON)

Appendix D: Recruitment Phone Script for Parents/Legal Guardians of Student Participants

(To be used only if the required number of participants is not recruited through schools and organizations and direct outreach to parents is necessary)

Script when calling parents from previously compiled list:

Good morning/afternoon/evening. Is this the <NAME FROM CALL LIST> household? My name is <NAME>, and I am calling from Educational Testing Service (ETS). We are conducting a study on behalf of the U.S. Department of Education’s National Center for Education Statistics, or NCES. We are recruiting students currently in grades 8 and 9 to sign up to participate in a research study supporting the development of new English language arts assessments and an interactive virtual environment for a project being sponsored by the National Assessment of Educational Progress or NAEP. [If parent asks what this is: This national assessment is administered to students in elementary, middle, and high schools throughout the United States.]

Are there any 8th or 9th grade students in your household?

Yes (Great! Let me give you some additional information about the study.)

No (Thank you, have a good day/evening.)

We are conducting research interviews to explore how students work through sample English language arts test questions and interact with a virtual environment to ensure that they are clear and understandable to a broad range of students. If selected to participate, your child will receive a <$25 Visa gift card (playtesting)/$30 Visa gift card (cognitive interviews)>. In addition, if transportation to the site is involved, you will receive a $25 Visa gift card to thank you for the time in bringing your child to and from the interview site.

The research interview will take place at Educational Testing Service in Princeton, NJ. It can be scheduled at your convenience and will last about <90 minutes (playtesting)/120 minutes (cognitive interviews)>. During the interview, your child will be asked to conduct information gathering and evaluation activities while exploring a virtual world on a computer, and think aloud about the process they use for answering those questions. NCES simply wants to know how students would use the virtual environment and answer the questions if they were included on an actual NAEP assessment.

These research interviews are an important stage of the NAEP test development process. By sharing their feedback with us, students can directly impact the future of this nationwide assessment by helping NCES evaluate the new types of tests that are being explored.

Do you believe your child would be interested in participating?

Yes (Proceed to protocol script below)

No (Thank you, have a good day/evening.)

Great! To ensure that we interview a broad mix of students, I have a few brief questions to ask about you and your child. It should only take a few minutes of your time. First, I want to collect some additional information about your child to identify if <HE/SHE> meets the participation requirements. The information I collect from you may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].(Skip to screener.)

Script when calling a parent who responded to e-mail or letter

Good morning/afternoon/evening. My name is <NAME>. I’m calling from Educational Testing Service (ETS) on behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education. I’d like to speak to <NAME>. Thank you for contacting us about the possibility of your child participating in a research interview to help develop innovative English language arts test questions for the National Assessment of Educational Progress, or NAEP. To ensure that we interview a broad mix of students, I have a few brief screening questions to ask about you and your child. It should only take a few minutes of your time. First, I want to collect some additional information about your child to identify if <HE/SHE> meets the participation requirements. The information I collect from you may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

Screener

(INTERVIEWER: Determine recruitment needs in advance. Record all responses. End screener at whatever point the parent or legal guardian gives a response that confirms that their group has already been sampled adequately. End screener script: Thank you for answering my questions. Based on the requirements of this study, we are not able to include your child in the interviews at this time.)

What is your child’s name?

What grade is your child currently in? (INTERVIEWER: DO NOT READ LIST)

 8th

 9th

 Other (Unfortunately, we are only interviewing students currently in grade 8 or grade 9 for this study. Thank you for your time. Have a good day/evening.)

What school does your child attend? (INTERVIEWER: Note if homeschooled)

____________________________________________________________

How old is your child?

What is your child’s gender?

What is your child’s race or ethnicity? (INTERVIEWER: Only read list if parent doesn’t immediately reply. Is your child…?)

Ethnicity:  Hispanic or Latino  Not Hispanic or Latino

Race (one or more of the categories below):

American Indian or Alaska Native
Asian
Black or African American
Native Hawaiian or other Pacific Islander
White or Caucasian

[If the respondent indicates they prefer not to disclose this information, the interviewer should say “That’s fine” and move on to the next question.]

Thank you. We would like to speak with your child to confirm if (he/she) is interested in participating in the research interview. It will only take a couple of minutes and then I will ask to speak with you again for another moment. Is your child available to speak with us now or should we call (him/her) back at another time?

 Yes, the child is available for screening questions  (SKIP TO SCRIPT FOR STUDENTS)

 No, please call back.

Is the phone number you gave me earlier the best number to call?

Other phone numbers: ____________________________

Is there a good time for me to call? When would be best for you?

INTERVIEWER: IF CHILD NEEDS TO RECEIVE A CALL BACK, RECORD NAME, PHONE NUMBER, DATE/TIME

Parent Name ____________________

Child Name ____________________

Child Phone ____________________

Date ____________________

Time ____________________

SCRIPT FOR STUDENTS

Good morning/afternoon/evening. My name is <NAME> and I’m calling from Educational Testing Service (ETS). We are doing an important study for the National Center for Education Statistics (NCES), part of the U.S. Department of Education.

We have spoken to your parent/guardian and we are pleased that you may be interested in helping us develop innovative English language arts test questions for a new test that is being developed by the National Assessment of Educational Progress, or NAEP. Before I tell you more about what you would be asked to do, I’d like to ask you one question first:

What grade are you currently in? (INTERVIEWER: DO NOT READ LIST)

 8th

 9th

 Other (Unfortunately, we are only interviewing students currently in grade 8 or grade 9 for this study. Thank you for your time. Have a good day/evening.)

Thank you. For the research interview, you will be asked to visit Educational Testing Service in Princeton and respond to a sample of English language arts test questions. Your responses will not be graded. The National Center for Education Statistics simply wants to know how you would interact with a virtual environment and answer the questions if they were to be included on an actual test. The questions focus on English language arts and are really interesting and fun to answer. You will be asked to answer the questions on a computer. Your input will help evaluate and improve the test questions and virtual environment. To thank you for participating, you will receive a <$25 Visa gift card (playtesting)/$30 Visa gift card (cognitive interviews)>. The interview will last about <90 minutes (playtesting)/120 minutes (cognitive interviews)> and will be recorded. Your interactions with the virtual environment will be video recorded as well but the data will only be used for research purposes. Your participation is totally voluntary. Without the permission of your parent or guardian, your interview responses and all other interview materials may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573]. With parental/guardian permission, some responses or clips from videos may be selected for use in research reports or presentations.

Would you like to participate?

 Yes

 No  (THANK AND TERMINATE)

If NO - Thank you for your time. I will need to speak to your parent/guardian to thank (him/her) for letting you take the time to speak with us.

TO PARENT: Your child has decided not to participate, but I would like to thank you very much for taking the time to speak with us today/this evening.

If YES - Thank you for agreeing to participate in this important research study. I will need to speak to your parent /guardian again. Is he/she available?

 Yes

 No

TO PARENT: Your child has indicated that he/she would be interested in participating in this study. We will be in touch with you within <TIME> to let you know whether or not your child has been selected to participate in this study, and to discuss scheduling and send you a consent form to fill out if he/she will be participating.

INTERVIEWER: IF CALLING PARENT BACK, RECORD NAME, PHONE NUMBER, DATE/TIME

Parent Name ____________________

Child Name ____________________

Parent Phone ____________________

Date ____________________

Time ____________________

Appendix E: Consent Form for Parents/Legal Guardian of Student Participants

PARTICIPATION CONSENT DOCUMENT

FOR PARTICIPATION IN RESEARCH

Dear Parent/Guardian:

Staff members at Educational Testing Service (ETS) and I are currently working on the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) Study, a federally funded project which is studying new types of technology delivered assessments. We are inviting 8th and 9th graders to participate in our study. For this project, a staff member from ETS will be meeting with students individually or in small groups for a study that lasts around <90 minutes (playtesting)/120 minutes (cognitive interviews) at the ETS location on Rosedale Road in Princeton, NJ. Parents will not be in the room with their child, and should plan to return to pick up their child at the end of the session. Students will receive a <$25 Visa gift card (playtesting)/$30 Visa gift card (cognitive interviews) for their participation. In addition, if transportation to the site is involved, your parent or legal guardian will receive a $25 Visa gift card to thank him or her for the time in bringing you to and from the <play testing/cognitive interview> site.

The project team will videotape students as they take part in problem-solving activities related to English language arts. They will be asked to interact with a virtual environment while answering questions about English language arts. Students will be asked to think aloud as they engage in these activities. Afterwards, they will answer a questionnaire about their experience with the activity.

Your child’s performance is not an indication of his or her abilities; it will inform us how the tasks work and help us to develop and refine our ideas for future tasks. Unless you give specific permission, your child’s responses to questions may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573]. Participation is voluntary and a student may withdraw from the study at any time.

We hope that you will give your consent for your child to participate in this important activity by signing the accompanying consent form. Students should bring the signed permission form with them when they come to ETS.

Please Note: Parents/Guardians are responsible for providing transportation for their children

to and from ETS’s Princeton campus, where the sessions will be held.

To check or confirm your appointment, or if you have further questions, please call (CONTACT) of ETS at (PHONE NUMBER) or email (EMAIL).

We thank you for considering supporting our research. Should you have any questions, please do not hesitate to contact me.

Sincerely,

Parent/Guardian Consent Form

Student’s name: ____________________________________________________

Please print student’s name

I am the parent/guardian of the child named above, and I hereby give my permission for my child to take part in the NAEP SAIL Study, including having the session video recorded for use by the project team. I understand that unless I give a separate consent for the information to be shared in other ways, my child’s information may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

Having read the accompanying letter, I voluntarily consent to my child’s participation in this study. I understand that my child’s participation is voluntary, my child may withdraw from the study at any time without prejudice, and that my child has the right to refuse to answer any questions he/she doesn’t feel comfortable responding to.

I understand that if I have any questions about any part of my child’s participation, I may call the Project Director, <SCIENTIST CONTACT INFORMATION>.

_____________________________________________ __________________________

Parent/Guardian Signature Date

Yes, I consent to my child participating in this research project, including the video capture of my child completing the assigned task and answering interview questions.

Shape2 No, I do not consent to my child participating in this research project.

________________________________________________________________________________________

In the future, we may have additional opportunities for students to participate in studies. If you are interested in being contacted about your child participating in future studies, please check the box below.

Shape3 I consent to being contacted about opportunities for my child to participate in future NAEP SAIL studies.

________________________________________________________________________________________

The data collected in this project will be valuable information for the larger cognitive learning field, and as such, your child’s responses or clips from these videos may be selected for use in research reports, presentations, and/or conferences. Your child’s name will not be identified, and you do not need to consent to your child’s recording being included in external research publications and/or research conferences in order for your child to participate in this study.

Shape4 Yes, I consent to my child’s data (video and responses) being included in external research publications and/or research conferences.

Shape5 No, I do not consent to my child’s data (video and responses) being included in external research publications and/or research conferences.

Please bring this completed form with you at the time of the

child’s scheduled appointment.

Appendix F: E-mail/Letter Interview Confirmation for Parents/Legal Guardians of Student Participants

IF SENT AS E-MAIL - Subject: Confirmation of <Play Testing/Cognitive Interview> Session

<DATE>

Dear <NAME>,

Thank you for agreeing to allow your child to participate in a study to give us feedback on the questions and simulation being developed for the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) project. During the session, your child will be asked to respond to sample test activities and interact with a virtual environment, which is being developed to assess knowledge of English language arts. The National Center for Education Statistics (NCES), part of the U.S. Department of Education, simply wants to know what your child thinks about the tests and how your child might answer questions on that test. Your child will work individually or in a small group of several students who will also be reviewing and reacting to the potential test questions. Your child’s feedback/responses will be noted, audio/video recorded, and used to help us revise and improve our tests.

Unless you give specific permission, your child’s responses to questions may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573]. Participation is voluntary and a student may withdraw from the study at any time.

Your child has been selected to attend a session at <START TO END TIME> on <DATE> located at the Princeton campus of ETS. To thank your child for <his/her> time, <he/she> will receive a <$25 Visa gift card (playtesting)/$30 Visa gift card (cognitive interviews)> as soon as <he/she> has finished the session. In addition, if transportation to the site is involved, you will receive a $25 Visa gift card to thank you for the time in bringing your child to and from the <play testing/cognitive interview> site.

Please confirm that this time slot still works for your child by <DATE>. We are attaching a consent form that must be signed and returned for your child to participate. Please fill it out and send it back to us, or have your child bring it with them to the <play testing/cognitive interview> site.

We look forward to meeting with your child and getting your child’s feedback. If you have any questions or if your child is unable to attend this session, please contact me using the information below.

Sincerely,

Educational Testing Service

Appendix G: Acknowledgement E-Mail to Parents/Legal Guardians of Students Not Selected to Participate

IF SENT AS E-MAIL - Subject: NAEP SAIL Study

<DATE>

Dear <NAME>,

Thank you for your interest in having your child be a part of the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) <SUBJECT> Study. Unfortunately, due to the number of students who were interested in participating, we are not able to ask your child to be a part of the study at this time. However, we anticipate that we will be conducting additional studies related to this project in the future. If you are interested in possibly being contacted about having your child participate at a later time if another opportunity becomes available, please let us know (see contact information below).

Thank you again for taking the time to respond to our participation request.

Sincerely,

Educational Testing Service

Appendix H: Thank You E-mail/Letter to Parent/Legal Guardian of Student Participants

IF SENT AS E-MAIL - Subject: Thank you for your Child’s Participation in a Research Study

<DATE>

Dear <NAME>,

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, Educational Testing Service (ETS) would like to thank you for allowing your child to participate in the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) research study. Thanks to your child’s participation, we are better able to refine, revise, and develop the simulation and questions that are part of this assessment.

Your child’s assistance helped us better understand how students approach these test questions and use the virtual environment. <HIS/HER> open, candid, and insightful comments were very helpful.

Again, thank you for your child’s assistance.

Sincerely,

Educational Testing Service

Appendix I: Thank You E-mail/Letter to School/ Organization/ Association of Student Participants

IF SENT AS E-MAIL - Subject: Thank you for your Participation in a Research Study

<DATE>

Dear <NAME>,

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, Educational Testing Service (ETS) would like to thank you for your <school’s/ organization’s/ association’s> participation in the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) research study. Thanks to the students at your <school/ organization/ association> we are better able to refine, revise, and develop the virtual environment and questions that are part of this assessment.

Your <school’s/ organization’s/ association’s> assistance helped us better understand how students approach the <SUBJECT> test questions. Students’ open, candid, and insightful comments were very helpful.

Again, thank you for your assistance.

Sincerely,

< COMPANY REPRESENTATIVE >

Educational Testing Service

Appendices – Tryouts

Appendix J: Site Recruitment E-mail/Letter (To be Sent to School Administrators/Staff Members)

IF SENT AS E-MAIL - Subject: Research Study Opportunity

<DATE>

My name is <CONTACT NAME> and I am contacting you from Educational Testing Service (ETS). We are conducting an important study for the National Center for Education Statistics (NCES), part of the U.S. Department of Education.

We are currently working on a new assessment being developed by the National Assessment of Educational Progress (NAEP) for the Survey Assessments Innovations Lab (SAIL) project, which may someday be delivered to students across the country. NAEP serves as the central means by which the United States measures student proficiency in core academic domains. It is vital for us to get student feedback while developing test questions.

We are looking for a school to work with us to try out this assessment with students in grades 8 or 9 to help us test new computer-delivered questions and simulations in English language arts. These voluntary tryouts will last no more than 90 minutes. Students will be screen recorded while responding to several <SUBJECT> test questions and use a simulation to help answer the questions. ETS staff will take notes of how students use the simulation, and use this information to revise and refine the questions.

Students will be informed that their participation is completely voluntary, and that they will not be evaluated. They will also be told they have a key role in helping us improve our tests and they can make a difference in the experience of thousands of students who may someday take these tests.

The school or schools we work with will need to be able to allow ETS researchers approximately 90 minutes of class time in a computer lab to administer the test to each participating class. This can be done in a single class period, or split across two class periods, depending on the schedule at your school. The school will be responsible for distributing and collecting consent forms, and for installing or allowing us to install screen capture software on the computers being used. As a thank you for participating, the school(s) we work with will receive $7.00 per participating student with signed consent forms, up to a maximum of 200 students (if all testing occurs at a single school). Participating schools should expect to be able to provide a minimum of 70 student participants.

If your school would be interested in participating in this important research project, or you have any questions or would like more information, please contact <CONTACT INFORMATION>.

Sincerely,

<NAME>

Educational Testing Service

Appendix K: Site Recruitment Phone Script (When Calling School Administrators/Staff Members)

Good morning/afternoon/evening. May I please speak to <NAME of contact at school/organization>?

We are currently working on a new assessment being developed by National Assessment of Educational Progress (NAEP) for the Survey Assessments Innovations Lab (SAIL) project, which may someday be delivered to students across the country. NAEP serves as the central means by which the United States measures student proficiency in core academic domains. It is vital for us to get student feedback while developing test questions.

We are looking for a school to work with us to try out this assessment with students in grades 8 or 9 to help us test new computer-delivered questions and simulations in English language arts. These voluntary tryouts will last no more than 90 minutes. Students will be screen recorded while responding to several test questions and use a various tools to help answer the questions. ETS staff will take notes of how students use the virtual environment, and use this information to revise and refine the questions.

The school or schools we work with will need to be able to allow ETS researchers approximately 90 minutes of class time in a computer lab to administer the test to each participating class. This can be done in a single class period, or split across two class periods, depending on the schedule at your school. The school will be responsible for distributing and collecting consent forms, and for installing or allowing us to install screen capture software on the computers being used. As a thank you for participating, the school we work with will receive $7.00 per participating student with signed consent forms, up to a maximum of 200 students (if all testing occurs at a single school). Participating schools should expect to be able to provide a minimum of 70 student participants. Would your school be interested in participating in this research study?

If NO: Thank you for your time. Have a wonderful day/evening.

If YES: Great, thank you very much. To ensure that we interview a broad mix of students and determine the best interview location, I have a few brief screening questions to ask you. It should only take a few minutes of your time.

Screener

[INTERVIEWER: Determine recruitment needs in advance. Look in public documents and school/organization website to find what grades a school/organization includes, the size of each grade, the SES, and racial/ethnic makeup of the student body. Record all responses to screener questions. End screener at whatever point the staff member gives a response that confirms that their group has already been sampled adequately. End screener script: Based on the requirements of this study, we are not able to include you in the interviews at this time.]

Do you have XX number [specify recruitment needs] of <GRADE> graders who might be available to participate in the study?
Would you be able to give ETS a room in which to conduct the study at your <ORGANIZATION>?

Would you be able to send flyers and consent forms to parents of potential student participants?

If NO: Thank you for your time. Have a wonderful day/evening.

If YES: Thank you for taking the time to answer the screening questions. Your school has students who are eligible to participate in the tryouts. Again, each session will last up to 90 minutes.

Schedule test date(s) and make arrangements to send consent forms to the school, and install screen capture software.

We may need to reach you by telephone. Is the number I called the best number to use?

Do you have any questions at this time?

[INTERVIEWER: Answer any questions.]

If you do have any questions, please call <PHONE NUMBER>.

Thank you again for agreeing to be part of this very important research project.

Appendix L: Recruitment E-mail/Letter (If Additional Out-Of-School Recruitment is Required - To be Sent to Teachers/Staff Members)

IF SENT AS E-MAIL - Subject: Research Study Opportunity

<DATE>

Dear <Teacher or Staff Member at After School Program>,

We are looking for students in grades 8 and 9 to help us test new computer-delivered questions for the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) research study. NAEP serves as the central means by which the United States measures student proficiency in core academic domains. It is vital for us to get student feedback while developing test questions.

In these sessions, students will interact with a virtual environment and answer questions that have been developed for the English language arts project. Students will be informed that their participation is completely voluntary, and that they will not be evaluated. They will also be told they have a key role in helping us improve our tests and they can make a difference in the experience of students who may someday take these or similar tests. ETS staff will record students as they observe the way in which students interact with the simulation and questions to evaluate what has been developed.

Student responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA 2002, 20 U.S.C §9573].

The sessions will be scheduled for 90 minutes, and students will be given $25 Visa gift cards for each session as a thank you for their time and as an incentive to help us. In addition, if transportation to the site is involved, a parent or legal guardian of each student will receive a $25 Visa gift card as a thank you for the time in bringing the child to and from the study site.

We would like your help in distributing flyers. We will hold sessions at ETS offices located in Princeton, New Jersey or at <LOCATION>. If you are interested and willing to help us recruit students for this effort, or would like additional information about the effort, please contact <staff member at ETS organizing the sessions>. We are attaching a flyer that can be shared with your group, and will be happy to provide you with printed copies if you prefer.

Thank you for your time and interest. If you have any questions, please contact <CONTACT INFORMATION>.

Sincerely,

Educational Testing Service

660 Rosedale Rd

Princeton, NJ 08541

Appendix M: Informational Flyer (If Additional Out-Of-School Recruitment is Required)

Can You Help Us Test Our Test?

Educational Testing Service (ETS) is looking for students in grades 8 and 9 to help us test new computer-delivered test questions for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. These questions are part of a new test being developed that deals with information gathering, processing, and evaluation skills. You will be interacting with a virtual environment and recorded as you answer test questions as part of a tryout of materials that have been developed.

The information we collect from you as you explore and react to our test will only be used to help improve the test. The information will not be shared with or used by any school or district.

These sessions will be held at <location of after school club or ETS location> on <note days of the week when sessions are scheduled and the window of time when these sessions will be held>. These sessions will last no more than 90 minutes.

You will get a $25 Visa gift card for your time in helping us to improve our tests. In addition, if transportation to the site is involved, your parent or legal guardian will receive a $25 Visa gift card to thank him or her for the time in bringing you to and from the tryout site.

Appendix N: Websurveyor Intake Form (If Additional Out-Of-School Recruitment is Required)

NAEP SAIL (SUBJECT) Study

Staff members at Educational Testing Service (ETS) and I are currently working on the NAEP (National Assessment of Educational Progress) SAIL (Survey Assessments Innovations Lab) research study, a federally funded project which is studying the effects of interactive computer delivery and content of NAEP SAIL assessments. We are inviting 8th and 9th graders to participate in our study. This will inform us how the tasks work and help us to develop and refine our ideas for future tasks.

The study will require that students be video recorded as they spend approximately 90 minutes using a computer simulation to answer questions about English language arts, and then answer some questions about their experience with the tasks

Students will receive a $25 Visa gift card for participation in the study. In addition, if transportation to the site is involved, the parent or legal guardian will receive a $25 Visa gift card to thank him or her for the time in bringing participant(s) to and from the <play testing/cognitive interview> site. Your child’s responses to questions may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573]. Participation is voluntary and a student may withdraw from the study at any time.

We would like to have a balanced group of students take part in the study. To that end, we’d like you to give us some information about your student by filling out the form on the following pages. We will let you know within <TIME> whether your student has been selected to take part in this study. If your child is not selected for this study, I hope you will consider letting him or her be a part of other studies in the future.

If you have any questions, please contact (CONTACT INFORMATION).

Thanks for your interest in this study!

(NAME)

Student Name (Last, First):

Parent Name (Last, First):

Parent Phone Number:

Parent E-mail Address:

What grade is your child in? ____8th ___9th

Student’s school:

*Student’s Age:

*Student’s Gender:

*Student’s Race/Ethnicity:

Ethnicity:  Hispanic or Latino  Not Hispanic or Latino

Race (one or more of the categories below):

American Indian or Alaska Native
Asian
Black or African American
Native Hawaiian or other Pacific Islander
White or Caucasian

(* Indicates that the question is voluntary.)

Possible times for 90-minute session: Please mark all the sessions that would work for your child(ren)

(It is possible that not all these times will be used.)

(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)
(DATE)	(TIME)	(TIME)	(TIME)

If some other time during these weeks would be better, what 90 minutes time frame would that be?

To complete submission of this form, please click the “Submit” button. You will be contacted within <TIME> to let you know whether or not your child has been selected to participate in this study.

(SUBMIT BUTTON)

Appendix O: Recruitment Phone Script for Parents/Legal Guardians of Student Participants (If Additional Out-Of-School Recruitment is Required)

(To be used only if the required number of participants is not recruited through schools and organizations and direct outreach to parents is necessary)

Script when calling parents from previously compiled list:

Good morning/afternoon/evening. Is this the <NAME FROM CALL LIST> household? My name is <NAME>, and I am calling from Educational Testing Service (ETS). We are conducting a study on behalf of the U.S. Department of Education’s National Center for Education Statistics, or NCES. We are recruiting students currently in grades 8 AND 9 to sign up to participate in a research study supporting the development of new English language arts test questions and interactive virtual environments for a project being sponsored by the National Assessment of Educational Progress or NAEP. [If parent asks what this is: This national assessment is administered to students in elementary, middle, and high schools throughout the United States.]

Are there any 8th or 9th grade students in your household?

Yes (Great! Let me give you some additional information about the study.)

No (Thank you, have a good day/evening.)

We are conducting tryouts to explore how students work through sample English language arts test questions and interact with a virtual environment to ensure that they are clear and understandable to a broad range of students. If selected to participate, your child will receive a $25 Visa gift card. In addition, if transportation to the site is involved, you will receive a $25 Visa gift card to thank you for the time in bringing your child to and from the interview site.

The tryouts will take place at Educational Testing Service or at <LOCATION>. It can be scheduled at your convenience and will last about 90 minutes. During the interview, your child will be asked to respond to sample science test questions on a computer. NCES simply wants to know how students would use the virtual environment and answer the questions if they were included on an actual NAEP assessment.

These tryouts are an important stage of the NAEP test development process. By sharing their feedback with us, students can directly impact the future of this nationwide assessment by helping NCES evaluate the new types of tests that are being explored.

Do you believe your child would be interested in participating?

Yes (Proceed to protocol script below)

No (Thank you, have a good day/evening.)

(Skip to screener.)

Script when calling a parent who responded to e-mail or letter

Screener

What is your child’s name?

What grade is your child currently in? (INTERVIEWER: DO NOT READ LIST)

 8th

 9th

 Other (Unfortunately, we are only interviewing students currently in grade 8 or grade 9 for this study. Thank you for your time. Have a good day/evening.)

What school does your child attend? (INTERVIEWER: Note if homeschooled)

____________________________________________________________

How old is your child?

What is your child’s gender?

What is your child’s race or ethnicity? (INTERVIEWER: Only read list if parent doesn’t immediately reply. Are you…?)

Ethnicity:  Hispanic or Latino  Not Hispanic or Latino

Race (one or more of the categories below):

American Indian or Alaska Native
Asian
Black or African American
Native Hawaiian or other Pacific Islander
White or Caucasian

[If the respondent indicates they prefer not to disclose this information, the interviewer should say “That’s fine” and move on to the next question.]

Thank you. We would like to speak with your child to confirm if (he/she) is interested in participating in the tryout. It will only take a couple of minutes and then I will ask to speak with you again for another moment. Is your child available to speak with us now or should we call (him/her) back at another time?

 Yes, the child is available for screening questions  (SKIP TO SCRIPT)

 No, please call back.

Is the phone number you gave me earlier the best number to call?

Other phone numbers: ____________________________

Is there a good time for me to call? When would be best for you?

INTERVIEWER: IF CHILD NEEDS TO RECEIVE A CALL BACK, RECORD NAME, PHONE NUMBER, DATE/TIME

Parent Name ____________________

Child Name ____________________

Child Phone ____________________

Date ____________________

Time ____________________

SCRIPT FOR 8th/9th GRADE STUDENTS

What grade are you currently in? (INTERVIEWER: DO NOT READ LIST)

 8th

 9th

 Other (Unfortunately, we are only interviewing students currently in grade 8 or grade 9 for this study. Thank you for your time. Have a good day/evening.)

Thank you. For the study, you will be asked to visit Educational Testing Service in Princeton or meet with someone at <LOCATION> and respond to a sample of English language arts test questions. Your responses will not be graded. The National Center for Education Statistics simply wants to know how you would interact with a virtual model environment and answer the questions if they were to be included on an actual test. The questions focus on English language arts and are really interesting and fun to answer. You will be video recorded as you answer the questions on a computer but the data will only be used for research purposes. Your input will help evaluate and improve the test questions and simulation. To thank you for participating, you will receive a $25 Visa gift card. In addition, if transportation to the site is involved, your parent or legal guardian will receive a $25 Visa gift card to thank him or her for the time in bringing you to and from the study site. The study will last about 90 minutes and will be recorded. Your interaction with the virtual environment will be video recorded as well. Your participation is totally voluntary. Your interview responses and all other interview materials may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

Would you like to participate?

 Yes

 No  (THANK AND TERMINATE)

If NO - Thank you for your time. I will need to speak to your parent/guardian to thank (him/her) for letting you take the time to speak with us.

TO PARENT: Your child has decided not to participate, but I would like to thank you very much for taking the time to speak with us today/this evening.

If YES - Thank you for agreeing to participate in this important research study. I will need to speak to your parent/guardian again. Is he/she available?

 Yes

 No

INTERVIEWER: IF CALLING PARENT BACK, RECORD NAME, PHONE NUMBER, DATE/TIME

Parent Name ____________________

Child Name ____________________

Parent Phone ____________________

Date ____________________

Time ____________________

Appendix P1: Consent Form for Parents/Legal Guardians of Student Participants (In-School Administration)

PARTICIPATION CONSENT DOCUMENT

FOR PARTICIPATION IN RESEARCH

Dear Parent/Guardian:

Your child’s school is participating in a pilot study for the National Assessment of Educational Progress (NAEP), for a federally funded project that is studying new types of technology delivered assessments being developed by Educational Testing Service (ETS). As part of this study, your child’s class will spend two class periods taking a test in which they undertake information-gathering activities designed to assess English language arts literacy as part of a virtual environment. After they finish the activity, students will complete a brief questionnaire about their experience.

Your child’s performance is not an indication of his or her abilities; it will inform us how the tasks work and help us to develop and refine our ideas for future tasks. All information obtained from your child may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

We hope that you will give your consent for your child to participate in this important activity by signing the accompanying permission form. Students should return the attached permission form to the classroom teacher by (DATE).

If you have any questions, please call (CONTACT) of ETS at (PHONE NUMBER) or email (EMAIL).

Should you have any questions, please do not hesitate to contact me.

Sincerely,

Parent Consent Form

Student’s name: ____________________________________________________

Please print student’s name

I am the parent/guardian of the child named above, and I hereby give my permission for my child to take part in the National Assessment of Educational Progress (NAEP) for the Survey Assessments Innovations Lab (SAIL) research study. I understand that my child will not be identified by name, and that my child’s information may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

I understand that if I have any questions about any part of my child’s participation, I may call the Project Director, <SCIENTIST CONTACT INFORMATION>.

Shape6 Yes, I consent to my child participating in this research project.

Shape7 No, I do not consent to my child participating in this research project.

_____________________________________________ __________________________

Parent/Guardian Signature Date

Appendix P2: Consent Form for Parents/Legal Guardians of Student Participants (If Additional Out-Of-School Recruitment is Required)

PARTICIPATION CONSENT DOCUMENT

FOR PARTICIPATION IN RESEARCH

Dear Parent/Guardian:

Staff members at Educational Testing Service (ETS) and I are currently working on the National Assessment of Educational Progress (NAEP) for the Survey Assessments Innovations Lab (SAIL) research study, a federally funded project that is studying new types of technology delivered assessments. We are inviting 8th and 9th graders to participate in our study. For this project, a staff member from ETS will be meeting with students for a study that lasts around 90 minutes at the ETS location on Rosedale Road in Princeton, NJ, or at <LOCATION>. Parents will not be in the room with their child, and should plan to return to pick up their child at the end of the session. Students will be given a $25 Visa gift card as a thank you for their participation. In addition, if transportation to the site is involved, the parent or legal guardian will receive a $25 Visa gift card to thank him or her for the time in bringing the participating student to and from the study site.

The project team will videotape students as they take part in problem-solving activities related to English language arts in a virtual environment. They will be asked to interact with a virtual environment on a tablet or computer while answering questions about English language arts. At the end of the test, they will be asked questions about their experience.

We hope that you will give your consent for your child to participate in this important activity by signing the accompanying permission form. Students should bring the signed permission form with them when they come to ETS.

Please Note: Parents are responsible for providing transportation for their children

to and from ETS’s Princeton campus, where the sessions will be held.

To check or confirm your appointment, or if you have further questions, please call (CONTACT) of ETS at (PHONE NUMBER) or email (EMAIL).

Should you have any questions, please do not hesitate to contact me.

Sincerely,

Parent Consent Form

Student’s name: ____________________________________________________

Please print student’s name

I am the parent/guardian of the child named above, and I hereby give my permission for my child to take part in the NAEP SAIL research study, including having the session video recorded for use by the project team. I understand that my child will not be identified by name, and that my child’s information may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

I understand that if I have any questions about any part of my child’s participation, I may call the Project Director, <SCIENTIST CONTACT INFORMATION>.

_____________________________________________ __________________________

Parent/Guardian Signature Date

Shape8 Yes, I consent to my child participating in this research project, including the video capture of my child completing the assigned task.

Shape9 No, I do not consent to my child participating in this research project.

________________________________________________________________________________________

Shape10 I consent to being contacted about opportunities for my child to participate in future NAEP SAIL studies.

Please bring this completed form with you at the time of the

child’s scheduled appointment.

Appendix Q: E-mail/Letter Interview Confirmation for Parents/Legal Guardians of Student Participants (If Additional Out-Of-School Recruitment is Required)

IF SENT AS E-MAIL - Subject: Confirmation of Tryout Session

<DATE>

Dear <NAME>,

Thank you for agreeing to allow your child to participate in a study to give us feedback on the questions and simulation being developed for the National Assessment of Educational Progress (NAEP) SAIL (Survey Assessments Innovations Lab) research project. During the session, your child will be asked to respond to sample test activities and interact with a virtual model that is being developed to assess knowledge of English language arts. The National Center for Education Statistics (NCES), part of the U.S. Department of Education, simply wants to know how your child might answer questions on that test. Your child’s feedback/responses will be noted, audio/video recorded, and used to help us revise and improve our tests.

All of your child’s responses will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by law [ESRA, 20 U.S.C §9573].

Your child has been selected to attend a session at <START TO END TIME> on <DATE> located at <LOCATION>. To thank your child for <his/her> time, <he/she> will receive a $25 Visa gift card as soon as <he/she> has finished the session. In addition, if transportation to the site is involved, you will receive a $25 Visa gift card to thank you for the time in bringing your child to and from the tryout site.

Sincerely,

Educational Testing Service

Appendix R: Acknowledgement E-Mail to Parents/Legal Guardians of Students Not Selected to Participate (If Additional Out-Of-School Recruitment is Required)

IF SENT AS E-MAIL - Subject: NAEP SAIL Study

<DATE>

Dear <NAME>,

Thank you again for taking the time to fill out our online screening form.

Sincerely,

Educational Testing Service

Appendix S: Thank You E-mail/Letter to Parent/Legal Guardian of Student Participants (If Additional Out-Of-School Recruitment is Required)

IF SENT AS E-MAIL - Subject: Thank you for your Child’s Participation in a Research Study

<DATE>

Dear <NAME>,

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, Educational Testing Service (ETS) would like to thank you for allowing your child to participate in our National Assessment of Educational Progress (NAEP) SAIL (Survey Assessments Innovations Lab) research study. Thanks to your child’s participation, we are better able to refine, revise, and develop the simulation and questions that are part of this assessment.

Your child’s assistance helped us better understand how students approach these test questions and use the simulation. <HIS/HER> help was very much appreciated.

Again, thank you for your child’s assistance.

Sincerely,

Educational Testing Service

Appendix T: Thank You E-mail/Letter to School/Organization/Association of Student Participants

IF SENT AS E-MAIL - Subject: Thank you for your Participation in a Research Study

<DATE>

Dear <NAME>,

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, Educational Testing Service (ETS) would like to thank you for your <school’s/ organization’s/ association’s> participation in the NAEP SAIL research study. Thanks to the students at your <school/ organization/ association> we are better able to refine, revise, and develop the virtual environment and questions that are part of this assessment.

Your <school’s/ organization’s/ association’s> assistance helped us better understand how students approach the <SUBJECT> test questions. The students’ help was very much appreciated.

Again, thank you for your assistance.

Sincerely,

< COMPANY REPRESENTATIVE >

Educational Testing Service

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	NATIONAL CENTER FOR EDUCATION STATISTICS
Author	joc
File Modified	0000-00-00
File Created	2021-01-22