National Assessment of Educational Progress (NAEP) Survey Assessments Innovation Lab (SAIL) Dynamic Assessments Cog Labs and Medium Scale Study

The National Center for Education Statistics (NCES) is conducting research to investigate how the design of test impacts students’ test taking motivation, and have contracted with researchers at Educational Testing Service (ETS) to design and administer a project called Dynamic Assessments.

Because your organization plays an active role in the community, we need your help.

NCES is looking for students in grades [6-10] (2020-2021 school year) to help us try out some new types of mathematics items that will help us better understand how students are able to interact with challenging mathematics content, and which could be used for future assessments. Student participation is vital in helping us develop the best questions possible.

The voluntary research interviews will be held virtually via <Zoom or other videoconference platform> and will last no more than 90-minutes. Participants will need a laptop as well as high-speed internet. During the interview with a researcher, each student will be asked to answer mathematics questions, with access to resources that will help the student answer them if they are challenging. There will be more than one researcher present during each research interview. While a parent does not have to be present during the interview, they can be. Student feedback is important for us to develop the best ways of assessing students’ mathematical understanding. Because we care about student privacy, we will assign each student a unique student identifier. Student names will not be linked to their answers.¹

The audio of the <Zoom or other videoconference platform> interview will be recorded and may include screen capture of their interactions with the items. However, no student faces will be recorded.

Each student will receive a $40 virtual gift card following the interview as a thank you.

Students play a key role in helping us improve our assessments. In fact, their participation enhances the experience of thousands of students all over the United States who may use these items in the future.

We would like to ask for your assistance in letting parents/legal guardians and students know how they can be involved in this important study. Attached is a letter which can either be e-mailed to parents, or printed out and provided to them if you are working with students in person. We appreciate your consideration of this request and invite you to please contact us with any questions regarding this project. We look forward to speaking with you if you have any questions!

Sincerely,

Educational Testing Service

Appendix B: Recruitment Email/Letter to Youth Organizations– Medium-Scale Study

[IF SENT AS EMAIL—Subject: Education Research Opportunity for Students in <GRADE> grade]

<DATE>

Dear <NAME>,

Because your organization plays an active role in the community, we need your help.

NCES is looking for students in [GRADE] (2020-2021 school year) to help us try out some new types of mathematics items that will help us better understand how students are able to interact with challenging mathematics content, and which could be used for future assessments. Student participation is vital in helping us develop the best questions possible.

The voluntary research study will not involve any interaction between students and researchers and will last no more than 45-minutes. Participants will need a laptop as well as high-speed internet. During the session, each student will be asked to answer mathematics questions, with access to resources that will help the student answer them if they are challenging. Because we care about student privacy, we will assign each student a unique student identifier. Student names will not be linked to their answers.¹

Students [will be able to participate at their convenience during a given window of time/will be given a chance to select a timeslot to participate.]

Each student will receive a $25 virtual gift card following the study as a thank you.

Sincerely,

Educational Testing Service

Appendix C: Recruitment Email/Letter to Parents or Legal Guardians of Student Participants (Under Age 18) - Cog Lab Study

[IF SENT AS EMAIL—Subject: Paid Education Research Opportunity]

<DATE>

Dear <NAME>:

We need your help.

NCES is looking for students in grades [6-10] (2020-2021 school year) to help us try out some new types of mathematics items that will help better understand how students are able to interact with challenging mathematics content, and which could be used for future assessments. Student participation is vital in helping us develop the best questions possible.

The <Zoom or other videoconference platform> interview will be audio recorded and may include screen capture of their interactions with the items. However, no student faces will be recorded.

Each student will receive a $40 virtual gift card following the interview as a thank you.

Because we care about your child’s privacy, we will assign him/her a unique student identifier. At no time will your child’s name be linked to his/her answers.¹

This study is voluntary and has many interviews available. Your child can be scheduled at a time that works best for you.

If your child is interested in this study, please <fill out an intake form at <LINK> OR email <CONTACT EMAIL>>. We look forward to speaking with you soon!

Thank you,

Educational Testing Service

Appendix D: Recruitment Email/Letter to Parents or Legal Guardians of Student Participants (Under Age 18) - Medium-Scale Study

[IF SENT AS EMAIL—Subject: Paid Education Research Opportunity]

<DATE>

Dear Parent or Legal Guardian:

[INCLUDE IF SENT BY ORGANIZATION NOT ETS: ETS has asked [ORGANIZATION] to share information about this opportunity with the parents of our students.]

NCES is looking for students who are in [GRADES] (2020-2021 school year) to help us try out some new types of mathematics items that will help us better understand how students are able to interact with challenging mathematics content, and which could be used for future assessments. Student participation is vital in helping us develop the best questions possible.

The voluntary research study will involve taking an assessment online, will not involve any interaction between students and researchers and will last no more than 45-minutes. Participants will need a laptop as well as high-speed internet. During the session, each student will be asked to answer mathematics questions, with access to resources that will help the student answer them if they are challenging. Because we care about student privacy, we will assign each student a unique student identifier. Student names will not be linked to their answers.¹

Students [will be able to participate at their convenience during a given window of time/will be given a chance to select a timeslot to participate.]

Each student will receive a $25 virtual gift card following the study as a thank you.

If your child is interested in this study, please <fill out an intake form at <LINK> OR call <CONTACT PHONE> OR email <CONTACT EMAIL>>. We look forward to speaking with you soon!

Thank you,

Educational Testing Service

Appendix E: Recruitment Web-Surveyor Intake Form—Link Provided in Emails/Letters to Parents/Legal Guardians Interested in Participating – Cog Lab Study and Medium-Scale Study

[Survey will be administered via a web-surveyor form hosted by Verint EFM. Separate intake form links will be used for each round, with one additional question included for the survey round, which is taking place first. Students will only be eligible to participate in one or the other.]

[Note that the research team will not be calling people as part of recruitment. If someone calls the researcher at the phone number provided on the recruitment letter and requests assistance registering their child, the following script will be used: “Thank you for calling us. I'm going to read out some questions for you so that I can fill out the intake form for your child. Please let me know if you have any questions as we go through”]

Student Name (Last, First): ____________________________________________

Student’s Grade (during the 2020-2021 school year): 6^th__ 7^th__ 8^th__ 9^th__10^th__

Student’s math course during the 2020-2021 school year: __________________

Student’s final math grade in the 2019-2020 school year (optional): _____________

Student’s School/District: _____________________________________________

Student’s Gender: ___________________________________________________

Student’s Ethnicity/Race:

Ethnicity:

 Hispanic or Latino  Not Hispanic or Latino

Race (one or more of the categories below):

 American Indian or Alaska Native

 Asian

 Black or African American

 Native Hawaiian or other Pacific Islander

 White

 Prefer not to answer

Is the student currently enrolled in any of the following types of mathematics summer courses? (select all that apply)

 Honor courses

 Remedial courses

 Other courses

Does the student participate in the National School Lunch program (or receive free or reduced price school lunch)?

Yes

No

Prefer not to answer

What is the total annual household income range based on the following ranges?

Less than $30,000

$30,000 to $99,999

$100,000 or more

Prefer not to answer

Does the student live in a rural, suburban, or urban area?

Rural

Suburban

Urban

Don’t Know

Prefer not to answer

Possible Time For 90-Minute Interview: [COG LAB STUDY ONLY]

Please mark all the interviews that would work for the student if he or she or they qualify to participate. Not all of these times will be used. We are trying to schedule a limited number of interviews that will be convenient for the largest number of participants.

<DATE>	<TIME>	<TIME>	<TIME>
<DATE>	<TIME>	<TIME>	<TIME>
<DATE>	<TIME>	<TIME>	<TIME>
<DATE>	<TIME>	<TIME>	<TIME>

If some other time during these weeks would be better, what 90-minute time frame would that be? Please include your time zone in your response.

[COG LAB SURVEY ONLY: If your child is not selected for participation in this study, would you be interested in being contacted about their participation in a follow-up study we expect to be conducting in the next few months?

If your student qualifies to participate, we will need to follow up with you for scheduling purposes. Please provide us with your contact information.

PARENT/LEGAL GUARDIAN FIRST NAME: ________________________________________

PARENT/LEGAL GUARDIAN LAST NAME: ________________________________________

EMAIL ADDRESS: _______________________________________________

MOBILE PHONE NUMBER: ________________________________________

ALTERNATE PHONE NUMBER: _____________________________________

To complete the submission of this form, please click the “Submit” button. You will be contacted within <TIMEFRAME> to let you know whether or not the student has been selected to participate in this study.

(SUBMIT BUTTON)

National Center for Education Statistics (NCES) is authorized to conduct NAEP by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622) and to collect students’ education records from education agencies or institutions for the purposes of evaluating federally supported education programs under the Family Educational Rights and Privacy Act (FERPA, 34 CFR §§ 99.31(a)(3)(iii) and 99.35). All of the information provided by participants may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). By law, every NCES employee as well as every NCES agent, such as contractors and NAEP coordinators, has taken an oath and is subject to a jail term of up to 5 years, a fine of $250,000, or both if he or she willfully discloses ANY identifiable information about participants. Electronic submission of participant’s information will be monitored for viruses, malware, and other threats by Federal employees and contractors in accordance with the Cybersecurity Enhancement Act of 2015. The collected information will be combined across respondents to produce statistical reports.

OMB No. 1850-0803 v.275 Approval Expires 7/31/2023

Appendix F: Consent Form for Parent/Legal Guardian of Student Participants (Under Age 18) - Cog Lab Study

[This will be attached to e-mail in Appendix H, and parents will sign, scan, and return by e-mail.]

PARENTAL CONSENT FORM

FOR PARTICIPATION IN RESEARCH

National Assessment of Educational Progress (NAEP) 2021 Dynamic Assessment Cognitive Interviews: Parent Consent Form

Summary: Your child is being asked to participate in a research study conducted by Educational Testing Service (ETS), which involves solving several online math problems which include resources that may help them solve the problems, over a 90-minute session which will be conducted via <ZOOM OR OTHER VIDEOCONFERENCE PLATFORM>. Please read the details below and ask questions about anything you do not understand.

1. Why is this research study being done?

The National Center for Education Statistics (NCES), which is part of the U.S. Department of Education, is conducting a study with students who are in grades 6-10 to look at how students use supporting resources when interacting with challenging mathematics content. Students’ participation may help the National Assessment of Educational Progress (NAEP) provide practice materials that better help teachers prepare their students for the NAEP assessments.

2. What is my child being asked to do?

As a participant in this research study, your child is being asked to do the following:

Each participant will work through an early version of mathematics questions with supporting resources on a computer via a virtual meeting platform. A researcher will ask what the student was thinking as he or she completed the questions, whether he or she thinks the questions are clear and understandable, and whether he or she has any suggestions on how to improve the questions or the system. The interviewer may help students with some of the questions.
Specific knowledge is not required, and participants’ responses will not be graded. There will be more than one adult on the remote call during each research interview.
ETS is not evaluating your child’s performance, i.e., participation in this research is not indicative nor a representation of the child’s ability. This study is being done to test how our questions work and gather information to make them better.
Your child’s answers will not be disclosed to his or her or their school and will not be linked to any personally identifiable information, such as the school’s name.

The amount of time your child will participate in this research study is no more than 90 minutes. The interview requires only one virtual meeting session. A parent or guardian is welcome to sit in while their child completes the study, but is not required to do so.

This research study will be conducted online via a virtual meeting platform.

3. Are there benefits to being in this research study?

Students will have an opportunity to interact with challenging online mathematical content, which may help prepare them for similar items they see in future assessments.

4. What are the risks?

There are no anticipated risks associated with participation in this research study.

5. How will my child’s data be used?

Data will be used for research purposes to inform question development. Your child will be assigned a unique student identifier and at no time will his/her name be linked to any of his/her answers. The study will not retain any personally identifiable information. Prior to the start of the study, students will be notified that their participation is voluntary and that the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Audio will be recorded for analysis, and may include screen capture of your student’s interactions with the assessment, but no faces will appear in the screen capture, and no other personally identifying information will be recorded or retained. The audio recording and/or screen capture will be destroyed after the release of the final report. Please note that in order to capture audio, the video must be recorded as well. The audio file can be separated from the video file post-interview, and any video portion containing faces or other identifying information will be immediately deleted.

Anonymized data may be used in future research studies; this de-identified data will only be accessible within ETS and will not be shared with outside entities. Research findings may also be shared at educational conferences and in academic publications, however, your child will not be individually identified.

6. How will my child’s data be maintained?

Data will be kept confidential and securely stored on ETS encrypted, password-protected computers and servers and/or secure, locked filing cabinets for up to 5 years, an ETS policy for auditing purposes.

7. What other options are there?

You may choose to not allow your child to participate in this research study.

8. What are my child’s rights and what else should I know as the parent of a research study participant?

Your child’s participation in this research study is completely voluntary, and your child has the right to withdraw from this study at any time and can end early if he/she wishes with the knowledge that if he/she withdraws early, he/she forgoes payment. All information gathered from your child up to the point of withdrawal will be deleted. Your child may choose not to take part in this research study at all, and your child also has the right to refuse to answer any questions that make them uncomfortable. You and your child also have the right to know specifically how your child’s data will be used.

9. Will my child receive a payment?

By participating in this research study your child will be paid $40 via virtual gift card. Your child will be paid if he/she completes the full 90-minute research study.

10. Who can answer questions about the study?

If you or your child have any questions or concerns about this research study, or if your child has any problems that occur from taking part in this research study, you or your child may contact the researcher [CONTACT] at [E-MAIL], who is coordinating this interview at ETS on behalf of NCES.

If you have any questions about your child’s rights as a research participant or if you believe your child has suffered an injury as a result of taking part in the research study, you or your child may contact the institutional review board, or IRB, that reviews and approves all research activities sponsored by ETS at IRB@ETS.ORG.

Statement of informed consent: I have read and understand this consent form and understand I can contact the researcher if I have any concerns. I also understand that the research study content to which my child will have access is confidential and the property of ETS.

Please select one of the following statements:

Yes, I allow my child to participate in this research.
No, I do not allow my child to participate in this research.

__________________________________________________

Parent/Guardian Signature Date

_________________________________________

Parent/Guardian Printed Name

_________________________________________

Print Child’s Name

Permission to Use Identifiable Media (Audio Recordings).

Please select one of the following statements:

I DO give permission to include my child’s voice on an audio-recording for use only by ETS staff, including use for assessment development, product development, research and/or training purposes. No names will appear on any materials.

I DO NOT give permission to include my child’s voice on an audio-recording for any purposes other than their participation in this research study.

Appendix G: Consent Form for Parent/Legal Guardian of Student Participants (Under Age 18) - Medium-Scale Study

[This will be attached to e-mail in Appendix I, and parents will sign, scan, and return by e-mail.]

PARENTAL CONSENT FORM

FOR PARTICIPATION IN RESEARCH

National Assessment of Educational Progress (NAEP) 2021 Dynamic Assessment Study: Parent Consent Form

Summary: Your child is being asked to participate in a research study conducted by Educational Testing Service (ETS), which involves solving several online math problems which include resources that may help them solve the problems, over a 45-minute session which will be conducted via <ZOOM OR OTHER VIDEOCONFERENCE PLATFORM>. Please read the details below and ask questions about anything you do not understand.

1. Why is this research study being done?

2. What is my child being asked to do?

As a participant in this research study, your child is being asked to do the following:

Each participant will work through an early version of mathematics questions with supporting resources on a computer.
Specific knowledge is not required, and participants’ responses will not be graded.
ETS is not evaluating your child’s performance, i.e., participation in this research is not indicative nor a representation of the child’s ability. This study is being done to test how our questions work and gather information to make them better.
Your child’s answers will not be disclosed to his or her or their school and will not be linked to any personally identifiable information, such as the school’s name.

The amount of time your child will participate in this research study is no more than 45-minutes. The study will take place in a single session.

This research study will be conducted online via a link which will be provided to students.

3. Are there benefits to being in this research study?

Students will have an opportunity to interact with challenging online mathematical content, which may help prepare them for similar items they see in future assessments.

4. What are the risks?

There are no anticipated risks associated with participation in this research study.

5. How will my child’s data be used?

6. How will my child’s data be maintained?

7. What other options are there?

You may choose to not allow your child to participate in this research study.

8. What are my child’s rights and what else should I know as the parent of a research study participant?

9. Will my child receive a payment?

By participating in this research study your child will be paid $25 via virtual gift card. Your child will be paid if he/she completes the full 45-minute research study.

10. Who can answer questions about the study?

Please select one of the following statements:

Yes, I allow my child to participate in this research.
No, I do not allow my child to participate in this research.

We may be interested in following up with some of the students involved with this study in the future. Do you consent to being contacted by ETS researchers if they are conducting a follow-up study to the current research? (Your answer to this will not affect your child’s ability to participate in this study).

Please select one of the following statements:

Yes, researchers may contact me again.
No, I would prefer not to be contacted in the future.

__________________________________________________

Parent/Guardian Signature Date

_________________________________________

Parent/Guardian Printed Name

_________________________________________

Print Child’s Name

Appendix H: Confirmation of Interview Email/Day-of Interview Reminder Email – Cog Lab Study

[Subject line when sent as a confirmation e-mail] RE: Confirmation of [STUDENT NAME]’s participation in Dynamic Assessment study

[Subject line when sent as a Day-of-Interview Reminder e-mail] RE: Reminder of [STUDENT NAME]’s participation in Dynamic Assessment study on [DATE] at [TIME]

Dear <NAME>:

Thank you for agreeing to participate in this mathematics study.

The interview will last about 90-minutes. We will begin the interview promptly at <TIME>, and it is important that you (picture ID required) and/or your child join the <Zoom or other videoconference platform> virtual interview 15-minutes early to check in.

This is a <friendly reminder OR confirmation> that your child has been scheduled to participate in a study at the following date/time:

DATE: <Day of Week, Month, Day, Year>

TIME: <XX:XX AM/PM, Time Zone of participant’s location>

GIFT CARD: Your child will receive a $40 virtual gift card to thank them for their participation.

LINK <Link to <Zoom or other videoconference platform> virtual meeting>

PLEASE COMPLETE THE FOLLOWING ITEM BEFORE THE RESEARCH INTERVIEW

Signed parent/guardian consent form (attached). You must scan or photograph it and email it back. Without a signed form, your child cannot participate in the interview and receive the $40 virtual gift card.

BRING THE FOLLOWING ITEMS TO THE RESEARCH INTERVIEW:

Parent/guardian valid picture ID—this is required for participation.
If your child uses eyeglasses for reading or watching TV, they should wear them during the study interview.

Thank you for volunteering in this very important research for NAEP. We look forward to seeing you. If you have any questions about the study, please contact <Education Testing Service CONTACT NAME> at <CONTACT INFORMATION>.

Sincerely,

<NAME>

Appendix I: Confirmation Email – Medium Scale Study

RE: Confirmation of [STUDENT NAME]’s participation in Dynamic Assessment study

Dear <NAME>:

Thank you for your interest in having your child to participate in this mathematics study.

The assessment will last about 45 minutes. Your child should plan to complete it in one sitting, and may do so any time between [DATE] and [DATE].

A parent/guardian consent form is attached to this e-mail. You must scan or photograph it and email it back. Without a signed form, your child cannot participate in the study and receive the $25 virtual gift card. Once we have received a signed consent form, we will send you a unique link that your child can use to participate.

If your child uses eyeglasses for reading or watching TV, they should wear them during the study.

Thank you for volunteering in this very important research for NAEP. We look forward to having your child participate. If you have any questions about the study, please contact <Education Testing Service CONTACT NAME> at <CONTACT INFORMATION>.

Sincerely,

<NAME>

Appendix J: Parent E-mail with Study Link – Medium Scale Study

RE: Link for [STUDENT NAME]’s participation in Dynamic Assessment study

Dear <NAME>:

Thank you for returning the consent form to have your child to participate in this mathematics study.

The assessment will last about 45 minutes. Your child should plan to complete it in one sitting, and may do so any time between [DATE] and [DATE].

The assessment can be accessed using the link below. This link is unique to your child, and should not be shared. Once your child signs into the assessment, they will be given instructions for completing the items.

If your child uses eyeglasses for reading or watching TV, they should wear them during the study.

Sincerely,

<NAME>

Appendix K: Thank You Email/Letter to Parent/Legal Guardian of Student Participants (Under Age 18)

[IF SENT AS EMAIL—Subject: Thank you—NAEP Research Study]

<DATE>

Dear <NAME>,

On behalf of the National Center for Education Statistics (NCES), which is part of the U.S. Department of Education, Educational Testing Service would like to thank you for allowing <CHILD’S NAME> to participate in the study for the National Assessment of Educational Progress (NAEP).

<CHILD’S NAME>’s participation helped us test out mathematics items and support materials currently under development. The assistance <he or she or they> provided, combined with input from other students, was incredibly useful and informative for improving the testing experience for students nationwide. We greatly appreciate you taking the time to help <CHILD’S NAME> log onto the [<Zoom or other videoconference platform> interview/assessment link] to participate. By now, you should have received an e-mail with a link to the gift card your child received for participating. If you have not received it, please contact [CONTACT NAME/E-MAIL].

For more information on NAEP, please visit the NCES NAEP website at: http://nces.ed.gov/nationsreportcard.

Again, thank you for your assistance.

Sincerely,

Educational Testing Service

Appendix L: Example Cognitive Lab Interview Tracking Schedule: Cognitive Interviews for Survey Questionnaires

An example tracking sheet to show how responses to screener questions (Appendix E) will be tracked in order to be used to determine the targeted sample, including diversification on key characteristics.

1 National Center for Education Statistics (NCES) is authorized to conduct NAEP by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622) and to collect students’ education records from education agencies or institutions for the purposes of evaluating federally supported education programs under the Family Educational Rights and Privacy Act (FERPA, 34 CFR §§ 99.31(a)(3)(iii) and 99.35). All of the information provided by participants may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). By law, every NCES employee as well as every NCES agent, such as contractors and NAEP coordinators, has taken an oath and is subject to a jail term of up to 5 years, a fine of $250,000, or both if he or she willfully discloses ANY identifiable information about participants. Electronic submission of participant’s information will be monitored for viruses, malware, and other threats by Federal employees and contractors in accordance with the Cybersecurity Enhancement Act of 2015. The collected information will be combined across respondents to produce statistical reports.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Stancavage, Fran
File Modified	0000-00-00
File Created	2021-01-13

National Assessment of Educational Progress (NAEP) Survey Assessments Innovation Lab (SAIL) Dynamic Assessments Cog Labs and Medium Scale Study

NCES System Clearance for Cognitive, Pilot, and Field Test Studies 2019-2022

Appendices NAEP SAIL Dynamic Assessments