NAEP Grade 4 Writing Computer-Based Assessment Usability Study, &The Assessment and Teaching of 21st Century Skills (ATC21S) Project Phase 2.2 Cognitive Interviews

Appendices A & B

• A. Phase 2.2 Cognitive Lab Recruitment and Consent Materials

  • Letter to Teacher – Recruiting Request

  • Text of Flyer

  • Parent/Guardian Permission form

  • Recruitment Procedures

• B. Phase 2.3 Pilot Test Recruitment and Consent Materials

  • Invitation to Teachers to Participate

  • Selected Teacher Consent Form

APPENDIX A: Phase 2.2 Cognitive Lab Recruiting Request to Teachers

Dear <Teacher’s Name>,

WestEd is asking your help with recruiting 6^th, 8^th, and/or 10^th grade students to participate in cognitive lab “think aloud” sessions for a project called Assessment and Teaching of 21^st Century Skills (ATC21S for short). The goal of the project is to develop computer-based assessments to measure critical 21st century skills such as team building, literacy in information and communication technology, and collaborative problem solving. WestEd, a non-profit educational research, development and service agency (www.wested.org) under contract with the National Center for Education Statistics (NCES) of the U.S. Department of Education, is implementing and managing the project in the United States. Other partner countries working on the ATC21S project are Australia, United Kingdom, Finland, Portugal, and Singapore.

The formative assessment has been developed by a team of developers from the University of California in Berkeley. Findings from the cognitive lab “think aloud” sessions will be used to make necessary modifications to the assessment.

The help we need from you involves:

• Announcing the cognitive lab “think aloud” opportunity to your students, and distributing the enclosed flyers to each student,

• Handing out the Parent Permission letter to students who express interest in participating, and instructing students to return signed Parent Permission letters to you until WestEd can retrieve them from you,

• Screening students who volunteer for basic computer and internet skills,

• Working with WestEd to schedule a two-hour cognitive lab session held on an after school time and date convenient to participating students, and arranging the use of a classroom with Internet connectivity to set up the cognitive lab. The WestEd team will bring the laptops and equipment.

• Allowing WestEd research team to run the cognitive lab session with necessary permissions from your school.

The WestEd team will disclose to participating students that the session will be audio taped and notes will be taken, both in the Parent Consent form, and before the session begins. Each of your student’s participation is voluntary and their responses may be used only for statistical purposes, and may not be disclosed or used in identifiable form for any other purpose except as required by law (20 U.S. Code, Sec. 9573).

For more information on the project, please contact Michael Timms, Ph.D. of WestEd at (510) 302-4214. If interested in helping with recruiting and setting up a

cognitive lab session, please contact Anna Araneta at aaranet@wested.org. If you cannot help with this request, we would appreciate suggestions of other teachers who teach your grade level, who might help.

Sincerely,

Michael Timms, Ph.D.

WestEd Science, Technology, Engineering & Mathematics

Attachments:

Recruitment Flyer for students & parents/guardians

Parent/Guardian Permission letter

APPENDIX A: Phase 2.2 Cognitive Lab Recruitment Flyer Text

Technology Project! Share Your Thoughts and Earn a Gift Card!

What is the project about?

The National Center for Education Statistics (NCES) of the U.S. Department of Education is involved in a project to develop computer based assessments that test how middle and high school students build teams, use technology in communicating and collaborating to solve problems. Under contract with the NCES, WestEd, a non-profit education development, research, and service organization is conducting “think aloud” sessions in your school on:

Date: ___________________

Time: ___________________

Classroom: ___________________

Students will complete computer-based tasks organized in ‘scenarios’ in the “think aloud” sessions. The purpose of the “think aloud” session is to track students’ perceptions and interaction with the tasks in order to fine-tune and improve the task items. The session will last approximately one and a half hours. A WestEd team will be on hand to help students get started and to ask questions.

Who can participate?

• Students in the 6^th, 8^th, & 10^th grades

• Students with experience using computers and navigating the Internet

What’s in it for students?

• A chance to explore the technology and skills you will need in higher education and in future careers

• A $10 amazon.com giftcard as a thank you for your participation

What do you have to do?

• Ask your parent’s permission and have them sign the Parent/Guardian permission letter from your teacher

• Submit the signed Parent/Guardian permission letter to your teacher

• Attend the scheduled session at your school posted above

• Complete the tasks in the time allotted.

Who to contact:

All information you provide may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, Section 9573). For questions about the project, please contact Mike Timms, PhD at mtimms@wested.org or call (510) 302-4214. If you are interested in participating in this project, let your teacher know.

APPENDIX A: Phase 2.2 Cognitive Lab Parent/Guardian Permission

Dear Parent/Guardian,

Your child is invited to participate voluntarily in a cognitive lab “think aloud” session conducted by WestEd, a not-for-profit educational research and service organization under contract with the National Center for Education Statistics (NCES), part of the U.S. Department of Education. The session is part of a project called Assessment and Teaching of 21^st Century Skills. The project is developing assessments designed to measure the application of “21^st Century Skills” such as teamwork, digital literacy, and collaborative problem solving in middle and high school students. The project is being conducted in six countries: Australia, United Kingdom, Finland, Portugal, Singapore, and the United States.

About the cognitive lab “think aloud” sessions

A “think aloud” is a process in which students are asked to complete the computer-based assessment(s), talking out their thoughts as they go, and answering questions the observing researcher asks them. The purpose of this “think aloud” session is to gather students’ insights and perceptions that could improve the computer-based assessment. Information gathered from the “think aloud” sessions will be used to make necessary modifications to the assessments. Conditions for the “think aloud” session are as follows.

• “Think aloud” sessions will take place at the student’s school.

• The sessions will take approximately two hours of after-school time.

• “Think aloud” sessions will be audio taped and attending WestEd team will take notes.

• Participants will use a unique ID and/or screen names that are not their real names.

• Participants will receive a gift card with a $10 value as thanks for their participation upon completion of the “think aloud” session. Parents who adjust their schedules to pick up their child after the session will receive a gift card with a $15 value.

Audio taping and documentation of the “think aloud” process

To capture participant feedback the session will be audio taped, enabled via a laptop computer or a digital recorder. Only the team working on the project will hear the audiotapes and no student names will be attached to the recordings. Notes will also be taken during the session as part of the documentation. The responses your child gives will be documented in a summary report of all participant responses on the same grade level, which will not identify your child individually. His/her responses will be used only for statistical purposes and will not be disclosed, or used in identifiable form for any other purpose except as required by law (20 U.S. Code, Section 9573).

Phase 2.2 Cognitive Lab Parent/Guardian Permission (cont.)

Permission

I hereby give my permission for (student’s name) ____________________________________in

grade ___________ to participate in the cognitive lab “think aloud” session scheduled in his/her school on

(date)_______________________________ (time) ____________________________.

______________________________________________________________ _______________________

Parent/Guardian name (PLEASE PRINT) Phone Number

______________________________________________________________ _______________________

Parent/Guardian Signature Date

I accept the invitation to participate in the Assessment and Teaching of 21^st Century Skills “think aloud” session, and I will return this signed consent to my teacher as proof of my interest. I understand that my participation is completely voluntary.

______________________________________________________________ _______________________

Student’s Signature Date

______________________________________________________________

Teacher who gave you this form

APPENDIX A: Phase 2.2. Recruiting Procedures

Cognitive Lab “Think Aloud” Recruiting Process

Contacting Teachers

The nine teachers who participated in a previous phase of the project in which they reviewed the early concepts for assessment items to validate that they were grade-level appropriate will be approached to invite their students to participate in the “Think Aloud” session. Having previewed early versions of the ICT assessment task concepts, the nine teachers are open to having their students participate. If some of these teachers cannot help recruit students, other teachers in the same schools will be approached to help.

Advertisement

To aid teachers in making an announcement about the cognitive lab “think aloud” sessions to their classes, a flyer (see attached) will be distributed to students. When a student expresses interest in participating, the teacher will give the student a parent/guardian consent form with instructions to return the signed consent form as soon as possible. The flyer will have information tailored for each scheduled session, grade level, and school.

The contractor will collect all signed parent/guardian consent forms.

Student Selection

With the help of teachers, cognitive lab participants will be selected based on the following:

• Students in the 6^th, 8^th, or 10^th grade,

• Students with basic computer experience and experience navigating the internet,

• Students with interest and time to make scheduled cognitive lab session

• Students who provide a signed parent/guardian consent form.

Recruitment Numbers

With the help of 9 teachers who recruit 4 students from grade levels they teach, the target number of recruited participants is 36 students:

• 12 students from 6^th grade

• 12 students from 8^th grade

• 12 students from 10th grade

Schedule

After-school hours and dates recommended by the teachers will be posted on the flyer.

Appendix B: Invitation to Teachers to Participate in the Pilot Test

Letter will be sent as an email:

Subject line: NCES of the US Dept. of Education needs your help!

Dear <Name of Teacher>,

The National Center for Educational Statistics or NCES, partnered with other countries, is conducting a project called “Assessment and Teaching of 21^st Century Skills” or ATC21S for short. At this time, formative assessments developed for the project hone in on literacy in information and communication technology, and collaborative problem solving skills.

WestEd, under contract with the National Center for Educational Statistics - is looking for 6^th, 8^th, & 10^th grade teachers who teach subjects in Mathematics, Science, Technology, or Language Arts to participate in a Pilot Test. The objective of the pilot is to test the feasibility of administering complex web-based assessments in a school setting, and test the data capture and measurement methods involved in interpreting the complex responses that these types of assessment will generate.

Only three (3) teachers: one each from 6^th, 8^th, and 10^th grades will be selected to pilot the assessments in their classes. Participants will receive $200 as thanks for their participation. Selection will be based on whether participants can:

• Attend a two hour online training to learn how to administer the assessment

• Schedule the piloting of assessments for 2 class periods (45 to 60 mins. each)

• Reserve a computer lab or classroom with one computer for each student

• Make sure the ATC21S assessment sites can be accessed through the school’s network; and work with the Technical Administrator to insure technical requirements for the pilot session are met

• Distribute and collect an “Opt-out” letter to Parent’s/Legal guardian prior to the scheduled pilot session. WestEd will provide this letter.

Please indicate your interest in participating by sending email to Anna Araneta aaranet@WestEd.org with the information below or fax form to 510.302.4242.

Teacher Name	School
Home / Mailing Address	School Address
City, State, Zip	School City, State, Zip
Home Phone	Work Phone
E-mail	Name of School Network Administrator

Appendix B: Phase 2.3 Pilot Test Recruitment selection letter & consent form

Dear <Teacher’s Name>,

You were selected to participate in a Pilot Test for an international project called “Assessment and Teaching of 21^st Century Skills”. Led in the United States by the National Center for Educations Statistics (NCES) of the U.S. Department of Education, the project seeks to ‘pilot test’ assessments that measure critical 21^st century skills such as literacy in information and communication technology and collaborative problem solving. WestEd, a non-profit educational research, development and service agency (www.wested.org) implements the project for NCES.

Participation in the Pilot Test is voluntary, and data collected during the Pilot Test will be used only for statistical purposes and will not be disclosed, or used in identifiable form for any other purpose except as required by law (20 U.S. Code, Section 9573).

A stipend of $200 will be paid to each participating teacher as thanks for their completion of Pilot Test activities.

Please indicate your willingness to participate by completing and sending in the attached consent form.

If you are unable to participate at this time, we would appreciate your referring other eligible teachers to us. For more information, contact Mike Timms at mtimms@wested.org, or Anna Araneta at aaranet@wested.org.

Sincerely,

Mike Timms, Ph.D.

WestEd Science, Technology, Engineering & Mathematics

Encl: Pilot Test Teacher consent form

Phase 2.3 Pilot Test Teacher Consent Form

By signing this consent form you agree to the following and consent to participate voluntarily:

• I have read the invitation to participate in the ATC21S Pilot Test and understand what is asked of me as a volunteer.

• I consent to participate in the training on how to administer the assessment items.

• I consent to insuring that each student has a computer that can access the assessment site during the Pilot Test session. I will get the help of the school network administrator to insure stable connectivity, if necessary.

• I consent to administering the pilot assessments to two class periods of my assigned students.

• I will distribute, collect and submit to WestEd the “Opt-out” forms returned from parents/guardians who do not want their child’s data used in this project.

• I understand that all information I or students provide may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, Section 9573).

• I understand I will receive $200 for completing all activities in the Pilot Test.

______________________________________________________ ______________

Teacher Signature Date

_________________________________________________________________________________________________________

I teach Grade Primary Subject

Please return signed consent form to: Anna Araneta, c/o WestEd, 300 Lakeside Drive, 25th Floor, Oakland, CA 94612 or by FAX at 510.302.4242

Appendix B: Phase 2.3 Pilot Test Parent/Legal Guardian “Opt-out” Letter

Dear Parent/Legal Guardian

<Teacher’s Name> who teaches your child’s <fill in grade> class is participating in a Pilot Test that involves administering a new computer based assessment to your child’s class on <1^st date> and <2^nd date>. The pilot test is part of a project called “Assessment and Teaching of 21^st Century Skills” or ATC21S for short, conducted by the National Center for Education Statistics (NCES), part of the U.S. Department of Education. On behalf of the NCES, WestEd, a research and service non-profit organization is implementing the project in schools.

Purpose. Critical skills such as literacy in information and communication technology and collaborative problem solving, are the focus of the assessments that were developed. The Pilot Test will allow assessment developers to modify the pilot version of the assessments before a final version is tested in more schools.

Confidentiality. Your student’s name will not be associated with the assessment results. No identifying information will be recorded, and information will be stored securely. All information may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, Section 9573). WestEd researchers will collect data to inform the development of the assessment; they are not evaluating any student’s performance.

Participation is voluntary. Participation in the project is voluntary. If you do not wish your student to participate, their data will not be included in the data sent to the assessment developers. They will still participate in the activity as part of their regular classroom instruction.

Please check the NO box if you do NOT want your child’s data included in the pilot assessment, sign and return to your child’s teacher no later than <deadline set by teacher>

 NO, I do not want my child’s data included in the ATC21S Pilot Test

__________________________________________________________________________

Child’s Name

__________________________________________________________________________

Parent/Guardian’s Name

__________________________________________________________________________ ______________

Parent Signature Date

ATC21S 2