Volume 1 NAEP Reading and Math Technology Based Assessment Survey Focus Groups 2014

National Center for Education Statistics

National Assessment of Educational Progress

Volume I

Supporting Statement

Focus Groups for Transitioning Reading and Math Survey Questionnaires to Technology Based Assessments

OMB# 1850-0803 v.96

February 7, 2014

Contents

1) Submittal-Related Information 2

2) Background and Study Rationale 2

3) Sampling and Recruitment Plans 3

4) Data Collection Process 4

5) Consultations Outside the Agency 5

6) Assurance of Confidentiality 5

7) Justification for Sensitive Questions 5

8) Estimate of Hourly Burden 6

9) Recruitment Costs 6

10) Costs to Federal Government 7

11) Schedule 7

1. Submittal-Related Information

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB# 1850-0803), which allows for NCES to conduct various procedures (such as focus groups) to test new methodologies, question types, or delivery methods to improve survey instruments and procedures.

2. Background and Study Rationale

The National Assessment of Educational Progress (NAEP) is a federally authorized survey of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is administered by NCES, part of the Institute for Education Sciences, in the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the various subject areas and to also collect survey questionnaire (i.e., non-cognitive) data from students, teachers, and school administrators to provide context for the reporting and interpretation of assessment results.

Over the next few years, NAEP will fully transition from paper-and-pencil assessments to technology-based assessments (TBAs). The current schedule for NAEP mathematics and reading assessments is to be operationally administered as TBAs in 2017 and piloted in 2016. Understanding the role of technology in reading and mathematics is crucial for a successful transition of the NAEP assessments to TBAs in order to capture the most relevant contextual factors.

To help inform the item development process for the survey questionnaires, data collection activities such as focus groups and cognitive interviews are planned prior to the 2016 pilot and 2017 operational assessments. Focus group research is especially important given unknown contextual factors associated with technology-based assessments. The main purpose of the proposed focus group research is to better understand the following constructs in the context of reading and mathematics:

• Familiarity with technology, especially as it relates to mathematics and reading (e.g., e-readers, tablets, software, apps, and Internet resources);

• Access to technology relevant for learning at home and at school, including patterns of access across student groups;

• Ways technology (especially tablets, smart phones, and Internet resources) change students’ experiences with text, numbers, and calculations;

• How the use of technology facilitates instruction and learning, both in and outside of the classroom; and,

• Shifts in the nature and type of skills required of students as they read and/or do math in technology-rich environments.

The focus group method is a qualitative, group approach to data collection. It involves holding group sessions guided by a moderator, who follows a topical outline containing questions or topics focused on a particular issue. The questions are purposefully open-ended in nature, incorporating several prompts to encourage a deeper, multifaceted exploration of an issue or topic. The method is useful for going beneath the surface of a response. As a research tool, focus groups are useful for understanding a targeted group’s views on complex social issues such as challenges and issues surrounding cultural issues and education.

The results from this study will be used to set priorities for new item development to ensure that the 2017 survey questionnaires capture the most relevant contextual factors. Following the focus groups, draft items will be written and undergo cognitive interview testing, prior to being pilot tested in 2016. The versions of the items for both the cognitive interviews and the pilot test will be submitted to OMB for review prior to the commencement of these activities.

Volume I of this submittal contains descriptions of the design and sampling, as well as burden, cost, and schedule information for the study. Volume II contains the protocols for the various focus groups. The appendices contain notifications, consent forms, screening checklists, informational flyers, phone scripts, and thank you documents.

3. Sampling and Recruitment Plans

Focus groups with students will be conducted face-to-face, and focus groups with teachers and school administrators will be web-based. CRP, a consulting firm experienced in conducting focus groups has been selected by NCES to conduct this study (See Section 5). To ensure that appropriate and relevant information is obtained, 80 respondents will participate across the respondent types and grades, as shown in Table 1.

Table 1. Sample Size for Focus Groups

Respondent Group	Grade 4	Grade 8	Total
Students (Primarily Reading Questions, Some Additional Math Questions)	8	8	16
Students (Primarily Math Questions, Some Additional Reading Questions)	8	8	16
Students Sub-total	16	16	32
Mathematics teachers	8	8	16
Reading teachers	8	8	16
Teachers Sub-total	16	16	32
School administrators	8	8	16
School Administrators Sub-total	8	8	16
Overall Total	40	40	80

Within these categories, CRP will recruit a mix of participants with respect to school type (e.g., public, private), community type (e.g., urban, rural), gender, race/ethnicity, familiarity with technology, and years of teaching/administrating experience to ensure a diverse sample.

Plan for Recruitment of Students

CRP will recruit students from the Washington, D.C. area. Students will participate in an in-person focus group at a location in Alexandria, VA, in order to equalize the travel distance for both prospective D.C. “urban students” and “rural students” in counties surrounding the focus group site.

First, CRP will send an email of introduction about the focus group research to various elementary and middle school principals in the D.C. area (see Appendix A) with the student flyers, (representative flyers in Appendices B and C), student demographic form (Appendix D), and parent/guardian consent form (Appendix I) attached. CRP will then follow up with the principal or a school administrator at each school via phone (transcript in Appendix E) to confirm their willingness to allow CRP to recruit students in their school. Principals that agree to participate will be asked to disseminate the flyers along with the consent form to students to bring to their parents/guardians. Principals, or their designee, will also be asked to select a subset of 4 to 6 students who volunteer to participate. The principal will be asked to complete the demographic form (Appendix D) for each student participant to ensure that the 4 to 6 students selected are a diverse sample as per the aforementioned criteria. Principals will then be asked to send the completed demographic forms to CRP via email.

Using the contact information provided by the principals, parents will first be sent a letter or email of introduction (Appendices F), and then a CRP staff member will follow up with the parent via phone (Appendix G). CRP will confirm participation, and the date and time of the focus group session (Appendix H). Parents/guardians will be required to sign informed consent forms for student participation prior to the focus group session (Appendix I).

Plan for Recruitment of Teachers and Administrators

Eight states will be randomly selected by CRP to participate. The largest population center or school districts (urban/suburban) and a random rural population area will be chosen from information derived from state websites. Teacher and school administrator recruitment will be augmented by utilizing national organizations’ databases of administrators and faculty.

Teachers and administrators will first be sent a letter or email of introduction and the Teacher/School Administrator consent form (Appendices J, K, and O), then a CRP staff member will follow up with the teachers and administrators via phone (Appendices L and M) and ask participants to provide demographic information to ensure a diverse sample is selected as per the aforementioned criteria. CRP will confirm participation, and the date and time of the focus group session (Appendix N). All participants will be required to sign informed consent forms for teacher/administrator participation prior to the focus group session (Appendix O).

4. Data Collection Process

Prior to the study, CRP will contact cooperating teachers, schools, and parents of participating students to confirm participation and make logistical arrangements. CRP staff experienced with focus groups will conduct the study. On the day of the focus group, participants will be asked to participate in focus group discussions (see Volume II). The focus groups are scheduled for 90 minutes. Participants will first be welcomed, introduced to the interviewer and the observer (if an in-room observer is present), and told that they are there to help answer questions about mathematics and reading learning and the role of technology in learning/instruction. Participants will be reassured that their participation is voluntary and that their responses will be used for research purposes only (see Section 6). As part of the introduction process, the moderator will explain to participants that their responses will be audio recorded. The moderator will ask each participant to speak clearly and distinctly so that each response will be captured by the recording. For the web-based teacher and administrator focus groups, the moderator will explain the technology and describe the tools the participants may use, such as muting their phone and asking questions.

The moderator from CRP will be tasked with keeping participants engaged by asking probe questions (see Volume II), soliciting responses from less talkative participants, and asking follow-up questions directed to the group (e.g., “what do you all think about… [repeat focus group probe]”, or “that’s interesting, could you tell me a little bit more about that”). Interviewers may also take additional notes during the student focus groups, including behaviors (e.g., the participant’s facial expressions indicated they are confused) and if extra time was needed to answer certain questions.

Analysis Plan

After the session, the notes and audio recording will be summarized to report themes and illustrative statements that will be analyzed by the NAEP questionnaire development team. The focus group results will be used to set priorities for new item development to ensure that the 2017 survey questionnaires capture the most relevant contextual factors.

5. Consultations Outside the Agency

Educational Testing Service (ETS)

ETS serves as the Item Development (ID) contractor on the NAEP project, developing cognitive and survey items for NAEP assessments. As such, ETS will be responsible for the management of all activities described in this package.

CRP, Incorporated (CRP)

CRP is a minority- and woman-owned professional services and management consulting firm located in Silver Spring, Maryland that provides technical consulting primarily in the fields of education and behavioral health. CRP is a small business and serves as a subcontractor to ETS on the survey item development work.

6. Assurance of Confidentiality

Participants will be notified that their participation is voluntary and that their answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (20 U.S.C. §9573)].

Written consent will be obtained from participants and from parents or legal guardians of students. Teachers and school administrators will be asked to complete and return the consent form to CRP via email, fax, or mail before the web-based focus group session. Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files and secured for the duration of the study and will be destroyed after the final report is completed.

The interviews will be recorded. The only identification included on the files will be the participant ID. The recorded files will be secured for the duration of the study and will be destroyed after the final report is submitted.

7. Justification for Sensitive Questions

Throughout the item, task, and interview protocols development process, effort has been made to avoid asking for information that might be considered sensitive or offensive. Reviewers have attempted to identify and minimize potential bias in questions.

8. Estimate of Hourly Burden

The estimated burden for recruitment assumes attrition throughout the process.¹ All focus groups will be scheduled for no more than 90 minutes. Table 2 details the estimated burden for the survey questionnaire focus groups.

Table 2. Estimated Burden for Recruitment of Focus Group Participants

Respondent	Hours per respondent	Number of respondents	Total hours
Principal or School Administrator for Student Recruitment
Initial contact	0.05	36	2
Follow-up & Identify students	1.0	24*	24
Sub-Total		36	26
Parent or Legal Guardian for Student Recruitment
Initial contact	0.05	96	5
Follow-up via phone	0.15	64*	10
Consent & Confirmation	0.15	32*	5
Sub-Total		96	20
Teacher and School Administrator Recruitment
Initial contact	0.05	144 (96+48)	7
Follow-up via phone or e-mail	0.15	96*(64+32)	14
Consent & Confirmation	0.15	48*(32+16)	7
Sub-Total		144	28
Participation (Focus Groups)
Grade 4 Students	1.5	16	24
Grade 8 Students	1.5	16	24
Teachers	1.5	32*	48
School Administrators	1.5	16*	24
Sub-Total		32	120
Total Burden	620 responses	308	194

*Subset of initial contact group

9. Recruitment Costs

To encourage participation and thank for their time and effort, a $25 VISA gift card will be offered to each participating student, plus a $25 VISA gift card to a parent or legal guardian bringing the student to and from the testing site, along with a thank you letter (see Appendix P) for allowing the child to participate in the study. Teacher and school administrator focus groups participants will be mailed a $40 VISA gift card and a thank you email (see Appendix Q) for taking part in the study.

10. Costs to Federal Government

The estimated cost for the survey questionnaire focus group activities in this submittal is $557,477 as delineated in Table 3.

Table 3. Estimate of Costs

Activity	Provider	Estimated Cost
Design, preparation, and analysis for mathematics and reading survey questionnaire focus groups	ETS	$ 77,000
Preparation and conduct of mathematics and reading survey questionnaire focus groups (including recruitment, incentive costs, data collection, analysis, and reporting)	CRP	$ 480,477

11. Schedule

Table 4 depicts the high-level schedule for the reading and mathematics focus group activities. The commencement of activities is contingent upon OMB approval.

Table 4. High-Level Schedule of Milestones

Dates	Activity
March–April 2014	Recruitment process
April-June 2014	Data Collection, Analysis, and Reporting

1 Assumptions for approximate attrition rates for direct participant recruitment are 33 percent from initial contact to follow-up, 50 percent from follow-up to confirmation. Note: The initial principal contact for student identification attrition rate is 33 percent from contact to confirmation.