National Assessment of Educational Progress (NAEP) Private School Focus Groups
Volume I
Supporting Statement
OMB #1850-0803 v.256
National Center for Education Statistics
Institute of Education Sciences
U.S. Department of Education
Washington, D.C.
October 2019
-------------------------------------
1. Submittal-Related Information 2
2. Background and Study Rationale 2
4. Consultations outside NCES 4
5. Assurances of Confidentiality 4
Appendix A: Private School Administrators and Teachers Recruitment Materials and Moderator’s Guides
Appendix B: Focus Group Materials (NAEP Statement and Private School Statements)
This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803), which allows NCES to conduct procedures to develop, test, and improve its data collection methodologies (e.g., cognitive interviews, focus groups, feasibility testing, etc.).
NAEP is a federally authorized survey by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622) of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, and civics. NAEP is conducted by NCES, within the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the different subject areas and collect survey questionnaire data from students, teachers, and principals to provide context for the reporting and interpretation of assessment results.
In order to report results for specific student groups, minimum response rates are required to ensure that the population representation of the sample is reliable. While response rates remain steady for public schools, NAEP faces an ongoing struggle of not being able to report data for private schools because minimum response rates are not met, especially among non-Catholic private schools, which include Conservative Christian, Lutheran, Non-Sectarian, Other Religious, and Unknown Affiliation schools. For example, non-Catholic private school response rates for the grade 4 mathematics and reading assessments dropped from 55 percent in 2011 to 40 percent in 2017, and from 58 percent in 2011 to 28 percent in 2017 for grade 8. NCES hopes to improve private school response rates for upcoming NAEP data collections by better understanding the factors that motivate participation and addressing the barriers to participation.
This request is to conduct four online focus groups to: (a) better understand private school administrators’ and teachers’ perceptions about the value of participating in NAEP and to identify both barriers and benefits associated with private school participation in NAEP; (b) identify informational needs and communication strategies that will help support recruitment and overcome barriers to private school participation; and (c) determine if and how messages should be customized for private schools. NCES aims to increase private school response rates for upcoming NAEP data collections, and the results from the focus groups will inform the recruitment and outreach strategies and materials tailored for private schools in NAEP 2021 and beyond.
On behalf of NCES, Hager Sharp will administer two online focus groups with private school administrators (i.e., heads of school, principals) and two online focus groups with private school teachers. Hager Sharp will utilize a screener (Appendix A) and work with an education-sector recruitment vendor, Fieldwork, to ensure that participants reflect diversity in geographic regions, urban/suburban/rural areas, private school affiliation (e.g., Catholic, Lutheran, Conservative Christian, Non-Sectarian), school level (e.g., elementary, secondary), and school size.
3.1 Research Questions
Key research questions to be explored in the focus groups include:
How do private school administrators (i.e., private school heads of school and principals) decide to participate in a voluntary survey or study, such as the National Assessment of Educational Progress (NAEP)?
What factors do they consider?
Whose opinions do they consider?
What value do private school administrators and teachers perceive NAEP has in education generally?
What value do private school administrators and teachers perceive NAEP has for private schools specifically?
What will motivate private school administrators to participate in NAEP? What will motivate private school teachers to support their school’s participation in NAEP? What benefits do private school administrators and teachers perceive come from participating in NAEP?
What are the main obstacles to private school participation in NAEP and what can we do to overcome them?
How does the value of the study and its contribution to the field of education factor into a private school administrator’s decision to participate?
What current or potential NAEP data do private school administrators and teachers find relevant, useful, and informative?
How can NAEP data be more user-friendly?
How does the format and duration of the study factor into a private school administrator’s decision to participate?
What messaging do private school administrators need to hear to motivate them to participate? What messaging do private school teachers need to hear to motivate them to support their school’s participation? What language and/or messages will help private school administrators and teachers understand the importance/value of NAEP and the importance/value of private school participation?
Are there specific endorsements that would influence private school administrators to participate in NAEP? Are there any endorsements that would make them less likely to respond?
After a more general guided discussion with focus group members about their perceptions of the benefits associated with participation in education studies and assessments, a collection of documents will be presented to participants to give an overview of the NAEP program and relevant participation details (e.g., testing time for students) and to elicit their feedback and suggestions. Those documents, as seen in Appendix B (NAEP and Private School Statements), contain text that has been pulled from various NAEP field publications, outreach materials, and online sources, including videos, fact sheets, brochures, recruitment letters, and web pages. The focus group moderator will read the NAEP statement to summarize the program, and the statement will be left on-screen for participants’ reference while asking for their perceptions of NAEP. The private school statements will be shared to elicit feedback on the credibility, relevance, and effectiveness of each statement.
Procedures
Under a contract to NCES, Hager Sharp developed screening questions for the recruitment of participants and will be responsible for the overall management of the focus groups and will work with an education-sector recruitment vendor, Fieldwork, to recruit participants for the online focus groups. For each online focus group, 10 participants will be recruited—a number that allows for the appropriate collection of information. Participants will reflect varying demographic characteristics, including geographic regions, urban/suburban/rural areas, private school affiliation (e.g., Catholic, Lutheran, Conservative Christian, Non-Sectarian), school level (e.g., elementary, secondary), and school size. Each focus group session will last approximately 90 minutes.
Focus group recruiters will contact by telephone people from their database and administer to them the recruitment screener (Appendices A-1 and A-2) to identify those who meet the eligibility criteria, namely that they are an administrator or teacher of a private school. To populate the database, Fieldwork reaches potential participants by sending out an email blast to individuals with specified demographics. The email includes a link to a programmed prescreener. Fieldwork phone screens those who respond to the link to qualify participants for the database. They use the same method of prescreening by posting the link to the prescreener in social media outreach. They tailor the social media outreach based on the demographics they want to reach, and then phone screen the people who respond to the link. They do not add anyone to their database without the individual’s permission. To recruit participants to the NAEP Private School focus groups, Fieldwork will decide who to call from their database based on criteria included in the recruitment screener (Appendices A-1 and A-2), filtering the potential participants using demographic characteristics that they have already gathered in the database.
The recruitment vendor will also distribute and gather tailored consent forms (Appendix A). Recruited participants for the online focus groups will be sent information in advance to enable them to dial in to the groups.
The recruitment procedure will ensure that:
All recruiting staff receive project-specific training before the start of any recruitment effort, including a discussion regarding the overall recruitment objectives, specific instructions on the critical importance of complete adherence to all NCES guidelines, protocols, and restrictions, and a section-by-section review of the screening instrument.
During intake, participants will be informed of the objectives, purpose, and participation requirements of the data collection effort, as well as the activities that it entails. The consent form (Appendix A) clearly identifies the objectives of the research and the agency conducting the research, and it also indicates that audio of the session will be recorded for analysis purposes only.
Along with identifying, contacting, and screening qualified research participants, ensuring their participation is equally as important. After screening participants, their contact information including telephone numbers and email addresses will be collected. After confirming that a participant is qualified, willing, and available to participate in the research project, they will receive a confirmation via email that includes the date, time, and dial-in information pertaining to their session.
Session Activities
Hager Sharp staff will administer all focus groups through an online platform, such as WebEx. After screening and securing 40 eligible participants for the 4 focus groups, they will schedule and administer the group sessions in November 2019. There will be a moderator and up to three Hager Sharp team members to observe and take notes on the participants’ comments and suggestions, which they will later compile into a summary report. Hager Sharp will also make an audio recording of the session.
After the focus group sessions, findings from the notes taken and from the audio recordings will be compiled and included in a comprehensive report that will describe the methodology, participants’ responses to the research questions outlined in Section 3.1, recommendations, and suggestions for next steps in order to inform potential revisions to NAEP outreach materials and strategies to more effectively recruit private schools for participation in the assessment. The audio recordings will be destroyed as soon as reports are finalized, and no personally identifiable information will be included in the reports.
Consultations outside NCES include a contractor with expertise in designing and administering focus groups, who will also take part in administering the study: Christina Nicols, MPH, MS, MS, Senior Vice President and Director of Planning, Research and Evaluation at Hager Sharp (1030 15th Street NW, Suite 600E, Washington, DC 20005).
All contractor staff working on the NAEP focus groups will sign the NCES Affidavit of Nondisclosure. Personal information of potential participants (e.g., name, address) will be collected for recruitment purposes only. On the data file used for analyses, respondents will be identified only by a unique study ID number assigned to each participant. Within 48 hours of respondents’ participation in the focus groups, the discussion notes will be edited, organized, and cleaned, and all identifiers will be stripped from the data set.
All computer files will be password-protected and hard copies will be locked in secure locations (e.g., data will be in locked file cabinets within locked offices). Only contract staff working directly on the data analysis portion of the project will have access to the data files. Once the report is finalized, all personally identifiable information will be destroyed. No individual names will be included in the reports.
The statement below will be presented in all written materials (e.g., letters, emails) and read at the start of the focus group sessions. Participants will also be informed that they can leave the discussion at any time.
NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). Your participation is voluntary and all of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).
Recruitment is estimated to take up to 10 minutes per private school teacher and private school administrator, and we anticipate needing to contact up to 60 private school teachers and 60 private school administrators to form the desired focus groups of a total of 20 diverse teachers and 20 diverse administrators. The focus groups will take approximately 90 minutes to conduct, and we have included an additional 30 minutes in the burden estimate for participants to dial into the groups. Table 1 provides the burden estimate for this study.
Table 1. Burden estimates for Private School Administrator and Teacher Focus Groups
Respondent group |
Number of respondents |
Number of responses |
Burden hours per participant |
Total burden hours |
Private School Administrators – Recruitment |
60 |
60 |
0.167 |
10 |
Private School Administrators – Focus Group Participation |
20* |
20 |
2 |
40 |
Private School Teachers – Recruitment |
60 |
60 |
0.167 |
10 |
Private School Teachers – Focus Group Participation |
20* |
20 |
2 |
40 |
Total Burden |
120 |
160 |
|
100 |
*Subset of initial contact group, not double counted in the total number of respondents.
Note: Total burden hours have been rounded to the nearest whole number
As in other NCES studies with school administrators and teachers, to encourage participation in the 90-minute session by busy private school administrators and teachers, who typically participate in NCES studies at low rates, we will offer each participant a $100 gift card to thank them for their time and effort. A monetary incentive is deemed necessary both to thank participants for their time and effort and to ensure that a diverse and reasonable cross section of participants who meet our screening requirements participate in the focus groups.
The estimated cost to prepare for, administer, and report the results of the NAEP focus groups is approximately $13,000. This cost includes salaried labor for contracting staff and other direct costs associated with conducting online focus groups with private school administrators and teachers.
The schedule of activities for the NAEP private school administrator and teacher focus groups is provided in Table 2.
Table 2. Schedule of activities for the NAEP private school administrator and teacher focus groups
Activity |
Tasks |
Date ranges |
Data collection |
Recruit participants |
October – November 2019 |
Online focus groups |
November 2019 |
|
Analysis |
Produce report |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |