Justification

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB# 1850-0803), which provides for NCES to conduct various procedures (e.g., focus groups, cognitive interviews, usability testing) to test new methodologies, question types, or delivery methods to improve survey and assessment instruments and study procedures, in order to improve the resulting data quality, utility, and study participant experience. This request is to test new questionnaire content through focus groups and cognitive interviews.

Background and Study Rationale

The National Assessment of Educational Progress (NAEP) is a federally authorized survey (the National Assessment of Educational Progress Authorization Act; 20 U.S.C. §9622) of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is conducted by NCES, which is part of the Institute of Education Sciences, within the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the different subject areas and collect survey questionnaire (i.e., non-cognitive) data from students, teachers, and principals to provide context for the reporting and interpretation of assessment results.

Socioeconomic status (SES) is a legislatively mandated reporting category in NAEP, and questions related to SES have been included in all past NAEP survey questionnaires. The current NAEP domain general (Core) Student Survey Questionnaires include five items used to collect information about household composition, parental education, and parental employment status.¹ The household composition item asks students to indicate whether people such as their mother or father live in their household, and the two education and two employment items specifically ask about the mother and father of the student. This approach stems from the well-established approach of evaluating student socioeconomic status through examination of parental income, education, and occupation, with adjustments made for the number of adults and children in the household.²

In NAEP, the items concerning parental education and employment assume students live in a nuclear family: a mother, a father, and children. These items are important for assessing the socioeconomic status of a student’s family, but the current approach may not be adequate for capturing this information from students who live in non-traditional households or do not live with their biological parents but with other caregivers. The five items administered to assess these SES factors should lead students to think about the adults with whom they live, regardless of whether or not they are the student’s biological parents.

We designed a two-phase study to develop new items that will capture household and caregiver information from students living in a broad range of households. In the first phase, focus groups will be conducted with students to develop a better understanding of how students talk and think about their living situations, caregivers, and household structures. Based on the results of the focus groups, draft survey items will be developed that reflect these living situations. In the second phase of the study, cognitive interviews will be conducted to assess whether these draft items adequately capture information about the students’ households, caregiver employment status, and caregiver occupation. The draft items will be modified based on the cognitive testing for subsequent pilot testing.

The focus group method is a qualitative, group approach to data collection.³ It involves holding group sessions guided by a moderator, who follows an outline containing questions or topics focused on a particular issue. The questions are purposefully open-ended in nature, incorporating several prompts to encourage a deeper, multifaceted exploration of an issue or topic. As a research tool, focus groups are useful for understanding a targeted group’s views on complex social issues, such as non-traditional household composition and the selected use of kinship terms. Transcripts of the audio-recorded sessions provide the researcher with rich narrative for each discussion, allowing the analysts to make both comparisons within each focus group as well as across the focus groups on the issues under discussion.

In cognitive interviews (often referred to as a cognitive laboratory study or cog lab), an interviewer uses a structured protocol in a one-on-one interview drawing on methods from cognitive science.⁴ The objective is to explore how participants think and what reasoning processes they use to work through tasks. A verbal probing technique will be used for this cognitive interview activity. With verbal probing techniques, the interviewer asks questions, as necessary, to explore additional issues that have been identified a priori as being of particular interest. This interview technique has proven to be productive in previous NAEP pretesting studies and will be the primary approach in the NAEP cognitive interviews described in this package.

Cognitive interview studies produce largely qualitative data in the form of verbalizations made by participants in response to the interviewer probes. Some informal observations of behavior are also gathered by a second observer who will be present in addition to the interviewer. Behavioral observations may include things such as nonverbal indicators of affect (i.e., emotional states such as frustration or engagement) and interactions with the task (e.g., ineffectual or repeated actions suggesting misunderstanding or usability issues). Cognitive interviews are important given that they help to identify potential problems with items. Such interviews can also inform item improvement.

The primary purposes of the two-phase study include:

Identify best practices for developing items concerning household composition and caregiver information for NAEP Student Survey Questionnaires (e.g., related to using grade-appropriate language, or representing content in a way that allows student participants in various household types to identify with the item/response choices).
Identify the people that should be mentioned in the NAEP household composition item in order to ensure the item is inclusive of students living in a broad range of household types.
Identify the additional questions about household composition that should be included in the NAEP core questionnaire to improve measurement of household composition and the resources available to a student’s household.
Identify the most informative questions that can be answered by students at grades 4, 8, and 12 about caregiver employment and/or occupation.

Volume I of this submittal contains descriptions of the design and sampling, as well as burden, cost, and schedule information for both the focus groups and cognitive testing phases of the study. Volume II contains the protocols for the focus groups and cognitive labs. The appendices contain notifications, consent/assent forms, screening checklists, informational flyers, phone scripts, and thank you documents.

Sampling and Recruitment Plans

ETS is the survey questionnaire developer for NAEP survey questionnaires and will be responsible for the overall management of the focus group and cognitive interview activities described in this package. Westat, subcontractor to ETS, will be responsible for recruitment and administering the focus groups and cognitive interviews (see Section 5).

Various resources will be employed to recruit student participants⁵ for both the focus groups and cognitive interviews, including:

existing participant databases from local focus group facilities;
targeted telephone and mail contact lists (i.e., lists that consist of individuals meeting basic criteria such as age or school grade);
school system research/assessment directors;
community organizations (e.g., Boys & Girls clubs, Parent-Teacher Associations, and limited on-site location-based and mass media recruiting); and
outreach/contact methods and resources (e.g., internet ads, flyers, canvassing, and having representatives available to talk to parents/legal guardians, educators, and community members at appropriate local community events, school fairs, etc.).

Each parents/legal guardian who responds to the outreach efforts or who is contacted by a facility recruiter will be given a broad description of the study (see Appendices A-B) and asked if he or she is interested in having his/her child participate. Screening procedures will then take place to see if the child is eligible to participate based on the child’s age, household composition, and other demographic factors. The subcontractor will document the information collected in the screeners using a tracking sheet which will be used to determine the targeted sample including diversification on key characteristics (see Appendix R for example tracking sheet). All individual demographic information collected will be used only for eligibility verification and an aggregated summary description of participants’ characteristics, and will be securely destroyed upon the completion of a report from this study. For children who are eligible to participate in a focus group, the parent/legal guardian will be given detailed information about the time and location of the group. For students who are eligible to participate in a cognitive interview, the recruiter will work with the parent/legal guardian to find a convenient time to conduct the interview. The parent’s or legal guardian’s contact information will also be obtained so that follow-up information can be mailed/emailed to him or her. Parents/legal guardians will be notified at the time of recruitment that the focus group and cognitive interview sessions will be audio-recorded.

To minimize the travel burden of students and parents/legal guardians, focus groups and cognitive interviews will be conducted in nearby venues that are convenient for the participants, such as local focus group facilities, community centers, or facilities of community-based organizations.

Focus Groups

The first phase of the study will include 12 focus groups, one per grade with a group of students in each of the following four household types:

Type 1: Students living with adults other than their biological parents (such as a grandparent or foster parent);
Type 2: Students living in extended family households with adults besides their parent(s);
Type 3: Students living with a single parent (and possibly additional adults); and
Type 4: Students dividing their time between two or more homes (e.g., of their divorced parents).⁶

Because of the marked differences in the ages and social development of the three student cohorts of interest, focus groups will also be divided by age of the participants. In addition, focus groups with the 8^th and 12^th graders will be divided by gender with the goal of fostering more open discussion about possibly sensitive issues.⁷ The locations for the focus groups will represent diverse geographic areas and suburban, urban, and rural areas, for example:

A large city of 2,000,000 people or more located in the East region of the U.S.
A mid-size city of approximately 600,000 people in the West region.
A small city of approximately 300,000 people in the Midwest region.
A small town/rural area of 25,000 people or less in the South.

Urban and suburban respondents will be recruited in the first three locations, and rural respondents in the fourth location, as shown in Table 1.

Table 1. Overview of Focus Group Locations and Student Household Types

Location	Grade 4 Students*	Grade 8 Students	Grade 12 Students
Site 1 – Large City	In two or more homes	Single parent [Boys only]	Adults other than biological parents [Girls only]
Site 2 – Mid-Size City	Extended family households	In two or more homes [Girls only]	Single parent [Girls only]
Site 3 – Small City	Adults other than biological parents	Extended family households [Girls only]	In two or more homes [Boys only]
Site 4 – Small Town/Rural Location	Single parent	Adults other than biological parents [Boys only]	Extended family households [Boys only]

* In the unlikely event that grade 4 students do not perform well in the focus group format (as evidenced by interviewer observations of negative student body language or other indicators of student discomfort), grade 4 students could participate in this phase of the study in one-on-one interviews instead of focus groups.

Westat’s recruitment process is outlined below:

Local recruiters from focus group facilities will send an email of introduction about the focus group research to (a) parents/legal guardians from their existing databases and (b) local community centers (Appendices A-D). The email of introduction will include an information brochure (Appendix E).
After receiving a contact of interest, a recruiter from the local facility will follow up with the parent/legal guardian (or student if age 18 or older) via phone (Appendices F-G) and ask them to provide demographic information to ensure that a diverse sample is selected as per the aforementioned criteria.
If the parent/legal guardian allows his/her child to participate, or if a student age 18 or older agrees to participate, the local facility will follow up to confirm participation and the date and time of the focus group discussion (Appendices K-L).
Parents/legal guardians (on behalf of students under 18) and students age 18 or older will be required to sign informed consent forms prior to the focus group. Students under 18 will be asked to provide verbal assent after reviewing the assent information brochure (Appendices H-J).

Cognitive Interviews

Westat will conduct 16 cognitive interviews each with 4^th, 8^th, and 12^th graders, for a total of 48 interviews. A diverse sample will be achieved by recruiting a mix of gender, race/ethnicity, urban/suburban/rural, and household type. No more than two students will be recruited from any school. Table 2 summarizes the numbers and types of cognitive interviews that are planned for these pretesting activities. A minimum of four respondents per household type per grade is recommended to identify major problems with an item. This also allows for a meaningful analysis of data to be conducted from exploratory cognitive interviews to test the usability of prototype questions. We will conduct cognitive interviews in two urban locations. These provide some variation by region, and approximately half of the interviews in each grade will be done in each locale. Westat’s recruitment process for the cognitive interviews will mimic the procedures described for the focus groups.

Table 2. Sample Size for Cognitive Interviews

Household Type	Grade 4 Students	Grade 8 Students	Grade 12 Students
Extended family households	4	4	4
Adults other than biological parents	4	4	4
Single parent	4	4	4
In two or more homes	4	4	4

Data Collection Process

Focus Groups

Each focus group will include 6-10 students and last a maximum of 60 minutes. This includes time for instructions and student introductions (maximum 10 minutes), conducting the discussion (40 minutes), and debriefing and/or time for additional questions/feedback from the participants (maximum 10 minutes). All focus groups will be moderated by a two-person team comprised of, at a minimum, a senior qualitative methodologist and a mid-level researcher. The focus groups with the 4^th graders will be smaller to allow each student more time to consider and respond to the moderator’s questions.

Prior to beginning the focus group, consent for participation and audio recording of the session will be obtained, and all study procedures will be reviewed with the parent or legal guardian who has brought their student under 18-years of age to the discussion. If 12^th graders under 18 have brought themselves to the interview, they must bring a signed parental consent form in order to participate. Student assent procedures will be reviewed with the participants to assure them that their participation is voluntary and that their responses will be used for research purposes only. Student verbal assent will be obtained prior to participation and prior to beginning audio recording of the focus group session. If audio recording is declined, the student will not be able to participate in the discussion.

The interview team will begin the session by asking each student to draw a pictorial representation of his/her place (or places) of residence, a symbol for and the first name of each person who lives in the residence(s), and the term the student uses to describe each person.⁸ A discussion will follow, in which the students will be asked to describe their living situation, identify the “most important person or persons” with whom they live, and describe the terms they use to refer to their parents or legal guardians. The moderators will encourage participants to talk about similarities and differences in terminology in order to understand the most common kinship terms for these important adult relationships in each of the specified living arrangements.

Participants will then be asked to describe how much they know about their important adults’ educational backgrounds, that is, their highest level of educational attainment relative to the categories currently being used in the NAEP survey (e.g., mother/father did not finish high school; graduated from high school, had some education after high school, graduated from college, I don’t know). The moderator will explore how much detail students in the different grade levels can offer about their parents’ or legal guardians’ work lives. The current NAEP survey asks simply if the “mother” and “father” are working or not, but it remains unclear how much information students may have about step-parents, aunts or uncles, and other household figures. Moreover, because an individual’s occupation is a reasonable proxy for socio-economic status, we will explore the most accurate, least burdensome ways to hone in on such occupational details.

After participating in the focus group, each student will receive an incentive (see Section 9) and be sent a thank you letter/email (Appendices O-P).

Focus Group Analysis Plan

Both the social diagrams (including the kinship terms) and transcripts from the discussion groups will provide the source data for analysis. Westat senior qualitative analysts will first review the transcript and drawings within each group (e.g., multi-generational households), looking for common terminology and key patterns of influence within each type of household configuration. For example, what are the characteristics of multi-generational households in which a grandparent or great aunt or uncle is identified by the student as being most influential? What contributes to a stepmother being more influential than the birth mother with whom the student also lives part-time? Once patterns are identified within the responses from each focus group, analysts will then examine the data for any patterns across the focus groups that might exist, especially by grade level.

The final study report will include a detailed discussion of the study methodology (including recruitment efforts, student characteristics, protocol development, and analytic approach), key findings, and conclusions and recommendations for next steps. In addition, based on the focus group discussions, draft questions will be developed to be tested in the cognitive interview phase of the project.

Cognitive Interviews

The cognitive testing sessions will last a maximum of 60 minutes with each student. This includes time for introductions (maximum 10 minutes), conducting the interview (40 minutes), and debriefing and/or time for additional questions/feedback from the participants (maximum 10 minutes). All student cognitive interviews will be conducted in person.

Using similar procedures to the focus groups, prior to beginning the interview, consent for participation and audio recording of the session will be obtained, and all study procedures will be reviewed with the parent or legal guardian who has brought their student under 18-years of age to the discussion. If 12^th graders under 18 have brought themselves to the interview, they must bring a signed parental consent form in order to participate. Student assent procedures will be reviewed with the participants to assure them that their participation is voluntary and that their responses will be used for research purposes only. Student verbal assent will be obtained prior to participation and prior to beginning audio recording of the interview. If audio recording is declined, the interviewer and observer will be prepared to take notes during the interview.

Participants will be asked to think about, answer, and reflect on the items developed in the focus group phase of the project. The interviewer will be tasked with keeping participants engaged by asking probe questions, soliciting responses from less talkative participants, and asking follow-up questions where appropriate (e.g., “That’s interesting, could you tell me a little bit more about that?”). Interviewers may also take additional notes during the in-person cognitive interviews, including notes on behavior (e.g., the participant’s facial expressions indicate they are confused) and whether extra time was needed to answer certain questions.

After participating in the cognitive interview, students will receive their incentive (see Section 9) and be sent a thank you letter/email (Appendices O-P).

Cognitive Interview Analysis Plan

The notes and audio recordings will provide the source data for analysis. Notes and audio recordings will be reviewed by grade, household composition, and across all participants to identify consistent themes and any notable participant reactions or responses. For example, do participants understand what the items are asking? How confident are t that they can provide the information being asked in the questions? Are they able to accurately and comprehensively describe their household by responding to these items? Are there any topics included in the items that make participants uncomfortable, or any questions participants do not want to answer? These observations will serve as the basis for revisions to the items administered during cognitive interviews.

A final study report will include a summary of study methodology (including recruitment efforts, student characteristics, protocol development, and analytic approach), key findings, illustrative statements, recommended revisions to the items administered in the cognitive interviews, and recommendations for next steps.

Consultations Outside the Agency

ETS serves as the Item Development contractor to NCES on the NAEP project, developing cognitive and survey items for NAEP assessments. As such, ETS will be responsible for the management of all activities described in this package.

Westat, will offer facilities, tools, and staff (senior qualitative methodologists and mid-level researchers) to collect and analyze both qualitative and quantitative data. Westat will recruit participants, moderate the focus groups, and administer the cognitive interviews.

Assurance of Confidentiality

The study will not retain any personally identifiable information. Prior to the start of the study, students will be notified that their participation is voluntary. As part of the study, students will be notified that the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Written consent will be obtained from parents or legal guardians of students under the age 18, or from students who are age 18 or older at the time of participation. Verbal assent will be obtained from all participating youth under the age of 18. Participants will be assigned a unique identifier (ID) at the time of recruitment, which will be created solely for data file management and used to keep all participant materials together. The participant ID will be separated from the participant’s name before the report from each stage of the study is finalized. The signed consent form, which includes the participant name, will be separated from the participant interview files. The focus groups and cognitive interviews will be audio recorded. The only identification included on the audio files will be the group ID (for focus groups) or the participant ID (for cognitive interviews). All consent forms, recordings, and individual records/notes will be secured for the duration of the study and will be destroyed after the final report is completed.

Justification for Sensitive Questions

Effort has been made wherever possible to avoid asking for information that might be considered sensitive or offensive. Given the nature of the study, some questions about sensitive topics will need to be asked, such as about the nature of relationships between household members, the number of people in the household, or how students allocate their time across multiple households. In the unlikely event that a student exhibits distress during participation, the distress protocol will be implemented and students will be provided with a list of resources at the end of the interview, should they have a need for any counseling as a result of the topic (see Volume II, Section 4). Reviewers have attempted to identify and minimize potential bias in questions.

Estimate of Hourly Burden

The estimated burden for recruitment assumes attrition throughout the process (attrition of approximately 33% from initial contact to follow-up and 50% from follow-up to confirmation is projected). All focus groups and cognitive interviews will be scheduled for no more than 60 minutes. Table 3 details the estimated burden for each phase of the study.

Table 3. Estimated Burden

Respondent	Number of respondents	Number of responses	Hours per respondent	Total hours
Parent or Legal Guardian for Student Recruitment (Focus Groups)
Initial contact	300	300	0.05	15
Follow-up via phone	200*	200	0.15	30
Consent & confirmation	100*	100	0.15	15
Participation (Focus Groups)
Grade 4 Students	28	28	1	28
Grade 8 Students	36	36	1	36
Grade 12 Students	36	36	1	36
Total Burden Focus Groups	400	700		160
Parent or Legal Guardian for Student Recruitment (Cognitive Interviews)
Initial contact	144	144	0.05	7
Follow-up via phone	96*	96	0.15	14
Consent & confirmation	48*	48	0.15	7
Participation (Cognitive Interviews)
Students	48	48	1	48
Total Burden Cognitive Interviews	192	336		76
Total Burden Estimate	592	1,036	-	236

*Subset of initial contact group

Note: Numbers have been rounded and therefore may affect totals

Recruitment Costs

To encourage participation and thank students for their time and effort, a $30 gift card from a major credit card company will be offered to each participating student. If a parent or legal guardian brings his/her child to and from the testing site, he/she will also receive a $30 gift card to thank him/her for the time and effort in transporting the child. Additionally, the parent or legal guardian will receive a thank you letter for allowing the student to participate in the study. Students 18 years of age or older will also receive a thank you letter for their participation.

This amount is higher than the customary $25 incentive used in many of the NAEP pretesting studies in order to improve effectiveness of recruiting students living in the four nontraditional household types that are the focus in this study. Unlike in previous NAEP survey questionnaire studies, where most students in grade 4, 8, and 12 qualify for participation, only a small percentage of students in these grades live with adults other than their parents (3.8%) or with a single parent (26.9%)⁹. Given that much fewer students will qualify to participate, we expect that a higher level of incentive will be needed to attract sufficient number and diversity of participants needed for this study, and to encourage the parents or legal guardians of students that do qualify to allow them to participate.

Costs to Federal Government

The estimated cost for the survey questionnaire focus group and cognitive activities in this submittal is $311,600 as delineated in Table 4.

Table 4. Estimate of Costs

Activity	Provider	Estimated Cost
Focus group and cognitive interviews development, project oversight, and review of all study materials and data	ETS	$93,313
Focus group and cognitive interviews design, preparation, administration (including recruitment, incentive costs, data collection, and documentation), and data analysis and reporting	Westat	$218,287
Total		$311,600

Schedule

Table 5 depicts the start and end dates of the focus group and cognitive testing phases.

Table 5. High-Level Schedule of Milestones

Activity	Dates
Recruitment and administration of focus groups	August to September 2017
Recruitment and administration of cognitive interviews	October to December 2017
Submission of study and item recommendations report to NCES	January 2018

1 Note: questions capturing parental education and employment status are included at grades 8 and 12, but not in the 2017 grade 4 student questionnaires because findings from the 2016 NAEP pilot indicated that fourth graders could not provide valid information on these questions.

2 As described in the white paper “Plans for NAEP Core Contextual Variables,” prepared for the National Assessment Governing Board (included in Tab06, Appendix A in: https://www.nagb.org/content/nagb/assets/documents/what-we-do/quarterly-board-meeting-materials/2014-5/may-2014-board.zip).

3 Roller, M.R. and Lavrakas, P.J. (2015) Applied Qualitative Research Design: A Total Quality Framework Approach. London: Guilford Press.

4 Blair, J. and Presser, S. (1993). Survey procedures from conducting cognitive interviews to pretest questionnaires: A review of theory and practice. In JSM proceedings, Survey Research Methods Section. Alexandria, VA: American Statistical Association. 370-375.

5 For students under age 18, parents/legal guardians will receive the various contact information.

6 Students that live in more than one home must spend at least 33 percent of their time living in each residence.

7 Note: A study design with equal numbers of focus groups among each sex is not necessary either for this subject matter or for the methodology. For both the 8^th and 12^th graders, we plan to hold two focus groups with boys and two with girls (for a total of 4 groups in each grade level) to encourage full and open discussion among the participants.

8 Sociograms (i.e., social diagrams), graph representations of a child’s social network, are often used to identify points of conflict (and potential resolution) between a student and his/her peers (Leung, BP and J Silberling (2006), Using Sociograms to identify Social Status in the Classroom. Journal of Contemporary School Psychology 11(57). Ferrandiz, I-M and BC Jimenez (2011) The Sociogram: The Analysis of Interpersonal Relationships in Higher Education. Journal of International Education Research. 7(5):9-14). Our approach relies upon literature showing that even young students have the skills to create a graphic representation of their social worlds (e.g., Philip, DN (2010) Social Network Analysis to Examine Interaction Patterns in Knowledge-Building Communities. Canadian Journal of Learning and Technology. 36(1)); we will simply restrict each participant’s network to those people with whom the student lives some or all of the time.

9 Data source: Forum on Child and Family Statistics (2016). Table FAM1.A: Family structure and children’s living arrangements: Percentage of children ages 0–17 by presence of parents in household and race and Hispanic origin. America’s children in brief: Key indicators of well-being (2016 ed.). Retrieved from https://www.childstats.gov/americaschildren/tables/fam1a.asp.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Debby E. Almonte
File Modified	0000-00-00
File Created	2021-01-22

Justification

Vol 1 NAEP SES Indicator Items Development Studies 2017 v2.docx

NCES Cognitive, Pilot, and Field Test Studies System

Justification

OMB: 1850-0803

Table of Contents

Submittal-Related Information

Background and Study Rationale

Sampling and Recruitment Plans

Data Collection Process

Consultations Outside the Agency

Assurance of Confidentiality

Justification for Sensitive Questions

Estimate of Hourly Burden

Recruitment Costs

Costs to Federal Government

Schedule