Cognitive Lab Summary

Appendix D. Summary of Key Findings from the Cognitive Interviews

During August and early September of 2013, we conducted 31 cognitive interviews about the Adult Training and Education Survey, and 23 about the After-School Programs and Activities Survey. Below is a summary of the key findings from this cognitive interview research.

Adult Training and Education Survey

Confusing skip patterns

Problem: Both the cognitive interviews and analysis of the NATES I pilot showed that some respondents were missing skip instructions while others skipped when they should not.

Response: We changed the ordering of response options to match the NHES:2012 where, whenever possible, the skip instruction flows off the first response option and an arrow directs non-skipping respondents to the next question. We also revised the formatting of instructions to make them consistent and distinguishable from question text. Lastly, we reordered several questions to reduce the amount of skips and to favor errors of commission over errors of omission.

Grids and Tables

Problem: The size and amount of text in the grids were difficult for respondents. Also, the number of things people have to look up in a table may be too much.

Response: We removed the grid for certificates and in the certification and training grids, we reduced the number of certifications and trainings we ask about. We simplified the text in the grids, deleted some grid questions, and reversed the column/row order. We shortened the table-look up categories and reformatted the layout so it was easier to reference the table.

Main certificate item

Problem: The question was too complex and wordy; use of “vocational” and “course of study” in particular were confusing some respondents.

Response: We simplified the text and it now tests at a 12^th grade reading level.

Noncredit section and job training

Problem: Other courses and job training sections are overlapping in respondents’ minds and they become confused as to how they differ from what they had answered about previously.

Response: The number of different types of training and education asked about are numerous. It is likely that reducing the number of types of training asked about will alleviate this problem. We will test a long and short version of the questionnaire in the split panel experiment. The short version of the questionnaire will contain only attainment items. The long form will contain all sections.

Overall survey burden

Problem: Respondents were skimming or skipping instructions at beginning of survey or within items.

Response: Changes made to simplify question wording and skip instructions, as well as format changes are expected to alleviate this problem.

Problem: Number of words used in item stems and/or response options is too high.

Response: We simplified wording in all questions where it was possible. For example, rather than repeat “this certification or license” in each question, we repeated it in the first follow-up question or two, and then referred to the credential as “it” or “this” in the remainder of the section.

Problem: The term “most recent” was confusing. Some respondents interpreted “most recent” to mean “recently”, e.g., in the last 6 months

Response: We changed the wording to refer to the “last” credential earned.

After-School Programs and Activities Survey

Public school choice question

Problem: Some parents were confused by the phrase “regularly assigned school or a school that you chose.” For example, a parent was confused because her son had special needs and went to a county program and was not sure how to respond. Other parents thought that this referred to the school that the child went to on a regular basis.

Response: We will test two different question wordings in the split panel experiment.

Questions asking about self-care and parent care

Problem: The wording of these questions was confusing and cognitively difficult for some parents.

Response: We revised the language and added an additional question for the skip pattern. We will examine the missing data rates and distribution on these items after the feasibility test.

Important factors in choosing care lists

Problem: Several parents had problems with these questions. Some parents were confused about what the categories should include. Some parents provided answers based on what they would look for in a program even though their child was not currently enrolled in a program and other parents made response errors. For example, one parent responded hypothetically because her child did not participate in any after-school programs; one parent referred to four years ago rather than the past school year; and another parent did not include sports as after-school activities, although it should have been.

Response: We added a skip pattern so that parents whose children who do not participate in programs or organized activities after school do not answer these questions.

Who looks after child after school list

Problem: Several parents had trouble with the time period reference about what “after school” meant. In the first round of cognitive interviews no end-time restriction was provided in questions that asked about after-school activities. This resulted in several different reporting periods that parents provided that were not consistent. Some would say until bedtime while others would say until about 6 or 7pm. In the second round of cognitive interviews an end-time was provided, from after school until 6pm. This resulted in parents providing feedback that 6pm may be too restrictive, since several activities can happen after 6pm until bedtime. This was evident with children in the later grades (7th and 8th). In addition to problems with the time reference period, parents found it difficult to answer about children’s activities during weekends because “they are so different” and “unstructured.”

We also found that some respondents would write in the "Other, specify" category that the child spent time at a relative’s house rather than selecting "At another home." Additionally, we found that some respondents were unsure about where to include time spent with step or foster parents.

Response: We added a time-frame (from the end of the school day until 8pm) to the questions and removed weekends. We now ask about weekend care separately. We changed “At another home” to “At another home, including a relative or care provider’s home.” We added the parenthetical (birth, adoptive, step, or foster) to the Mother and Father categories.

After school activities list

Problem: There was confusion over response categories stemming from the long list of items. Parents tried to fit the activity that their child participated in into the first available category and then had to change answers when they read the correct category lower down on the list. It was unclear to some parents where they should include that their child “listens to music” after school. There was also confusion over the reference period of “after-school.” Additionally, respondents expressed list fatigue.

There was some confusion about what should be included in the "playing indoors" response option. Some respondent were uncertain of what would be included. A respondent asked if that includes playing Xbox. Playing Xbox is included as part of response option that comes later in the list.

Response: We reordered the lists to put the most common items first and added a reference period for after-school (from the end of the school day until 8pm). We consulted with the TRP on where to put “listen to music” and TRP members recommended against including this with music lessons and did not make a recommendation to add a new category. We will examine missing data rates and response distributions from the feasibility study to evaluate fatigue. We changed the wording to “Playing indoors, like playing board games and playing with toys inside.”

Charge or fee for programs questions

Problem: Respondents typically had issues responding if there was no cost for the program or activity. There was no skip pattern that allowed them to skip out.

Response: We added a skip pattern so that skips respondents who do not have any out-of-pocket costs for the program or activities do not answer this question.

Household composition questions

Problem: One parent who completed the question wondered whether she should count her son who is in college. She decided to include him since he is still counted for taxes and comes home regularly. Several parents were unclear about how the sampled child was counted. Several other parents responded to the question with their household’s total number of members ignoring the text in the parenthesis that indicated the sum should match the total of the prior question.

Response: We added instructions to exclude the sampled child from the counts.

Additional issues

Problem: Some parents were confused by the phrase “last week (or the most recent typical week)” in the activities questions and thought that the questions were asking about two separate time frames. Additionally, some parents reported that some activities only take place for part of the year so it was challenging to figure out how to respond for a typical week.

Response: We removed the parenthetical text from the question stem and added it to the italic instruction text after the question

3