2016 National Household Education Surveys Program (NHES) Parent and Family Involvement Survey (PFI) and Early Childhood Program Participation Survey (ECPP) Cognitive Interviews

Attachment 1c NHES 2016 PFI & ECPP Cognitive Interviews Protocol

Interview ID:		Date of Interview:
Parent First Name:		Interviewer
Age of Child:

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is suggested content with which the interviewer should be thoroughly familiar in advance. Text in brackets [ ] presents instructions for the interviewer.

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the parent]:

You are here today to help us out with a new survey that we are working on for the U.S. Department of Education. This survey asks about your child’s schooling [and finding and choosing child care for your child for parents with children 5 and younger]. I will ask you to take the survey as if you were taking a real survey. There are no right or wrong answers, and it’s ok if you do not know some of the answers. As you are answering the questions I will ask you about your answers to the questions. This will help us make the survey better and clearer for other parents just like you.

Because the information you provide is so important, I am going to be taking notes while you are answering the survey and while we are talking [INSERT IF THERE IS AN OBSERVER: and my colleague will also be observing our session today].

Also, this interview will be recorded so researchers can review the recording later. Any information that refers to you or your family personally, like your name or name of your child’s school, will not be included in our report, and your responses will be used for research purposes only.

Remember, you are not being tested; we want you to tell us what you think about the survey.

If at any time you want to stop, just let me know.

[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions about what I just said? [Answer any questions the parent may ask.]

PROVIDE RESPONDENT WITH A COPY OF THE INFORMED CONSENT FORM. ASK THE RESPONDENT TO READ THE FORM (OR READ IT TO HIM/HER, DEPENDING ON RESPONDENT PREFERENCE), ANSWER ANY QUESTIONS, AND HAVE THE RESPONDENT SIGN THE FORM. LEAVE A SEPARATE COPY OF THE FORM WITH THE RESPONDENT.

IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.

STEP 2: THINK-ALOUD PRACTICE

While you fill out the survey I want you to talk about how you are answering the questions. Hearing you talk about how you figure out your answers to the questions will help me understand how to make the questions better. I am more interested in how you arrive at your answers than in the answers you give me. To help you get started on thinking aloud, we are going to do a practice activity.

I’m going to give you a piece of paper with a question in it. I’m going to ask you to read out loud the question and think aloud as you decide on your answer.

Give respondent handout with think aloud practice.

How many windows are there in the house or apartment where you live?

Please tell me what you are thinking while you are figuring out your answer.

Probe as needed for detail: How did you arrive at that answer? What were you visualizing when you were deciding on that answer? Can you tell me more about what you are including in your count?

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about your answers to the questions in the survey.

HAND RESPONDENT A COPY OF THE QUESTIONNAIRE AND A PEN OR PENCIL.

STEP 3: COMPLETION OF THE QUESTIONNAIRE

Now I would like you to complete the questionnaire. I’d like you to think aloud while you decide on your answer. Please read aloud anything you would have read to yourself if I were not here.

Remember that I am very interested in hearing you talk about what you are thinking as you figure out your answers to the questions.

REMEMBER TO TELL THE PARENT THROUGHOUT THE QUESTIONNAIRE THAT THEY ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO YOUR QUESTIONS ARE REALLY HELPFUL.

Useful Probes

Can you tell me in your own words what that question was asking?

What does the word [term] in this question mean to you?

How easy or difficult is it to remember [topic]?

How easy or hard was it to choose an answer?

Tell me what you are thinking?

How did you come up with that answer?

You said [answer]. Can you tell me more about that?

[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?

[If you pick up on a visual cue like a furrowed brow or a puzzled look]: “Tell me what you are thinking.”

[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old question before answering this question]: “Tell me what just happened. Tell me about what you were just doing.”

Screener
ASK THE RESPONDENT TO FILL OUT THE ENTIRE SCREENER. REMIND THEM TO THINK ALOUD IF THEY ARE NOT DOING SO.
Q3. What is the person’s month and year of birth? [If the respondent is having difficulty with the month and/or year] How did you come up with your answer?
Q5. Is this person currently in…? How did you come up with your answer? DO NOT PROBE ON WHAT RESPONDENT THINKS “HOMESCHOOL” MEANS SINCE IT MIGHT INFLUENCE THEIR RESPONSES ON HOMESCHOOL ITEMS IN THE SURVEY.
Child’s Schooling
ASK QUESTIONS AND USE PROBES ONLY AS NECESSARY
Instructions: It is not necessary to probe on the instructions. Note if the respondent has any problems if they read them.
Q1-Q2: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q3. Is it his/her regularly assigned school? [Ask only if responded to this question] How did you decide on your answer? What does “regularly assigned school” mean to you? SHOW THE RESPONDENT THE ALTERNATIVE ITEM ON ANOTHER PIECE OF PAPER. Here is another related question we are thinking about asking. How would you answer this question? Is it his/her district-assigned school? What does “district assigned” mean to you?
Q4-Q13: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q14. Is he/she currently enrolled in any high school Advanced Placement (AP) classes? What were you thinking of when you answered this question? What do “Advanced Placement (AP) classes” mean to you? What grade-level are the students who usually take AP classes in your child’s school district?
Q15-Q21: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q22. Some students take school-related courses for credit over the Internet. Is this child receiving any instruction this way? How did you decide on your answer?
Q23. Is that instruction [internet] provided by any of the following places?   How did you decide on your answer? [If chooses a response option or “Someplace else:”] Tell me more about that. How does your child access the classes? Where is he/she when accessing the classes? Who provides the class?
Q24: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q25. What is the main reason this child took school-related courses over the internet?    How did you come up with your answer? Can you tell me more about why your child took the course(s) over the internet? How did you find out about the course(s)? Does your child get credit for the course(s)?
Q26. Some parents decide to educate their children at home rather than send them to a public or private school. Is this child being schooled at home instead of at school for at least some classes or subjects?   In your own words, tell me what this question is asking about. How did you determine whether to answer yes or no? [If respond “yes,”] Do not probe on the meaning of homeschool until Q30. [If respond “no” and if not evident from think aloud to previous questions] What does “homeschool” mean to you? Make a note if the respondent did not skip properly, but do not probe on skip pattern.]
Q27. Which of the following statements best describes your homeschooling arrangement for this child?   How did you come up with this answer? [Do not probe on the meaning of homeschool until Q30. Check to see if respondent skipped properly, but do not probe on skip pattern.]
Q28. How many hours each week does this child usually go to a public or private school for instruction? Do not include time spent in extracurricular activities. How did you come up with your answer? [Check to see if respondent filled in this question although they indicated that the child is homeschooled.]
Q29. There are many different reasons that parents choose to homeschool their children. Did your family choose to homeschool this child because:   How did you determine how to answer yes or no to each of these questions [a-j]? Tell me more what you were thinking when answering these questions. [Do not probe on the meaning of homeschool until Q30.]
Q30. Of the reasons your family chose to homeschool this child, which one would you say is the most important to you? How did you come up with your answer? [If not evident from think aloud to previous questions] What does “homeschool” mean to you?
Q31. Thinking about all years this child has been homeschooled, which of the following subject areas has this child been taught during his or her home instruction? How did you decide what to mark? How do you cover these subjects during home instruction? Is there anything missing?
Child’s Health
Q32-Q35: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q36. Are these services provided by any of the following sources? How did you decide on your answer?
Q37. Are any of these services provided through an Individualized Education Plan (IEP) or services plan? How did you come up with your answer? What does a “services plan” mean to you?
Child’s Background
Q38-Q43: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q44. Does this child live at this address and another address (for example, because of a joint custody arrangement)? How did you come up with your answer? In your own words, what is this question asking? [If answered No] What does joint custody arrangement mean to you?
Q45. If yes, does this child spend most of the time at this address, at another address or equal time at both addresses? How did you come up with your answer? [If needed] Can you describe the arrangement your child has?
Q46. What language does this child speak most at home? It is not necessary to probe on this question. Note the respondent’s response, especially for those who are parents of ELL students. Record in your notes whether the respondent reads the response options below the top two responses.
Your Household
Q48. How many of the following people live in this household with this child?   Tell me more about what you were thinking when you answered this question? Who are you including in the counts? In your own words, what is this question asking?
Q49. Enter the total number of people living in this household with this child (This number should be equal to the sum of a through k above).   How did you come up with your answer? Who are you including in the counts?
Child’s Family
Q59. What is the highest grade or level of school that this parent or guardian completed? [Only if respondent chooses “high school diploma or equivalent”, “vocational diploma after high school”, or “some college, but no degree”:] How did you come up with your answer? What does “vocational diploma after high school” mean to you?
Q60. Is he or she currently attending or enrolled in a school, college, university, or adult learning center, or receiving vocational education or job training? In your own words, what is this question asking?

THE REMAINING QUESTIONS ARE FOR PARENTS WHO ALSO HAVE A CHILD UNDER AGE-5, NOT YET IN KINDERGARTEN, AND WHO IS IN A CARE ARRANGEMENT.

Tell the respondent the following:

The remaining questions are about your other child who is under age 5, not yet in kindergarten and in a care arrangement. How old is this child? What type of care arrangement does this child have? [Record this in your notes.]

Please continue and answer the remaining questions. Please remember to think aloud.

Finding and Choosing Care for your Child
Q61-64: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q65. What was the primary reason for the difficulty finding care? Please tell me more about how you chose your answer. Can you tell me more about that?
Q66. How important was each of these reasons when you chose the child care arrangement or program where this child spends the most time? 66h. Ratings on a website? How did you decide what to mark? In your own words, what is 57h asking? Could you provide some examples? 66i. Recommendations from friends and family? How did you decide what to mark? In your own words, what is 57i asking? 66j. Accreditation of the care provider (e.g. from the National Association for the Education of Young Children or the National Early Childhood Program Accreditation)? How did you decide what to mark? In your own words, what is 57j asking? 66k. The religious orientation of the program? How did you decide what to mark? In your own words, what is 57k asking? 66l. Your city or state’s Quality Rating and Improvement System (QRIS)?   How did you decide what to mark? In your own words, what is 57l asking? What does QRIS mean to you? [If applicable:] How did you learn about the QRIS? How did you use it? Do you remember what it is called for your state or area—that is, which QRIS did you use?
Q67. Which of the following best describes this program’s curriculum? How did you come up with your answer? [If respondent chose Montessori:] Tell me what Montessori means to you. [If respondent chose Creative Curriculum]…and what about Creative Curriculum?

FINAL QUESTION

Do you have anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention?

STEP 4. THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making this survey better.

Interviewer Instructions: Collect materials and escort parent out of the interviewing room and to the lobby. Please give the respondent incentive envelope [$40 gift card], and thank them for their time and effort.

9