Volume I:
Request for Clearance for the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011) Spring Fourth-Grade Cognitive Interviews and Timing Tests
OMB# 1850-0803 v.95
The Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011) is a survey that focuses on children’s early school experiences beginning with kindergarten and continuing through the fifth grade. It includes the collection of data from parents, teachers, school administrators, and nonparental care providers, as well as direct child assessments. Like its sister study, the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K),1 the ECLS-K:2011 is exceptionally broad in its scope and coverage of child development, early learning, and school progress, drawing together information from multiple sources to provide rich data about the population of children who were kindergartners in the 2010-11 school year. The ECLS-K:2011 is sponsored by the National Center for Education Statistics (NCES) within the Institute of Education Sciences (IES) of the U.S. Department of Education. The Educational Testing Service (ETS) serves as the subcontractor developing the study’s child assessments.
Fall and spring data collections in the study children’s kindergarten, first-, second-grade years were conducted for NCES by Westat. In the latest clearance package for the study, the spring 2014 third-grade data collection, recruitment for the fourth-grade (spring 2015) data collection, and tracking for the fifth-grade (spring 2016) data collection were approved in December 2013 (OMB No. 1850-0750 v.15).
Clearances for studying the ECLS-K:2011 cohort were granted for the fall 2009 field test data collection, fall 2010 and spring 2011 kindergarten national data collections, fall 2011 and spring 2012 first-grade national data collections, fall 2012 and spring 2013 second-grade national data collections, and the spring 2014 third-grade national data collection (OMB No. 1850-0750). Several generic clearance requests for testing various components of the study have also been approved (OMB 1850-0803).
The ECLS-K:2011 is the third in a series of longitudinal studies of young children sponsored by the U.S. Department of Education that examine child development, school readiness, and early school experiences. Like its predecessors, the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K) and the Early Childhood Longitudinal Study, Birth Cohort (ECLS-B), the ECLS-K:2011 will provide a rich and comprehensive source of information on children’s early learning and development, transitions into kindergarten and beyond, and progress through school for a cohort of young children. Coming more than a decade after the inception of the ECLS-K, the ECLS-K:2011 will allow cross-cohort comparisons of two nationally representative kindergarten classes experiencing different policy, educational, and demographic environments. Also, the longitudinal nature of the ECLS‑K:2011 will enable researchers to study children’s cognitive and physical growth and socioemotional status, as well as relate trajectories of growth and change to variation in home and school experiences in the early grades. The ECLS-K:2011 data can be used by policymakers, educators, and researchers to consider the ways in which children are educated in our nation’s schools and to develop new effective approaches to education.
This is a request for clearance to conduct cognitive interviews and timing tests to inform the instruments planned for the ECLS-K:2011 spring fourth-grade data collection. Specifically, the goals of these interviews are 1) to identify and correct problems with ambiguity in or misunderstanding of new and revised items for the spring fourth-grade teacher questionnaires and parent interview2 and 2) assess the length and burden of the draft teacher questionnaires and parent interview through timing tests.
NCES has contracted Westat to conduct the spring fourth-grade data collection for the ECLS-K:2011, including these cognitive interviews and timing test activities.
Cognitive interviews (described below) will be conducted via telephone to evaluate new and revised items in several of the draft ECLS-K:2011 spring fourth-grade instruments, specifically:
Subject-specific (i.e., reading, math, or science) child-level teacher questionnaires (included in appendix A). In the national fourth-grade data collection, these questionnaires will be completed by teachers who teach reading, math, and/or science to at least one sampled child. The questions that will be cognitively tested are indicated on the cover sheet of the appendix.
Parent interview (included in appendix B). In the national fourth-grade data collection, this instrument will be administered primarily via telephone3 to the parents of the sampled children. The questions that will be cognitively tested are indicated on the cover sheet of the appendix.
Most items from the teacher questionnaires and parent interview have been fielded over multiple rounds of the ECLS-K and ECLS-K:2011 and, therefore, will not be tested in these cognitive interviews. The new items being tested in the cognitive interviews have been included in the questionnaire to address issues that were recently identified by the ECLS-K:2011 Technical Review Panel (TRP) as important areas for research. These items were developed using feedback from the TRP and experience from the ECLS-K fifth-grade and ECLS K:2011 third-grade instruments.
During the cognitive tests, participating teachers and parents will be asked to discuss their interpretation of selected items, to discuss the process they would use to obtain an accurate response to the selected items, and to provide suggestions for item and response category wording for any unclear items or items that are difficult to answer. (Appendix C includes the cognitive interview protocols for teachers and parents.) In addition, teacher respondents will be asked to briefly review the full questionnaire to understand the general context in which the tested questions are asked.
Telephone interviews (described below) will also be conducted to determine the overall timing of each draft study instrument as well as the timing of different sections of several of the spring fourth-grade instruments, specifically the:
Teacher background questionnaire (included in appendix D). In the national fourth-grade data collection, this instrument will be completed by all teachers of sampled children.
Subject-specific teacher questionnaires.
Parent interview.4
Teacher and parent respondents who have not participated in the cognitive interviews will be asked to participate in the timing tests. Teacher respondents will be asked to fill out the entire teacher background questionnaire and one subject-specific child-level questionnaire and to note both the time taken to complete each questionnaire section and the total time to complete the questionnaire. Parent timings will be conducted by telephone by a trained interviewer who will note the time taken to administer each section of the interview, as well as the total interview time. Both teacher and parent respondents in the timing tests will then be asked via telephone a few follow-up questions about their experience. (Appendix C includes the timing interview protocols for teachers and parents.)
The request to conduct the national spring fourth-grade data collection will begin the clearance process in the summer of 2014. The ECLS-K:2011 collections are authorized under 20 US Code section 9543, which states that the purpose of NCES is “to collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations.”
Cognitive Interviews with Teachers. The cognitive interviews will be conducted as one-on-one telephone interviews between a responding teacher and an experienced qualitative interviewer. Teachers will complete the interviews on their own time (i.e., outside of their normal work/school day). Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions and to respond to a series of questions about the items he or she just answered. This approach includes asking probing questions, as necessary, to clarify points that are raised in the think-aloud comments.
Cognitive interviews will be conducted with 24 fourth-grade teachers (8 teachers who teach reading, 8 who teach math, and 8 who teach science). Although this is a purposive sample, we will attempt to recruit teachers from among elementary schools with various characteristics, e.g., from public (including charter) and private schools and rural and urban schools located throughout the country.
Recruitment of teachers will be completed by Shugoll Research, a third-party vendor. It will start with a national database that includes hundreds of thousands of households across the country. Information is kept on file for database members that includes presence of children in the household, age of children, occupation, and other demographics such as age, gender, and area of residence. Information about occupation will be used to identify elementary school teachers who can be recruited for a cognitive interview. This database will used in conjunction with a purchased list, which contains a list of names and home telephone numbers of fourth-grade teachers across the nation.
Based on the past experience of Shugoll Research, approximately twice as many teachers than are needed will have to be recruited to meet the target number of respondents with an incentive amount of $25. The level of effort needed for recruitment is why both the national database and the purchased list will be used to obtain teacher respondents. Thus, we anticipate contacting approximately 48 teachers to obtain 24 responding teachers. During recruitment, a recruiter will explain the purpose and anticipated length of the cognitive interview and ask the respondent if he or she would like to participate. The recruitment burden is anticipated to be about 5 minutes, on average, per teacher contacted (see appendix E for the teacher recruitment script).
After speaking with study staff and agreeing to participate in a cognitive interview (using the recruitment script found in appendix E), teacher respondents will be sent a confirmation email containing the details of their interview appointment (see appendix E). They will then be mailed a letter explaining the purpose of the cognitive interviews (appendix F) and a copy of the fourth-grade subject-specific child-level teacher questionnaire (appendix A) to review prior to the cognitive interview. The subject-specific child-level questionnaire covers a wide range of topics relevant to fourth-grade children and the instruction they receive, including the behavior of students in the children’s classes, their peer relationships, the topics taught in their class for the specific subject being asked about, and the activities their teacher uses to teach the subject-specific skills and content being asked about. A cover letter will instruct respondents to take 5 minutes to briefly review the entire questionnaire prior to the cognitive interview. While the interviews will focus only on selected items, reviewing the entire questionnaire will provide some additional context for the respondent.
During the actual cognitive interviews, trained interviewers from Shugoll Research will follow a prewritten protocol (appendix C) that asks respondents to answer selected questions and includes some follow-up questions on specific aspects of the tested items. Interviewers may deviate from the protocol in order to address specific issues or anomalies in the respondents’ verbal reports. At the conclusion of the interview, interviewers will ask the teachers for their opinion on the length of the questionnaire.
After the completion of roughly half of the interviews, Westat and NCES will meet to discuss the preliminary results and determine if any of the questionnaire items should be altered to address comments made in the completed interviews. If needed, the protocol will then be updated with the changes to the items and the revised items will be administered to the remaining teacher respondents.
Currently, there are 11 teacher questions to be cognitively tested, although each question has several sub-parts. The items to be tested include a measure of school liking, an evaluation of the child’s peer group and social emotional development, the frequency of electronic communication from the school to parents, the frequency that certain subject-specific skills and concepts are taught in the classroom, and the frequency that certain instructional methods are used in the classroom. Cognitive interviews with teachers are expected to last about an hour on average. Thus, the total burden per cognitive interview, including 5 minutes to review the questionnaire, is anticipated to be about 65 minutes for teachers.
Cognitive Interviews with Parents. As with teachers, the cognitive interviews with parents will be conducted as one-on-one telephone interviews between a responding parent and an experienced qualitative interviewer. Cognitive interviews will be conducted with 25 parents of fourth-grade children.
Using the recruitment script found in appendix E, parents will be recruited using the same national database as is used for the teacher recruitment. If needed to supplement the database, advertisements for respondents will be posted on Westat’s website.
In order to secure the needed number of participants, approximately twice as many parents will be recruited. Thus, we anticipate contacting about 50 parents to obtain 25 responding parents. During recruitment, a recruiter from Shugoll Research will explain the purpose and anticipated length of the interview and ask the respondent if he or she would like to participate using the recruitment script. Interviewers will make an appointment with participating parents and a confirmation email will be sent containing the details of their interview appointment (see appendix E).
As during the teacher cognitive interviews, during the interview with parents interviewers will follow a protocol (appendix C) which leads the parents through several questions. Parents will be asked how they arrived at their response, as well as some follow-up questions about each item. Interviewers may deviate from the protocol in order to address specific issues or anomalies in the respondents’ verbal reports.
There are currently 15 questions to be cognitively tested in the parent interview, including items on frequency of computer use, family rules for internet use, and an evaluation of the child’s peer group. The interviews are expected to take about 60 minutes per parent, with an additional 5 minutes of recruitment time. An incentive of $25 will be offered to parent respondents in order to recruit them and encourage their completion of a lengthy telephone interview.
As with the teacher cognitive interviews, Westat and NCES will meet to discuss the results of the parent cognitive interviews after about half of them have been conducted. Question wording may change during the testing period in response to suggestions from parents who are interviewed earlier in the testing period. Items that are working as expected may be removed from the protocol for the remainder of the interviews.
Timing Tests with Teachers and Parents. In order to determine the average length of time needed to complete the hard-copy teacher questionnaires and telephone parent interview, timing tests will be conducted with eighteen fourth-grade teachers (6 from each subject area) and eighteen parents of fourth-grade children. The teachers and parents will be recruited from the same channels described above; however, the respondents for the timing tests will not be the same respondents participating in the cognitive interviews. During recruitment, interviewers will make an appointment with participating parents to conduct the timing test and with teachers to collect timing information once they have completed the questionnaires. Recruitment scripts and confirmation emails can be found in appendix E.
As with the cognitive interviews, in order to ensure that enough interviews are completed, twice as many respondents as needed will be recruited. Thus, 36 teachers and 36 parents will be recruited for the timing tests.
For the timing tests, each teacher will be asked to complete the teacher background questionnaire (appendix D) and one subject-specific child-level questionnaire (appendix A), which is consistent with current plans for national data collection. Once recruited, teacher respondents will be sent a cover letter, the relevant Questionnaire Timings Form (appendix F), and the relevant questionnaires. They will be asked to complete the instruments, recording the start and end times for each section in the questionnaires, as well as the total time for completion of each questionnaire. Staff from Shugoll Research will then follow-up with the respondents briefly on the phone to collect their timings and ask a few follow-up questions about their experience completing the questionnaires. However, respondent’s actual responses to the items and the completed questionnaires will not be collected; respondents will be asked to dispose of the questionnaires after their follow-up call. Completion of both questionnaires is expected to take about 60 minutes per teacher, plus an additional 5 minutes of recruitment time and 10 minutes of follow-up on the telephone with an interviewer.
The timing tests with parents will be conducted as telephone interviews by experienced interviewers. Interviewers will make an appointment with participating parents and conduct the interview on the telephone using a hard-copy paper version of the parent interview (appendix B), following the proper skips in the interviewer instructions. Timings will be recorded by the interviewer for each section and for the interview overall. At the conclusion of the interview, a few follow-up questions will be asked. These follow-up questions can be found in appendix C. Parents’ responses for the timing tests are not relevant to this component of the fourth-grade pilot tests (beyond any questions or areas of confusion that may arise) and therefore will not be saved by interviewers. Completion of the timing test, including the time for the follow-up questions, is expected to take about 60 minutes per parent, plus an additional 5 minutes of recruitment time. Table 1 presents information on the sample for the cognitive interviews and timing tests.
Table 1. Samples for Fourth-Grade Cognitive Interviews and Timing Tests
Instrument |
Approach |
Number of participants |
Teacher Background Questionnaire |
Timing tests |
18 teachers |
Subject-specific Child-level Teacher Questionnaires |
Cognitive interviews |
24 teachers (8 for each subject) |
|
Timing tests |
18 teachers (6 for each subject) |
Parent Interview |
Cognitive interviews |
25 parents |
|
Timing tests |
18 parents |
In preparation for the fourth-grade data collections, a Technical Review Panel (TRP) meeting was held in November 2013. The members of the panel included experts in child development, teacher education and classroom activities, education policy, and the learning environment. During the TRP meeting, the experts made recommendations for items to be added to the fourth-grade hard-copy instruments and parent interview. Some of these items come from existing scales and have not been suggested for cognitive testing because of their successful use in other studies. The items proposed for the fourth-grade cognitive interviews are newly-written items or items from unpublished scales; all reflect the TRP’s recommendations. Table 2 shows the members of the 2013 TRP.
Table 2. TRP Panelists
|
Members of the 2013 TRP |
|
Robert H. Bradley, Ph.D. Family & Human Dynamics Research Institute Arizona State University |
David Dickinson, Ed.D. Vanderbilt, Peabody College |
Megan McClelland, Ph.D. Human Development & Family Sciences Hallie E. Ford Center for Healthy Children & Families Oregon State University |
Carol Connor, Ph.D. Department of Psychology Arizona State University |
George Farkas, Ph.D. School of Education University of California, Irvine |
Greg Roberts, Ph.D. The Meadows Center for Preventing Educational Risk University of Texas at Austin |
Rob Crosnoe, Ph.D. Department of Sociology and Population Research Center University of Texas at Austin |
Gary Ladd, Ed.D. Sanford School of Social and Family Dynamics Arizona State University |
Judy Snow, Ed.M. State Assessment Director Montana Office of Public Instruction |
Recruitment of teachers and parents for the cognitive interviews will begin with a national database of thousands of households across the country. The database is used as a starting point because it contains households of all demographic types interested in participating in research. As needed, this database will be supplemented with a purchased list of fourth-grade teachers.
Teachers and parents participating in the timing tests will be recruited through the same networks that are being used for cognitive interview recruitment. To attract respondents to participate and to thank them for their time and effort, participating teachers and parents will also receive $25.
We anticipate that recruitment of respondents will prove to be difficult due to the length of the interviews (or time to complete the questionnaires in the case of the teacher timing tests). In addition, we anticipate that many of the interviews will need to occur in the evening or on weekends, as these will be conducted during the respondents’ personal time. In order to thank respondents for their time, a $25 incentive will be provided. This amount is low compared to industry standards—typically, these types of respondents are compensated $75 to $150 for a 30- to 60-minute interview.
In the written materials and during recruitment, respondents will be informed that their participation is voluntary and that all information they provide may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002), 20 U.S.C., § 9573]. Confirmation that they consent to participate will be obtained from all respondents at the time of the interview. No personally identifiable information will be maintained after the interview analyses are completed. With respondent permission, interviews will be audio recorded for later analysis. If the respondent indicates that he/she does not want to be audio recorded, only written notes will be taken. The recordings and notes will be destroyed at the conclusion of the ECLS-K:2011.
Table 3 below summarizes the estimated burden for each respondent type. The estimated burden totals 170 respondents, 255 responses, and 104 burden hours.
Table 3. Estimated Burden for Fourth-Grade Cognitive Interview and Timing Tests
Respondent |
# Contacted for Recruitment |
Recruitment Time (minutes) |
Recruitment Burden (hours) |
# of Participants |
Time per task (minutes) |
Burden per Task (hours) |
Total Estimated Burden |
Teacher cognitive interviews |
48 |
5 |
4 |
24 |
65 |
26 |
30 hours |
Teacher timing tests |
36 |
5 |
3 |
18 |
70 |
21 |
24 hours |
Parent cognitive interviews |
50 |
5 |
4 |
25 |
60 |
25 |
29 hours |
Parent timing tests |
36 |
5 |
3 |
18 |
60 |
18 |
21 hours |
There is no direct cost to the respondent.
The cost to the government to conduct the cognitive interviews is $91,718.
Recruitment for the cognitive interviews and timing tests is expected to begin as soon as possible in February 2014, after OMB approval is received. The results of the cognitive interviews will be discussed immediately following the completion of the cognitive interviews, and then a report summarizing the findings will be finalized in May 2014. Table 4 presents the proposed schedule.
Table 4. Schedule for Fourth-Grade Cognitive Interviews and Timing Tests
Task |
Date |
Recruitment for Cognitive Interviews Begins |
by 2/20/14 |
Train Cognitive Interview Staff |
by 2/27/14 |
Data Collection |
2/28/14 - 3/28/14 |
Conference call to discuss preliminary results |
Week of 3/10/14 |
Summation and recommendation memo |
4/9/14 |
Fourth-Grade Cognitive Interviews and Timing Tests Report |
|
Draft 1 Field Test Report |
4/23/14 |
Final Field Test Report |
5/19/14 |
1Throughout this package, reference is made to the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99. It is referred to as the ECLS-K. The new study for which this submission requests approval is referred to as the ECLS-K:2011.
2 Because most of the questions in the School Administrator Questionnaire (SAQ) have been used in multiple rounds of data collection, the SAQ will not be included in the cognitive testing. However, a new set of items on the mode of communication between the school and parents is being considered for inclusion in the fourth-grade SAQ, and there is a desire to cognitively test them. Rather than conduct a separate set of interviews with administrators just for these items, the questions will be included in the cognitive interviews with teachers, since teachers will be able to respond to and discuss these items. This set of items (included in Appendix A) will be printed on a separate sheet and included in the package sent to math and science teacher respondents prior to the interviews.
3 In-person interviews will be completed with parents who do not have telephones, are hard to reach by telephone, or who prefer an in-person interview.
4 Timing tests of the SAQ are not being conducted because timings for instrument completion were collected as part of the pilot test conducted in spring 2013. A component of that pilot test asked school administrators to complete the paper SAQ before completing the instrument online; the goal was to determine if the paper instrument took less time and was easier to complete. Respondents indicated in debriefing questions how long it took to complete the paper questionnaire and online instrument.
| File Type | application/msword |
| File Title | Volume I: |
| File Modified | 2014-02-12 |
| File Created | 2014-02-11 |