OMB-ACF Q&A July 2011

PRA Review: Q-CCIIT Project

PRA Review: Q-CCIIT Project

Response to Comments on Supporting Statements:

Part A

1. What are anticipated next steps with the Q-CCIIT? Does ACF expect to use it in any of its studies planned for 2013 or 2014? Are there other non-ACF studies for which its use in the next few years is anticipated?

The Q-CCIIT measure is designed to be used in early care and learning settings serving infants and toddlers, both by the Early Head Start program and by the wider child care research and provider community. One of the project's tasks is the development of a plan for sustainability and dissemination of the measure to support its usefulness. This plan is anticipated to be completed in 2013 after psychometrics on the Q-CCIIT measure have been documented.

2. Has ACF thought about how the rating procedure would fit with rating approaches for older children who might be in the same setting? Not that the issue should drive the design, but the more compatible the measurement approach across age ranges, the more likely that they will be done properly by field staff.

The Q-CCIIT measure is specifically designed to focus on caregiver interactions with infants and toddlers, which may require approaches that are different from interactions with older children. The Q-CCIIT measure will use key developmental constructs and a scale range similar to other measures, which may facilitate the administration of multiple measures in mixed-age settings. Although the areas of caregiver interaction (caregiver support for the domains of child development) parallel many of those for older children, the levels and descriptors of caregiver behaviors will be based on the developmental needs of infants and toddlers rather than older children in the setting. Thus, the Q-CCIIT measure does not have a direct extension for older children.

3. This seems like an unusually precise estimate. It is also on the long side for a focus group. Why precisely 1 hr and 55 min?

This approximately two-hour time frame was indicated in order to allow for introductions and explanation of the purposes of the focus group, in addition to an activity (designed to elicit evidence of the face validity of the ordering of items) and time for discussion. The session will conclude with a 5-minute demographic questionnaire for focus-group participants on topics such as type of setting (center-based or FCC), work experience, and race/ethnicity, so that we can capture the range of respondent characteristics.

4. Is ACF going to capture information about the amount of exposure the child has had to the care setting and care givers in the configuration of both with the Q-CCIIT measurements are taken? This will not be a perfect test, but might help control for noise in the relationship between quality measures and child development measures.

Yes, we agree, and we will be collecting information about exposure to the caregivers and setting.

5. Have these been benchmarked against direct measurements of different dimensions of child development?

Yes, these parent questionnaires were selected because they have evidence of associations both with other observational measures as well as with direct assessments of children. For example, according to Squires, Clifford, and Twombly (personal communication, 2007) domain scores from the ASQ:IT were significantly correlated (r = .75 - .94, p < .01) with scores from the Battelle Developmental Inventory, 2nd Edition (BDI-2), a direct assessment.

Total scores on the CDI were significantly related to the PLS total (r = .63) while the CDI Words and Gestures subtest has demonstrated relations ranging from .51 to .87 with the PPVT-III, the PLS, the Reynell DLS, and the Language Sample Number of Different Words. The CDI Words and Sentences subtest was correlated .40 to .88 with the Bayley Expressive subtest, the EOWPVT, the SICD-R, the PLS, PPVT, and Reynell DLS expressive subtest (Fenson et al., 2007; Fenson, Pethick, Renda, & Cox, 2000; Kisker et al., 2003).

BITSEA scores correlated significantly with evaluator ratings of behavioral competence and predicted CBCL and ITSEA problem and competence scores one year later (Briggs-Gowen, Carter, Irwin, Wachtel, Cicchetti, 2004). Further, ITSEA competence scores related positively to the Vineland composite score (r = .58) and ITSEA Externalizing scores related negatively to the Vineland (r = -.26; Carter, Briggs-Gowan, Jones, and Little, 2003). Carter and colleagues (Carter, Briggs-Gowen, Jones, & Little, 2003) also found significant relations between the ITSEA domains and the CBCL (.21 to .73); the Colorado Child Temperment Inventory (CCTI; .08 to .57); and the Beck Anxiety Inventory and CES-D (.12 to .29).

6. Is the Q-CCIIT itself being fielded both times? This could help isolate change in competencies for children who change settings over the 6 month period, and for children in settings that change substantially over 6 months. Discussion on page B-4 indicates that providers may be rescored during the followup collection, but the issue is not completely clarified.

The Q-CCIIT observations will be completed at only one time point, with child competencies measured both concurrently with those observations and at a 6-month follow-up. The research question we seek to answer is whether the quality of the setting is related to child growth over this 6-month interval. The repeat observations mentioned on page B-4 are for test-retest reliability purposes, in order to check the stability of the instrument. They are scheduled to occur approximately 2 weeks after the initial observations, and we do not expect substantial change in the composition of classrooms over this 2-week period.

Also, be sensitive to including competencies that are for children up to 42 months old. The sample is designed to measure relationships for children from birth to 36 months. By the of the 6 month lag, some children who were 36 months old during the first set of measures will be 42 months old.

For the purposes of measuring child competencies, we propose to include children up to age 30 months at the initial time point, who would be no greater than 36 months at the 6-month interval. Our focus is on children in the infant/toddler age range, not on older children.

7. What are the advantages and disadvantages of using a probability based sample design within the sites? It seems that there is an attempt to mimic features like stratification, but without the real benefits. Related, as designed, it seems likely that there will be selection bias (on the low end of the quality spectrum) which undetected could undermine some of the benefits of the project.

Our sampling plan is designed to yield a range of settings with important characteristics of interest, such as program-based, community-based, center-based, family child care homes, age-specific, and mixed age. Unfortunately, we are not aware of information that would be available at the level of a geographic location that would allow us to construct a sampling frame and stratify more formally. We will make every effort to obtain access to settings across the quality spectrum, but we may in fact have difficulty gaining access to lower-quality settings, as is common in other studies of child care. If there are national, regional, or local databases with such information available through new studies, we would be happy to make use of them in our sampling approach.

8. Given its leadership on the Early Childhood Longitudinal Study, Birth Cohort, and the increased utility of developing the Q-CCIIT for use in multiple studies, we would like ACF to include National Center for Education Statistics staff Chris Chapman and/or Gail Mulligan in future consultations on the Q-CCIIT.

We appreciate this suggestion and will identify the appropriate person from NCES to invite.

9. Why the preference for gift cards over cash or checks for each of the incentives in this section?

In our experience, gift cards are a preferred approach to respondent incentive payments over checks or cash. They do not require a SSN, do not expire, do not require check-cashing fees, can be carefully tracked, and are well-received by respondents.

10. Please justify this amount. It seems high to us by about 2-fold for an observation.

We will modify our respondent payment to caregivers for setting observation and completion of background questionnaires to $25 per observation.

11. Please change to “assurance” or similar term.

We will make this change from “guarantee” to “assurance.”

Part B

1. Is the target universe up to 36 months old? Typo? Seems inconsistent with Part A.

Although the target population of interest is from birth through 36 months of age, for the purposes of measuring child competencies, we will focus on children who are up to 30 months at the time of observation, and follow-up at a 6-month interval.

2. Would classrooms be rescored on Q-CCIIT during the retest? How will situations be handled where children were in setting at time 1 but not time 2? Footnote 4 on page B-7 provides a procedure, but it is unclear how those leaving care during the 6 month period might affect the analysis or power of the ultimate test/pretest sample.

The repeat observations mentioned on page B-4 are for test-retest reliability purposes, in order to check the stability of the instrument. They are scheduled to occur approximately 2 weeks after the initial observations, and we do not expect substantial change in the composition of classrooms over this 2-week period.

3. Please elaborate on why these are the right targets based on the literature and also on policy considerations.

Given the strong and continuing influence of the family on child development, classroom observations typically exhibit weak correlations to child outcomes. Although higher correlations are found when disadvantaged samples are examined (Burchinal, Roberts, Riggins, Zeisel, Neebe, and Bryant 2000), the influence of the family advantages (and possibly peer effects) is strong, In addition, continuity of care is not widely practiced and infants and toddlers may be in care for varying amounts of time. Frequently, infants and toddlers are in a particular classroom fewer than 6 months before being moved to the next setting.

Bivariate relations between parent- or teacher-report measures and classroom observation measures are generally less than 0.15. Direct assessments have a stronger, though still weak, relation with classroom observations. Within the NICHD-SECC sample (705<n<435), the zero-order correlations of the ORCE quality score with the BSID MDI were 0.12 at 15 months and 0.25 at 24 months (NICHD-ECCRN 2000). At 36 months, the correlation of the ORCE quality score with the Bracken was 0.23, and the correlations with the Reynell Vocabulary Comprehension and Expressive Language were 0.25 and 0.16, respectively. The parent report measures (CDI, CBCL) employed in this sample had weaker relations with the ORCE (zero-order correlations range from 0.02 to 0.17 across the time points) (NICHHD-ECCRN 2000; Burchinal, personal communication, February 1, 2011).

4. How were the design effects derived? Given that the samples are designed to capture pretty slight levels of correlation, this design effect may not matter that much unless it is essential that correlations in the .1 range be detectable.

The design effects presented in this table, and incorporated into the effective sample size and minimum detectable correlation calculations, reflect the impact of within-classroom clustering on the variance of estimates. The formula for the design effect is 1 + ICC (n-1), where the ICC is the intraclass correlation coefficient, and n is the average number of children per classroom. The ICC is the proportion of the total variance that is accounted for by between-classroom (as opposed to within-classroom) variation. If there were no clustering effect within classrooms, then there would be no homogeneity of children within classrooms, and ICC = 0 and the design effect = 1. As the ICC increases, it reflects some level of homogeneity of children within classroom, meaning that the effective sample size of children is less than the nominal sample size. We assumed an ICC value of 0.1 in these calculations.

5. From ACF’s perspective, is this instrument’s use limited to validating the Q-CCIIT or will it have utility even once the assessment is finalized?

If the latter, are there plans to determine if the 45 minute instrument could in any way meaningfully be paired back to 5 minutes for insertion into other studies? Something like this is not central for this particular study, but the analysis could be useful for others.

The parent questionnaire is intended solely for validation of the Q-CCIIT measure.

Response to Comments on Instruments:

1. Please fix the question stem on the two race questions on the parent form.

We will make the requested changes. Please supply the new template for race questions; we have been following a previously approved template.

2. Also, why is the Agency seeking additional information on the Pacific Islander category?

We will remove the “other specify” line under Pacific Islander. If it is possible to include, we would like to replace it with “Another race (please specify).”

3. The questions provided were not formatted as they will need to be for self-administered format. The spreadsheets also do not include the required information on burden, confidentiality, etc. Are better reproductions available now? Please provide.

We provided the content of the questionnaires using the spreadsheet format to allow for ease of amendment, as needed. Once the final questions are decided upon, we will prepare these for self-administered format, and include additional information such as the Paperwork Reduction Act statement, OMB number, burden, and assurances of confidentiality.

4. Please provide the supplemental materials that will go with this collection, such as letters and recruiting scripts.

Please note that Appendix E contains Frequently Asked Questions (FAQs), caregiver recruitment and consent materials, and parent notification and consent materials. If needed, we can also supply recruitment scripts used for telephone calls.

References

Briggs-Gowan, M. J., Carter, A. S., Irwin, J. R., Wachtel, K., & Cicchetti, D. V. (2004). The brief infant-toddler social and emotional assessment: Screening for social-emotional problems and delays in competence. Journal of Pediatric Psychology, 29(2), 143-155.

Burchinal, M. R., & Roberts, J. E. (2000). Relating quality of center-based child care to early cognitive and language development longitudinally. Child Development, 71(2), 339.

Carter, A.S., Briggs-Gowan, M.J., Jones, S.M., & Little, T.D. (2003). The Infant-Toddler Social and Emotional Assessment (ITSEA): Factor structure, reliability, and validity. Journal of Abnormal Child Psychology, 31(5), 495-514.

Fenson, L., Marchman, V.A., Thal, D.J, Dale, P.S., Reznick, J.S., & Bates, E. (2007). MacArthur-Bates Communicative Development Inventories, User’sGuide and Technical Manual, Second Edition. Baltimore: Paul H. Brookes Publishing.

Fenson, L., Pethick, S., Renda, C., & Cox, J.L. (2000). Short-form versions of the MacArthur Communicative Development Inventories. Applied Psycholinguistics, 21(1), 95-115.

Kisker, E.E., Boller, K., Nagatoshi, C., Sciarrino, C., Jethwani, V., Zavitsky, T., Ford, M., & Love, J. (2003). Resources for measuring services and outcomes in Head Start programs serving infants and toddlers. Washington, D.C.: Mathematica Policy Research.

National Institute of Child Health and Human Development Early Child Care Research Network. (2000). The relation of child care to cognitive and language development. Child Development, 71(4), pp. 960-980.

7