LO12-QUEX-5 Baley3 Template

A.1 OIRA TEMPLATE DESCRIPTION

NCS Formative Research Template for OIRA Clearance

TO BE COMPLETED BY FIELD CONTRACTOR:

LOI #: LOI2-QUEX-5

Title of Formative Research: Bayley-3 Short Form for the NCS

Participating Institutions: WESTAT

Recruitment Study Arms:

SME: Gitanjali Taneja

COTR: Gitanjali Taneja

Purpose of the Study

This study has two purposes. The first purpose is to examine the feasibility, acceptability, and cost effectiveness of implementing a short form of the Bayley Scales of Infant and Toddler Development, Third Edition (i.e., the Bayley-3 Short Form) at 6-, 12-, 18-, 24- and 36-months in the National Children’s Study (NCS). The second purpose is to collect data in order to develop standardized norms for the Bayley-3 Short Form using Item Response Theory (IRT) analysis so that Bayley-3 Short Form IRT scores can be generated for each child as if the full Bayley-3 had been administered. The item content for the Bayley-3 Short Forms has previously been determined by conducting IRT analysis of the publisher dataset so the present study is intended to demonstrate feasibility and to generate a scoring system so that Bayley-3 Short Form scores will emulate the scores of the full Bayley-3.

Benefit to NCS Vanguard or Main Study

The Bayley Scales of Infant and Toddler Development, 3^rd edition (Bayley-3, Bayley, 2006), is generally regarded as the gold standard for early childhood developmental assessment. Of key consideration for the NCS is that the Bayley-3 provides norm-based measurement of children’s cognitive, language (receptive and expressive language), and motor (gross and fine motor) skill development. Having the Bayley-3 Short Form available to the NCS would substantially reduce participant burden and field costs by reducing time in the field and field training time. In addition the Bayley-3 Short Forms can serve as strong baseline measures to anchor the NCS, enable future comparisons with various outcome measures, and to parse the effects of various exposures on children’s development. Prior versions of the Bayley have also been shortened using the same procedures as with Bayley-3, see the Early Childhood Longitudinal Study-Birth Cohort (ECLS-B) 2 year psychometric report (chapter for the development of the short form of the Bayley-2).

In examining developmental growth it is strongly advisable to have multiple data points using the same measure, in order to conduct growth curve modeling and to examine children’s developmental trajectories. Having multiple age versions will also provide the NCS with the flexibility to obtain strong baseline measures at any of the data collection points, for example perhaps a Bayley-3 Short Form cannot be obtained at 6- or 12 months for a particular child (or group of children) but can be obtained at 24 months. This instrument is also intended to be feasible for reliable administration in the field by data collectors with no background in neurodevelopmental assessment or child development.

Average length of session across the age-specific short forms ranges from 45 to 65 minutes (which includes warm-up time to build rapport with the child and for the caregiver to review the consent form) with longer testing times likely for above average and gifted children. The item content for the Bayley-3 Short Form was developed along the lines of the item content for the short form of the Bayley-II that was developed for the 9- and 24-month data collections of the Early Childhood Longitudinal Study-Birth Cohort (ECLS-B). It is no longer advisable to use the ECLS-B short forms because the standardized norms of the Bayley-II are outdated and would give erroneous results, as well as the age groups not matching well to the NCS’s needs. In addition, the items in the Bayley-3 were revised on the basis of the most recent research in infant and toddler development. In ECLS-B, development of the short Bayley-2 saved approximately 15 minutes at 9-months, and up to 30 minutes at 24-months. If similar time reductions are achieved with Bayley-3 Short form, this would yield considerable cost savings and a reduction in burden for the NCS.

It is estimated that the average face-to-face interaction time between the infant and examiner of the Bayley-3 Short form will be 30 minutes at 6 months, 40 minutes at 12 months, and 50 minutes for older ages. Prior to administration of the instrument, mothers will be administered consent; also during this time data collectors will also be working to establish a rapport with the child. An additional 15 minutes for rapport building with the infant/toddler and for review of the consent form has been added to the timing estimates for the entire session.

The original Bayley-3 was developed using a sample of 2,300 infants and children across 17 age groups; this sample was rigorously stratified to represent the population of infants and toddlers in the United States. Stratification variables included gender, age, child race/ethnicity, level of education of the parent, and region of the country. With the exception of region of the country, the same stratification variables will be used in the development of the Bayley-3 Short Form IRT scores and standardized norms. Region of the country is less important than other factors for the development of norms and including it in this study would be cost prohibitive.

Study Design

This study will take place in a series of steps, with each subsequent stage dependent on the demonstration of feasibility, acceptability, and cost effectiveness of the administration of the Bayley-3 Short Form. At each step we will obtain timings on each of the subtests (e.g., cognitive, language, and motor), review descriptive statistics to ensure that there is sufficient variability to uncover potential systematic group differences, compute estimates of reliability (using classical test theory, i.e., alpha), and collect information on cost per administration.

The first step is to design and implement a small study to demonstrate the feasibility, acceptability, and cost effectiveness of developing a short form of the Bayley-3 at 6- and 12-months. These ages were selected for two major reasons. First, a short Bayley-3 should be obtained as a baseline measure as early as possible in the study. Second, these ages were chosen because of chronological proximity of NCS data collection events. As part of this step we will obtain timings on each of the subtests (e.g., cognitive, language, and motor), compute estimates of item reliability (using classical test theory, i.e., alpha), and collect information on cost per administration. Data will be collected from 125 participants at each age group, 6- and 12-months.

If appropriate feasibility, acceptability, and cost effectiveness is demonstrated in the early months of age, then we will proceed. The next step will involve a sample of 125 24-month olds and 125 36-month-olds. After feasibility, acceptability, and cost effectiveness have been demonstrated at 24- and 36-months of age, we will then add a sample of 125 18-month olds.

If these feasibility, acceptability, and cost effectiveness are demonstrated, then we will augment the data collection with an additional 375 participants. This step will add 75 cases at each age, bringing the total sample size to 1000 with 200 participants at each age. Two hundred participants at each age provides the minimum number of cases per group required for Item Response Theory analysis in order to achieve the goal of generating IRT standardized norms.

Item Response Theory

As a rule of thumb, Item Respnse Theory (IRT) requires 250 data point per item in order to obtain item parameter and thereby generate IRT norms and IRT standard score¹. Because many Bayley items overlap across the ages, only 200 cases per age are required. Collecting enough cases to conduct IRT norms is a necessary final step in order to equate the Bayley-3 Short Form to the long Bayley-3. Equating the Bayley-3 Short Form to the full form of the Bayley-3 is a necessary step because it will enable the NCS to claim that a score on the Bayley-3 Short Form is the same “as if” the full Bayley-3 had been administered. The equating of the short forms with the full Bayley-3 will enable the comparison of Bayley-3 Short Form scores with the publisher’s full Bayley-3 standardized norms.

Development of IRT standard scores for the short forms of the publisher’s full-length Bayley-3, Cognitive, Language and Motor Scales, will rest on national norms provided by the publisher. Publisher standardized norms development is described in the publisher’s technical manual. The publisher standardization sample (N=1700) used a quota system based on Census data with geographic region, race/ethnicity, child gender, and parent education level as stratifiers. This is the standard procedure that most publishers use for test development. The Bayley pilot study will use a similar procedure for recruitment in an attempt to be as much like the publisher sample as possible. However, due to cost considerations, we will be unable to stratify by geographic region. We will, however, stratify on the remaining key variables of child gender, parent education, and race/ethnicity to ensure sufficient variability for Item Response Theory (IRT) analysis.

The Bayley pilot study does not need to be a nationally representative or random sample. Instead, the publisher’s standardization sample serves as an external anchor that will allow us to use IRT analysis to place our scores on the same scale metric used by the publisher, and the publisher’s standardization sample is approximately representative of the US infant population at the time the norms were developed.

The Bayley pilot study’s requirements include a sufficient number of observations (IRT rule of thumb is 250 observations per data point, or item) with sufficient variability in order to calibrate our scores to the publisher scores. We then equate our short forms to coincide with the publisher’s (full) Bayley scale metric. After our scores have been calibrated and equated to the publisher’s data, we then estimate what an individual child’s full Bayley score would have been, based on the (short form) scores that we have. We then use the publisher’s lookup table to obtain the full set of corresponding standardized scores to each observation record. IRT standard scores will be obtained for the Cognitive, Language, and Motor Scales of the short forms of the Bayley-3 for the 6-, 12-, 18-, 24- and 36-month age ranges.

Proprietary Agreement

The Bayley tool is a proprietary assessment. Westat has a licensure agreement with Pearson that will not require NIH to engage into a licensure agreement for this tool. Pearson used Item Response Theory (IRT) to develop the Bayely-3. In order for Westat to enter into a licensing agreement with Pearson, they had to specify procedures and analyses in the license application. Pearson reviewed and agreed to the proposed procedures.

The short Bayley is a derivative of the Bayley-3 and is therefore Pearson intellectual property. This was also the case for the short form of the Bayley-2 that was developed for the ECLS-B. The data that we obtain from this project are NOT the property of Pearson. Please see text from license agreement below, and please note that “Main Study” does not refer to the National Children’s Study Main Study, but rather the study to develop the Bayley short form.

Text from the License Agreement:

1.2 "Intellectual Property (IP) Rights" means all intellectual property rights and interests including, without limitation: (i) all copyrights and copyrightable subject matter, including any and all worldwide applications, registrations, renewals and extensions thereof and all rights of reproduction and publication, rights to create derivative works and all of the rights incident to copyright ownership; (ii) all trade secrets and confidential information, all technology, ideas, know-how and proprietary processes and formulae; (iii) all inventions, designs, models, mask works, patents and pending patent applications; (iv) all trademarks and pending trademark applications applicable to the Test(s); and (v) all causes of action heretofore and hereafter accrued in favor of the owner of such intellectual property rights for infringement of any one or all of the aforesaid intellectual property rights. For clarification, IP Rights do not include any rights relating to any participant data (participant responses) collected by Licensee as part of the Main Study.

1.3 "Main Study" means the study involving the administration and Use of the Test(s) for the Research Project.

1.4 "Research Project" means the study titled, "Bayley-3 Short Form".

In summary, we will recruit a total of up to 1050 non-NCS participants (the goal being 1000 cases plus up to 50 additional cases to allow for “unusable data”), for n = 200 at five age ranges, 5 month to 7 months 30 days, 11 months 0 days to 13 months 30 days, 16 months 0 days to 20 months 30 days, 22 months 0 days to 27 months 30 days, and 33 months 0 days to 43 months 30 days.

The study design involves a single one-time-visit (up to 65 minutes maximum at the older ages because the test requires more items to cover children’s expanding skills). The age-appropriate Bayley-3 Short Form will be administered to children (with parents present) by trained staff. The estimated time for the child's participation on the Bayley-3 Short Form is 30 to 50 minutes, depending on the child's age, plus up to an additional 15 minutes in order to establish rapport with the child and allow time for the parent or guardian to review the consent form and ask any questions. The entire session should take less than 45 to 65 minutes, on average.

We will obtain informed consent from parents or legally authorized representatives for their child’s participation (Attach 2 Exemplar Consent Form). Parents will be asked to provide standard demographic information about the child (e.g., gender, age, parent-identified race/ethnicity and parent education and income level) during a short five-minute phone screener (Attach 1 Exemplar Telephone Screener). The demographic information will be used in the psychometric analysis to rule out any item bias and to help ensure that the Bayley-3 Short form is the psychometric equivalent of the full Bayley scales. The age-appropriate Bayley-3 short form will be administered to children (with parents present) by trained staff.

Target Respondents

Our initial target is 125 6-month olds and 125 12-month olds, followed by 125 24-month olds and 125 36-month olds, and 125 18-month olds, as part of our work to establish the feasibility, acceptability, and cost effectiveness of administering the Bayley-3 Short Form. Our ultimate goal is to obtain a convenience sample of 1000 non-NCS participants (200 6-month olds, 200 12-month olds, 200 18-month olds, 200 24-month olds, and 200 36-month olds) who are demographically similar to the children in the publisher’s standardization set. The publisher’s standardization sample is a quota sample stratified on key demographic variables. The key demographic variables are: child age (6-, 12-, 18-, 24-, and 36-months); child gender; and, three levels of parent education (less than high school, high school, and some college and above). Because the sample for 6-, 12-, 18-, 24-, and 36-months is smaller than that of the publisher, it is not reasonable to expect this to match the stratified quota sample exactly. However, we will attempt to include sufficient diversity in the children recruited to enable us subsequently to examine the singular effects of child age, child gender and parent education on children’s outcome scores. The age-appropriate Bayley-3 Short Form will be administered to children (with parents present) by trained staff. We also will collect information on the parent-identified race/ethnicity of the child and household income. Although household income is not one of the key stratification variables, this information will be required when IRT analyses are conducted to examine differential item functioning and differential test functioning (which would indicate any bias in test items or in the test itself).

We will take steps to ensure that children recruited are not enrolled in any of the National Children’s Study data collections by including a question in our telephone eligibility screener that asks whether the child is an NCS participant.

Our population is normal-healthy infants and toddlers who were born full-term.

• Inclusion Criteria: Infants and toddlers who match the age ranges of each age-specific Bayley-3 Short Form will be included: infants and toddlers from 5 months 0 days to 7 months 30 days, 11 months 0 days to 13 months 30 days, 16 months 0 days to 20 months 30 days, 22 months 0 days to 27 months 30 days and 33 months 0 days to 43 months 30 days. Infants and toddlers included will also meet the following criteria: no medical complications at birth or currently and not diagnosed or receiving treatment for mental, physical or behavioral difficulties.

• Exclusion Criteria: Infants and toddlers who are outside the age windows of our 6-, 12, 18-, 24-, and 36-month Bayley-3 Short Forms are not eligible for participation. Also excluded are infants and toddlers who were born earlier than 36 weeks of gestation or later than 42 weeks, even if their chronological age falls within our age windows. Non-English speaking families will not be eligible because the publisher norms are standardized on an English-speaking sample. Non-English speaking will be determined on the basis of primary language spoken in the home. Current NCS participants will not be eligible.

Sample Size Calculation: The general rule of thumb for conducting Item Response Theory analysis will be followed, which is that 250 data points per item are required in order to obtain reliable item parameters (the item difficulty parameter and the item discrimination parameter), as well as to ensure the IRT reliability of the finished product. A traditional power analysis is not appropriate. Because some items are present in more than 1 age set (e.g., some items at the high end of ability at 6-months are also present at the low end of ability at 12-months), it will be possible to obtain the required 250 data points per item with only 200 children in each age group. A smaller sample size would result in unreliable data that would decrease the IRT internal consistency reliability of the Bayley-3 Short Form.

Our planned screening and sample size will provide sufficient cases per cell to conduct basic reliability analyses (alpha) and to enable basic analyses (e.g., Analysis of Variance) examining the main effects of age, gender, and parent education on children’s scores and on timings. For example, gender may be associated with differential timings for the various subtests. If the Bayley-3 Short Form promises to be a reliable measure, it would be expected that scores obtained would resemble scores obtained across groups as found in the literature. When children in all age groups are recruited, we will attempt to recruit participants sufficiently diverse to enable us, to the extent possible, to replicate the demographic stratification of the publisher’s standardization sample.

Method of Recruiting:

To obtain this sample, Westat will purchase names and telephone numbers of the families of age-appropriate children from a marketing firm (e.g., InfoUSA). Parents will be contacted by mail (Attach 3 Exemplar Cover Letter) initially with telephone follow-up by a research assistant within 7 days. Because this telephone call is for the purpose of government-sponsored research, it will be exempt from “Do Not Call” limitations. The cover letter and accompanying flyer (Attach 4 Exemplar Flyer) with study contact information are in the Cover Letter and the Flyer.

If necessary, we will also recruit children through announcements on the Westat web page, on WesInfo (Westat’s intranet) by posting the flyer and, possibly, the NCS Facebook page, at the discretion of the Project Office. If necessary, flyers will also be placed in local child care centers, libraries, and other locations that focus on children (e.g., community recreation centers), as well as community newsletters and newspapers. Private Health Information (PHI) will not be used to identify or recruit potential participants.

It is anticipated that parents or guardians of potential participants, who are recruited from flyers or local media, will contact the research assistant (TBD) by telephone. If convenient for the parent/guardian at that time, a screening questionnaire will be completed by telephone to see if inclusion criteria for study participation are met. In addition, the parent will be asked to provide demographic information to ensure that our sample will match the publisher’s standardization dataset demographics. The demographic information provided by parents in the phone screener will also be used in the psychometric analysis to rule out item bias and to help ensure that the Bayley-3 Short form is the psychometric equivalent of the full Bayley-3 scales. If it is not convenient to complete the screener at that time, a follow-up telephone call with the parent will be scheduled. At the end of the screener, the research assistant will say (per the instructions at the end of the screener): “I will forward this information to our study staff and they will see if your child is eligible to participate.” The results of the screener will be discussed with the PI and if study-eligible, then the research assistant will call the parent or guardian to report that the child is eligible and a testing session can be scheduled. The wording for this is provided at the end of the screener, as well as for ineligibility.

^Confidentiality: Westat will abide by the terms of our Data Use Agreement, which should reference all formative research efforts involving the collection or management of NCS restricted-use data. Westat has an approved NCS Data Use Agreement and Security Plan.

The screening questionnaire information and the completed Bayley-3 Short Form scoring sheets, which contain the de-identified demographic information, will be kept in a locked file cabinet in a locked office. Each subject will be given a unique subject ID number and demographic information from the screening questions will be translated into numerical information. Item-level test data is scored as 1 or 0. The contact information (parent/caregiver address and phone number) will be kept in a separate spreadsheet which only will be accessible to the PI and a research assistant. The screening information and Bayley-3 Short Form item level data will be in a separate spreadsheet with will include no identifiable information and all cases will be identified by the study ID number.

^IRB Approval: Local IRB clearance for this activity has been obtained by Westat. Please see the attached IRB Protocol (Attach 5) and the IRB approval letter (Attach 6).

The Bayley tool is a proprietary assessment. Westat has a licensure agreement with Pearson that will not require NIH to engage into a licensure agreement for this tool.

Incentives: To thank them for their time, study participants will receive a $25 monetary incentive upon completion of the assessment session. The study participants also will receive an age-appropriate book.

Sensitive Questions: We will not ask sensitive questions as a component of this study.

Proposed Project Schedule: We will begin this project upon receipt of all regulatory approvals.

Data Collection Burden:

Estimates of Annual Hour Burden –Bayley-3 Short Form for the NCS

Data Collection Activity	Type of Respondent	Estimated Number of Respondents	Estimated Number of Responses per Respondent	Average Burden Per Response (in hours)	Estimated Total Annual Burden Hours
Screening Phone Call 6-months	Parent/Guardian	250	1	5/60	21
Bayley Short Form Session 6-months	Child	200	1	45/60	150
Screening Phone Call 12-months	Parent/Guardian	250	1	5/60	21
Bayley Short Form Session 12-months	Child	200	1	55/60	183
Screening Phone Call 18-months	Parent/Guardian	250	1	5/60	21
Bayley Short Form Session 18-months	Child)	200	1	65/60	217
Screening Phone Call 24-months	Parent/Guardian	250	1	5/60	21
Bayley Short Form Session 24-months	Child	200	1	65/60	217
Screening Phone Call 36-months	Parent/Guardian	250	1	5/60	21
Bayley Short Form Session 36-months	Child	200	1	65/60	217
TOTAL		2250			1089

Note: 15 minutes added to each Short Form session to account for rapport building

Annualized Cost to Respondents -- “Bayley-3 Short Form for the NCS”

Data Collection Activity	Type of Respondent	Estimated Total Annual Burden Hours	Hourly Wage Rate	Respondent Cost
Screening Phone Call 6-months	Parent/Guardian	21	$10.00	$210.00
Bayley Short Form Session (6-months)	Child	150	$10.00	$1500.00
Screening Phone Call 12-months	Parent/Guardian	21	$10.00	$210.00
Bayley Short Form Session (12-months)	Child	183	$10.00	$1830.00
Screening Phone Call 18-months	Parent/Guardian	21	$10.00	$210.00
Bayley Short Form Session (18-months)	Child)	217	$10.00	$2170.00
Screening Phone Call 24-months	Parent/Guardian	21	$10.00	$210.00
Bayley Short Form Session (24-months)	Child	217	$10.00	$2170.00
Screening Phone Call 36-months	Parent/Guardian	21	$10.00	$210.00
Bayley Short Form Session (36-months)	Child	217	$10.00	$2170.00
TOTAL				$10890.00

Please check here after ensuring that all calculations have been verified

Estimated Costs: Staff Hours: 2178

Supervisor Hours: 545

Attachments: Attach 1 Exemplar Screener, Attach 2 Exemplar Consent Form, Attach 3 Exemplar Cover Letter, Attach 4 Exemplar Flyer, Attach 5 Bayley 3 Approved Protocol, Attach 6 IRB Approval Letters, Attach 7 Cover Sheet for Instruments

Please check here after ensuring that the OMB #: 0925-0661 and Expiration Date: 06/30/2015 have been inserted as first-page headers on each proposed instrument.

Please check here after ensuring that the following OMB burden statement has been inserted as a first-page footer on each proposed instrument.

Public reporting burden for this collection of information is estimated to average 55 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0661). Do not return the completed form to this address.

Appendix 1. Maximum NCS Incentives, by Study Activity and Impact on Participants (Approved by OMB 1/5/12)
Data Collection Activity Characteristics	Initial NCS Vanguard Study	NCS Recruitment Substudy and Formative Research
		Phase 1	Phase 2	Formative Research
Time for encounter	3 hours	0.5 to 1 hour	0.5 to 1 hour	0.5 to 1 hour
Sensitivity of questions	Sensitive, including sexual activity	Few sensitive questions	Few sensitive questions	Few sensitive questions
Physical measures	Yes	No	No	Yes*
Environmental specimens	Yes	No	Yes	Yes*
Biospecimens	Yes	No	Yes	Yes*
Participant observation	Yes	No	No	No
Monetary incentive, per visit	$100	$25	$25 for the group of study questionnaires, plus $25, in total, for any bio-specimens collected during a contact and, where appropriate for environmental specimens	$25, in total, for any bio-specimens collected during a contact. For questionnaires, or any environmental specimens – up to $25 when deemed necessary
Non-monetary incentives (tote bags, post its, key chains, etc.)	In addition to the monetary incentive, non-monetary incentives valued at $25 or less may be offered to participants	As an alternative to the monetary incentive, NCS logo gifts valued at $25 or less may be offered to the participants in lieu of cash or local incentives not exceeding $25 in value and deemed non-coercive by local IRBs	In addition to the monetary incentive, NCS logo gifts valued at $25 or less may be offered to the participants if these are deemed acceptable by local IRBs	Instead of monetary incentives, NCS logo gifts valued at $25 or less may be offered to the participants if these are deemed acceptable by local IRBs

1 Michele Zimowski, Eiji Muraki, Robert J. Mislevy and R. Darrell Bock. BILOG-MG. (USER'S MANUAL). Introduction, p. 1 - 24.

 To be completed before project proposal is submitted for OIRA clearance.

Send completed forms to ncs@mail.nih.gov Page 7