Responses to OMB on BSF 36-month Follow-up - 6-9-08

Draft responses to OMB questions on BSF 36-Month Follow-Up

PART A

1. OMB noted that during the draft phase of this ICR, there was mention of a 15-month follow-up survey that was currently being conducted. We would like to know if ACF has obtained any preliminary results from this first follow-up and more specifically, would like to know what (if any) operational issues (e.g. any problems with the questionnaire; any issues with the sensitive question portion of the survey; if the length of the survey was deemed by participants to be too burdensome, etc.) were observed and how ACF intends to address these issues during the second follow-up. If there is a write-up available, OMB would appreciate a copy.

Our contractor is closely monitoring the 15-month survey to assess if any questions in the survey are problematic or if there are other operational issues. We were particularly interested in using this monitoring information to improve, as needed, upon the structure and content of the 36-month survey instrument. There is no formal write-up of the results of that monitoring, but contractor staff have reviewed the data looking for logic problems and questions that are not functioning as expected. They have not found any problems with the questions in the instruments. Debriefings with the interviewers have not revealed concerns from respondents concerning the length of the survey and only 2% of interviews that were broken-off were not later resumed (hence not suggesting a problem with burden).

With regard to sensitive questions, Table 5 in question 11 of Part A includes the findings from our analysis of non-response to sensitive questions. We have been pleased to find that the non- response rate has been quite low for most of these questions, suggesting that most respondents understand and are comfortable responding to somewhat sensitive questions in the 15-month survey. Below is an excerpt from Table 5 with the non-response finding highlighted.

Whether the BSF partner is the parent of other children born after random assignment (Question FS52)

This question will enable us to examine BSF’s potential impact on multiple partner fertility. Multiple partner fertility has been shown to have negative consequences for child well-being, reducing financial and other support from parents and increasing children’s exposure to unrelated adults, which can increase the risk of child maltreatment (McLanahan and Sandefur 1994; Radhakrishna et al. 2001; Carlson and Furstenburg 2006; Harknett and Knab 2005). This question has been used on follow-up surveys conducted as part of the Fragile Families and Child Wellbeing Study and the first follow-up Building Strong Families survey. In our experience with the first follow-up survey for BSF thus far, nonresponse is less than 1 percent for this item.

2. pg. 8: Did ACF ever consider collecting information on the parent’s relationship at the time of conception, meaning that prior to becoming pregnant, were the two individuals dating or just take part in a one-night-stand? How long the individuals knew each other and the nature of their relationship just prior to conception may have an effect on their ability and/or desire to maintain a strong relationship following.

We agree that this is useful information. The BSF program enrolls couples during pregnancy or within three months of the birth of their child. Couples complete a baseline form (OMB clearance number 0970-0273) at the time of enrollment that includes questions concerning the status and quality of their relationship at the time they entered the sample. In particular, we ask both mothers and fathers, “How long did you know (MOTHER/FATHER) before this pregnancy?” We also ask both mothers and fathers whether they considered the pregnancy unwanted or mistimed. In addition, we ask them a series of eight questions covering several key dimensions of relationship quality, including intimacy, supportiveness, commitment, and communication. At baseline, we also asked couples whether they were living together and asked them to rate their perceived likelihood of marrying their BSF partner. Analysis of the baseline data indicate that, on average, couples report having known each other for 3.5 years; almost three-quarters were cohabiting at the time they enrolled in BSF; and over two-thirds rated their likelihood of marriage to each other as high.

(3) Pg. 9: Please provide further insight as to how the “presence and severity of asthma” provides elicit information about the warmth present in the parent-child relationship. OMB understands that this is typically used as an indicator of child’s health, but we would like further explanation as to how this particular indicator is correlated to family structure.

Both the “presence and severity of asthma” and the “warmth present in the parent-child relationship” will be examined analytically as separate and distinct constructs, and we did not intend to imply that these constructs were directly related.

ACF proposes to examine the effect of the BSF intervention on focal children’s cognitive, socio-emotional, and physical development. As noted, the presence and severity of asthma before age three is an important indicator of child health and this indicator of child health has been used as a proxy for general child health in other research (for example, Liu & Heiland, 2007). Childhood asthma is also associated with contextual factors including socio-economic status (Neidell, 2004;), stressful life events (including parental relationship conflicts; Klinnert et al., 1994), and stress experienced by a caretaker (Wright et al., 2002), all of which may be influenced by the BSF intervention. There is also evidence that the presence and severity of asthma is related to family structure. For example, analysis from the Fragile Families Study indicates that asthma early in childhood was related to family composition with a higher incidence of asthma diagnosis and asthma-related emergencies during infancy for children of unmarried parents (Harknett, 2005). Because the intent of the BSF intervention is to assist unmarried couples in strengthening their relationship, effects on childhood asthma may be detectable.

(4) Pg. 25: Please provide greater information on what ACF means when it states that “all respondents will be informed that their identity will be kept private…” More specifically, does this statement apply to keeping spousal comments/remarks confidential from his/her partner? What specific efforts is ACF making with regards to maintaining spousal confidentiality?

When telephone interviewers introduce themselves to the sample member on the phone, they explain that the information sample members provide will be kept private We have also revised the introduction to the survey to correspond with our advance letters, which emphasize the voluntary nature of the study and that, to the extent possible, the privacy of individuals will be maintained. (Revised instrument is attached for your review.)

Further, all interviewers and locators are trained not to reveal information about a sample member to anyone else, including partners and spouses. If one partner or spouse requests information about the other, interviewers and locators are trained to politely refuse and state that it would violate our commitment of privacy to all persons in the study.

(5) Pg. 29: It states that the interviewing process will start in 2007 and is expected to be completed in 2011. In light of this sentence, can ACF please confirm the protocol and timeline used for interviewing both parents? How far apart (in terms of time) could spousal interviews potentially be? OMB would like a copy of the overall project schedule for this information collection.

The timing of the interview for this survey corresponds to when the focal child is reaching the age of 36 months. Both mothers and fathers will be eligible to be interviewed during the period in which the child’s age ranges from 35 to 41 months. Therefore the maximum amount of time between a mother and father interview will be 6 months. If a sample member (mother or father) is not reached by 41 months, an interview will no longer be pursued because (1) the meaning and the developmental appropriateness of the child data changes as the children become older and (2) it has been our experience that after 6 months of repeated attempts to contact a sample member locating efforts yield very few interviews relative to the resources expended.

The survey data collection schedule is below.

(6) Pg. 29: Why are the time estimates for mothers and fathers different?

On average, we expect the mother’s interviews will be five minutes longer than father’s interviews because, in most cases, the mother is asked additional questions related to child outcomes that are not included in the father’s interview. Sample members’ responses to questions coupled with logical skip patterns embedded in the survey determine who is asked each set of questions. For example, questions about the child’s behavior and family routines are asked of a mother if she lives with the child all or most of time. If a father indicates that the father lives with the child all or most of the time and the mother does not, then the father is asked these questions. While in some cases, fathers may be the ones answering the additional questions, our expectation is that most of the time the mothers will be in the best position to answer about the child. Therefore, we estimate the burden to be slightly higher for mothers.

PART B

(7) pg. 1 Given that there are only seven sites, please provide further detail as to why you would only choose 5 or 6? If ACF chooses to go to all seven sites, then no generalizations have to be made. Additionally, can ACF please elaborate on what methods it will use to select which sites it will attend?

The 36-month follow-up telephone survey will include all sites. Because of cost constraints, the in-home direct assessments will be conducted for a portion of the research sample. We do not intend to generalize the in-home assessment findings to sites not included in the in-home direct assessment component. Rather, the findings will be attributed to the specific sites from which the data are collected. Because the aim of the BSF study is to determine whether a well-designed and well-implemented intervention can have the intended impacts on unmarried parents and children, the findings will provide useful policy and program information but will not be intended as general statements about the effectiveness of a broad array of similar programs.

We will select sites for the in-home assessment data collection based on two factors. The first is the quality of program implementation, as measured by the rate at which couples attend the prescribed group workshop sessions that are the core of the intervention. The second factor will be an early estimate of impacts on relationship status (whether couples are still together) based on available 15-month follow-up data. We anticipate that one or two sites will be excluded based on these factors. In this way, we will utilize scarce project resources for the in-home direct assessment data collection at the sites where there are indicators that the program implementation was strong and the program shows signs of yielding impacts on an important outcome of interest.

SURVEY

(8) Does Mathematica have the full SSN and DOB for each individual participating in this study? If so, why? Why is the verification so critical that you would need to ask sensitive personal identifying information questions such as DOB and SSN?

The data elements were collected through the baseline data collection (OMB clearance number 0970-0273). These data are used only when necessary to confirm the survey interviewer is speaking with the correct person and to locate individuals when all other methods have been exhausted.

Verification is a critical part of the interviewing process. In a study such as this, when couples separate and find new partners, there can be a tendency among sample members to want to substitute their new partners for their study partners, sometimes in an effort to obtain the additional respondent payment, but more often because they prefer to avoid offending the current partner or do not want to be associated with their former partner. In either case, to ensure that we are indeed collecting data from the correct individual, a system of verification is utilized. As part of this verification, we ask, before conducting the interview, for the purported sample members to provide their name and DOB, and if there is any discrepancy with our records, the sample members are then asked to provide the last four digits of their SSN, and finally, if necessary, street address at the time of enrollment.

(9) pg. 2: The interviewer says that the survey will take approximately 45 minutes, yet in the supporting statement is says that it would take approximately 53 minutes. Please explain the discrepancy.

The instrument was drafted before we had final estimates of the administration time. The introduction to the survey has been revised to indicate that the survey will take approximately 50 minutes to complete. The revised instrument is attached for your review.

(10) What is the relevance of collecting the child’s exact birthdate? For the purposes of this study, wouldn’t simply the year and possibly the month suffice?

When possible, we collected information on the child’s birth date during the baseline (OMB clearance number 0970-0273) and 15 month telephone surveys (OMB clearance number 0970-0304). In those cases, we do not ask about the child’s birth date during the 36 month survey; however, for sample members who were pregnant at the time of enrollment and did not complete the 15 month survey, we would ask about the child’s birth date during the 36 month survey in order to gather this information for all focal children. The child’s exact birth date (including the day) is necessary to calculate the child’s precise chronological age as a basis for determining the correct starting point for the Peabody Picture Vocabulary Test (PPVT-4; Dunn & Dunn, 2006) during the direct child assessment. The child’s exact age is also needed to obtain the correct normative scores for this assessment.

(11) What would be one of the reasons that the parent did not conduct a 15 month interview? Will the lack of data for this particular parent or set of parents at 15 months affect the outcomes of this study if ACF chooses to include them in the 36 month follow-up?

A parent may not conduct a 15-month (first-follow-up) interview for many different reasons. The correct contact information for the parent may not be available, the parent may not respond to calls, or the parent may refuse to participate in the survey. As parents’ circumstances and perceptions of surveys and the BSF program can change, they may respond to one wave of a survey but not to another.

Our experience is that although the response rate to the second wave of a survey is much lower among those sample members who did not respond to the first wave of a survey than among those sample members who did respond to the first wave, some non-responders to the first wave do respond to later waves. Hence, attempting to interview all sample members for the 36-month second follow-up interview—including those who did not respond to the 15-month interview—will increase the total number of completed interviews and the overall response rate.

We will attempt to complete the 36-month survey with those who did not complete the 15-month survey because most analyses of the 36-month interview data will not require any data from the 15-month interview and hence would benefit from the additional data provided by the 15-month interview non-responders. Further, the 36-month interview collects some information, such as data on marriage formation and dissolution, that can be used to determine whether parents were married or not at 15 months after random assignment, even if they did not complete the 15-month survey. Hence, data from the 36-month survey can enhance the analysis of marriage at 15 months, by providing information on those who did not respond to the 15-month survey. Further, a few questions pertaining to baseline events or characteristics are asked on the 15-month survey. If the parent did not respond to the 15-month interview, these questions are asked on the 36-month interview. This approach ensures that we have these data for parents who respond to either of the follow-up surveys.

A response to the 36-month survey when there was no response to the 15-month survey will not negatively affect the analyses. There are standard analysis techniques that will be applied to ensure that, even when there is survey non-response, data can be generalized to the full sample. Specifically, the contractor will adjust the 15-month survey data and 36-month survey data separately for survey non-response, so that the findings from respondents of each survey can be generalized to the full sample of parents. Further, the contractor will also test the robustness of this analytical comparison by examining the sensitivity of the 36-month findings when restricting the 36-month sample to only those parents who also responded to the 15-month survey.

(12) pgs. 38-40: What is the likelihood of one or both parents saying that they interact with their child fairly frequently and have a ‘close relationship’, even if this is not the case? Similarly, on pages 41-42, the parent is asked about their disciplinary habits and may potentially downplay how he/she disciplines the child. How does ACF intend to verify—as best as it can—the truthfulness of the respondents answers?

The question of whether self-report results in primarily “socially desirable” responses is a common issue in survey analyses. One way this is addressed is through examination of the extent of variation of responses to the selected items in other surveys that have used the same or very similar items. The items relating to the quantity and quality of the parent-child relationship on pages 38-40 have been included in surveys in numerous large scale studies including Fragile Families, the Early Head Start Research and Evaluation Project, the Early Childhood Longitudinal Study-Birth Cohort (ECLS-B), the New Chance evaluation, and the Jobs Opportunity and Basic Skills programs evaluation of the National Evaluation of Welfare-to-Work Strategies (JOBS-NEWSS). Responses to these items in previous studies have shown variation, indicating that parents do not all provide the same response to these items (for example, although reading to a child on a daily basis is known to have many benefits, not all parents respond that they read to their child on a daily basis). We, therefore, have confidence that the items will generate variation in the BSF survey.

Similarly, the items relating to disciplinary habits on pages 41-42 are drawn from the Conflict Tactic Scale: Parent Child Version (CTSPC; Straus et al., 2003). The authors of the CTSPC offer evidence of variation in parental responses to these and similar items (for example, see Wauchrope & Straus, 1990). Researchers have not found that parent’s self-report about disciplinary habits is affected by tendencies to provide socially desirable responses (for example, Sugarman & Hotaling, 1995).

The in-home component will provide us the opportunity to obtain independent assessment on some measures. For example, the Three-Bag parent-child interaction task will be coded by independent observers examining constructs related to the quality of the parent-child relationship including aspects of supportive parenting (for example, warmth) and unsupportive parenting (for example, intrusiveness), and these codes can be examined in relation to parent responses to the items on relationship quality on page 40.

(13) Pg. 44: References to asthma. Please provide a detailed explanation as to how asthma is an appropriate outcome measure for the purpose of this study.

As discussed in our response to Question 3, in the interest of examining child development outcomes including outcomes in the physical domain, the presence and severity of asthma before age three is included as a proxy for child health (along with a standard question on the child’s general health status, CH1, that has been used in other large studies including the Early Head Start Research and Evaluation Project, ECLS-B, and Fragile Families).

There are also indications from the Fragile Families study that the presence and severity of asthma early in childhood is associated with a number of factors that the BSF intervention may directly or indirectly influence including: the family’s socio-economic status, the presence of stressful live events in the home, caregivers’ experience of stress, and parent relationship status.

(14) page 53: The question about the spouse/partner lists a mix of positive and negative behaviors.  Conversely, the question on about the child page 45 lists only positive behaviors while the question about the child on page 46 lists only negative behaviors.  The question on page 55 contains all positive behaviors about the spouse/partner except one.  Given that all of these are rather long lists that could lend themselves to “straight lining,” why don’t the last two sets of items use the mixed approach as does the first question?

Both the positive child behavior items on page 45 and the negative child behavior items on page 46 are drawn from standard scales [the Social Interaction subscale of the Preschool and Kindergarten Behavior Scales-Second Edition (PKBS-2; Merrell 2002) and the Behavior Problems Index (BPI; Zill, 1985) respectively]. The intention of each scale is unique. The PKBS-2 is used to assess the construct of empathy and the BPI is used to measure externalizing and internalizing behavior problems in children. In an effort to maintain the psychometric integrity of these scales, we have not changed the arrangement of items.

The couple relationship items on page 55 follow the same arrangement as in the BSF 15-month follow-up survey. An analyses of response frequencies from that survey does not indicate evidence of “straight lining.” Further, to help avoid “straight lining” during long item sets such as those on pages 45, 46 and 55, phone interviewers are trained to repeat the response categories for approximately every third item to allow these categories to remain salient for the respondent.

(15) pg. 108: Is it appropriate to interview one parent right after the other, especially—if for the sake of a person’s safety and well-being—it is imperative that their partner not know who they were speaking with?

Given the sensitive nature of couples’ relationships, attention to safety is a priority in the BSF study. Sample members are only enrolled as a couple and couples are only accepted into the sample if both members of the couple sign a consent form knowing that both are agreeing to participate in the study and will later be interviewed. Further, before enrollment, program sites screen couples in both the treatment and control groups for indications of domestic violence, and couples that fail the screen are excluded from the study. During the intervention, program staff continue to assess couples in the treatment group for indications of domestic violence. These features of the overall evaluation design limit risks associated with the survey administration.

In addition, care has been taken in the design of the telephone interview instrument to protect privacy of respondents and thus avoid interruptions and disruptions from anyone else who might be present. All survey items have been carefully selected so that sample members do not articulate sensitive information in their responses. This means that another person who is in the room cannot discern what the responses mean or the questions the respondent is answering. For example, rather than being asked an open-ended question about relationship quality, the sample member is asked to respond by agreeing or disagreeing with a series of statements that are read to the sample member over the phone. This kind of protection is essential, because even if we attempted to administer interviews only when the respondent is alone, there would certainly be instances in which that condition cannot be achieved.

Interviewers are also trained to protect sample members’ privacy and safety in several additional ways:

• Before proceeding with an interview, interviewers ask the sample member if this is a convenient time to begin. If the sample member indicates that it is not a good time (for any reason), the interviewer will accommodate by providing the sample member with the option of rescheduling the interview or calling a toll-free call-in number to complete the survey at a time that is more convenient for them. One reason could be that the sample member feels some anxiety about responding in the presence of others (partner, other family member, or others).

• When telephone interviewers introduce themselves to the sample member on the phone, they explain that the sample member does not have to answer any questions that make him or her feel uncomfortable. During the interview, if the interviewer perceives that the sample member is uncomfortable, the interviewer will remind the sample member that that he or she does not have to respond to any question. Further, all interviewers and locators are trained to listen for cues that indicate if it may not be a good time for the respondent or that he/she might not be able to answer questions freely. In such situations, the interviewers are trained to offer the respondent the option of rescheduling the interview for a different time or having the respondent call a toll-free number to complete the interview at their convenience.

• Thus the interviewer gives repeated signals to the sample member that it is acceptable to stop the interview and continue under more favorable circumstances.

• Finally, if at any time a partner or spouse requests information about the other member of the couple, interviewers and locators are trained to politely refuse and state that it would violate the privacy of the other person.

(16) Neither the parental survey nor the child assessment protocol has an area to indicate whether the child has been diagnosed or shows signs/symptoms of a physical and/or mental ailment that would effect the child’s interactions with his/her parents, as well as his/her performance on the direct assessment. How does ACF plan to address this potential issue that would affect the observational aspects of the study?

In-home observation staff will be instructed to note signs of physical and mental ailments that could affect data quality of the in-home direct assessment. However, with regard to the implications for the study, because couples are randomly assigned to the BSF intervention or control group, we expect that the incidence of physical and mental ailments will be randomly distributed across the two groups being compared and will therefore not differentially affect impacts.

9 02/04/21 5:57 PM