PACT follow up supporting statement A - 11.22.13 - clean

U.S. Department of Health and Human Services Administration for Children and Families Office of Planning, Research, and Evaluation Aerospace 7th Floor West 901 D Street, SW Washington DC 20447 Project Officers: Nancye Campbell and Seth Chamberlain

Parents and Children Together (PACT) Evaluation (0970-0403): OMB Supporting Statement for the Follow-up Surveys Part A: Justification September 2013

CONTENTS

1. Circumstances Making the Information Collection Necessary 1

a. Background 1

b. Overview of the Evaluation 2

c. Prior Submissions and Current Request 3

2. Purpose and Use of the Information Collection 6

3. Use of Improved Information Technology 6

4. Efforts to Identify Duplication and Use of Similar Information 7

5. Impact on Small Businesses or Other Small Entities 7

6. Consequences of Not Collecting Information or Collecting Information Less Frequently 7

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 7

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency 8

9. Explanation of Any Gift to Respondents 8

10. Assurance of Privacy Provided to Respondents 8

11. Justification for Sensitive Questions 9

12. Estimates of Annualized Burden Hours and Costs 13

13. Estimates of Other Total Cost Burden to Respondents and Record Keepers 17

14. Cost to the Federal Government 17

15. Explanation for Program Changes or Adjustments 17

16. Plans for Tabulation and Publication and Project Time Schedule 17

a. Plans for Tabulation 17

b. Time Schedule and Publications 19

17. Reason(s) Display of OMB Expiration Date Is Inappropriate 20

18. Exceptions to Certification for Paperwork Reduction Act Submissions 20

REFERENCES 21

1. Circumstances Making the Information Collection Necessary

This information collection request (ICR) is for clearance to collect information for the Parents and Children Together (PACT) Evaluation, which is evaluating a subset of Responsible Fatherhood (RF) and Healthy Marriage (HM) grants authorized under the Claims Resolution Act of 2010 (Public Law 111-291).

The evaluation is being undertaken by the U.S. Department of Health and Human Services, Administration for Children and Families (ACF), and is being implemented by Mathematica Policy Research and its partner, ICF International.

a. Background

The past several decades have witnessed sweeping changes in family structure. In 1980, 77 percent of children lived with two married parents; by 2010, this figure had fallen to only 66 percent (U.S. Census Bureau 2011). Families have also become more complex: nearly one in five fathers now has children with more than one woman (Guzzo and Furstenberg 2007). While many children do well living with only one parent, research suggests that on average children do better when they have two involved parents (McLanahan 2009). These changes in family structure, their attendant consequences for children, and recent changes in welfare policy set the stage for new investments in programs aimed at strengthening families and in policy research on fatherhood and marriage.

As one response by the federal government, the Deficit Reduction Act of 2005 created the RF and HM grant programs, authorizing ACF to provide up to $50 million for RF grants and $100 million for HM grants each year from 2006 to 2010. This funding represented an “unprecedented financial commitment by the federal government to support marriage and fatherhood programs” (U.S. Government Accountability Office 2008). Under this act, awards were made to 226 grantees to provide three RF services (responsible fatherhood classes, marriage and relationship education, and economic stability services) or one or more of eight HM services, which include parenting classes, marriage and relationship education, and economic stability services.

The Claims Resolution Act of 2010 reauthorized this grant program, evenly allocating the $150 million between RF and HM funding ($75 million for each). New three-year grants were awarded in September 2011 to 55 RF and 60 HM grantees. The PACT Evaluation will provide documentation of the operations of a subset of these grant programs, the characteristics and life experiences of those who apply for services in the selected programs, and, for some study grantees, an assessment of the impact of the programs on a range of outcomes.

Few rigorous studies of RF programs have been conducted to date. Of the 60 impact and implementation studies of programs for low-income fathers included in a recent systematic review of the evidence (Avellar et al. 2011), only 13 used a rigorous evaluation design. These rigorous program evaluations generally showed at least one statistically significant favorable impact, but most did not result in a compelling pattern of positive impacts.

With regard to HM evaluations, beginning in 2002, ACF sponsored two large-scale, multi-site evaluations utilizing random assignment of enrolled couples to evaluate the effects of programs offering healthy relationship and marriage skills and supportive services to unmarried parents having a child together (Building Strong Families [BSF]) and of similar services for low-income married couples with children (Supporting Healthy Marriage [SHM]). ACF also sponsored an evaluation of community-wide healthy marriage programming (Community Healthy Marriage Initiative [CHMI]) utilizing a quasi-experimental design. Interim results from the BSF evaluation showed no significant differences when data from all eight programs were aggregated; however, in site-specific analyses, a positive pattern of impacts in one site and a negative pattern in another were observed (Wood et al. 2010). After three years, across the eight programs, BSF had no effect on the quality and stability of the couples’ relationship, the couples’ co-parenting relationship, family stability, or economic well-being of children. Impact analyses also identified a modest reduction in children’s behavior problems and small negative effects on some aspects of father involvement (Wood et al. 2012). Interim impact findings from the SHM evaluation showed a consistent pattern of small positive effects after twelve months on aspects of the couples’ relationship, including marital happiness, warmth and support, and positive communication; however, SHM did not affect marital stability after twelve months (Hsueh 2012). Final results from SHM, reporting on impacts 30 months after program enrollment, are forthcoming. Results from the CHMI evaluation indicated that 24 months after implementation of healthy marriage services in target communities, there were no significant differences in participation in healthy marriage services or awareness of healthy marriage messaging (Bir et al. 2012).

b. Overview of the Evaluation

The PACT Evaluation is addressing a number of research questions, including:

• What are the net impacts of the interventions on key outcomes (e.g. relationship status; relationship or marital stability; quality of relationships; attitudes and expectations regarding their relationship or marriage; parenting attitudes and behaviors; measures of child well-being and development (e.g., cognitive, social, emotional, health); adult well-being; and economic outcomes for families)? What are the net impacts for different sub-groups within the research sample?

• What are the issues and challenges in designing, implementing and operating interventions to meet stated program goals and objectives?

• What are the characteristics of lead organizations and partners?

• What are the features and characteristics of the interventions and the context within which they are provided? Are they operated as planned?

• What are the characteristics of individuals/couples targeted by interventions?

As indicated in prior submissions, work under PACT will be carried out in stages with different types of information collection in each stage. Thus, clearance is being requested in stages as work progresses. As previously presented, and as currently detailed in Appendix A, the PACT Evaluation uses three interrelated evaluation strategies: (a) experimental impact; (b) implementation; and (c) qualitative evaluations.

These three strategies are combined into two types of multi-component evaluations:

• impact evaluations, complemented with implementation and qualitative evaluations, will be conducted in a subset of grantees to provide rigorous estimates of the effectiveness of the studied programs and information about their operating contexts; and

• implementation and qualitative evaluations (without impact evaluations) will be conducted in a separate subset of grantees which present some particular feature of program design or target population that warrants detailed study, but which would not support an impact evaluation (for example, if power analyses indicate that sample size is inadequate).

For ease of communication, as in the prior ICRs these two types of evaluations are called “impact evaluations” and “implementation/qualitative only evaluations,” respectively.

c. Prior Submissions and Current Request

Tables A.1 and A.2 list the data collection instruments approved-to-date, as well as those for which we seek approval now. The names and numbers for these instruments are carried throughout the Supporting Statements and Appendices.

The first submission, approved on April 20, 2012, provided clearance for discussions with grantees [instrument (1)] that may be considered as sites in the evaluation. OMB Control number 0970-0403 was set for the evaluation.

The second submission, approved on October 31, 2012, provided clearance for the introductory script, baseline survey, and management information system to be used with RF programs selected to be in the evaluation [instruments (2), (3), and (6)].

The third ICR was originally submitted in February 2013. On July 8, 2013, clearance was provided for the (4) HM introductory script, (5) HM baseline survey, and (7) HM study management information system to be used with HM programs selected to be in the evaluation. On August 27, 2013 clearance was provided for nine additional data collection instruments for use in the impact evaluations [instruments (8)-(15)]; and two new instruments for use in implementation/qualitative only evaluations [(16)-(18)].

This fourth ICR, requests clearance for a follow-up survey for use with RF programs selected for the evaluation [instrument (19)] and a follow-up survey for use with HM programs selected for the evaluation [instrument (20)]. Both surveys will collect data for the impact evaluations.

Table A.1. PACT Impact Evaluation (Experimental Impact Complemented with Implementation and Qualitative Evaluations) Data Collection Instruments

	Site Selection	Experimental Impact	Implementation		Qualitative
			MIS	Additional Implementation Data Collection Instruments1
Responsible Fatherhood Grantee Evaluation	(1) Selecting study grantees (discussion with program and partner organization staff) – APPROVED APRIL 20, 2012	Baseline: (2) Introductory script (for program staff to discuss with program applicants) (3) Baseline survey (for study participants) BOTH APPROVED OCTOBER 31, 2012 (19) Follow-up RF survey (for study participants): SUBMITTED IN THIS ICR	(6) RF study MIS – APPROVED OCTOBER 31, 2012	(8) Semi-structured interview topic guide (for program staff) (9) On-line survey (for program staff) (10) Telephone interview guide (for program staff at referral organizations) (11) On-line Working Alliance Inventory (for program staff and participants) (12) Focus group discussion guide (for program participants) (13) Telephone interview guide (for program dropouts) APPROVED AUGUST 27, 2013	(14) In-person, in-depth interview guide (for program participants) (15) Telephone check-in guide (for program participants) APPROVED AUGUST 27, 2013
Healthy Marriage Grantee Evaluation		Baseline: (4) Introductory script (for program staff to discuss with program applicants) (5) Baseline survey (for study participants) BOTH APPROVED JULY 8, 2013 (20) Follow-up HM survey (for study participants): SUBMITTED IN THIS ICR	(7) HM study MIS –APPROVED JULY 8, 2013		Not anticipated

¹ Note that instruments (8) through (13) contain variations which are appropriate to RF and HM programs.

Table A.2. PACT Implementation/Qualitative Only Data Collection Efforts

	Site Selection	Implementation/Qualitative Instruments
Responsible Fatherhood Grantee Evaluation	(1) Selecting study grantees – APPROVED APRIL 20, 2012	“Hispanic RF sub-study” (for RF grantees with a focus on Hispanic populations) (16) Semi-structured interview topic guide (for program staff) (17) Focus group discussion guide (for program participants) (18) Questionnaire (for program participants in focus groups) APPROVED AUGUST 27, 2013
Healthy Marriage Grantee Evaluation		To Be Determined

2. Purpose and Use of the Information Collection

The information to be obtained through the PACT Evaluation is critical to understanding the current field of RF and HM programs—the services they provide, the context in which they are provided, the experiences of their participants, and their effectiveness. This information can be used to inform decisions related to future government investments in this kind of programming as well as the design and operation of such services.

The purpose of each approved information collection instrument was discussed in previous ICRs. The purpose of each newly proposed information collection instrument is described below – numbering matches that in Tables A.1 and A.2.

• (19) Follow-up survey of study participants in RF programs. The RF follow-up survey, administered by telephone, will collect data on outcomes for fathers in the program and control groups in the impact evaluation of RF programs. The survey will be administered approximately 12 months after the father was randomly assigned. The surveys will collect data on the fathers’ involvement with their children, the quality of their relationship with the children and their parenting; their relationship with the mother of one of their children; the fathers’ employment, earnings, and job readiness, criminal justice involvement, mental health, and attitudes; and the receipt of services similar to those offered by the RF program. The data collected by this survey will be used to estimate impacts of the RF programs.

• (20) Follow-up survey of study participants in HM programs. The HM follow-up survey, administered by telephone, will collect data on outcomes for couples in the program and control groups in the impact evaluation of HM programs. The survey will be administered approximately 12 months after the couple was randomly assigned. The surveys will be administered separately to the male and female member of the couple. Data will be collected on the status and quality of the couples’ relationships, parenting behaviors, employment, earnings, and job readiness, mental health, attitudes and the receipt of services similar to those offered by the HM program. The data collected by this survey will be used to estimate impacts of the HM program.

Appendix D and Appendix Eprovide a question-by-question justification for the questions included in the follow-up survey instruments.

3. Use of Improved Information Technology

The follow-up survey will use computer assisted telephone interviewing (CATI). The CATI system reduces respondent burden by automating skip logic and question adaptations that allow interviews to progress from question to question without having to refer back to previous answers to determine whether a follow-up question should be asked or phrasing should be adjusted to properly apply to a respondent’s circumstances. The evaluator will preload data from the baseline survey and embed appropriate skip logic to further reduce respondent burden. CATI minimizes interviewer error through control over the question logic, consistency checks, and probes, and it eliminates the need to call back respondents to obtain missing data since inconsistencies in responses are corrected during the interview process.

The CATI system facilitates survey tracking because of its capability to produce timely reports on screening and interview outcomes, nonresponse rates, and interviewer productivity. CATI improves interviewer supervision through the use of audio and video monitoring. The autodialer, linked to the CATI system, almost eliminates dialing error and improves interviewer efficiency. The automated call scheduler manages interviewer assignments by scheduling and rescheduling calls to ensure that they are made according to the optimal calling patterns, that all appointments are kept, and that cases requiring special attention or fluency in languages other than English are routed to the appropriate interviewers.

4. Efforts to Identify Duplication and Use of Similar Information

There are no other evaluations of ACF-funded RF and HM grantees ongoing, thus the PACT Evaluation is not duplicative of other efforts. Likewise, there are no other sources of information that would allow us to answer the specific questions regarding the effectiveness of ACF-funded RF and HM programs. Within the evaluation, we do propose to use measures (e.g., in the baseline survey for HM program participants) that have successfully been used in prior studies involving similar populations and programs.

5. Impact on Small Businesses or Other Small Entities

No small businesses that are not RF/HM grantees or their partners are expected to be involved in data collection. In the case that RF/HM grantees or their partners are small entities, instruments have been tailored to minimize burden and only collect critical evaluation information.

6. Consequences of Not Collecting Information or Collecting Information Less Frequently

The purpose of each information collection instrument included in this submission is described in Item A2, above. Not collecting information using these instruments would limit the government’s ability to document and report on the kinds of activities implemented with federal funds, the characteristics and views and life circumstances of fathers and couples who seek these services, and the effectiveness of the programs.

The follow-up surveys collect some information that is similar to the information collected on the baseline survey. Collecting data on an outcome measure, such as father involvement, both before random assignment (i.e., at baseline) and also at follow-up, increases the precision of the estimates of the impacts of the program on those outcomes.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances for the proposed data collection.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

In accordance with the Paperwork Reduction Act of 1995, the public was given an opportunity to review and comment through the 60-day Federal Register Notice, published on May 28, 2013 (78FR 102, document number 2013-12588, pp. 31942-31943). A copy of this notice is attached as Appendix F. The notice provided certifis for public comment. No substantial changes in burden are proposed over those proposed in the 60-Day FRN. One comment was received objecting to the entire undertaking; it did not include any specific comments on the proposed information collection.

9. Explanation of Any Gift to Respondents

We propose to provide a $25 appreciation to participants for completing the follow-up survey (see Table A.3.). Studies have shown that providing an appreciation may decrease the potential for nonresponse bias (Singer and Kulka 2002). Studies show that appreciation payments are effective at decreasing nonresponse rates for people with lower educational levels (Berlin et al. 1992) and low-income and nonwhite populations (James and Bolstein 1990), which helps to provide better estimates. Note, the amount proposed for the follow-up surveys is a slight increase from the amount ($10) provided to participants at baseline for completing follow-up surveys. Compared to at baseline, slightly larger gifts are offered to respondents for the follow-up surveys to ensure high response rates. Attrition from surveys tends to increase overtime due to mobility of participants, as well as study fatigue; higher incentives are needed to continue to ensure participant responses. This amount is consistent with other studies to decrease nonresponse bias (e.g., Building Strong Families).

Table A.3. Proposed Appreciation for Participants for Data Collection Involvement

Data Collection Activity	Length of Activity (minutes)	Proposed Appreciation (per Participant)
(19) Follow-up RF survey (for RF study participants)	45	$25
(20) Follow-up RF survey (for RF study participants)	45	$25

10. Assurance of Privacy Provided to Respondents

Respondents will be informed that the identifying information they provide will be kept private as provided by the Confidentiality Certificate issued by the Eunice Kennedy Shriver National Institute of Child Health and Human Development on January 3, 2013 (a copy is provided in Appendix G). All consent forms that are given to study participants will include assurances that the research team will protect their privacy to the fullest extent possible under the law (the consent forms are presented in Appendix H and Appendix I). The interviewers will begin the survey by stating that participation in the survey is voluntary and that the respondent need not answer any questions that makes him or her feel uncomfortable.

The contractor will take the following specific measures to protect respondents’ privacy:

• Training interviewers in privacy procedures. The RF and HM follow-up surveys will be administered by telephone interviewers at the evaluator’s Survey Operations Center (SOC). Interviewers will be seated in a common supervised area. Interviewing staff will receive training that includes general SOC security and privacy procedures as well as project-specific training that includes explanation of the highly private nature of this information, instructions to not share it or any personally identifiable information (PII) with anyone not on the project team, and warnings about the consequences of any violations. After receiving training, these staff members sign privacy and nondisclosure agreements.

• Restricting and logging access to the sample management system (SMS). Some data elements from the baseline survey data will be entered into an SMS to locate sample members for the follow-up survey. This is a sequel server database housed on an encrypted server. A hierarchical architecture will be used to assign user rights to specific individuals who will be able to access the system and enter information only at their own location. All activity in the system will be logged. Unless otherwise required by ACF, the information stored in the SMS will be destroyed when no longer needed in the performance of the project.

In addition to these study-specific procedures, the evaluator has extensive corporate administrative and security systems to prevent the unauthorized release of personal records, including state-of-the-art hardware and software for encryption that meets federal standards and other methods of data protection (e.g., requirements for regular password updating), as well as physical security that includes limited key card access and locked data storage areas.

11. Justification for Sensitive Questions

Some sensitive questions are necessary in a study of programs designed to affect personal relationships and employment. In the follow-up surveys, all respondents will be informed that their identity will be kept private and that they do not have to answer questions that make them uncomfortable. Table A.4 describes the justification for the sensitive questions included in the RF follow-up survey and Table A.5 provides the justification for sensitive questions in the HM follow-up survey. Although these questions are sensitive, they have commonly, and successfully, been asked of respondents similar to those who will be in this study (for example, in the Fragile Families and Child Wellbeing Study, the Building Strong Families Study, and the Early Head Start Research Evaluation Project). Additionally, many of these questions are asked at baseline and have been previously approved by OMB (see Table A.4 and Table A.5 for specific notation of which questions have previously been approved by OMB).

Table A.4. Justification for Sensitive Questions in (19) RF Follow-Up Survey

Question Topic	Justification
Methods of discipline used with focal child by respondent	These items measure the use of mild to harsh disciplinary practices. These measures will enable us to determine whether the PACT sites’ emphasis on conflict management and parenting skills leads to a reduction in the use of harsh discipline techniques among participants. These items are drawn from the Conflict Tactics Scale: Parent Child Version (CTSPC; Straus et al. 2003). The CTSPC is well validated, shown to have good internal consistency, and has been used in large-scale longitudinal surveys, including the National Survey of Child and Adolescent Well-Being.
Earnings	A key goal of PACT RF programs is to improve couples’ economic stability. The outcomes of an individual employed when he/she enters the program may be very different than those of an individual who enters without employment. The survey asks whether the respondent worked in the past three months and, if so, the rate of pay and average weekly hours worked. This information will be used to calculate earnings. Questions on earnings are asked on many surveys including the Building Strong Families survey (Wood et al. 2010). In this survey, only 0.4 percent of mothers and 0.1 percent of fathers did not respond to the earnings questions. The RF Baseline Survey, previously approved by OMB, contains questions related to earnings.
Involvement with the criminal justice system	Recent research suggests that a history of incarceration and involvement with the criminal justice system may be fairly common among men in the PACT target population (Pearson et al. 2011). Incarceration has major negative effects on child and family well-being, including reducing the financial support and other types of support adults can provide to their partners, children, and families, thus, documenting the incidence is important. Further, because the PACT RF sites encourage men to become more responsible, we want to explore whether the programs had any effect on criminal involvement. Similar questions have been included in other large national studies, such as the Fragile Families and Child Wellbeing Study, the National Job Corps Study, and the Building Strong Families Study. In the Building Strong Families survey, nonresponse was less than 1 percent for these items. These questions are currently being asked of participants on the RF Baseline Survey and have previously been approved by OMB.
Symptoms of depression	Parental depression has been shown to have adverse consequences for child outcomes (Downey and Coyne 1990, Gelfand and Teti 1990). To measure depressive symptoms, we will use eight items from the Patient Health Questionnaire (PHQ-9), which was designed as a diagnostic instrument for depression but can also be used to measure subthreshold depressive disorder in the general population (Martin et al. 2006). The PHQ-9 has been shown to be reliable and valid in diverse populations and has been used in clinical settings to measure symptom improvement and monitor treatment outcomes (Kroenke, Spitzer, and Williams 2001; Löwe et al. 2004). Findings from telephone administrations of the instrument have been shown to be similar to in-person assessments (Pinto-Meza et al. 2005). The PHQ-8 includes eight of the nine items from the PHQ-9; it has been shown to be a useful measure of depression in population-based studies (Kroenke et al. 2009).

Table A.5. Justification for Sensitive Questions in (20) HM Follow-Up Survey

Question Topic	Justification
Methods of discipline used with focal child by respondent	These items measure the use of mild to harsh disciplinary practices. These measures will enable us to determine whether the PACT sites’ emphasis on conflict management and parenting skills leads to a reduction in the use of harsh discipline techniques among participants. These items are drawn from the Conflict Tactics Scale: Parent Child Version (CTSPC; Straus et al. 2003). The CTSPC is well validated, shown to have good internal consistency, and has been used in large-scale longitudinal surveys, including the National Survey of Child and Adolescent Well-Being.
Whether respondent or his or her PACT partner have cheated	Infidelity has been found to be a major obstacle to marriage for unwed parents (Edin and Kefalas 2005). The curricula used by the PACT sites addresses this in different ways, including discussing the importance of fidelity and trust in building healthy relationships and marriage. Several large surveys have included similar questions concerning infidelity, such as the Study of Marital Instability Over the Life Course, the Louisiana Fragile Families Study, and the Baseline Survey of Family Experiences and Attitudes in Florida. These questions were also used in the Building Strong Families 15- and 36-month follow-up surveys and had low nonresponse rates. Note: these questions are also included in the HM Baseline survey and were approved by OMB.
Whether respondent has been isolated, controlled, or physically or sexually assaulted by his or her PACT partner	A goal of PACT is to improve relationship quality so that participants have healthy relationships. A key characteristic of a healthy romantic relationship is one that is not marred by violence or intimidation. These questions are drawn from the Supporting Healthy Marriage survey and the revised Conflict Tactics Scale (CTS2), the most widely used tool for measuring domestic violence in research studies (Straus and Douglas 2004). The CTS2 has been well validated and shown to have good internal consistency (Straus et al. 1996). Versions of these CTS questions have been used on many surveys, including the National Family Violence Survey, the National Violence Against Women Survey, the Building Strong Families 15- and 36-month follow-up surveys, and surveys conducted in six states as part of the ASPE-funded TANF Caseload Project. The SHM questions were successfully administered to a large, multi-site sample of low-income married couples with children. These items were also previously approved by OMB, as they are on the HM Baseline survey.
Earnings	A key goal of PACT HM programs is to improve couples’ economic stability. The outcomes of an individual employed when he/she enters the program may be very different than those of an individual who enters without employment. The survey asks whether the respondent worked in the past three months and, if so, the rate of pay and average weekly hours worked. This information will be used to calculate earnings. Questions on earnings are asked on many surveys including the Building Strong Families survey (Wood et al. 2010). In this survey, only 0.4 percent of mothers and 0.1 percent of fathers did not respond to the earnings questions. Questions related to earnings are currently on the HM Baseline Survey and have previously received OMB approval.
Symptoms of depression	Parental depression has been shown to have adverse consequences for child outcomes (Downey and Coyne 1990, Gelfand and Teti 1990). To measure depressive symptoms, we will use eight items from the Patient Health Questionnaire (PHQ-9), which was designed as a diagnostic instrument for depression but can also be used to measure subthreshold depressive disorder in the general population (Martin et al. 2006). The PHQ-9 has been shown to be reliable and valid in diverse populations and has been used in clinical settings to measure symptom improvement and monitor treatment outcomes (Kroenke, Spitzer, and Williams 2001; Löwe et al. 2004). Findings from telephone administrations of the instrument have been shown to be similar to in-person assessments (Pinto-Meza et al. 2005). The PHQ-8 includes eight of the nine items from the PHQ-9; it has been shown to be a useful measure of depression in population-based studies (Kroenke et al. 2009).

12. Estimates of Annualized Burden Hours and Costs

The estimated reporting burden for the data collection instruments included in the PACT study is presented in Tables A.6 through A.8.

For cost calculations in all of the following tables, we estimate that the average hourly wage for staff at the grantee organizations is the average hourly wage of “social and community service managers” taken from the U.S. Bureau of Labor Statistics, National Compensation Survey, 2010 ($27.86). The average hourly wage of program applicants is estimated from the average hourly earnings ($4.92) of study participants in the Building Strong Families Study (Wood et al. 2010). These average hourly earnings are lower than minimum wage because many study participants were not working. We expect that to also be the case for many PACT study participants.

Previously Approved Burden

Table A.6 summarizes burden and costs for previously approved instruments. The burden and costs for all these instruments are annualized over three years, meaning the total number of respondents over the three year period has been divided by three to determine the annual number of respondents and calculate annual burden estimates.

Table A.6. PREVIOUSLY APPROVED AND ONGOING – Estimates of Burden and Costs for the PACT Evaluation

Activity/Respondent	Annual Number of Respondentsa	Number of Responses per Respondent	Average Burden per Response (hours)	Total Annual Burden Hoursa	Average Hourly Wage	Total Annualized Cost
SITE SELECTION
(1) Selecting study grantees1 Discussions/ program and partner organization staff	50	1	1	50	$29.34	$1,467
IMPACT (COMPLEMENTED WITH IMPLEMENTATION AND QUALITATIVE EVALUATIONS)
Responsible Fatherhood Grantee Impact Evaluation
(2) Introductory script2
a) Program staff	30	70.2	0.167	351	$27.86	$9,779
b) Program applicants	2,105	1	0.167	351	$4.92	$1,726
(3) Baseline survey2
Study participants	2,000b	1	0.5	1,000	$4.92	$4,920
Healthy Marriage Grantee Impact Evaluation
(4) Introductory script3
(a) Program Staff	30	70.2	0.167	351	$27.86	$9,779
(b) Program Applicants	4,210	1	0.167	703	$4.92	$3,454
(5) Baseline survey3
Study participants	4,000 c	1	0.5	2,000	$4.92	$9,840
Responsible Fatherhood and Healthy Marriage Grantee Implementation Evaluation
(6) RF study MIS2
Program staff	30	2,533	0.03	2,533	$27.86	$70,569
(7) HM study MIS3
Program staff	30	3,400	0.03333	3,400	$27.86	$94,724
(8) Semi-structured interview topic guide4
Program staff	250	2	1.033	517	$27.86	$14,404
(9) On-line survey4
Program staff	250	2	0.5	250	$27.86	$6,965
(10) Telephone interviews (with staff at referral organizations) 4
Program staff at referral organizations	50	1	0.5	25	$27.86	$697
(11) On-line Working Alliance Inventory4
1) Program staff	50	20	0.167	167	$27.86	$4,653
2) Program Participants	1,000	1	0.167	167	$4.92	$822
(12) Focus group guide4
Program participants	600	1	1.5	900	$4.92	$4,428
(13) Telephone interviews 4
Program participants (program dropouts)	150	1	0.25	38	$4.92	$187
Responsible Fatherhood Grantee Qualitative Evaluation
(14) Guide for in-person, in-depth interviews4
Study participants	32	3	2	192	$4.92	$945
(15) Check-in call guide4
Study participants	32	4	0.167	21	$4.92	$103
IMPLEMENTATION/QUALITATIVE ONLY
Responsible Fatherhood Grantee Implementation Evaluation – Grantees with a Focus on Hispanic Populations
(16) Semi-structured interview topic guide4
Program staff	42	1	1.5	63	$27.86	$1,755
(17) Focus group guide4
Program participants	20	1	1.5	30	$4.92	$248
(18) Questionnaires4
Program participants in focus groups	20	1	0.333	7	$4.92	$34
Total				13,116		$241,149

^a Burden estimates are annualized over three years.

^b Note that the 2,000 “study participants” are part of the 2,105 “program applicants,” as the study participants will all begin as program applicants. (Five percent of program applicants are not expected to agree to participate in the study and complete the baseline survey, thus there are 5 percent more program applicants than study participants.) The 2,000 study participants do not represent 2,000 individuals in addition to the 2,105 program applicants.

^c Note that the 4,000 “study participants” are part of the 4,210 “program applicants,” as the study participants will all begin as program applicants. (Five percent of program applicants are not expected to agree to participate in the study, thus there are 5% more program applicants than study participants.) The 4,000 study participants do not represent 4,000 individuals in addition to the 4,210 program applicants.

¹ Approved April 20, 2012.

² Approved October 31, 2012.

³ Approved July 8, 2013.

⁴ Approved August 27, 2013

Burden for Data Collection Efforts Covered by this ICR

Table A.7 summarizes burden and costs for the two newly requested instruments associated with the impact study (which, as stated in A1, contains experimental impact evaluations complemented with implementation and qualitative strategies). The burden and costs for all these instruments are annualized over three years, meaning the total number of respondents over the three year period has been divided by three to determine the annual number of respondents and calculate annual burden estimates (see Appendix K for additional information on the calculation of burden).

Table A.7. CURRENT REQUEST – Estimates of Burden and Costs for the RF and HM Follow-up Surveys

Instrument Respondent	Annual Number of Respondentsa	Number of Responses per Respondent	Average Burden per Response (hours)	Total Annual Burden Hoursa	Average Hourly Wage	Total Annualized Cost
IMPACT
Responsible Fatherhood Grantee Impact Evaluation
(19) RF Follow-up survey
Study participants	1,600	1	0.75	1,200	$4.92		$5,904
Healthy Marriage Grantee Impact Evaluation
(20) HM Follow-up survey
Study participants	3,200	1	0.75	2,400	$4.92		$11,808
Total				3,600			$17,712

^a All burden estimates are annualized over three years.

Combined Total Burden

Table A.8 summarizes the total estimated reporting burden and costs for the previously approved and currently requested burden. If the current request is approved, 16,716 hours and $258,861 would be approved for the PACT study.

Table A.8. Estimate of Burden and Cost for the PACT Evaluation – TOTAL Burden Request

Data Collection	Total Annual Burden Hoursa	Total Annualized Cost
Previously Approved and Ongoing (from Table A.6)	13,116	$241,149
Current Request – Impact Evaluation (from Table A.7)	3,600	$17,712
Total	16,716	$258,861

^a Burden estimates are annualized over three years.

13. Estimates of Other Total Cost Burden to Respondents and Record Keepers

These information collection activities do not place any additional costs on respondents or record keepers.

14. Cost to the Federal Government

As reported in PACT’s initial ICR for field data collection, the total cost of the PACT study to the federal government is estimated to be $22,075,787. Since the study will last five years, the total cost over this three year request is $13,245,472 and the annualized cost to the federal government is $4,415,157.

15. Explanation for Program Changes or Adjustments

This submission is for additional data collection under the Parents and Children Together evaluation and therefore increases total burden under OMB Control number 0970-0403.

16. Plans for Tabulation and Publication and Project Time Schedule

a. Plans for Tabulation

The impact analysis will estimate the effectiveness of the RF and HM grantee programs in the evaluation. The goal of the impact analysis is to compare observed outcomes for program participants (fathers for RF programs and couples for HM programs) with outcomes for members of a control group who were not permitted to participate in the programs. We will use the experience of the control group as a measure of what would have happened to the program group fathers and couples in the absence of the program. Random assignment of fathers or couples to a program and a control group ensures that the two groups of couples do not initially differ in any systematic way on any characteristic. Any observed differences in outcomes between the program and control group couples can therefore be attributed to the program.

Differences of means or proportions in outcomes between the program and control group will provide unbiased estimates of the impacts of the program. More precise estimates will be obtained using regression models to control for random differences in the baseline characteristics of program and control group members. In their simplest forms, these models can be expressed by the following equation:

(1) Y = X΄ß + δ P + e,

where:

Y is an outcome variable

X is a vector of control variables (including an indicator for each site)

β is the vector of regression coefficient for the control variables

δ is the measure of the impact of the RF or HM program

P is an indicator that equals 1 for program group members and 0 for control group members

e is a random error term that is assumed to have a mean of zero conditional on X and P, and is interpreted as the unobserved factors that affect Y.

The statistical techniques used to estimate the regression‑adjusted impacts depend on the form of the dependent variable, Y. If the dependent variable is continuous, then ordinary least squares techniques will produce unbiased estimates of the parameter δ. However, if the dependent variable is binary—for example, whether the couple is married—then consistent parameter estimates can be obtained by using logit or probit maximum likelihood methods.

Control variables in the vector X will include any variables that may affect the outcome that are not affected by the intervention. Hence, X could include the characteristics of the father or couple for which data are collected on the baseline surveys [instruments (3) and (5)]. For example, X will include the extent to which the father is involved with his children at baseline (in the RF program models) and the quality of the couple’s relationship at baseline (in the HM program models).

We will estimate impacts for individual programs (as sample size allows) and for all the RF programs together and all the HM programs together. Estimates of the impacts by site will be obtained by introducing interaction terms in the regression model (1) that is the product of the program group indicator (P) and an indicator of membership in the grantee of interest (G_i):

(2) Y = X΄ß + δ P + γ_i* G _i * P + e.

The estimated impact of the program for grantee i is given by (γ_i + δ).

Some fathers may benefit from RF programs more than others and some couples may benefit from HM programs more than others. To address how the programs benefit different subgroups of the population, impacts for key subgroups of the population will be estimated. Subgroups of interest include those defined by characteristics at baseline. For the RF program models, these include demographic characteristics (such as age and race/ethnicity), measures of father involvement, employment, well-being, and criminal involvement at baseline. For the HM program models, these include demographic characteristics (such as age and race/ethnicity), measures of the quality of the couple’s relationship, as well as measures of wellbeing, and employment.

Estimates of impacts by subgroup will be obtained by introducing an interaction term in regression model (1) that is the product of the program group indicator (P) and an indicator of membership in the subgroup of interest (Sub_i):

(3) Y = X΄ß + δ P + θ_i* Sub_i * P + e.

The estimated impact of the program for members of the subgroup of interest is given by (δ + θ_i ).

b. Time Schedule and Publications

The PACT study is expected to be conducted over a five-year period beginning on September 30, 2011. Discussions with sites began after April 20, 2012, when the first ICR approval was received (for such discussions to begin). This ICR is for three years. Table A.9 provide the expected schedule for the study.

Table A.9. Schedule for the Evaluation

Activity	Date
Selection of grantees for inclusion in evaluation Intake period for impact evaluation Report on early findings on implementation Implementation mid-term report Report on Hispanic RF Sub-study First report on qualitative interview analysis Second report on qualitative interview analysis Final implementation report Final impact report Final report on qualitative interview analysis	Spring 2012 to spring 2013 December 2012–August 2014 Fall 2013 Spring 2014 Winter 2013/2014 Winter 2014/2015 Fall 2015 Winter 2016 Summer–fall 2016 Summer–fall 2016

In addition to the planned reports described above, PACT provides opportunities for analyzing and disseminating additional information through special topics reports and research or issue briefs on an as requested basis. We will also provide a restricted use data file for others to replicate and extend our analysis.

17. Reason(s) Display of OMB Expiration Date Is Inappropriate

All instruments will display the expiration date for OMB approval.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

REFERENCES

Avellar, Sarah M., Robin Dion, Andrew Clarkwest, Heather Zaveri, Subuhi Asheer, Kelley Borradaile, Megan Hague Angus, Timothy Novak, Julie Redline, and Marykate Zukiewicz. “Catalog of Research: Programs for Low-Income Fathers.” Washington, DC: The evaluator Policy Research, June 2011.

Axinn, William G., and Lisa D. Pearce. “Innovations in Life History Calendar Applications.” Social Science Research, vol. 28, 1999, pp. 243–264.

Berlin, Martha, Leyla Mohadjer, Joseph Waksberg, Andrew Kolstad, Irwin Kirsch, D. Rock, and Kentaro Yamamoto 1992 An experiment in monetary incentives. Pp. 393-398 in Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association.

Bir, Anupa, Robert Lerman, Elise Corwin, Brian MacIlvain, Allison Beard, Kelly Richburg, and Kevin Smith (2012). Impacts of a Community Healthy Marriage Initiative, OPRE Report # 2012–34A, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Coffey, A., B. Holbrook and P. Atkinson (1996) 'Qualitative Data Analysis: Technologies and Representations', Sociological Research Online, vol. 1, no. 1, <http://www.socresonline.org.uk/ 1/1/4.html>.

Dion, Robin M., Sarah A. Avellar, and Elizabeth J. Clary. “The Building Strong Families Project: Implementation of Eight Programs to Strengthen Unmarried Parent Families.” Final report submitted to the U.S. Department of Health and Human Services, Office of Planning, Research and Evaluation, Administration for Children and Families. Washington, DC: The evaluator Policy Research, May 2010.Glasner, Tina, and Wander van der Vaart. “Applications of Calendar Instruments in Social Surveys: A Review.” Quality & Quantity, vol. 43, 2009, pp. 333–349.

Downey G., and J.C. Coyne. “Children of Depressed Parents: An Integrative Review.” Psychological Bulletin, vol. 108, 1990.

Edin Kathy, and Maria Kefalas. Promises I Can Keep: Why Poor Women Put Motherhood Before Marriage. Berkeley, CA: University of California Press, 2005.

Edin, Kathryn, and Al Young. September 27, 2012. Personal communication.

England, Paula, and Kathryn Edin (eds). Unmarried Couples with Children. The Unfolding Lives of New Unmarried Urban Parents. New York, NY: Russell Sage Foundation, 2007.

Fagan, J, Palkovitz, R, Roy, K, and Farrie, D (2009).
Pathways to Paternal Engagement: Longitudinal Effects of Risk and Resilience on Nonresident Fathers
Developmental Psychology, 45:1389-1405.

Fitzgerald, M, Roy, K, Anderson, E, and Letiecq, B (forthcoming). The Effect of Depressive Symptoms on Low-Income Men in Responsible Fathering Programs.


Gelfand, D.M., and D.M. Teti. “The Effects of Maternal Depression on Children.” Clinical Psychology Reviews, vol. 10, 1990.

Guzzo, Karen Benjamin, and Frank F. Furstenberg, Jr. “Multipartnered Fertility Among American Men.” Demography, vol. 44, 2007, pp. 583–601.

Hsueh, JoAnn, Desiree Principe Alderson, Erika Lundquist, Charles Michalopoulos, Daniel Gubits, David Fein, and Virginia Knox (2012). The Supporting Healthy Marriage Evaluation: Early Impacts on Low-Income Families. OOPRE Report 2012-11. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

James, Jeannine M., and Richard Bolstein.1990 The effect of monetary incentives and follow-up mailings on the response rate and response quality in mail surveys. Public Opinion Quarterly 54:346-361.

Johnson, Earl, Ann Levine, and Fred Doolittle. Fathers’ Fair Share: Helping Poor Men Manage Child Support and Fatherhood. New York, NY: Russell Sage Foundation, 1999.

Kroenke, Kurt, Robert L. Spitzer, and Janet B.W. Williams. “The PHQ-9: Validity of a Brief Depression Severity Measure.” Journal of General Internal Medicine, vol. 16, no. 9, 2001, pp. 606–613.

Kroenke, Kurt, Tara W. Strine, Robert L. Spitzer, Janet B.W. Williams, Joyce T. Berry, and Ali H. Mokdad. “The PHQ-8 as a Measure of Current Depression in the General Population.” Journal of Affective Disorders, vol. 144, no. 1, 2009, pp. 163–173.

Löwe, B., J. Unützer, C.M. Callahan, A.J. Perkins, and K. Kroenke, “Monitoring Depression Treatment Outcomes with the Patient Health Questionnaire-9.” Medical Care, vol. 42, no. 12, 2004, pp. 1194–1201.

Martin, A., W. Reif, A. Klaiberg, and E. Braehler, “Validity of the Brief Patient Health Questionnaire Mood Scale (PHQ-9) in the General Population.” General Hospital Psychiatry, vol. 28, no. 1, 2006, pp. 71–77.

Pinto-Meza A., A. Serrano-Blanco, M.T. Peñarrubia, E. Blanco, and J.M. Haro, “Assessing Depression in Primary Care with the PHQ-9: Can It Be Carried Out over the Telephone?” Journal of General Internal Medicine, vol. 20, no. 9, 2005, pp. 738–742.

Marsiglio, William, Sally Hutchinson and David Nibert. Sex, Man, and Babies: Stories of Awareness and Responsibility. New York: New York University Press, 2002.

McLanahan, Sara. “Fragile Families and the Reproduction of Poverty.” Annals of the American Academy of Political and Social Science. vol. 621, no. 1, 2009, pp. 111–131.

Palkovitz, Rob. Involved Fathering and Men’s Adult Development: Provisional Balances. Mayway, NJ: Lawrence Erlbaum Associates, Inc., 2002.

Pearson, Jessica, Lanae Davis, and Jane Venohr. “Parents to Work!” Denver: Center for Policy Research, February 2011.

Roy, Kevin and Omari Dyson. “Making Daddies into Fathers: Community-Based Fatherhood Programs and the Construction of Masculinity for Low-Income African-America Men.” American Journal of Community Psychology, vol. 45, no. 3–4, 2010, pp. 139–154.

Roy, Kevin and Omari Dyson. Gatekeeping in context: Babymama drama and the involvement of incarcerated fathers. Fathering: A Journal of Theory, Research, and Practice about Men as Fathers, 3, 289-310, 2005.

Roy, Kevin (2006) Father Stories: A Life Course Examination of Paternal Identify Among Low-Income African American Men. Journal of Family Issues, 27:31-54,

Roy, K and Vesely, C (2009). Caring for “The Family’s Child”: Social Capital and Kin Networks of Young Low-Income African American Fathers in The Myth of the Missing Black Father, R. Coles and C. Green (eds), New York: Columbia University Press.

Singer, E., and R.A. Kulka. “Paying Respondents for Survey Participation.” Ann Arbor, MI: The University of Michigan, Institute for Social Research, Survey Research Center, 2002.

Straus, Murray A., and Emily M. Douglas. “A Short Form of the Revised Conflict Tactics Scales, and Typologies for Severity and Mutuality.” Violence and Victims, vol. 19, no. 5, October 2004.

Tach, Laura, and Kathryn Edin. “The Relationship Context of Young Disadvantaged Men.” The ANNALS of the American Academy of Political and Social Science, vol. 635, 2011, pp. 76–94.

Tach, Laura, Ronald Mincy, and Kathryn Edin. “Parenting as a ‘Package Deal’: Relationships, Fertility, and Nonresident Father Involvement Among Unmarried Parents.” Demography, vol. 47, 2010, pp. 181–204.

U.S. Census Bureau. “Family Structure and Children’s Living Arrangements.” Available at [http://www.childstats.gov/americaschildren/famsoc1.asp.] Accessed December 24, 2011.

U.S. Government Accountability Office. “Healthy Marriage and Responsible Fatherhood Initiative: Further Progress Is Needed in Developing Risk-Base Monitoring Approach to Help HHS Improve Program Oversight.” Washington, DC: U.S. Government Accountability Office, September 2008.

Weiss, R. S. (1994). Learning from strangers: The art and method of qualitative interview studies. New York: The Free Press.

Wood, Robert G., Sheena McConnell, Quinn Moore, Andrew Clarkwest, and JoAnn Hsueh. “Strengthening Unmarried Parents’ Relationships: The Early Impacts of Building Strong Families.” Princeton, NJ: Mathematica Policy Research, May 2010.

Wood, Robert G., Quinn Moore, and Andrew Clarkwest. “BSF’s Effects on Couples Who Attended Group Relationship Skills Sessions: A Special Analysis of 15-Month Data.” Princeton, NJ: Mathematica Policy Research, May 2011.

Wood, Robert G., Quinn Moore, Andrew Clarkwest, Alexandra Killewald, and Shannon Monahan (2012). The Long-Term Effects of Building Strong Families: A Relationship Skills Education Program for Unmarried Parents, OPRE Report # 2012-28A, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services (OPRE).

Yin, R. (1994). Case study research: Design and methods (2nd ed.). Thousand Oaks, CA: Sage Publishing.