OMB package - FINAL 11.12.2008

Compassion Capital Fund Program Evaluation

Follow-up Survey

Administration of Children and Families

Office of Planning, Research and Evaluation

September 30, 2008

Part A: Justification

A.1 Explanation of Circumstances That Make Collection of Data Necessary

This information collection request is seeking approval to continue an information collection (follow-up survey) that was initially approved by OMB in December 2005 with an expiration date of December 31, 2008 (OMB No: 0970-0293). It is expected that all sample members will have been sent an initial follow-up survey before the December 31, 2008 expiration date of the currently approved survey form. However, based on prior experience, ACF and contractor staff expect that a substantial number of sample members will not complete the survey on time and may need to have their web survey link kept open or another hardcopy survey form sent to them after December 31, 2008. In order to attain a high response rate to the follow-up survey and obtain valid and reliable data for analyses, additional follow-up to obtain completed surveys after December 31, 2008 will be necessary.

Background

Historically, faith-based and community organizations have played valuable roles in providing social services to individuals and families in need. Recently, the federal government has focused attention on the unique position of faith-based and community organizations (FBCOs) to address local needs by delivering critical services. However, despite their contribution, it has often been difficult for FBCOs to access government funding to support the services they provide.

Recognizing the critical role faith-based and community organizations play in their communities, President George W. Bush launched a national initiative to expand opportunities for FBCOs to compete for federal funds through the establishment of the White House Office of Faith-Based and Community Initiatives and Faith-Based Centers in ten federal Cabinet agencies, including the Department of Health and Human Services (HHS). As long as FBCOs achieve results with their social services, the President and his administration are committed to providing them with the fullest opportunity permitted by law to compete for federal funds.

The Compassion Capital Fund (CCF), a key component of the President’s Faith-Based and Community Initiative, was established by a Congressional appropriation in 2002. The CCF program is operated by the Administration for Children and Families (ACF), Office of Community Services (OCS), within HHS. The CCF Demonstration program provides grants to intermediary organizations to work with FBCOs to improve their organizational capacity. Intermediaries provide support to FBCOs in three ways: direct funding, through sub-awards; technical assistance (TA); and workshops/training. In keeping with the agency’s goals and interest in program performance, ACF awarded a contract to evaluate the effects of the CCF Demonstration program. The contract was awarded to Abt Associates, Inc. and their subcontractor, Branch Associates, Inc. The contract includes three evaluation components: a Retrospective Study, an Outcome Study, and an Impact Study. Information collection for the Retrospective Study, the Outcome Study, and the Impact Study baseline has been completed. Administration of the follow-up survey for the Impact Study is currently underway.

The Impact Study will assess the effect of intermediary services on the organizational capacity of the FBCOs they assist through the CCF grant. The Impact Study involves ten CCF-funded intermediary organizations selected in the FY 2006 award year and the 455 FBCOs that were identified by the intermediaries as eligible for their services. These FBCOs were randomly assigned to either a program group that could receive a sub-award or one-on-one technical assistance services from the intermediaries or to a control group that did not. The contractor obtained baseline data from each of these FBCOs prior to random assignment. Through the follow-up survey, information will be collected from FBCOs to assess change and improvement in various areas of organizational capacity. The impact of sub-award and technical assistance services provided by intermediaries will be determined by comparing the changes in organizational capacity among organizations in the program group and those in the control group.

The follow-up survey is sent out to FBCOs 15 months after they were selected to receive services. The selection process by the intermediaries extended over about six months. That schedule means that the 15-month follow-up survey is mailed out to a group of FBCOs each month between July and December 2008. All FBCOs will have been sent an initial follow-up survey before the December 31, 2008 expiration date of the currently approved information collection. However, based on prior experience, ACF and contractor staff expect that a substantial number of FBCOs will not complete the survey on time and will need to have their web survey link kept open or another hardcopy survey form sent to them after December 31, 2008.

A.2 How the Information Will Be Collected, by Whom, and For What Purpose

The follow-up survey is a self-administered form to be completed by Executive Directors (or other senior managers) of the FBCOs in the study and may be completed via the web or in hard copy transmitted through the mail. The survey is administered for ACF by Abt Associates, Inc.

The survey asks for information about basic organizational characteristics, areas of service, and organizational capacity in areas such as financial management, funding development, human resources, leadership and staff development, governance, technology, recordkeeping, and community engagement. The information from the survey will provide the basis for analysis of the impacts of CCF intermediary grantees’ services on changes and improvements in various areas of organizational capacity among faith-based and community organizations.

The Impact Study is intended to address the following research questions.

1. What type and quality of services are the FBCOs receiving from CCF-funded intermediaries?¹

2. What are the results of the services at the FBCO level? How effective is this service in increasing organizational capacity among FBCOs? What is the net impact of the service?

3. What types of services did FBCOs receive as part of CCF?

4. In what areas of organizational capacity improvement have intermediary services been most effective? What approaches and practices of intermediaries are associated with the greatest gains in FBCO organizational capacity?

5. In what areas of organizational capacity have FBCOs shown the greatest improvement? Among FBCOs served by CCF intermediaries, what characteristics are associated with the greatest achievements in capacity building and improvement?

A.3 Use of Improved Information Technology to Reduce Burden

Respondents will have the option of completing a web-based survey. The use of this technology reduces burden by automatically applying skip patterns based on the responses provided and, where appropriate, pre-populating some fields. Those who do not have access to the technology needed to complete the survey electronically or who merely prefer to complete a pencil-and-paper version of the survey will be mailed a hard copy version.

A.4 Efforts to Identify and Avoid Duplication

The information to be collected by this data collection does not currently exist in a systematic format. While intermediary grantees submit progress reports that include some information about changes in organizational capacity among FBCOs served, they do not have information about changes among the control group members.

A.5 Efforts to Minimize Burden on Small Business or Other Entities

No small businesses will be involved as respondents. Respondents are nonprofit faith- and community-based organizations. Effort has been made to minimize burden by limiting the range of topics included on the survey to those determined most critical for assessing the effects of the Compassion Capital Demonstration program and by offering a web-based survey mode.

A.6 Consequences of Less-Frequent Data Collection

This submission requests continuation of one round of follow-up data collection. The follow-up data collection is required to evaluate the effects of the services funded through the CCF program. If the follow-up data collection activity is not continued beyond December 31, 2008 we may not obtain responses from a sufficiently high number of sample members.

A.7 Special Circumstances Requiring Collection of Information in a Manner Inconsistent with Section 1320.5(d)(2) of the Code of Federal Regulations

There are no special circumstances associated with this data collection.

A.8 Federal Register Comments and Persons Consulted Outside the Agency

In accordance with the Paperwork Reduction Act of 1995, ACF published a notice in the Federal Register announcing the agency’s intention to request OMB approval of continuation of the follow-up data collection. The notice was published on August 5, 2008, and provided a 60-day period for public comments. The second, 30-day notice, has been submitted for publication.

The follow-up survey was developed by the Abt/Branch research team, comprising: Dr. Larry Orr, Principal Investigator (now retired); JoAnn Jastrzab, Project Director; Suzanne Klein, Deputy Project Director; and Dr. Carrie Markovitz, Statistician. Dr. William Ryan, a research fellow at Harvard University’s Hauser Center for Nonprofit Organizations, played a key role in the development of survey questions to measure changes in organizational capacity. Feedback on the draft surveys was solicited from Scott Hebert, an Abt Associates analyst specializing in volunteerism and community service; Amy Sherman, Senior Fellow at the Welfare Policy Center of the Hudson Institute; Thomas Sander, Executive Director of the Saguaro Seminar: Civic Engagement in America; program staff from the Compassion Capital Fund Program in the Administration for Children and Families; and three intermediary grantees and six FBCO sub-grantees.

Because it is important that early and late cohorts of FBCOs in this study be administered the same survey, no changes to the content or format of the survey are sought.

A.9 Payments to Respondents

We do not plan to offer payments to respondents.

A.10 Assurance of Confidentiality

The data collection plan for this study has been reviewed and approved by Abt Associates’ internal Institutional Review Board (IRB). Every effort will be made to maintain the privacy of respondents, to the extent provided by law. No personal identifiers other than name of the person completing the survey are obtained. The remaining information collected is about the organization.

The following procedures for this study during data collection, data processing, and analysis activities will be followed:

• All respondents included in the study sample will be informed that the information they provide will be used by the contractor and ACF only for the purpose of this research. Reports produced will summarize findings across the full sample and sub-groups and will not associate responses with a specific individual or organization, unless required by law.

• Information that identifies individuals will not be reported to anyone outside the ACF and contractor study team, except as required by law.

• To ensure data security, all individuals hired by Abt Associates and Branch Associates are required to adhere to strict standards and sign an oath of confidentiality as a condition of employment.

• Hard-copy data collection forms will be delivered to a locked area at Abt Associates Inc. for receipt and processing. Abt Associates maintains restricted access to all data preparation areas (i.e., receipt, coding, and data entry). All data files on multi-user systems will be under the control of a database manager, with access limited to project staff on a “need-to-know” basis only.

• Individual identifying information will be maintained separately from completed data collection forms and from computerized data files used for analysis. No respondent identifiers will be contained in any public use files made available from the study, and, to the extent provided by law, no data will be released in a form that identifies individuals or organizations.

A.11 Questions of a Sensitive Nature

The questions included on the information collection instrument do not involve sensitive topics.

A.12 Estimates of Respondent Burden

The original information collection request was submitted in 2005 before the final number of organizations to be included in the Impact Study was known. The original package estimated that there would be 800 respondents to the follow-up survey for the Impact Study. However, the final number of organizations in the Impact Study is 455.

Exhibit 1 presents the estimate of the reporting burden for respondents.

Exhibit 1 – Annual Burden Estimate

Instrument	Total number of respondents	Number of responses per respondent	Average burden hours per response	Total burden hours
Follow-up Survey	455	1	.42 hour (25 minutes)	191
Total Estimated Annual Burden Hours	455			191

A.13 Estimates of the Cost Burden to Respondents

There are no annualized capital/startup or ongoing operation and maintenance costs associated with collecting the information. Other than their time to complete the surveys, which is estimated in Exhibit 2, there are no direct monetary costs to respondents.

Data Collection Sources	Number of Respondents	Minutes Per Respondent	Estimated Cost Per Houra	Costs per Respondent	Total Burden (Costs)
Follow-up Survey (Impact Study)	455	.42 hours (25 minutes)	$23.44	$9.85	$4,482
Total	455		$23.44	$9.85	$4,482

A.14 Estimates of Annualized Government Costs

The information collection activity and associated forms have been developed in the performance of the U.S. Department of Health and Human Services Contract Number: 233-02-0088. The total cost to the Federal government for the Compassion Capital Fund Evaluation is $2,990,534. Of that total, approximately $2,065,198 will be used for the data collection activities and subsequent analyses and report writing. The annualized cost over the three year period for this aspect of the impact study is $688,399.

A.15 Changes in Hour Burden

This submission to OMB is for request for approval for an information collection instrument for the CCF Impact Study that will expire before the information collection activities are completed. The initial submission estimated the total number of respondents for the Impact Study would be 800 faith-based and community organizations, resulting in 336 burden hours. We are estimating that about 455 of those organizations may need additional follow-up to complete the survey beyond the current expiration date of instrument, equating to 191 burden hours remaining. Based on the original request (and approval of) 583 hours, this reduces the overall burden request by a reduction of 393. This reduction is the result of previously approved collections having been completed and their instruments no longer needed by ACF.

A.16 Time Schedule, Publication, and Analysis Plan

The schedule shown below in Exhibit 4 displays the sequence of activities under the evaluation project related to the information collection activities, analyses and reporting.

Exhibit 4 Time Schedule
Activities and Deliverables	Date
Instrument Design Baseline Data Collection	August 2005 January 2007 – November 2007
Follow-up Data Collection Data Analysis Reporting	July 2008 – January 2009 February–June 2009 July–September 2009

A.17 Display of Expiration Date for OMB Approval

ACF is not requesting a waiver for the display of the OMB approval number and expiration date on the data collection instruments.

A.18 Exceptions to Certification Statement

This submission does not require an exception to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).

Part B: Collection of Information Employing Statistical Methods

This evaluation is designed with the understanding that the goal of capacity building programs for faith-based and community organizations is to equip participating organizations to acquire a sustainable stream of resources (including money, knowledge, and talent). These increased resources, combined with clear goals and plans, will enable FBCOs to deliver effective and increased services to people in need.² More specifically, this capacity includes four broad areas: management and leadership, program services, sustainable funding, and community engagement. The CCF Evaluation Impact Study will use statistical methods to examine these major areas, which are covered in the survey instrument.

The analysis for the Impact Study will assess what difference CCF services make among FBCOs compared to what happens within a control group of FBCOs in the absence of CCF services (the net impact). The analysis will compare changes in the organizational capacity and other characteristics of FBCOs receiving CCF services with those experienced by a like group of organizations that applied for, but did not receive those services. To the extent sample size allows, findings will be presented separately for sub-groups of FBCOs based on relevant characteristics such whether they designate themselves as faith-based or secular, the length of time the organization has existed, the size of the operating budget, and whether there is a paid or non-paid Executive Director. The analysis will include the calculation of impact estimates and descriptions of the types of FBCOs in the study and participating intermediaries’ approaches to capacity building service delivery.

B.1 Respondent Universe and Sampling Methods

Sample Selection

ACF’s contractor, Abt Associates Inc. of Cambridge, MA and its subcontractor, Branch Associates Inc. of Philadelphia, PA, are responsible for the design and administration of the survey.

The CCF Impact Study includes all the Intermediaries awarded CCF Demonstration program grants in FY 2006. The sample for the Impact Study was obtained by having these Intermediaries identify all FBCOs to which they would be providing sub-awards (financial assistance) or intensive (10 hours or more) technical assistance (TA). There were a total of 455 FBCOs identified as the universe of eligibles. These FBCOs completed a baseline survey and then were randomly assigned to a program group, to receive services from the Intermediary grantees, or to a control group which was not eligible for sub-awards or intensive TA for the period of the study. Exhibit 1 presents the minimum detectable effect size that can be expected with different sample sizes.

Exhibit 1 Minimum Detectable Effect Size a Based on 80 percent power at the .05 significance level (two-tail test), 80 percent survey response rate.
Sample Size, Treatment Group	Sample Size, Control/Comparison Group	Minimum Detectable Effect Size (MDES)a
100	100	.39
200	200	.28
250	250	.25
300	300	.23
400	400	.20
500	500	.18

B.2 Information Collection Procedures

The 15-month survey is being administered to all FBCOs in the study, both those in the program group and in the control group. The survey is to be completed by a knowledgeable program director or staff member and may be completed on-line or in hard-copy. FBCO contact persons are sent both an email message with the web link for the survey and a mailed package including a hard copy that may be completed and returned by mail. Contractor staff are assigned to work with specific groups of FBCOs, to serve as liaisons to be the point of contact for any questions or problems related to completion of the survey. The liaison staff will make follow-up contacts with non-respondents via email, phone and mail on an established schedule to help ensure high response rates.

B.3 Methods to Maximize Response Rates

We have a multi-pronged strategy to help maximize response rates. First, in order to help ensure that the survey is sent to a viable address/contact person, we will obtain any available updated contact information on the FBCOs in the sample from the CCF Intermediary grantees. Further, a staff person on the research team will be assigned to address questions and report on response rates from specific FBCOs. Follow-up email and phone contacts will be made by the designated liaison staff member on a regular schedule after the deadline for completion of the survey has passed. These follow-up contacts will be initiated within 2 weeks of the completion deadline date. For those still not responding, we will send a letter signed by the CCF program manager at ACF stressing the importance of their responses for the study.

By instituting these procedures to follow-up and address non-response on a timely basis, we expect to achieve an 80% response rate on the follow-up surveys.

B.4 Test of Procedures

This request is for continuation beyond the current expiration date of December 31, 2008 for the 15-month follow-up survey. The initial instrument was drafted and reviewed by the contractors and HHS/ACF staff. The instrument was pre-tested with respondents from three intermediaries and six FBCOs receiving CCF services during the 2004 program year. These pre-tests included interviews with respondents to obtain information on the time associated with the survey, the clarity of the questions, and the ability of respondents to provide the information requested. Prior to final approval by OMB, minor modifications to the length, content, and structure of the survey were made based on the results of the pre-test interviews.

B.5 Individuals Consulted on Statistical Aspects of Design

The plans for statistical analyses for this study were developed by Abt Associates Inc., primarily. The team was led by JoAnn Jastrzab, Project Director; Dr. Larry Orr, Principal Investigator; and Dr. Carrie Markovitz, Statistician. In addition to Abt staff, Dr. William Ryan, a research fellow at the Hauser Center for Nonprofit Organizations at Harvard University, was consulted on the statistical analysis plans. Contact information for these individuals is provided below.

JoAnn Jastrzab

Abt Associates Inc.

55 Wheeler St.

Cambridge, MA 02138

617-349-2372

Dr. Larry Orr

Abt Associates Inc.

4550 Montgomery Ave.

Bethesda, MD 20814

301-634-1724

Dr. Carrie Markovitz

Abt Associates Inc.

4550 Montgomery Ave.

Bethesda, MD 20814

301-634-1807

Dr. William Ryan

The Hauser Center for Nonprofit Organizations

John F. Kennedy School of Government, Harvard University

79 John F. Kennedy Street

Cambridge, MA 02138

617-496-5675

1 This question will be answered using information gathered through the Impact Evaluation Process Study (OMB No. 0970-0341) and the Intermediary Survey (OMB No. 0970-0316).

2 This formulation of nonprofit capacity is consistent with, for example: P. Brinkerhoff, Mission-Based Management (Dillon, CO: Alpine Guild, Inc., 1994); P. Drucker, Managing the Nonprofit Organization: Practices and Principles (New York: HarperBusiness, 1992); M. Hudson, Managing Without Profit: The Art of Managing Third Sector Organizations (London: Penguin Books, 1999); and C. W. Letts, W. P. Ryan, and A. Grossman, High-Performance Nonprofit Organizations (New York: John Wiley & Sons, Inc., 1999).

Part A: Justification 8