CONTENTS (continued)
Chapter Page
Understanding Barriers and Successful Strategies for Faith-Based Organizations in Accessing Grants
OMB Supporting Statement for Data Collection
CONTENTS
Chapter Page
a. justification 1
1. Need and legal basis 1
2. information users 10
3. use of information technology 10
4. efforts to avoid duplication of similar
information 11
5. SMALL BUSINESSES OR ENTITIES 12
6. consequences of less-frequent collection 12
7. special circumstances 13
8. federal register notice and outside consultation 13
9. payments or gifts to respondents 13
10. confidentiality of the data 14
11. sensitive questions 14
12. estimate of burden 15
13. capital costs 15
14. cost to the federal government 16
15. reasons for program or burden changes 16
16. schedule and plans for tabulation and
publication 16
17. approval not to display the expiration date for
omb approval 19
18. exceptions to certification statement 19
B. collection of information employing statistical
methods 20
1. respondent universe and sampling methods 20
2. PROCEDURES FOR THE COLLECTION OF INFORMATION 23
3. methods to maximize response rates and deal with
nonresponse 27
4. tests of procedures or methods to be undertaken 30
5. individuals consulted on statistical aspects and
individuals collecting and/or analyzing the data 31
APPENDIX A: TELEPHONE SURVEY INSTRUMENT a.1
APPENDIX B: UNDERSTANDING BARRIERS AND SUCCESSFUL
STRATEGIES FOR FAITH-BASED ORGANIZATIONS
IN ACCESSING GRANTS: IN DEPTH TELEPHONE
INTERVIEW GUIDE FOR SURVEY RESPONDENTS B.1
APPENDIX c: UNDERSTANDING BARRIERS AND SUCCESSFUL
STRATEGIES FOR FAITH-BASED ORGANIZATIONS
IN ACCESSING GRANTS: FOCUS GROUP GUIDE
FOR FEDERAL GRANT MANAGERS C.1
APPENDIX d: UNDERSTANDING BARRIERS AND SUCCESSFUL
STRATEGIES FOR FAITH-BASED ORGANIZATIONS
IN ACCESSING GRANTS: focus group guide
for federal grant reviewers d.1
APPENDIX e: UNDERSTANDING BARRIERS AND SUCCESSFUL
STRATEGIES FOR FAITH-BASED ORGANIZATIONS
IN ACCESSING GRANTS: section 301 of the
public health service act (42 u.s.c.241) E.1
APPENDIX f: UNDERSTANDING BARRIERS AND SUCCESSFUL
STRATEGIES FOR FAITH-BASED ORGANIZATIONS
IN ACCESSING GRANTS: federal register
notice f.1
APPENDIX g: UNDERSTANDING BARRIERS AND SUCCESSFUL
STRATEGIES FOR FAITH-BASED ORGANIZATIONS
IN ACCESSING GRANTS: advance letter:
telephone survey g.1
APPENDIX h: UNDERSTANDING BARRIERS AND SUCCESSFUL
STRATEGIES FOR FAITH-BASED ORGANIZATIONS
IN ACCESSING GRANTS: advance letter:
IN-DEPTH INTERVIEW H.1
APPENDIX I: UNDERSTANDING BARRIERS AND SUCCESSFUL
STRATEGIES FOR FAITH-BASED ORGANIZATIONS
IN ACCESSING GRANTS: ADVANCE LETTER:
FOCUS GROUPS Notice I.1
APPENDIX J: UNDERSTANDING BARRIERS AND SUCCESSFUL
STRATEGIES FOR FAITH-BASED ORGANIZATIONS
IN ACCESSING GRANTS: focus group
participant information forms j.1
The Office of the Assistant Secretary for Planning and Evaluation (ASPE) at the U.S. Department of Health and Human Services (DHHS) is requesting clearance for data collection instruments to be used in conducting the Understanding Barriers and Successful Strategies for Faith-Based Organizations in Accessing Grants (“FBO Grant Access”) study. The purpose of this data collection is to collect information from grant applicants and others to identify potential underlying barriers that prevent FBOs from having equal access to DHHS grant programs for which they are eligible. DHHS will use this information to gain a better understanding of perceived barriers FBOs face in accessing grant funds, identify common strengths and weaknesses of grant applications from FBOs, and consider strategies for ensuring equal access for FBOs that seek grant funds.
Faith-based organizations (FBOs) have long been providers of various health and social services in their communities. Beginning in the mid-1990s, desire emerged at the federal level to expand client choice of federally supported social services providers to include FBOs. “Charitable choice” provisions in the Temporary Assistance for Needy Families legislation were followed by similar provisions in the Department of Labor Welfare-to-Work Grants Program, as well as in the DHHS Community Service Block Grant and certain Substance Abuse and Mental Health Services Administration (SAMHSA) programs. DHHS has recognized that FBOs have an inherent community-based infrastructure for providing health and social services and has encouraged them to compete for federal grants.
In 2001, the White House issued several executive orders designed to reduce barriers to federal funding of social services through FBOs without requiring them to neutralize their religious character or provide services in a secular fashion. In response to one of these orders, DHHS, along with four other cabinet-level departments, conducted an internal audit to identify existing barriers to participation by FBOs in its discretionary grant programs. Specific barriers identified by those audits included (1) perceptions among federal officials that close collaboration with religious organizations was legally suspect, (2) exclusion of FBOs from grant competitions without a legal basis, and (3) excessive restrictions on religious activities within federal grant programs (White House Office of Faith-Based and Community Initiatives 2001).
Since 2001, DHHS has tried hard to reduce regulatory and administrative barriers for FBOs seeking grant funds and to educate FBOs on the federal grant-making process. As a result, DHHS has documented increases in the number of grants and amount of funding FBOs receive. Federal grants to FBOs increased by 82 percent between FY2002 and FY2005; discretionary grant funding awarded by DHHS to FBOs increased from 6.7 percent in FY2004 to 7.4 percent in FY2005 (White House Office of Faith-Based and Community Initiatives 2005).
Despite these gains, however, a significant gap still exists in the rate at which FBOs receive federal discretionary grant awards compared with secular organizations. Among 30 DHHS grant programs reviewed by the department’s Center for Faith-Based and Community Initiatives (CFBCI) in FY2005, 20 percent of the applicants were FBOs and 62 percent were secular nonprofits. Of all awards, however, only 14 percent were made to FBOs, while 64 percent went to secular nonprofits. Only 20 percent of FBO applicants received an award, compared with 30 percent of secular nonprofits; these organizations also received nearly nine times more funding than FBOs.
One of the current objectives of DHHS is to emphasize faith-based and community solutions by expanding faith-based and community partnerships providing effective health and human services, and by increasing the commitment to faith-based and community organizations. This data collection will address these objectives by providing a better understanding of the barriers FBOs face to accessing DHHS grant funds. While the data discussed above suggest lower success rates and award amounts for FBO applicants compared with secular nonprofits, little is known about the perceived underlying barriers that prevent FBOs from obtaining awards from DHHS grant programs for which they are eligible. A full and accurate understanding of such barriers cannot be developed without information from the FBOs themselves. The perceived barriers FBOs face in obtaining DHHS grant funds may be different from those identified by DHHS staff through their 2001 audit; for example, barriers could include fiscal reporting requirements, difficulty understanding grant requirements, or challenges with grant writing. DHHS hopes to identify through this data collection the factors that contribute to the low ratio of FBO to secular awardees. It will do so by collecting information from FBOs that have applied for grants, and from the grant managers and reviewers who evaluate their applications.
To identify possible barriers, ASPE wants to collect information and feedback about and from FBOs that have applied for DHHS discretionary grants—including those that have won grants and those that have not. ASPE also desires information from grant managers and reviewers about their approaches and experiences working with applications submitted by FBOs. To collect such information and consider its implications, ASPE has contracted with Mathematica Policy Research, Inc. (MPR) to conduct the FBO Grant Access Study.
Since its 2001 audit, DHHS has conducted several internal assessments of FBO access to discretionary grants, and has taken several steps within the department to reduce overt barriers. The FBO Grant Access Study now provides an opportunity for the department to collect information from FBO applicants themselves on perceived or underlying barriers that remain. This information may lead to improvements in the quality of future FBO applications, and hence more equitable access to federal funds, and improved client choice of social providers, consistent with federal policy. Consistent with this focus, ASPE has identified two main priorities for the study:
Identifying remaining potential underlying or perceived barriers to federal grant access for FBOs
Identifying approaches and strategies FBOs have used to compete successfully for federal discretionary grant funds, and which other applicants might emulate to overcome potential barriers
Several DHHS Operating Divisions offer discretionary grants. This study will include five divisions: (1) the Administration for Children and Families (ACF), (2) the Health Resources and Services Administration (HRSA), (3) SAMHSA, (4) the Centers for Disease Control and Prevention (CDC), and (5) the Administration on Aging (AoA). Three divisions—ACF, HRSA, and SAMHSA—will be the focus of the study, because they offer most of the grant programs for which FBOs are eligible and have applied, and are most likely to have available grant management data. However, to the extent it is relevant and available, information from CDC and AoA will also be used.
We will pursue the objectives of the FBO Grant Access Study by focusing on 8 main research questions, using data sources indicated in Table 1 and described in subsections c and d:
What are the characteristics of the competitive grant programs available to FBOs? What Operating Divisions house them, and what program areas do they represent? What is their purpose? How large is the pool of available funding, and what is the range of available grant award amounts? What are their selection criteria? How are applications reviewed and recipients selected?
What are the characteristics of FBOs that have applied for discretionary grants? What types of FBOs apply for funds (national networks, independent nonprofits, or congregations)? What is their geographic location (such as such as state, region, and rural/urban setting)? What programs and services do they propose? How experienced are they providing these programs or services? What target groups do they serve? What other sources of funding have they sought, and what has their success been? How do they learn about federal grants and the process for applying? How do they develop their federal grant application packages?
What are the outcomes of FBO grant applications? How many FBOs received federal grants for which they applied? What was the amount of the grant award? What proportion of FBO applicants receive grants? How do these results vary among Operating Divisions and by grant programs?
What are the characteristics of successful FBO applicants and their applications? What types of FBOs were selected to receive grant awards? How do they differ from those FBOs not selected? How do the content and quality of their applications differ? How do their proposed programs and services differ? What strategies or approaches have successful awardees used that distinguish them from those that are unsuccessful?
What obstacles, if any, to applying for or being awarded funds do FBOs perceive or have they experienced? How do FBOs view their own capacity for designing worthy program approaches and developing successful grant applications? Which capacities do they feel they lack? What sources of grant writing assistance or other capacity-building support, if any, are available to them? Which have they used, and why? What other resources or types of support do they feel could help to improve their future grant applications?
What feedback resulting from unsuccessful applications have FBOs received, and how helpful has it been? What information or guidance do federal grant managers and reviewers typically provide to grant applicants not selected for funding? What feedback do FBOs feel they need most to improve future federal grant applications?
How do federal grant reviewers and managers view applications from FBOs? Are grant reviewers and managers able to identify applications submitted by FBOs? Are they knowledgeable about current federal guidelines, regulations, and procedures concerning FBO eligibility for funding? What training do they receive, if any, in these areas? Do they have any underlying issues or concerns about faith-based programs or applicants that might affect how FBO applications are reviewed and rated?
Are there perceived differences by grant managers and reviewers in the quality of grant applications submitted by FBOs? What are some typical strengths and weaknesses of grant applications? What strategies or approaches have successful awardees used that distinguish them from those that are unsuccessful? What are some perceived obstacles faced by FBOs in submitting applications? Are these strengths, weaknesses, and obstacles any different from other types of applicants? Do grant managers and reviewers think that FBOs face any special obstacles applying for federal grants, or that changes in federal grant application or review processes might be necessary to ensure equal access to grant funds for FBOs?
To address study research questions, the FBO Grant Access Study will collect and analyze data from several sources. Clearance is currently being requested for three main data collection activities: (1) a survey of FBO grant applicants, (2) follow-up semistructured telephone interviews with a subgroup of survey respondents, and (3) two focus groups: one with grant managers who oversee the grant review process, and one with grant reviewers. Table 1 links each research question to one or more of these data collection activities.
Telephone Survey of FBO Applicants. We will conduct a telephone survey of 250 FBOs that applied for DHHS discretionary grant funds in FY2006, including FBOs that were awarded grant funds and those that were not. We expect each survey to last about 30 minutes. The survey respondent will be the person at the FBO who is most familiar with the process of developing the 2006 grant application. The survey contains six main sections: (1) respondent screener, (2) characteristics of the organization, (3) knowledge of federal grant opportunities, (4) experience applying for grants, (5) strengths and capacities of organizations, and (6) respondent characteristics (Appendix A).
TABLE 1
FBO Grant Access Study research questions and Data Sources
|
Quantitative Sources |
|
Qualitative Sources |
|||
Research Questions |
FBO Applicant Administrative Data Review |
Telephone Survey of FBO Applicants* |
|
FBO Application Review |
Telephone Interviews with Survey Respondents* |
Focus Groups with Grant Managers and Reviewers* |
1. What are the characteristics of the competitive grant programs available to FBOs? |
X |
|
|
|
|
|
2. What are the characteristics of FBOs that have applied for discretionary grants? |
X |
X |
|
|
|
|
3. What are the outcomes of FBO grant applications? |
X |
|
|
|
|
|
4. What are the characteristics of successful FBO applicants and their applications? |
X |
X |
|
X |
X |
|
5. What obstacles, if any, to applying for or being awarded funds do FBOs perceive or have they experienced? |
|
X |
|
|
X |
|
6. What feedback resulting from unsuccessful applications have FBOs received, and how helpful has it been? |
|
X |
|
X |
X |
X |
7. How do federal grant reviewers and managers view applications from FBOs? |
|
|
|
X |
|
X |
8. Do grant managers and reviewers perceive differences in the quality of grant applications submitted by FBOs? |
|
|
|
X |
|
X |
* Data collection activity for which clearance is required and being sought.
Follow-up Semistructured Telephone Interviews with Survey Respondents. We will interview a subsample of 20 survey respondents in more depth to gain a better understanding of their experiences applying for federal grants in 2006. The interviews will be 45 minutes each. The interview guide uses open-ended questions to follow up in depth on topics addressed in the survey. We will learn more about the respondent’s role in the organization and experience preparing grant applications, why the organization decided to apply for the grant, and who was involved in preparing the application. We will ask respondents to tell us in detail how they went about planning and preparing the application, their positive and negative experiences in doing so, and the helpfulness of feedback received from DHHS on their applications. We will also ask respondents for their views on the challenges FBOs face in seeking federal grant funds, their recommendations on improving the review process, and their recommendations for other FBOs seeking federal grants (Appendix B).
Focus Groups with Grant Managers and Reviewers. We have prepared two focus group guides—one for grant managers and another for grant reviewers (appendixes C and D). Grant managers are DHHS staff who oversee the review process for a specific grant program, including selecting review panel members, supervising the review panels, and making the final selection of awardees. Grant reviewers serve on review panels and rate the quality of grant proposals; grant reviewers include DHHS staff, as well as people outside DHHS and the federal government. Our current plan is to conduct separate 90-minute focus group sessions for managers and reviewers. The questions in these guides are designed to help us understand the review process and the makeup of review panels in detail, including the extent to which reviewers identify FBO applications and their experiences reviewing applications from FBOs. We also ask about their perspectives on the strengths and weaknesses of FBO applications, the nature of the feedback provided to applicants, their recommendations for improving the review process with respect to FBOs, and their advice for future FBO applicants.
To help answer the research questions described earlier, and to minimize the data collection burden on survey, interview, and focus group participants, we will also review administrative data on FBO grant applications from several sources. To provide OMB with more complete background information on the study, we describe these data sources here.
The Center for Faith-Based and Community Initiatives Database. We will use the FY2006 version of this database, which contains information on grant applications and outcomes collected by DHHS’s CFBCI from the DHHS Operating Divisions. MPR has already reviewed the FY2005 database, which contains information on about 8,000 grant applications (including those from FBOs and non-FBOs) for about 30 discretionary grant programs. The database contains (1) the applicant’s name, city, and state; (2) the type of applicant (FBO or non-FBO); (3) the applicant DUNS number, if provided in the application; (4) the Operating Division offering the grant for which they are applying; (5) the grant name; and (6) whether or not the applicant received a grant.
Grant Management Databases. Additional information about each applicant and the application is contained in grant management databases maintained by each Operating Division. Although the usefulness and availability of grant management data may vary across divisions, MPR will work with ASPE to contact DHHS Operating Divisions and obtain information on FBO applicants from their databases.
Individual Applications and Grant Review Records. MPR will select a sample of up to 20 FBO grant applicants for an in-depth review of the information on their applications available in the grant management systems. Documents included in the review are likely to include the grant applications, reviewer comments, grant review summary score sheets, and acceptance or denial letters.
These data collection activities are authorized by Section 301 of the Public Health Service Act (42 U.S.C. 241) (Appendix E).
ASPE anticipates that MPR will use the information obtained through the proposed data collection to produce two policy briefs and a comprehensive final report at the end of the project. These documents will be used by DHHS staff, especially staff at ASPE and CFBCI, staff at the White House Office for FBCI, and staff from FBOs interested in applying for DHHS discretionary grants. MPR will also use the information to provide oral briefings for DHHS staff, as well as staff at ASPE and CFBCI. ASPE expects that policymakers and other federal staff will use the information in three main ways:
To consider whether policy or procedural changes should be made to alleviate disparities in grant access for FBOs
To disseminate successful approaches and strategies FBOs have used to obtain federal discretionary grant funds
To determine whether and what steps grant managers and reviewers may need to take to avoid disadvantaging FBOs that apply for federal grant funds
Because MPR will conduct the FBO survey by telephone, no automated data collection techniques will be used. MPR considered fielding a self-administered mail or Web-based survey rather than a telephone survey in order to reduce respondent burden. However, a mail or Web survey would be more likely to require extensive telephone followup to obtain accurate and complete information and achieve a high response rate. Web-based surveys could also disadvantage potential respondents, who might include volunteers with limited access to the Internet, whereas all applicants have provided telephone contact information. A telephone survey will allow interviewers to work with each FBO to identify the best possible respondent—ideally the person who prepared the FY2006 grant application. In some cases, more than one respondent within an organization may be needed to answer all the questions in the survey. Based on these considerations, MPR determined that a telephone survey would be the most efficient mode for collecting accurate information and achieving a response rate of at least 85 percent. Moreover, a telephone survey is less expensive than a Web-based one and, because it minimizes the need for respondent callbacks, likely to be the least burdensome for respondents.
The in-depth telephone interviews and focus groups are qualitative data collections. Therefore, electronic, mechanical, and other technological techniques are not applicable.
DHHS has conducted several internal assessments to review regulatory and administrative barriers to accessing federal grants and related issues for faith-based organizations. Analysis of internal, administrative data suggests lower success rates and funding percentages among faith-based applicants as compared to secular non-profit organizations. However, little is known about the experiences and perspectives of faith-based organizations themselves regarding underlying or perceived barriers preventing them from participating more fully in DHHS grant programs for which they are eligible. Collecting information directly from faith-based organizations that have previously applied for such funding will both complement and supplement the internal efforts underway at DHHS to ensure equitable access to federal resources for these important social service providers.
There is no similar, prior, or ongoing data collection being conducted that duplicates the efforts of the proposed data collection. The survey, in-depth interviews, and focus groups do not ask for information that can be obtained through the administrative records discussed previously. In fact, this is the first time DHHS will obtain information directly from FBO staff about the grant application process, thus filling a gap in knowledge.
Some FBOs in the sample may be small entities. To understand the experiences of a wide range of FBOs in applying for federal grants, including small organizations, we believe it is important to include small entities in the data collection. By limiting the telephone survey to 30 minutes and the in-depth interview to 45 minutes, we are collecting from each organization the minimum amount of information required to address the study’s research questions and prepare the policy briefs, reports, and oral briefings to be produced. Therefore, we have not developed alternative, short forms for administration to small entities.
This data collection will provide the first opportunity for DHHS to collect information from recent FBO grant applicants on potential underlying or perceived barriers. Without information from the telephone survey, in-depth interviews, and focus groups, it would be very difficult for policymakers and other DHHS staff to learn about the experiences of FBOs applying for federal discretionary grants. Moreover, without a systematic data collection strategy, efforts to identify, contact, and interview people in the various FBOs that applied for grants would likely be burdensome for federal and FBO staff. It is unlikely that the process would be systematic and comprehensive and thus provide the information needed for considering whether additional policy or procedures changes are needed to alleviate disparities in access to grants. Federal staff would have to rely on anecdotal information from a small sample of FBO applicants, rather than systematically collected data from a representative sample. Thus, users of the information might draw inaccurate conclusions and possibly perpetuate unknown or unintended barriers to FBOs, which could further limit FBOs’ access to federal resources and limit client choice of social services providers. Although DHHS has access to existing administrative data, this information does not fully assess potential barriers to faith-based applicants. The data collection involved in the current study, however, will allow us to provide systematic, representative, and unbiased information directly from the FBO applicants themselves to inform decisions about how to improve the grant-making process.
There are no special circumstances.
A request for comment on the proposed data collection activities and instruments was published in the Federal Register on Feb. 26, 2007 (Vol. 72, No. 37, p. 8383). A copy of the notice is provided in Appendix F.
Discussion of responses to the Federal Register notice will be described here.
During the preparation of the study design and data collection instruments, we engaged the professional counsel of several people:
Alan Hershey, MPR
Dr. Joshua Haimson, MPR
Anne Ciemnecki, MPR
Dr. John Orr, Professor Emeritus, University of Southern California
None.
We will not offer payments or gifts to respondents. However, we did offer a cash incentive of $25 to FBOs that participated in the pretest of the survey. We offered this incentive for two reasons. First, some pretests took longer than the actual survey, because in addition to using the pretests to determine the time necessary for completing the survey, we also requested feedback on participant understanding or interpretation of some survey items, thereby lengthening the pretest compared to the actual survey. Second, since data collected during the pretest will not be part of the project’s survey sample, pretest participants did not have the personal satisfaction of contributing directly to study findings. This could have reduced their motivation for participating in the pretest, as compared to those who will be asked to participate in the survey itself. The incentives were offered to pretest participants in order to compensate them for their time and satisfaction. Any pretest participants drawn from the CFBCI database were drawn without replacement, so that they will not be selected for the survey sample.
Data confidentiality is not relevant for this study because respondents are organizations, not individuals. Federal guarantees of privacy or confidentiality are not applicable to organizations. While information about grant applications and about the grant review process will be collected from people participating in the survey, follow-up interviews, and grant manager/grant reviewer focus groups, these people will be speaking to us in their capacities as representatives of their organizations. They will be discussing organizational characteristics and processes.
We are not asking any questions of a sensitive nature. The telephone survey asks the respondent to identify the specific religious affiliation of the organization, if there is one. We will use this information to describe more fully the range of faith groups and denominations with which FBOs participating in the survey are affiliated. However, respondents will not be asked about their personal religious affiliation.
Table 2 presents the number of respondents, the number of responses per respondent, the average burden hours per respondent, their average hourly wage rate, and the total annual burden hours and costs for the data collection. Data for the FBO Grant Access Study will be collected through an FBO telephone survey, in-depth telephone interviews, a focus group with grant managers, and a focus group with grant reviewers. We estimate the total respondent burden for the entire study to be 170 hours. Our burden estimate for the FBO survey is 125 hours, or about half an hour per respondent. We estimate respondent burden for the in-depth telephone interviews to be 15 hours, or about 45 minutes each for 20 respondents. We estimate respondent burden for the two focus groups to be 15 hours each; we anticipate that each group will last for 1.5 hours and will have 10 participants. Our estimates are based on pretests of the FBO survey and our experience using similar in-depth interview and focus group protocols in other studies, combined with respondent wage rates obtained from the U.S. Department of Labor.
TABLE 2
annual burden and
cost estimates
Instrument |
Number of Respondents |
Number of Responses per Respondent |
Average Burden Hours per Response |
Average Hourly Wage Rate |
Total Burden Hours |
Total Cost to Respondents |
FBO Survey |
250 |
1 |
0.5 |
$20.44 |
125 |
$2555.00 |
FBO In-Depth Interview |
20 |
1 |
0.75 |
20.44 |
15 |
306.60 |
Grant Manager Focus Group* |
10 |
1 |
1.5 |
N/A |
15 |
0.00 |
Grant Reviewer Focus Group |
10 |
1 |
1.5 |
30.00 |
15 |
450.00 |
Estimated Total Annual Burden Hours/Cost |
|
|
|
|
170 |
$3311.60 |
*Grant managers participate as part of their regular employment activities, so their hourly wages are not reflected in the average hourly wage rates or total costs to respondents, although their time is included.
There are no capital costs associated with this data collection.
All costs for conducting the FBO Grant Access Study will be borne by the research contract between DHHS and MPR (Contract No. 100-03-0017/HHSP23320060000003T). The total estimated cost to the federal government of this data collection is $340,000. This includes all costs for study design, instrument development, data collection, analysis, and reporting.
This is a new project that incorporates new research.
Findings from this study will be reported in three ways: (1) policy briefs, (2) a final report, and (3) oral briefings. These materials will be targeted primarily to policymakers and other federal staff to help them determine whether policy or procedural changes need to be made to alleviate grant access disparities. The final report will be available to the public and is likely to be of interest to a variety of audiences within DHHS and other federal agencies. ASPE also considers the FBO community to be an important audience for the report, and will make the report available on its website.
Policy Briefs. At two stages during the study, MPR will write succinct briefs that distill initial study findings and present them in nontechnical language. The first brief, to be produced in September 2007, will summarize interim findings from the FBO survey. It will include new or significant descriptive information on FBO grant applicants, information on barriers to grant access as reported by survey respondents and gleaned from analysis of the survey data, and information on the differences between successful and unsuccessful FBO applicants. The brief will also identify potential implications of these findings for DHHS and the FBO community.
The second brief, scheduled for production in January 2008, will summarize interim findings from the in-depth interviews with FBOs and focus groups with DHHS grant managers and reviewers. The brief will describe the approaches and strategies for planning and completing grant applications used by successful applicants, underlying factors affecting the ability of FBOs to implement such strategies, the perceived quality and competitiveness of grant applications submitted by FBOs compared with non-FBO applicants, characteristics of the grant-making process that affect awards to FBOs, and the potential implications of these findings for DHHS and FBOs.
Final Report. The MPR project team will prepare a final report that addresses the study’s research questions and summarizes findings from all its components. The report will contain an executive summary, an introduction, and sections on the methodology, limitations, and key findings. It will also identify factors that affect grant outcomes and their implications, including underlying and perceived barriers, and will suggest general options or approaches for addressing potential barriers. The final report will be produced in February 2008.
Oral Briefings. MPR will brief ASPE and other DHHS staff at key interim time points and at the end of the study. Project staff will conduct a briefing on each policy component to provide federal staff with quick-turnaround information about what we are learning. At the end of the project, we will provide a comprehensive briefing on key findings from all aspects of the study and its implications for DHHS and the grant-making process.
To analyze data collected through the FBO survey, we will calculate descriptive and summary measures, such as frequencies, ranges, means, and variances. We will create tables that present statistics in the aggregate and for subgroups of FBOs identified for the study. We will examine these results and empirically identify any additional subgroups that emerge from the data. We will then create cross-tabulations to compare means among subgroups, including testing for the statistical significance of differences—for example, between successful and unsuccessful applicants or types of FBO. We will then use appropriate analytic techniques, such as correlation or logistic regression, to test the associations between applicant characteristics, such as size, experience, or reported barriers, and grant award outcomes and to test hypotheses about whether certain factors are significantly associated negatively or positively with award outcomes. Outcomes may be categorical, such as obtaining/not obtaining a grant or obtaining a grant amount within certain ranges, or continuous, such as the dollar value of a grant award.
We will also analyze data collected through the in-depth interviews with survey respondents, and the focus groups with grant managers and reviewers. This analysis will include three steps. First, we will review our writeups of each interview and focus group and organize the information in these documents by study research question (enumerated in Table 1). Next, we will write a summary that describes, compares, contrasts, and synthesizes the information we collected from each source, organized by research question. Finally, for each research question, we will compare the information collected from each source, and draw overall study conclusions.
See Table 3.
TABLE 3
FBO GRANT
ACCESS STUDY PROJECT SCHEDULE
Task and Deliverable or Activity |
Deliverable Due Date |
Task 3: Data Collection Instruments and OMB Package |
|
Draft instruments |
December 8, 2006 |
Draft OMB package |
January 5, 2007 |
Final OMB package ready for public comment |
January 26, 2007 |
Task 4: Administrative Data Review |
|
Summary memo |
May 25, 2007 |
Task 6: Selecting In-Depth Interview and Focus Group Participants |
|
Selection criteria and list of recommended interview and focus group participants |
August 31, 2007 |
Task 8: Interim Briefs and Presentations |
|
Draft survey brief |
September 14, 2007 |
Final survey brief and presentation |
September 28, 2007 |
Draft qualitative brief |
January 4, 2008 |
Final qualitative brief and presentation |
January 18, 2008 |
Task 9: Final Report |
|
Outline |
November 9, 2007 |
Draft report |
February 1, 2008 |
Final report |
February 29, 2008 |
Task 10: Final Briefing |
|
Briefing materials |
March 7, 2008 |
Briefing |
March 14, 2008 |
|
|
All data collection instruments will display the OMB expiration date.
Exception to the certification statement is not requested.
This submission requests clearance for four data collection activities to be conducted as part of the FBO Grant Access Study: (1) a survey of 250 FBOs that applied for DHHS discretionary grants in FY2006, (2) in-depth interviews with a subsample of 20 FBOs that participated in the survey, (3) a focus group with DHHS grant managers, and (4) a focus group with DHHS grant reviewers. In this section we describe the respondent universe and proposed sampling methods for each of these data collection activities.
We will draw our sample of FBOs to participate in the survey from the universe of FBOs that applied to one of 30 DHHS discretionary grant programs in FY2006, as documented in a database created by CFBCI. This database contains information on grant applicants, their applications, and outcomes, as reported to CFBCI by the DHHS Operating Divisions. Although the FY2006 database is not yet available, MPR has already reviewed the FY2005 database, which contains information on about 8,000 grant applications—including those from FBOs and non-FBOs—for the 30 discretionary grant programs. Of those, about 1,400 are applications received from FBOs. The database contains (1) the applicant’s name, city, and state; (2) the type of applicant (FBO or non-FBO); (3) the applicant DUNS number, if provided in the application; (4) the Operating Division offering the grant for which they are applying; (5) the grant name; and (6) whether or not the applicant received a grant.
From the universe of FY2006 FBO grant applicants, MPR will select for the survey a stratified random sample of FBOs. We will draw a sample of 294 FBO grant applicants and seek a response rate not less than 85 percent, for a total of 250 completed instruments. We will oversample FBOs that received grant awards. This allocation will allow comparisons between successful and unsuccessful applicants and will also provide a large enough sample to permit statements to be made about applicants as a whole. Therefore, before sample selection, we will stratify the sample frame on whether the application was successful, to ensure that each group is represented in the sample to the extent called for by the design.
We will also consider stratifying on other characteristics of FBO applicants available in the database, such as the DHHS Operating Division to which they applied, the grant program, or the location of the FBO (such as geographic region) so that the groups they define will be proportionately represented in the sample. Characteristics such as these may be related to success of the application, opinions about the process, or both. Before determining whether to do this additional stratification, we will examine the quality and completeness of the information that can be used to stratify applicants. We will also consider the relative sizes of such subgroups.
We will select for in-depth interviews a purposive sample of 20 FBOs that participated in the FBO survey. A purposive sample that includes FBOs with a range of selected characteristics deemed to be important for the study, rather than a representative sample of survey respondents, will be more useful for collecting specific data needed to ensure that we can address the study’s main research questions. We will review survey frequencies and emerging issues, such as potential grant access barriers or application strategies, to develop interviewee selection criteria in consultation with the APSE Task Order Monitor. Potential sample selection criteria include:
Whether or not applicant received grant award
Type of FBO
Type of social services provided or grant program applied for
Amount of funding sought
Experience providing social services prior to grant application
Organizational capacity (such as number of paid staff, volunteers, size of annual budget, number of members, training and experience of staff)
Types of barriers to obtaining grant funds identified in the survey instrument
Once criteria are identified, MPR will create cross-tabulations for all characteristics of interest and sort survey respondents into desired categories. If more than one observation meets all criteria for any desired set of characteristics, we will choose one or more respondents from that group. If any desired category is null, we will select an observation having the largest proportion of key characteristics for that category. We will select 35 potential respondents: an initial group of 20 representing the range of respondents we wish to interview, as well as a backup group of 15 who can be added to the sample if we cannot contract those in the initial sample, or in case they choose not to participate or are unable to complete the interview.
We will draw our sample of grant managers and reviewers from the universe of grant managers and reviewers that participated in selecting grant recipients for the DHHS discretionary grant programs included in the 2006 CFBCI database. We will use two overarching criteria to select focus group participants. First, we will seek participants representing a wide range of DHHS Operating Divisions, grant programs, experience, and personal backgrounds. Second, we will use evidence emerging from the study to prioritize such organizational and personal characteristics so that we include participants who can provide information on (1) the most influential factors determining grant outcomes, (2) program or grant areas deemed most important to the study, and (3) potential underlying or perceived barriers to grant access. Specific criteria could include:
DHHS Operating Division
Grant program area or specific program
Grant characteristics (amount, whether established or new program, other)
Grant history of division or program (such as proportion of awards made to FBOs and non-FBOs previously)
Length and/or variety of grant-making experience
Sector represents (such as federal staff, nonprofit staff, FBO staff)
Area of substantive expertise
Once specific selection criteria have been identified in consultation with ASPE, MPR will take several steps to identify possible focus group participants. We will request lists of (1) grant management staff, and (2) grant review panels that were active during the FY2006 grant period. We will also request information on the characteristics of interest for each person on the list, to the extent such information is known or available from agency records. We will sort these people based on selection criteria and work with the ASPE Task Order Monitor to select categories or specific people who can balance representativeness and particular knowledge of experience desired for the study.
As stated previously, we will draw a sample of 294 FBO grant applicants and seek a response rate not less than 85 percent, for a total of 250 completed surveys. Assuming that the ratio of successful to unsuccessful applicants in FY2006 is similar to that of FY2005, we expect that 148 cases will be allocated to unsuccessful applicants and 102 to the successful. Because the population from which the sample of successful applicants is selected is much smaller than the population of unsuccessful ones (just 20 percent of FBO applicants received grant awards in 2005), this allocation will result in roughly equal effective sample sizes for the two groups. For a 0/1 variable with an expected value of 50 percent, this sample will yield 95 percent confidence intervals of about ±7.5 percentage points around estimates of characteristics for each group and about ±6.14 percentage points for the sample as a whole (Table 4). It will also provide a high probability of detecting differences of 15 percentage points between the successful and unsuccessful applicants.1
TABLE 4
EXPECTED PRECISION
OF PROPOSED SAMPLE FOR THE FBO GRANT ACCESS STUDY
|
|
|
|
Half-Width 95 Percent Confidence Intervala for P = |
|
|
|
Population |
Sample |
Effective Sample |
50 Percent |
20 Percent |
Minimum |
Total Sample |
1,246 |
250 |
255.41 |
6.14 |
4.92 |
|
Unsuccessful Applicant |
997 |
148 |
173.80 |
7.46 |
5.96 |
|
Successful Applicant |
249 |
102 |
172.78 |
7.48 |
5.98 |
|
Other Subgroup (50 percent) |
623 |
125 |
127.70 |
8.71 |
6.96 |
|
Other Subgroup (35 percent) |
415 |
83 |
84.80 |
10.71 |
8.56 |
|
Contrast A-Bb |
|
|
|
10.56 |
|
15.08 |
Note: Proposed sample: stratified with oversampling of successful applicants.
a In percentage points.
b Contrast successful and unsuccessful applicants.
Survey administration will include several operations. First, MPR will draw the sample and obtain contact information for sample members. After survey interviewers are trained, MPR will release the sample to its Survey Operation Center in phases, with the timing dependent upon the rate at which interviewers actually make contact with respondents and complete interviews. MPR will send an advance letter that describes the study and its importance and encourages sample members to participate (Appendix G). The letter will be personalized and will stress the importance of individual participation for obtaining the most useful information possible for the study. It will also include a toll-free number that sample members can call to ask questions or to complete the survey. Advance letters will be mailed before each portion of the sample is released, so that calls can follow receipt of the survey notification. MPR will track survey contacts and completions and then follow up as necessary to obtain adequate response rates. An additional set of 100 sample members will be held in reserve and will be released as needed if response rates fail to meet 85%. Alternately, if response rates are higher than 85%, fewer sample points will be released. Should the response rate fall below 80%, we will conduct a nonresponse analysis. Finally, survey data will be entered into a data file from which analysis files can be created.
MPR will conduct 10 hours of project-specific training during which telephone interviewers will learn about the purposes of the study, planned uses of the data, and methods for gathering information. Training will include question-by-question instruction on the instrument, along with a discussion of commonly asked questions and approved responses. To ensure that all staff follow consistent procedures and do everything possible to achieve a high response rate, we will address possible challenges, such as potential difficulties identifying respondents. For quality control purposes, supervisors will carefully monitor interviewer performance during the course of the study, providing guidance and retraining as necessary. Special telecommunications equipment at the Survey Operation Center will allow supervisors to monitor live interviews.
As mentioned earlier, we will interview a subsample of 20 survey respondents in more depth to gain a better understanding of their experiences applying for grants in 2006. Prior to each in-depth, follow-up telephone interview, members of the study team will review the informant’s survey responses to develop a brief profile and to select or tailor particular interview questions based on survey responses. MPR will then send an advance letter that describes the study and its importance and encourages sample members to participate in the in-depth, follow-up phone interview (Appendix H). The letter will be personalized and will stress the importance of individual participation for obtaining the most useful information possible for the study. It will include a toll-free number that sample members can call to ask questions or to complete the interview. A senior researcher will then contact the respondent to schedule an interview, conduct the interview, and write up notes on the information collected, organized by discussion topic and research question. These writeups will be combined with the pre-interview profile for each informant.
Understanding the content and characteristics of successful grant applications, and how FBOs fare in the grant review process, is an important element of the study. To obtain this information, we will talk directly with those who manage and conduct grant reviews for HHS. We will hold two focus groups: one with grant managers, and one with people who have served on a grant review panel.
The focus group meetings will be conducted during working hours at a convenient location in Washington, DC. Each focus group recruit will be sent an advance letter that describes the study and its importance. The letter will include a toll-free number that sample members can call to ask questions (Appendix I). Recruits will also receive a follow-up letter thanking them for their willingness to participate and providing the information they will need to attend the group. Participants will be asked to arrive 10 minutes before the focus group begins, and to complete a short form asking for basic information (Appendix J). Use of this brief form will eliminate the need to ask focus group participants to provide information on their backgrounds during the focus group discussion.
One member of the research team will moderate the focus groups, and another will take notes. MPR will make a digital audio recording of all focus groups.
To maximize response rates for the telephone survey, we will use several strategies. As described below, these include (1) locating and contacting the most knowledgeable informant, (2) using tested survey items respondents can clearly understand and efficiently address, and (3) implementing proven sample recruitment and refusal avoidance procedures. In addition interviewers will be carefully trained in administering the survey items and in dealing with any potential obstacles that arise. Experienced senior staff members will supervise and monitor survey operations and step in when needed to help assure completion.
Contacting the right person is essential to achieving desired survey response rates. Contact information for sample members will be obtained from Operating Division grants management databases. These databases typically identify multiple representatives for each applicant, such as the grant writer, program director, and executive director. Therefore we have a range of people from which we can select the respondent most knowledgeable about the items in the survey. A screener at the beginning of the survey helps identify alternative respondents if the key informant is no longer with the organization or unavailable to answer the survey. Furthermore, during the survey there are several opportunities for respondents to provide name(s) of additional contact person(s) if the original respondent is unable to answer survey questions. In our pretest, we were able to identify and contact informants successfully, including obtaining contact information for individuals who had changed their position or location, and obtaining their cooperation. Specialized locating staff and resources will be utilized when necessary to boost our ability to contact sample members.
Having an effective and efficient survey instrument is a second key to maximizing the rate of survey completions. We have selected survey questions based not just on their relevance to the study, but also on their length, clarity, and directness. The questions included in the survey use plain, coherent, and unambiguous terminology. Many have been successfully administered as part of prior surveys. Sources of survey items include the Faith Communities Today (FACT) Survey conducted in 2000 and 2005, the 2002 Los Angeles Nonprofit Human Services Study, and the 2005 National Survey of Congregations. Items were also adapted from 2005 DHHS Staff Survey on Barriers to American Indian, Alaska Native, and Native American Communities Access to DHHS programs, conducted by ASPE. We have pretested the survey instruments, inviting questions and feedback from pretest respondents, and we subsequently revised questions and interviewer instructions to improve the ease of answering survey and eliminate overlap or duplication across survey items.
For a variety of reasons, completing the survey may be more challenging for some sample members. To contend with problems that arise, MPR’s Survey Operations Center has long experience using specialized staff and techniques to recruit sample members, and to convert incompletes and refusals to completed surveys. We mail advance letters and offer a toll-free 800 number for participants to call and either schedule or conduct their interview. Call backs are scheduled and made if respondents are called away from or interrupted during their first interview before being able to complete the survey. Followup letters are sent to sample members not reached within a limited time period or who do not complete the survey within a designated period. Experienced staff members make telephone and email contacts with those who may be busy or reluctant to participate, to encourage their participation and allay any concerns. Pretest respondents were enthusiastic about participating in the survey, as they felt that its topic was valuable to them, so we expect that many sample members will be highly motivated to participate in the survey.
Some sample members may not be eligible to participate in the survey. Grant applicants do not self-identify as FBOs, but instead have been identified as FBOs by HHS staff when they compile the database that serves as our sample frame. Therefore, before administering the survey to any sample member, we must identify whether the applicant’s organization fits within the study’s operational definition of an FBO or considers themselves to be an FBO (section B of the survey; if the organization is not an FBO the survey is immediately terminated and the organization is not part of our sample). To replace such ineligibles, when the survey sample of 294 applicants is drawn, we will also draw a supplemental sample of 100, which can be used to replace ineligibles.
If for any reason the telephone survey response rate falls below 80 percent, we will conduct a nonresponse analysis using information from the administrative data sources at our disposal. These data sources are quite rich; as described earlier the sampling frame alone includes the applicant’s geographic location, the HHS Operating Division and specific grant program to which they applied, whether or not they received grant awards, and the amount of their award. Additional data from Operating Division grants management databases to be used in our administrative data analysis will provide even further details. We will compare respondents and non-respondents across the dimensions available in the data and, if there are statistically significant differences, either make adjustments in our statistical analysis of the survey data to correct for bias, or disclose and discuss potential limitations of the analysis due to response-nonresponse differences, in the report.
To maximize the response rate for both the telephone survey and the in-depth follow-up phone interview, MPR will contact sample members at various times during the normal work day and ask them to schedule a time to complete the survey. For sample members who do not complete surveys within two weeks of our initial contact attempt, we will send a second letter explaining why participation in the study is important and asking the recipient to call our toll-free telephone number and complete the survey promptly. Sample members who still do not respond after receiving the follow-up letter will be re-contacted. MPR has staffed the project with personnel who possess the range of technical skills necessary to provide expert guidance to interviewers and respond to their questions. Staff include survey researchers, senior researchers, and a senior sampling statistician.
To maximize the response rate for the focus groups, MPR will provide each recruit with an advance letter and a follow-up letter thanking them for their interest and outlining all the relevant information they will need to attend the group. MPR will obtain permission from supervisors of grant managers and reviewers to participate during working hours and will provide a convenient location for the focus groups to meet. On the day before each focus group meets, we will place telephone calls to each recruit, politely reminding them of the day and time of the group, and asking them to contact MPR if they have an emergency and cannot attend.
MPR has pretested the phone survey with nine FBO respondents that vary along several dimensions. Pretest respondents were drawn from the database used to draw the survey sample, and were selected from the pool of applicants not included in the survey or reserve sample. MPR purposively selected pretest respondents who represented a range of FBO types and characteristics. Although many of the questions have been successfully administered as part of prior surveys, we used the pretest to assess respondent identification procedures, ease of administration, instruction clarity (such as skip patterns), adequacy of response categories, flow and order of questions, average interview length, and overall respondent burden. We revised the survey as necessary, based on the results of the pretests.
This study is being conducted by Mathematica Policy Research, Inc. (MPR), under contract to the Office of the Assistant Secretary for Planning and Evaluation (ASPE), U.S. Department of Health and Human Services. The project director is Ms. Debra A. Strong, the principal investigator is Ms. Diane Paulsell, and the survey director is Dr. Martha Bleeker—all MPR employees. The project team consulted with Dr. John Hall, senior statistician at MPR, about the sampling approach for this study. Ms. Wilma Tilson, ASPE Task Order Monitor, will receive, review, and approve all contract deliverables. Contact information is provided below.
Debra A. Strong, Mathematica Policy Research, Inc., 609-750-2001
Diane Paulsell, Mathematica Policy Research, Inc., 609-275-2297
Martha Bleeker, Mathematica Policy Research, Inc., 609-275-2269
John Hall, Mathematica Policy Research, Inc., 609-275-2357
Wilma Tilson, Office of the Assistant Secretary for Planning and Evaluation, 202-205-8841
1 In the table, we computed minimum detectable differences (MDDs) allowing for 80 percent power. In other words, there is an 80 percent probability of detecting a true difference at least as large as the MDD.
| File Type | application/msword |
| File Title | MEMORANDUM |
| Author | Marjorie Mitchell |
| Last Modified By | DHHS |
| File Modified | 2007-04-27 |
| File Created | 2007-04-27 |