OMB Control Number:
Expiration Date:
Grantee Discussion Guide
Purpose: As a part of the PACT Evaluation we are attempting to identify healthy marriage and responsible fatherhood grantee programs that can provide valuable information to practitioners and policymakers within multiple levels of government as well as in other sectors. We will also need ongoing information about the operations of grant programs selected to be included in the evaluation.
We are seeking to obtain information about and from organizations that are operating programs under awarded healthy marriage and responsible fatherhood grants from the Office of Family Assistance within the Administration for Children and Families (ACF).
Use of information: The first stages of inquiry will ultimately lead to the identification of grantee programs to be considered for the PACT evaluation. Later stages of information collection will be used to document the actual operational experiences of agencies and participants over the course of the grant period.
Authority of collection: Mathematica Policy Research, Inc is conducting the PACT Evaluation on behalf of ACF. This evaluation is part of the Healthy Marriage and Responsible Fatherhood initiative. ACF obtained Office of Management and Budget approval for the collection of this information. We want to speak with you because ACF has asked us to learn more about your program.
Privacy: The information you provide will be discussed internally among the evaluation team as well as with federal staff in the Administration for Children and Families. Descriptive program information may be included in reports. However, to the extent permitted by law, individuals’ identifying information will not be disseminated publicly.
Voluntary Participation: Providing information for the purposes described is voluntary.
Estimated time: This discussion is expected to take an hour.
DISCUSSION GUIDE
Program Name:
Program Location:
Sponsoring Organization:
Individual(s) Interviewed: (names and titles)
Date of Communication:
To the extent information is not available from other sources (e.g., grant application), the following are guidelines for points for discussion. The discussions will be tailored to the specific persons interviewed and the programmatic context.
A. Individual Information
1. *What are your current position and role(s)?
Probes: What role do you have in the organization? Is that the same role you have in the new (healthy marriage/responsible fatherhood program)? If not, what is your role in the new program?
2. *How long have you been with the organization? How long have you been in your current position or role?
3. *Have you had experience doing similar things as required by your position/role in the (healthy marriage/responsible fatherhood) program? Describe.
B. Organization
1. Mission of organization and the range of programs or services provided through the organization, including but not limited to the (healthy marriage/responsible fatherhood) program just funded.
Probes: How many different types of programs or services are provided through the organization? Are these direct services to individuals or families?
2. When was the organization originally established? How long has the organization been providing each of these kinds of services?
Probes: If multiple types of services provided: which type of service does the organization have the longest history providing? Which services are newer additions? Specifically, has organization provided healthy marriage or responsible fatherhood (or closely aligned) services previously? If yes, for how long? If not these kinds of services directly, has the organization worked with types of couples/fathers expected to be involved in (healthy marriage/responsible fatherhood) program and to achieve similar outcomes?
3. How many staff work for the organization in total? Other than administrative support, what types of staffing categories or positions are there? Are these the same types of categories/positions that will be used in the new (healthy marriage/responsible fatherhood) program or were new categories or types of positions proposed/created?
Probes: If new, what staffing categories are there? Why did you think new categories were needed? Are qualifications different from existing positions? Explain. Did existing staff have the opportunity to transition to new program positions? Did they?
Is the direct manager of the new (healthy marriage/responsible fatherhood) a new hire or someone who was already in the organization? If positions not filled, what is the plan (internal or external hire for manager, supervisors, staff) and qualifications for the position?
4. Specifically related to the newly funded (healthy marriage/responsible fatherhood) program, are all positions filled? If not, which are vacant? Are there other staff positions within the organization that will be needed to support the grant program (although not funded through the grant)? What are those positions? Are they currently filled?
Have steps been taken to fill vacancies? Describe status for each vacancy.
Probes: How long does it usually take to fill vacancies? Are you finding there are sufficient numbers of qualified candidates applying? Expected fill dates?
5. Please describe the types of administrative support positions in the organization and their roles, and, particularly, the administrative supports for the grant program.
Probes: What types of support are there for information, budget and organizational management (data systems)? Are there supports for direct service workers (e.g., clerical support, data systems, etc.)?
6. Please describe the types of information or reports you/managers receive on program operations? What are your plans related to the new (healthy marriage/responsible fatherhood) program? On what frequency is information received/reviewed?
Probes: What level of management oversight exists currently and is planned for new program? Are there staffing and systems in place to support/facilitate management oversight of significant areas of operations (staff performance; participant enrollment, documentation of couple/father participation, quality/fidelity of service provision, receipt of supports, etc.)? Is there a management information system in place? Describe elements, reports produced, etc. What extent of modification is needed to support new grant? What is timeline?
If not automated systems, what paper systems exist, what data are compiled, on what schedule, etc?
7. What are key partner organizations in community? What is history and type of relationship (referral, joint services, etc.)? Specifically, is there history working with employment sector? With child support enforcement? What is current status of partner involvement or agreements?
Probes: What partner organizations are critical to the grant program success? Are agreements about roles and responsibilities and other agreements in place with all? If not all, which ones are in place? What is timeline for remaining agreements? What major issues have arisen? What has been experience to date with partners?
Any additional/replacement agencies or organizational identified as needed? What steps have been taken to engage them?
8. Are there other key issues related to your organization that are important to understand in relation to the new (healthy marriage/responsible fatherhood) program?
C. Program Design and Operation
Has this organization been recipient of a federal grant in the past? If yes, please describe the program funded and the experience.
Probes: Were there any aspects of administering the grant that were particularly challenging – from programmatic and fiscal perspective?
Why did you/organization apply for the new (healthy marriage/responsible fatherhood) grant? What are the overall goals of the new (healthy marriage/responsible fatherhood) program? What led to the proposed approach?
Probes: Who was involved in proposing the program approach or design of the program? How do you see this fitting in with the overall mission of the organization? How do you see the new program fitting in with other services you offer? How do you see it fitting in within the community?
*Please provide a “walk through” of the new program from the perspective of a new (couple, father) learning about and entering the program? Please begin with how (couples/fathers) will learn about the (healthy marriage/responsible fatherhood) services available and then what they will experience.
Probes: What are marketing strategies to make program services known in community as well as identify potential eligibles? Are there any identified referral partners (other agencies who can refer eligibles to program)? What happens at first encounter with (couple/father)?
What happens when (couple/father) first come to organization location? What is the pattern of involvement generally expected to “complete” the program/achieve objectives for each couple/father?
For each component: activities/content, staffing, frequency, duration, definition of completion; setting/location where component is delivered; supports available (e.g., transportation reimburse/payment); type of participants (if specialized); staff responsible for activities within component; other key elements.
*Are there specific curricula to be used within various components?
Probes: Which curriculum (commercial, home-made)? Is it used in whole or in part (which parts)? What was the rationale for selection/development of curricula? Is there any prior research/evidence on the curricula? How will staff be trained in its use?
5. Please provide information on the population to be served – demographics, needs, and challenges.
Probes: What are the demographics of the target population (race/ethnicity, gender, education levels, SES, urbanicity, etc.) in the proposed service area? What is the source of the information on population (local data or extrapolation for other level) – reliable/reasonable? Are program design features well aligned with target population characteristics?
Will you target couples/individuals with specific characteristics for any program components?
6. Please provide your estimates of your target numbers for 1) recruitment, 2) active participants; and 3) completion. Estimated number of couples/fathers to be served (monthly/yearly)? How did you arrive at this number? What are your expectations of what will be required to achieve this level of active participation (e.g., extent of outreach needed to achieve enrollment targets; etc.)?
Probes: What is the basis for estimates? What data exist regarding number of “eligibles” in the targeted service area? What level of experience supports the assumptions/plans? Are staffing plans aligned with levels of recruitment and retention efforts desired? Have target goals for enrollment and participation per component been set? Are challenges to achievement being realistically considered with options and alternatives at the ready? Any plans/actions for engaging no-shows or drop-outs?
Experience to date: recruitments per attempts; participation levels per enrollee; completion rates? Extent targets met to date? Adjustments made, if any?
7. Are there to be cohorts of enrollees? What number will be required to begin individual cohorts (e.g., classes, etc)?
Probes: What level of enrollment (per week/month) is needed to support program model – start of new classes, etc? Are current plans (including staffing) adequate to have only reasonable delay before active engagement of participants will begin?
8. *What will constitute “completion” of the program for participants?
Probes: Is there a clearly defined sequence of activities/steps that will be undertaken by couples/fathers in the program? What level of participation is expected to be required of couples/fathers (number of hours per week, over the full program)?
9. *What contributes to participant retention in program services so they will obtain the intended “dosage” of services? And conversely, what are chief causes of dropping out or limited involvement? What strategies are planned/operational to achieve high levels of participation and completion?
Probes: Does the organization focus on retention and have methods of documenting and monitoring attendance/participation? Does the program design or operational approach include methods to support and promote active participation toward “completion?” What barrier removal strategies are planned/ in place (e.g., transportation assistance, etc.)? Any incentives provided to couples/fathers to encourage participation? What are staff roles/responsibilities related to retention and active engagement?
10. *What, if any, support services are provided (e.g., costs related to child care or transportation related to participation, referrals to other services, meals, etc.) to facilitate participation or meet critical needs?
Probes: Does the program model include supports to address potential barriers to active participation, related participant needs, or to meet emergencies? What are levels of payments/reimbursements per participant for each type of support? How have referral systems been established? Will there be direct staff involvement in establishing referrals and follow-up (versus providing list or name/number)?
11. *Is there a management information system in place to support the healthy marriage/responsible fatherhood program? What is job description and qualifications of person who manages the system? What information is to be entered? Who will enter various types of information? Are there other methods outside the automated systems used to record staff activities and participation?
*What types of information is provided back to staff? What information/reports are produced for supervisors and managers? On what schedule? How is information used? Any current issues or challenges with the system or its use (e.g., isn’t flexible enough to meet administrative needs, staff do not routinely/timely input)?
Probes: Are all relevant data entered into single system or are there multiple places data are stored (e.g., hard copy case files, case notes, hard copy attendance rosters, etc.)? If there is no current automated system, willingness to implement a system if provided?
12. What are the physical locations from which services will be provided?
Probes: All services at one location? Are different types of services offered at different locations? Are services provided in multiple locations in proximity to target population? Some services provided at partner agency locations? Will any services be provided in homes/work places of participants?
13. *During what hours are services provided (e.g., business hours, evenings, weekends)? Which services at which times?
Probes: Do you anticipate any challenges in offering services during these times (from staffing or participants)?
14. *Has the program faced any major challenges to date? What was outcome? Are there any specific challenges you are anticipating?
Probes: Start-up challenges? Implications? Any unique challenges given the target population or community context? Methods or approaches to address challenges considered or implemented?
15. *Please tell me anything else we should know about this program and its operations.
D. Staffing
*Please describe how the program management and staffing is structured?
Probes: What is the organizational structure? Is this similar to management and staffing structure within other parts of the organization? If not, what is different and why? For the healthy marriage/responsible fatherhood grant program, what are the specific numbers and staff positions?
What types and numbers of staff will perform recruitment and enrollment duties?
Probes: Are staff to be full-time, part-time, or contracted personnel? What is the rationale for the type of positions and how it will be filled? What are the qualifications for positions? What are specific expectations about what staff will have to do to identify and recruit couples/fathers? What experience does organization have with this approach? Why will this approach work? Any challenges to date/anticipated? Plans to address challenges?
What types and numbers of staff will perform other roles and duties required in the program model – for each component and overall?
Probes: For each: Are staff to be full-time, part-time, or contracted personnel? What is the rationale for the type of positions and how it will be filled? What are the qualifications for positions? What experience does organization have with this staffing approach? Any challenges to date/anticipated? Plans to address challenges?
What are the 1) manager to supervisor and 2) supervisor to staff ratios planned or actual – for different component parts of the program?
Probes: What are the staffing and management structures for the grant program?
What has been your experience in finding qualified persons and filling positions for new grant program?
What has been your experience with staff retention within your organization in last 2 years? In general, is turnover greater in some types of positions than others? Reasons for this? Methods for addressing turnover? How is coverage handled while positions are vacant? What is involved when a position is vacated/needs to be filled?
Probes: What has been your experience with finding persons with the qualifications initially established for the positions? Any changes made in the qualifications for any positions based on experience? Are there qualities that have been shown to be especially important for different types of positions? Are staff positions and qualifications written?
What are the procedures/processes that must be followed to fill vacancies? How long does the process itself take (how many people in the chain must act)?
What approach are you taking for training for staff? *Describe training for the new program (who is trained, frequency, who provides training, topics covered). Have any issues related training arisen to date?
Probes: Do the plans differ by types of staff positions? Plans for training beyond an initial period?
*Please tell me anything else we should know about staffing and management.
E. Funding
What, if any, funds do you anticipate may support the (healthy marriage/responsible fatherhood) program beyond the OFA grant funds? Are there “in kind” partner supports that are critical to the program design?
Probes: What are your current sources of funding? Have any of these sources changed? What do you anticipate are future funding sources? How long have you received funding from these other sources? What determines whether the funding continues (required re-application, performance, etc.)? Are there MOUs or similar agreements in place related to partner supports and contributions?
Overall organizational budget and project budget –for initial and subsequent years. Any changes from what was included in the original grant application (e.g., based on any disallowance in final budget review process, etc.)?
Probes: Provide breakdown of costs by major categories (i.e. for staff, materials, training, rent, etc.). For programs with direct employment component, include funding for any wage subsidies or similar supports.
Any other information we should know about funding for this program.
F. Community Context
What are the characteristics of the community in which the program is operating?
Probes: What are urbanicity, population size, SES, race/ethnicity, marriage rates, out of wedlock birth rates, employment/unemployment rates, major employers? Are there major religious affiliations, practices, or influences? Are there major issues or challenges facing the community? Source of information provided?
*What do you think are the attitudes in the community related to the topic of healthy marriage or responsible fatherhood?
Probes: What are the prevailing attitudes and beliefs? How might they affect your goals and objectives? Are there major differences among different population groups?
*Please describe how community features and characteristics influence your program approaches and design?
Probes: Did the community context influence: where the services are provided, the inclusion of any specific or special components, the inclusion of any specific partners, other aspects?
*Please provide a brief description of key partners.
Probes: Describe types of organizations, history of partnerships, types of services partners provide to community, specific services/roles they will play in new program. Is there a formal agreement regarding partnership on the new program? What is the frequency and methods of communication between you and partners, between all partners? Which staff have key levels of communication/interaction?
*What has been experience with partnership interactions on new program to date?
Probes: Are partnerships functioning as envisioned or agreed upon? What challenges have been experienced to date? Solutions to challenges identified? Any partners dropped or new partners added? Explain role of new partners or how function to be filled by dropped partners addressed?
*Please tell me about anything else we should know about this community and partners and interactions with the program.
G. Other Programs and Services in Community
*Are there other healthy marriage/responsible fatherhood programs and services available in this community? Or other types of programs with similar goals and objectives as the grant program? Describe them.
Probes: Please provide their name, location, community sponsorship, services provided, characteristics of population served, what are the needs that the programs are addressing, perceived impact and effectiveness of other programs on target population and community. Size of program in terms of participants served?
*Anything additional about other programs or efforts with related or similar goals that we should know about in this community?
*Considering your program and the other programs you have mentioned, please tell me about unmet needs in this community related to family stability and responsible fatherhood.
H. Conclusion
*What are major activities you will be undertaking in the next six months? What do you see as the major 1) challenges to accomplishing that as planned and 2) facilitating factors that will support progress and goal achievement?
Participation in Evaluation
We will be reviewing information from a number of programs, after which we may want to follow-up with you about participating in this evaluation. I will now explain more about the evaluation.
As indicated in the Funding Opportunity Announcement for the grant, ACF is undertaking a federal evaluation of selected healthy marriage and responsible fatherhood grant programs. We will provide summaries of the information we obtained from you to ACF staff. ACF will make the decisions regarding which grantees to include in the evaluation and which to include in the impact evaluation component.
Some programs selected to be included in the evaluation will work closely with us and ACF staff to put in place procedures to allow us to conduct an impact evaluation of the selected programs. An impact evaluation requires the establishment of two groups of couples or fathers with similar characteristics. This is accomplished through a process called random assignment which is a lottery like system that assigns couples/fathers who volunteer for the program to the program or to a control group (which will not be in the program/receive program services). Members of both groups will be tracked and interviewed over time to document their circumstances and outcomes. Some programs will be included in the evaluation but will not establish a control group.
Over the course of the evaluation, we will work closely with all grantees in the evaluation to understand and document the implementation and operation of each program and to understand managers’, staff and participants’ perspectives of their experiences and to share information and lessons learned among grantees and provide feedback and support in operating the programs.
Examples of Program Materials to Obtain
Program Plans/Reports
Organization chart
Program Flow/Logic model
Outreach or recruitment materials
Program newsletters
Relevant data/reports
MOU example
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | APPENDIX B |
| Author | Seth F. Chamberlain |
| File Modified | 0000-00-00 |
| File Created | 2021-01-29 |