Instrument 2 - FaMLE Impact Report Table Shells

Final Impact Evaluation Report Tables Template for Healthy Marriage and Responsible Fatherhood Grantees

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this collection is 0970-0356; this number is valid through 6/30/2021.  Public reporting burden for this collection of information is estimated to average 10 hours, including the time for reviewing instructions, gathering and maintaining the data needed, reviewing the collection of information, and revising it. This collection of information is voluntary for individuals, but the information is required from Grantees.

NOTE: In all tables and figures in this template, example information is included in italics. Please use as a guide and remove before completing tables/figures. Please use a regular font in your final tables. Instructions for completing all tables are included in the impact report instructions document.

Table Il.1. Description of intended intervention and counterfactual components and target populations

Component	Curriculum and content	Dosage and schedule	Delivery	Target Population
Intervention
Relationship skills workshops	Healthy relationships curriculum: Understanding partner’s perspectives; avoiding destructive conflict; and communicating effectively	20 hours, with 2-hour sessions occurring twice a week, or 4-hour sessions occurring every Saturday	Group lessons provided at the intervention’s facilities by two trained facilitators in every session	Low-income married couples
Economic stability workshops	Resume preparation; interview and communication skills; appropriate work attire; financial literacy	Monthly 2-hour workshops	Workshops are provided by one facilitator	Individual members of the couple who need job search assistance
Counterfactual
Economic stability workshops	Resume preparation; interview and communication skills; appropriate work attire; financial literacy	Monthly 2-hour workshops	Workshops are provided by one facilitator	Individual members of the couple who need job search assistance

^{Notes: [Anything important to note about the information above]}

Table II.2. Staff training and development to support intervention and counterfactual components

Component	Education and initial training of staff	Ongoing training of staff
Intervention
Relationship skills workshops	Facilitators are male and female and hold at least a bachelor’s degree and received four days of initial training.	Facilitators receive a half-day of semi-annual refresher training in the intervention’s curricula from study staff.
Economic stability workshops	Facilitators are male and female and hold at least a bachelor’s degree and received two days of initial training.	Facilitators receive a half-day of semi-annual refresher training in the intervention’s curricula from study staff.
Counterfactual
Economic stability workshops	Facilitators are male and female and hold at least a bachelor’s degree and received two days of initial training.	Facilitators receive a half-day of semi-annual refresher training in the intervention’s curricula from study staff.

^{Notes: [Anything important to note about the information above]}

Table IV.1a. Individual sample sizes by intervention status (Only use for studies with individual-level assignment; if your design uses cluster-level assignment, skip this table and use Table IV.1b instead)

Number of individuals	Intervention sample size	Comparison sample size	Total sample size	Total response rate	Intervention response rate	Comparison response rate
Assigned to condition	1a	1b	1c (=1a + 1b)	N/A	NA	N/A
Contributed a baseline survey	2a	2b	2c (=2a + 2b)	=2c/1c	=2a/1a	=2b/1b
Contributed to first follow-up survey (timing)	3a	3b	3c (=3a + 3b)	=3c/1c	=3a/1a	=3b/1b
Contributed to first follow-up (timing) (accounts for item non-response and any other analysis restrictions)	4a	4b	4c (=4a + 4b)	=4c/1c	=4a/1a	=4b/1b
Contributed to second follow-up survey (timing)	5a	5b	5c (=5a + 5b)	=5c/1c	=5a/1a	=5b/1b
Contributed to second follow-up (timing) (accounts for item non-response and any other analysis restrictions)	6a	6b	6c (=6a + 6b)	=6c/1c	=6a/1a	=6b/1b

^{Notes: [Anything important to note about the information above]}

Reminder from instructions: For rows that account for item non-response and other analysis restriction, note that you may have very different sample sizes for two outcomes of interest because of very different rates of missing data for the outcomes. If this is the case, please add a row for each outcome in each time period, as needed. Indicate in a table note to which outcome the sample sizes apply. For example, if you have two primary outcomes and there was very different response rates on the items needed to construct these outcomes, you should include two rows for “Contributed to first follow-up (accounts for item non-response and other analysis restrictions)”: one for the analysis sample for outcome one and one for the analysis sample for outcome two.

Table IV.1b. Cluster and individual sample sizes by intervention status (Only use studies with cluster-level assignment; if your design uses individual-level assignment, skip this table and use Table IV.1a instead)

Number of:	Intervention sample size	Comparison sample size	Total sample size	Total response rate	Intervention response rate	Comparison response rate
Clusters
Clusters: At beginning of study	1a	1b	1c (=1a + 1b)	N/A	NA	N/A
Clusters: Contributed at least one individual at baseline	2a	2b	2c (=2a + 2b)	=2c/1c	=2a/1a	=2b/1b
Clusters: Contributed at least one individual at first follow-up (timing)	3a	3b	3c (=3a + 3b)	=3c/1c	=3a/1a	=3b/1b
Clusters: Contributed at least one individual at second follow-up (timing)	4a	4b	4c (=4a + 4b)	=4c/1c	=4a/1a	=4b/1b
Individuals in non-attriting clustersa
Individual: At time that clusters were assigned to condition	5a	5b	5c (=5a + 5b)	N/A	NA	N/A
Individual: Who consented	6a	6b	6c (=6a + 6b)	=6c/5c	=6a/5a	=6b/5b
Individual: Contributed a baseline survey	7a	7b	7c (=7a + 7b)	=7c/5c	=8a/5a	=8b/5b
Individual: Contributed to first follow-up survey (timing)	8a	8b	8c (=8a + 8b)	=8c/5c	=9a/5a	=9b/5b
Individual: Contributed to the impact analysis at first follow-up (timing) (accounts for item non-response and any other analysis restrictions)b	9a	9b	9c (=9a + 9b)	=9c/5c	=9a/5a	=9b/5b
Individual: Contributed to second follow-up survey (timing)	10a	10b	10c (=10a + 10b)	=10c/5c	=10a/5a	=10b/5b
Individual: Contributed to the impact analysis at second follow-up (timing) (accounts for item non-response and any other analysis restrictions)b	11a	11b	11c (=11a + 11b)	=11/5c	=11a/5a	=11b/5b

^a For all rows in this section, do not include individuals from clusters that dropped (attrited) over the course of the study. For example, if you randomly assigned 10 clusters (5 to each condition), and one intervention group cluster (e.g. school) dropped from the study, you would only include individuals in this section from the 9 clusters that did not drop from the study. Because the cluster-level response rate in the above rows already captures that dropped cluster, you do not need to count individuals from the lost clusters in your individual-level response rates.

^b See guidance in section IV.A for defining your analytic sample(s).

Reminder from instructions: For rows that account for item non-response and other analysis restriction, note that you may have very different sample sizes for two outcomes of interest because of very different rates of missing data for the outcomes. If this is the case, please add a row for each outcome in each time period, as needed. Indicate in a table note to which outcome the sample sizes apply. For example, if you have two primary outcomes and there was very different response rates on the items needed to construct these outcomes, you should include two rows for “Contributed to first follow-up (accounts for item non-response and other analysis restrictions)”: one for the analysis sample for outcome one and one for the analysis sample for outcome two.

Table IV.2. Outcome measures used for primary impact analyses research questions (this template includes example data in italics, as an example for you to consider for your own report)

Outcome measure	Description of the outcome measure	Source	Timing of measure
Marriage status	The outcome measure is a yes/no response taken directly from the question in the survey, “Are you currently married?”	Local follow-up survey	6 months after intervention ends
Level of affection	The outcome measure is a scale (value range 1 to 5) calculated from both partners’ responses as the average of five survey items measuring support, intimacy, commitment, trust, and friendship. Cronbach’s alpha (if applicable): [enter number]	Local follow-up survey	6 months after intervention ends

Notes: [Anything to note about the information above]

Table IV.3. Outcome measures used for secondary impact analyses research questions (this template includes example data italics, as a sample for you to consider for your own report)

Outcome measure	Description of outcome measure	Source	Timing of measure
Level of affection	The outcome measure is a scale (value range 1 to 5) calculated from both partners’ responses as the average of five survey items measuring support, intimacy, commitment, trust, and friendship. Cronbach’s alpha (if applicable): [enter number]	nFORM exit survey	At post-test (immediately after intervention ends)

Notes: [Anything important to note about the information above]

Table IV.4. Summary statistics of key baseline measures and baseline equivalence across study groups, for individuals/couples completing [Outcome measure at survey follow-up period]

Baseline measure	Intervention mean (standard deviation)	Comparison mean (standard deviation)	Intervention versus comparison mean difference (p-value of difference)	Effect size (optional)
Female (%)
Race/ethnicity (%)
Hispanic
Non-Hispanic White
Non-Hispanic Black
Non-Hispanic Asian
Socioeconomic status
Outcome measure 1 (range: 1 to 5)
Outcome measure 2
Sample size

Notes: p-values are include in parentheses. Effect sizes are calculated by dividing the differences in means by the standard deviation of the comparison group. [Anything else important to note about the information above]

Reminder from instructions: Please present an equivalence table for each analytic sample (sample on which impacts are estimated) being used to answer the primary research questions.

Table V.1. Covariates included in impact analyses

Covariate	Description of the covariate
Age	Age (in years) as of the baseline data collection
Baseline marital status	Marital status (1 = married; 0 = not married) as of the baseline data collection

Notes: [Anything to note about the analysis.]

Table V.2. Post-intervention estimated effects using data from [Survey follow-up time period] to address the primary research questions

Outcome measure	Intervention mean or % (standard deviation)	Comparison mean or % (standard deviation)	Intervention compared to comparison mean difference (p-value of difference)	Effect size (optional)
Outcome 1
Outcome 2
Sample Size

Source: [Name for the Data Collection, Date. For instance, first follow-up surveys administered 12 months after the program.]

Notes: p-values are included in parentheses. Effect sizes are calculated by dividing the differences in means by the standard deviation of the comparison group. [Anything to note about the analysis. See Table IV.2 for a more detailed description of each measure and Chapters IV and V.B for a description of the impact estimation methods]

Table V.3. Differences in means between intervention and comparison groups estimated using alternative methods

Outcome	Benchmark approach	No covariate adjustment	Name of sensitivity approach 2	Name of sensitivity approach 3
Primary Research Questions
Outcome 1
Outcome 2
Outcome 3

Source: [Name for the Data Collection, Date. For instance, Follow-up surveys administered six to eight months after the program.]

**/*/+ Differences are statistically significant at the .01/.05/.10 levels, respectively.

Notes: [Anything to note about the analysis]

Table V.4. Post-intervention estimated effects using data from [Survey follow-up time period] to address the secondary research questions

Outcome measure	Intervention mean or % (standard deviation)	Comparison mean or % (standard deviation)	Intervention compared with comparison mean difference (p-value of difference)	Effect size (optional)
Outcome 1
Outcome 2
Sample Size

Source: [Name for the Data Collection, Date. For instance, first follow-up surveys administered 12 months after the program.]

Notes: p-values are included in parentheses. Effect sizes are calculated by dividing the differences in means by the standard deviation of the comparison group. [Anything else to note about the analysis. See Table IV.3 for a more detailed description of each measure and Chapters IV and IV.B for a description of the impact estimation methods.]

VIII. APPENDIX TABLES AND FIGURES TO SUPPLEMENT FINAL IMPACT REPORT

Appendix A: Logic Model (if applicable)

[Paste logic model here]

Appendix B: Data, Sample, and Measures (if applicable)

Table B.1. Data used to address implementation research questions

Implementation element	Research question	Data source	Timing/frequency of data collection	Party responsible for data collection
Fidelity	Were all intended intervention components offered and for the expected duration?	Workshop sessions in nFORM	All sessions delivered	Intervention staff
Fidelity	What content did the clients receive?	Fidelity tracking log or protocol; attendance logs; session observations	Every session for fidelity tracking and attendance logs; two times a year for session observations	Intervention staff for fidelity tracking and attendance logs; study staff for session observations
Fidelity	Who delivered services to clients?	Staff applications; hiring records; training logs	One time X months after start of implementation; annually	Intervention staff
Fidelity	What were the unplanned adaptations to key intervention components?	Adaptation request; work plan; 6-month progress report; annual progress report	Annually; ad hoc	Intervention staff; study staff
Dosage	How often did clients participate in the intervention on average?	Workshop sessions and individual service contacts in nFORM; attendance logs	All sessions delivered	Intervention staff
Quality	What was the quality of staff–participant interactions?	Observations of interaction quality, using protocol developed by study staff	X percentage of sessions selected at random for observation	Study staff
Engagement	How engaged were clients in the intervention?	Observations of engagement, possibly using an engagement assessment tool; ratings from facilitator fidelity logs; engagement ratings from participant satisfaction surveys	Y percentage of sessions selected at random for observation	Study staff
Context	What other HM/RF programming was available to study participants?	Interviews with staff from partnering agencies in the community; survey items on baseline and follow-up assessments; websites of other agencies in the community providing HM/RF programming	Once a year; ad hoc	Study staff
Context	What external events affected implementation?	Interviews with community/county representatives; list of site/school closures	Once a year; ad hoc	Study staff

Note: We use “clients” in the questions above to avoid redundancy.

Table B.2. Key features of the impact analysis data collection

	Data source	Timing of data collection	Mode of data collection	Party responsible for data collection	Start and end date of data collection
Intervention	Intervention group study participants	Enrollment (baseline) End of intervention (8 months after enrollment)	In-person online survey (nFORM)	Program staff	September 2016 through January 2020
		3 months after the end of the intervention (11 months after enrollment) 6 months after the end of the intervention (14 months after enrollment)	Telephone survey	Evaluation staff	August 2017 through March 2021
Counterfactual	Comparison group study participants	Enrollment (baseline) 8-month follow-up	In-person online survey (nFORM)	Program staff	September 2016 through January 2020
		11-month follow-up 14-month follow-up	Telephone survey	Evaluation staff	August 2017 through March 2021

[Paste updated CONSORT diagram from you Analysis Plan here.]

Appendix C: Compromised RCT, Attrition, and/or Baseline Equivalence of original RCT sample (if applicable)

C.1 If your study was originally an RCT but had to construct equivalent groups using a QED approach because of high attrition, lack of baseline equivalence, and/or another issue that compromised the random assignment, please describe the details in this appendix. Table C.1 can be used to summarize baseline equivalence of the RCT sample.

Table C.1. Baseline equivalence of the original RCT intervention and control groups

Baseline measure	Intervention mean (standard deviation)	Comparison mean (standard deviation)	Intervention versus comparison mean difference (p-value of difference)	Effect size
Female (%)
Race/ethnicity (%)
Hispanic
Non-Hispanic White
Non-Hispanic Black
Non-Hispanic Asian
Outcome measure 1 (range: 1 to 5)
Outcome measure 2
Sample size

Notes: p-values are include in parentheses. Effect sizes are calculated by dividing the differences in means by the standard deviation of the comparison group. [Anything else important to note about the information above]

C.2. For both QEDs and RCTs that had to construct equivalent groups, please describe the approach to constructing equivalent groups in this appendix.

7