SSA - GenIC SIMR Eval Plan Template

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Strengthening the Implementation of Marriage and Relationship Programs (SIMR)

Formative Data Collections for ACF Research

0970-0356

Supporting Statement

Part A

December 2020

Submitted by:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4th Floor, Mary E. Switzer Building

330 C St., SW

Washington, D.C. 20201

Project Officers:

Samantha Illangasekare

Shirley Adelstein

Part A

Executive Summary

• Type of request: This information collection request is for a generic information collection under the umbrella generic, Formative Data Collections for ACF Research (0970-0356).

• Description of request: We are seeking clearance to collect information from Healthy Marriage and Relationship Education (HMRE) grantee programs about the design of their local evaluations using an evaluation plan template. Strengthening the capacity of grantees and local evaluators to conduct high quality, rigorous evaluations is a key objective of the Strengthening the Implementation of Marriage and Relationship Programs (SIMR) study. We do not intend for this information to be used as the principal basis for public policy decisions.

• Time sensitivity: We would like to distribute the evaluation plan template and instructions as soon as possible – ideally beginning in December 2020 – to allow grantee programs sufficient time to complete the template before they begin local evaluation data collection in April 2020.

A1. Necessity for the data collection

The Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services requests permission to collect information from grantees funded by the Office of Family Assistance (OFA) about their local evaluation designs. The study team will provide a standardized template for grantees to use to document their evaluation plans. The completed plans will be reviewed by the study team for the purpose of identifying design strengths and weaknesses, determining whether the plans meet standards of rigor, developing recommendations for improvement, and informing subsequent technical assistance (TA). The SIMR study team will use these completed plans to design TA to strengthen the capacity of grantees and local evaluators to conduct high quality, rigorous evaluations.

Study background

Since 2005, Congress has authorized dedicated funding for discretionary grants from the OFA to promote HMRE programs. For more than 10 years, ACF’s Office of Planning, Research, and Evaluation (OPRE) has led a sustained effort within the federal government to develop, document, and evaluate HMRE programs, particularly those serving low-income populations. Within ACF, OPRE and OFA have a long and proven commitment to supporting research on healthy relationships and family stability through federally funded HMRE programs. Through several prior and ongoing evaluations, ACF has sought to assess and improve the effectiveness of HMRE programs designed to develop the skills people need to form and sustain stable, high quality relationships.

In 2019, OPRE launched the SIMR study to 1) understand implementation challenges faced by HMRE programs and test strategies to address these challenges, and 2) support HMRE programs’ local evaluation through TA. The efforts to support local evaluations through TA will position grantees to contribute to the field’s understanding of varying HMRE program approaches, contexts, and target populations, while answering questions important to the local community. To support their success, we will provide TA tailored to each grantee’s needs while working to build their evaluation capacity and helping grantees (1) describe how they implemented their HMRE programs and (2) generate evidence about the programs’ effectiveness. The influence of SIMR will extend beyond this project as participating grantees become better prepared to evaluate the effectiveness of subsequent programs. ACF contracted with Mathematica and Public Strategies to implement the SIMR project.

As specified in the HMRE Funding Opportunity Announcements (FOAs), grantees are required to work with ACF to refine, improve, pilot test and pre-test, and make necessary changes to the evaluation design and methods proposed in their grant application. The current generic information collection (GenIC) request involves asking HMRE grantees who plan to conduct a local evaluation to complete a local evaluation plan template that documents the design of their proposed evaluation.

Legal or administrative requirements that necessitate the collection

There are no requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose of survey and data collection procedures

Overview of purpose and approach

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for research and evaluation (0970-0356):

• Inform the development of ACF research

• Inform the provision of technical assistance

The purpose of the current information collection request is to seek approval to ask grantees to document their local evaluation plans using a standardized template (Appendix A) and accompanying instructions (Appendix B), which will be disseminated by the study team. The study team will distribute the local evaluation plan template and accompanying instructions to grantee programs during the HMRE grantees’ planning period, which is currently ongoing through March 2021. The information collection will take place upon OMB approval through March 2021. This evaluation plan template is critical for the study team to undertake a standardized review process of documented evaluation plans to identify their strengths, determine whether the plans meet standards of rigor, develop recommendations for improvement, and inform subsequent TA. The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision maker, and is not expected to meet the threshold of influential or highly influential scientific information.

Guiding questions

The study team will use the completed evaluation plans to answer three questions:

1. To what extent does each evaluation plan describe the evaluation design with sufficient detail and demonstrate the grantee’s capacity to implement the plan?

2. What are the evaluation plans’ strengths and challenges, and what are the barriers to implementation?

3. What are any additional areas that could need evaluation TA as grantees advance to the evaluation start-up phase?

Study design

This data collection effort will be a preliminary step to collect information that will inform: (1) the evaluation TA that will be provided by the study team to grantees throughout the grant period, and (2) OPRE and OFA about each grantee’s capacity to successfully implement the proposed evaluation. The study team will collect and review completed evaluation plans from all grantees that are proposing an evaluation (43 of the 55 funded grants).

The evaluation team will provide grantees with a standardized template (Appendix A) and instructions for completing it (Appendix B), outlining all aspects of the local evaluation plan that must be fully described by the end of the planning period. Upon OMB approval, the study team will send an email to the program leader(s) specified in each grant application (see Appendix C, SIMR Evaluation Plan Email Template). The email will introduce the template (Appendix A), the instructions (Appendix B), and the timeline for completing the plan. Because the FOA states that local evaluation plans must be developed and refined as a condition of funding, we expect a response rate of 100 percent among those grantees selected for the data collection activities outlined in this package.

Grantees will use the instructions for completing the plan to understand how to populate the evaluation plan template. The instructions will give them guidance about the type of information to include in each section of the final evaluation plan along with any population-specific details they should consider in their write-ups. The evaluation plan template outlines each section and calls for information about each of the 16 evaluation design areas specified in the HMRE FOAs.¹ The evaluation team organized the 16 evaluation design topics specified in the FOA into six sections in the template, plus references and appendices. Table 1 cross-walks these topics specified in the FOAs to their locations in the evaluation plan template.

Table 1. Cross-walk of 16 evaluation design topics in FOA to evaluation plan template

Topic in FOA	Section in evaluation plan template
(1) Background (2) Research questions (3) Relation to program logic model (4) Hypotheses	Section I. Research questions (Parts AD)
(5) Research design (6) Methods to develop research groups (7) Sample (8) Lead staff (9) Ongoing grantee and local evaluator coordination	Section II. Research design and staff (Parts AE)
(10) Data collection	Section III. Data collection (Part A)
(11) Data analysis	Section IV. Data analysis (Part A)
(12) Privacy (13) IRB/protection of human subjects (14) Data (15) Data archiving and transfer	Section V. Privacy and data security (Parts AD)
(16) Dissemination	Section VI. Dissemination (Part A)
References	Section VII. References
Plans may include (1) logic model (or theory of change) for the program, (2) curriculum vitae for Principal Investigator/Project Director, and (3) Federal-wide Assurance (FWA)	Section VIII. Appendices (Parts AC)

The study team will use three methods to provide support through TA while grantees complete the plans: (1) group TA calls, including webinars and Q&A sessions; (2) one-on-one TA to grantees on an as-needed basis only, and (3) an evaluation TA help desk. For all webinars and calls, the study team will invite the grantee’s project director and key staff, the local evaluator, and the family assistance program specialist from OFA.

Group TA calls. The study team will lead up to six 60-minute webinars during the planning period on topics covered in the evaluation plan template. The study team will maintain some flexibility in the planned approach to the webinars to tailor some of them to specific needs of grantees. In addition to the webinars, the study team will lead three Q&A calls, which will be open forums for grantees to ask specific questions about their evaluation design issues or completing the plan.

One-on-one TA. As the need arises, a member of the study team may also meet with grantees on a case-by-case basis.

Evaluation TA help desk. The study team will provide a help desk in the form of a monitored email address that grantees can reach out to with clarifying questions or requests for resources. This email address will be intended for simpler grantee queries that do not require a call or more in-depth conversation.

Grantee teams will complete the plans by March 2021. Each of the final evaluation plans will undergo a review process conducted by the study team. Using a standardized process, the team will review the grantees’ evaluation plans to succinctly document key strengths and weaknesses in the proposed evaluation designs, identify opportunities for additional TA, and provide feedback to ACF about each grantee’s evaluation plan.

There are no quantitative components for the current phase of the SIMR project and this request.

Universe of data collection efforts

The data collection for the current request will occur via the local evaluation plan template (Appendix A), which will include an instructional document (Appendix B) to help grantees complete their templates.

Data collection activity	Instrument(s)	Respondent, content, purpose of collection	Mode and duration
Virtual	Local Evaluation Plan Template	Respondents: Program leaders Content: Description of local evaluation plan Purpose: This document collects details on grantee program local evaluation plans in a standardized manner, allowing more efficient review and TA from the study team. It includes instructions with a description of all data elements captured in the local evaluation plan template and any population-specific details they should consider including in their plans.	Mode: Email Duration: 8 hours

A3. Improved information technology to reduce burden

The burden on grantees is minimal, and the study team plans to use improved information technology wherever possible. The grantees can use their original applications for the grant to inform and populate the many parts of the final evaluation plan. Local evaluation plans will be collected virtually, and will not require any in-person follow-up.

A4. Use of existing data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The study team will encourage HMRE grantees to draw on information that was already in their original grant applications to complete the local evaluation plans. The local evaluation plan template will also encourage the grantees to provide details about the plans that they did not include in their applications.

A5. Impact on small businesses

We expect most of the programs in the study will be small, nonprofit organizations. The burden for respondents will be minimized by restricting the required elements of the plan to those previously specified in the FOA.

A6. Consequences of less frequent data collection

Without the collection of these standardized local evaluation plans, the study team would be unable to complete a review of grantee evaluation plans or identify additional TA opportunities.

A7. Special circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal register notice and consultation

Federal register notice and comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. This notice was published on October 11, 2017, volume 82, number 195, page 47212, and provided a 60-day period for public comment. During the notice and comment period, no substantive comments were received.

Consultation with experts outside of the study

The larger SIMR study has engaged several experts in HMRE programming and research for consultation to the study team and ACF on multiple occasions throughout 2020. However, for this portion of the study, we do not expect we will have to consult experts outside the study.

A9. Tokens of appreciation

No tokens of appreciation for respondents are proposed for this information collection.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally identifiable information

This data collection effort does not include collecting personally identifiable information.

Assurances of privacy

As specified in the contract, the contractor shall protect respondent privacy to the extent permitted by law and will comply with all federal and departmental regulations for private information. The contractor shall ensure that all its employees, subcontractors (at all tiers), and employees of each subcontractor who perform work under this contract/subcontract are trained on data privacy issues and comply with the above requirements. Respondents are not considered human subjects, but they will still be informed of all planned uses of data, that their participation is voluntary, and that some of the information they provide may be shared with OPRE to help us design the study.

Data security and monitoring

No information will be given to anyone outside of the SIMR study team and ACF. The local evaluation plans submitted by grantees will be stored on Mathematica’s network, which is accessible only to the study team.

A11. Sensitive questions

There are no sensitive questions in this data collection.

A12. Burden

Grantee program leaders will review instructions and complete the local evaluation plan template.

Estimated annualized cost to respondents

Instrument	Total/annual number of respondents	Number of responses per respondent	Average burden hours per response	Annual burden hours	Average hourly wage	Total annual cost
Local Evaluation Plan Template	43	1	8	344	$34.20	$11,764.80
Estimated annual burden: total				344		$11,764.80

Total annual cost

To compute the total estimated annual cost, the total burden hours were multiplied by the estimated average hourly wage for local program directors (see table above). According to the Bureau of Labor Statistics’ Current Population Survey 2019, the median hourly wage for full-time employees over age 25 with a bachelor’s degree or higher is $34.20.

A13. Costs

There are no additional costs to respondents.

A14. Estimate annualized costs to the federal government

The total estimated cost to the federal government for the data collection activities under this current request will be $122,186.80. This includes personnel effort plus other direct and indirect costs.

Cost category	Estimated costs
Instrument development and OMB clearance	$23,042.39
Review and TA support of local evaluation plans	$99,144.41
Total costs over the request period	$122,186.80

A15. Reasons for changes in burden

This is an individual information collection request under generic clearance 0970-0356.

A16. Timeline

The information collected under this request will be used to strengthen the capacity of grantees and local evaluators to conduct high quality, rigorous evaluations. Upon OMB approval, the dissemination of materials and the collection of local evaluation plans will take place through March 2021. There are no plans to publish the information collected under this request.

A17. Exceptions

All instruments will display the expiration date for OMB approval. No exceptions are necessary for this information collection.

Attachments

Appendix A, SIMR Local Evaluation Plan Template

Appendix B, SIMR Local Evaluation Plan Instructions

Appendix C, SIMR Local Evaluation Plan Dissemination Email

1 These 16 areas are specified in Appendix G of the FRAMEWorks FOA (pp. 9396) and READY4Life FOA (pp. 98-102).