Process_ROI Supporting Statement Part A

The population of older Americans is growing in size and diversity, along with the complexity of their service needs. In 2017, 51 million people (16 percent of the U.S. population) were 65 or older; this number is projected to reach 95 million (23 percent) in 2060.¹ Many of these adults will have unmet health care and social service needs, which will require coordinated care across a range of services, including access to nutritious meals, transportation, preventive health care, home and community-based care, social interaction, support for family caregivers, and advocacy to help maintain older adults’ health, safety, dignity, and legal rights.

The Older Americans Act (OAA) helps to address these needs and promote the independence and well-being of adults over the age of 60 to ensure their needs are adequately met. The OAA consists of seven titles, each with a focus on an essential need or addressing specific vulnerable sub-populations of older Americans. Title II of the OAA (Appendix A) designates the Administration on Aging (AoA), an agency of the Administration on Community Living (ACL) within the U.S. Department of Health and Human Services (HHS), with directing funds to states to implement OAA programs. Section 201 also “authorizes research and evaluation activities that support the objectives of this Act—including evaluation of new and existing programs and interventions authorized by this Act; and research on and assessment of the relationship between programs and interventions under this Act and the health outcomes, social determinants of health, quality of life, and independence of individuals served under this Act” (42 U.S.C. 3011). ACL contracted with Mathematica, a non-partisan public policy research firm, to conduct a process evaluation to learn how agencies collaborate to improve the lives of the older adults they serve and how to measure the effectiveness of their efforts with the goal of strengthening their reach and impact.

A.2. Indicate how, by whom, and for what purpose the information is to be used.

This is a new information collection request. The data collected through the activities summarized in this request will be used by ACL to better understand how agencies collaborate to improve the lives of older adults they serve, and the nature of these relationships as they have adapted to the COVID-19 pandemic. It will also inform a report recommending approaches for measuring the return-on-investment (ROI) of programs supported through the OAA.

The OAA’s mission is carried out by a set of federal, state, and local organizations, collectively known as the national Aging Network. The network’s core components include the AoA, which leads the federal administration of OAA grant programs; State Units on Aging (SUAs) that develop state plans for providing direct services, advocacy and support systems for older adults; Area Agencies on Aging (AAAs), which provide services to elders in specific planning and service areas designated by the SUAs; and Native American tribal organizations that operate under Title VI to provide nutrition, caregiver services, and supports to older American Indians, Alaska Natives, and Native Hawaiians. The OAA also funds Aging and Disability Resource Centers (ADRCs), which serve as one-stop service centers for information and assistance on long-term services and supports for older adults and people with disabilities. Nearly three-fourths of ADRCs are operated by AAAs.

A.2.1. Overview of the evaluation

The process evaluation will address four sets of questions that focus on the Aging Network’s structure at the federal, state, and local levels. By investigating these questions, ACL can better understand how the various agencies collaborate, how states differ in their structure, how agencies work together, and potential strategies for evaluating ROI of ACL programs. The evaluation questions are:

How is the Aging Network structured and how does it operate at the local, state, and federal levels? How does it target its services to vulnerable groups, and what data are collected about activities and outcomes?
How does the Aging Network measure and improve the quality of services available/provided? What is the network’s role in identifying and responding to emerging needs?
How do the various levels of the Aging Network work together, with whom do they partner, and how do they collaborate?
How does the Aging Network measure successful practices and identify areas for improvement?

To answer these questions, we are conducting literature reviews and seeking expert input from a Technical Advisory Panel, as well as two activities for which we are seeking OMB approval: 1) web-based survey of Aging Network members comprised of SUAs, AAAs, and tribal aging agencies, and 2) key informant interviews with Aging Network members who have evaluated ROI. Table A. 1 summarizes how the data to be collected will be analyzed to address the major questions.

Table A.1. How data will be used, by data collection activity

Data collection instrument

How the data will be used

1. Aging Network survey

We will conduct descriptive analyses to, 1) describe the Aging Network’s characteristics, and 2) describe variation in coordination and processes for agencies within the network with different characteristics.

We will create visualizations, such as geographical maps and network graphs to show how agencies in the Aging Network interact with one another.

2. Key informant interview guide

We will analyze the data to identify common themes and elaborate on findings observed in the survey data about how agencies conceptualize and measure ROI.

A.2.2. Overview of the data collection

Aging Network survey. We will solicit participation in the Aging Network Partnership and Effectiveness web survey (Appendix B) to agencies in collaboration with the national organizations that represent them, ADvancing States, which represents state units on aging and disability, and USAging, which represents Area Agencies on Aging. Before fielding begins, the national organizations will include an announcement about the upcoming survey in existing communication channels, such as e-newsletters and social media posts (Appendix C). At the same time, we will work with ACL to notify their regional contacts and ask that they similarly endorse participation to their SUAs. Survey fielding will begin with an advance letter and email (Appendix D) to SUAs that will include a description of the study purpose, the survey topics, and Mathematica’s collaboration with ADvancing States and USAging, as well as a request to endorse participation to the AAAs in their state. Following the advance notices, we will send an email invitation with the survey link (Appendix E) and stagger reminders over the field period. These reminders will include targeted emails (Appendix F), letters (Appendix G), and calls (Appendix H). The survey of AAAs and tribal aging agencies will lag behind the SUA survey by 2 weeks, allowing time for the SUAs to first pass along information about the study to their AAAs. Reminders for this survey will be similar to the SUA survey (i.e., emails, letters, calls), but from Mathematica and USAging rather than ADvancing States.

Key informant interviews. To supplement the survey, we will hold key informant interviews (Appendix I) which will provide additional insight about how members of the Aging Network are assessing ROI, including costs and benefits used in ROI calculations, successes and challenges to evaluating ROI, and lessons learned that could benefit other agencies seeking to conduct their own assessment of ROI. Participants will be selected from a subset of those who participated in the web survey.

A.3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or forms of information technology. Also, describe any consideration of using information technology to reduce burden.

Information technology has been incorporated into the data collection whenever possible to reduce respondent burden. The Aging Network Partnership and Effectiveness survey will be web-based, so it can be accessed from a computer or mobile device, allowing for the greatest ease of access. The web-based survey will enable respondents to complete the data collection instrument at a location and time of their choice, and its built-in editing checks and programmed skips will reduce the level of response errors and allow respondents to complete the survey as quickly as possible. The study team will conduct key informant interviews by telephone or video conference using available platforms, such as WebEx or Zoom.

A.4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purpose described in Question 2 above.

There is no similar data collection effort that has produced the information ACL needs. Although ACL has periodically conducted surveys of the Aging Network, prior surveys do not capture the depth of information this evaluation seeks related to coordination and collaboration in the network focused on meeting emergent needs of seniors such as those that arose during the COVID-19 pandemic, nor have they focused on the Aging Network’s knowledge, use, and application of ROI. Every effort has been made to avoid duplication.

A.5. If the collection of information impacts small businesses or other small entities, describe any methods used to minimize burden.

Because the Aging Network survey is a census of a diverse set of organizations that are part of it, some AAAs and tribal aging agencies will be large and some are expected to be small entities. The study team will minimize burden for respondents by programming logic into the web instrument so respondents can complete it as quickly as possible. For example, if a small entity indicates they have not participated in an assessment of ROI, the instrument will skip subsequent items related to the assessment. For small entities participating in key informant interviews, the study team will limit interview length to the minimum required and conduct interviews at times convenient for informants.

A.6. Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

The proposed data collection activities will take place one time. If the data are not collected, ACL will not have a thorough understanding of the collaborations among the Aging Network as its members have adapted to meet needs brought about by COVID, or sufficient information to make informed decisions about measuring ROI.

A.7. Explain any special circumstances that would cause an information collection to be conducted in a manner:

requiring respondents to report information to the agency more often than quarterly;
requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;

requiring respondents to submit more than an original and two copies of any document;
requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years;
in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;
requiring the use of a statistical data classification that has not been reviewed and approved by OMB;
that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or that unnecessarily impedes sharing of data with other agencies for compatible confidential use; or
requiring respondents to submit proprietary trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.

There are no special circumstances.

A.8. As applicable, state that the Department has published the 60 and 30 Federal Register notices as required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB.

A.8.1. Federal Register Notice and Comments

A notice of the proposed information collection and an invitation for public comment was published in the Federal Register on August 30, 2021, Volume 86, Number 165, pages 48428-48429. ACL did not receive any public comments during the 60-day FRN. A 30-day notice published in the Federal Register on February 8, 2022, Volume 87, Number 26, pages 7183-7184. N

A.8.2. Consultations Outside of the Agency

Consultations about the data requested and ways to minimize burden have taken place with the following individuals outside the agency:

Table A.2. Individuals providing consultation on data collection design

Name	Degree	Title	Organization
Marisa Scala-Foley	M.G.S.	Director, Aging and Disability Business Institute (ADBI)	USAging
Traci Wilson	Ph.D.	Director of research	USAging
Meredith Hanley	M.S.W.	Director, Community Capacity Building	USAging
Erica Lindquist	M.A.	Senior director, Business Acumen	ADvancing States
Damon Terzaghi	M.S.A.	Senior director, LTSS policy	ADvancing States
Kari Benson	M.P.A.	Director	Minnesota Division of Aging and Adult Services
Larry Curley	M.P.A.	Director	National Indian Council on Aging
Sarah Galvan	J.D.	Senior Staff Attorney	Justice in Aging
Suzanne Kunkel	Ph.D.	Executive Director	Scripps Gerontology Center, Miami University
Daniel Schoeps		Director, Purchased Long Term Services and Supports	U.S. Department of Veterans Affairs
Mary Ann Spanos	B.S.	Director	Chautauqua County Office for the Aging
Lucy Theilheimer	M.A.	Chief Strategy and Impact Officer	Meals on Wheels America

A.9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees with meaningful justification.

There are no incentives for this data collection.

A.10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy. If personally identifiable information (PII) is being collected, a Privacy Act statement should be included on the instrument.

There are no assurances of confidentiality with no PII being collected. All respondents’ information will be kept private and not disclosed to anyone but the analysts conducting this research. The individuals from agencies participating in this study will be assured that the information they provide will not be released in a form that identifies them. Individual agency responses will be grouped with others in published evaluation reports.

During the life of the project, electronic data will be maintained on secured, password-protected computer servers. Survey respondents will have a unique ID number, and analysis will be conducted on datasets that include only respondent ID numbers. Only approved contractor staff will have access to the data.

A.11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior or attitudes, religious beliefs, and other matters that are commonly considered private.

There are no sensitive questions in the Aging Network survey or key informant interview protocol.

A.12. Provide estimates of the hour burden of the collection of information.

The public entities affected by this study are State/Territory, local, and tribal governments, including SUAs, local AAAs, and tribal agencies on aging. At the time of this submission, there are potentially 874 respondents and 226 burden hours.² The table below shows sample sizes, estimated burden, and estimated annualized cost of respondent burden for each part of the data collection. The estimate of respondent cost is based on the burden estimates and utilizes the U.S. Department of Labor, Bureau of Labor Statistics, May 2020 National Occupational and Wage Estimates (http://www.bls.gov/oes/current/oes_nat.htm). The hourly wage for respondents’ time is valued at $33.46, which is the median hourly rate for Social and Community Service Managers, plus $33.46 per hour to account for the value of benefits and overhead (based on 100% of the hourly value).

Table A.3. Estimated annualized respondent hour and cost burden

Data collection activity	Annual number of respondents	Number of responses per respondent	Total number of responses	Average burden per response (in hours)	Annual estimated burden hours	Average hourly wage	Annual monetized burden costs
Aging Network survey	864	1	864	0.25	216	$66.92	$14,454.72
Key informant interview guide	10	1	10	1	10	$66.92	$669.20
Total	874	Varies	874	0.26 (weighted mean)	226	$66.92	$15,123.92

A.13. Provide estimates of the total annual cost burden to respondents or record keepers resulting from the collection of information, (do not include the cost of any hour burden shown in items 12 and 14).

Total Annualized Capital/Startup Cost:

Total Annual Costs (O&M):

Total Annualized Costs Requested:

There are no capital and start-up, or ongoing operation and maintenance costs associated with this information collection.

A.14. Provide estimates of annualized cost to the Federal government.

The annualized government costs include the costs associated with the contractor conducting the project and the salary of the assigned FNS project officer. The total cost to the Federal government for all tasks is $499,973, or $166,658 on an annualized basis for three years. This information collection also assumes a total of 600 hours of labor performed by federal staff at an average hourly rate of $51.33. Including 100% for fringe benefits and overhead, the total for Federal employee time equals $60,396 over the life of the contract. Federal employee pay rates are based on the General Schedule and locality payment for the Washington, D.C., Metropolitan Areas provided by Office of Personnel Management for 2020 (https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2020/DCB_h.pdf).

A.15. Explain the reasons for any program changes or adjustments reported.

This is a new information collection that will add 226 burden hours and 874 responses to the OMB inventory as a result of program changes.

A.16. For collections of information whose results are planned to be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used.

The plans for analysis will use the data collected through the Aging Network survey and key informant interviews to describe and visualize collaboration within the Aging Network, and to inform recommendations to measure ROI.

Descriptive analysis of the Aging Network. We will use a mixed-methods approach involving quantitative and qualitative analyses of survey and interview data:

Quantitative analysis (univariate). We will first produce descriptive statistics for each type of agency from the closed-ended variables in the web survey data. For categorical variables, we will estimate the percentage of agencies that responded in each category. For continuous variables, we will either present the mean or median value of the distribution, or partition the distribution into multiple ranges and estimate the percentage of agencies with values within each range.
Quantitative analysis (cross-tabular). Cross-tabulations will enable us to understand the variation in coordination and processes for agencies within the network with different characteristics.
Qualitative analysis. We will review the responses to the open-ended variables in the survey and identify commonalities and differences in themes across respondent types. We will develop a coding system to classify the major themes by (1) question topic and (2) type of respondent (SUA, AAA, ADRC, tribal agency). In addition to analyzing the survey-based qualitative data, the analysis team will review the notes from key informant interviews to identify common themes and highlight quotations that best illustrate them to include in the final report.

Visual representations of the Aging Network. We will create a series of visualizations, which may include geographical maps, network graphs, or heat maps, to show how agencies in the Aging Network interact with one another. To develop these visualizations, we will supplement information obtained during the literature review with data collected in the Aging Network survey and key informant interviews to ensure we have sufficiently detailed information on agency roles and responsibilities, and the types and frequency of interactions between agencies within the network. We will standardize information collected across these sources to determine the direction and magnitude of interactions.

Recommendations on measuring ROI of Aging Network model. We will use the information gathered from our review of the literature and the data collected from the Aging Network through the web survey and key informant interviews to prepare a report outlining up to three recommended approaches for measuring the ROI of the programs administered by the Aging Network. The report will describe the scope of costs and benefits encompassed by each approach, the data sources needed to measure them, and the analytic methods used to determine their values. We expect the recommended approaches will vary mainly in how broadly the benefits of the programs are defined. For example, a restrictive approach might be limited to measurable reductions in health care spending (such as fewer hospitalizations), whereas an expansive approach might include estimated net gains in quality-adjusted life years.

A.17. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

Project Timeline

Activity	Schedule
Conduct Aging Network data collection	2/1/2022-5/31/2022
Analyze data	4/1/2022-6/30/2022
Prepare data files and documentation	7/1/2022-9/30/2022
Prepare visual representation of Aging Network	10/1/2022-3/30/2023
Recommend approaches for measuring ROI	4/1/2023-7/30/2023
Prepare report and stakeholder briefing	12/1/2022-8/29/2023

A.18. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

The agency plans to display the expiration date for OMB approval of the information collection on all instruments.

A.19. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification statement.

1 Administration for Community Living. “Profile of Older Americans.” Washington, DC: ACL, 2018. Available at https://acl.gov/aging-and-disability-in-america/data-and-research/profile-olderamericans.

2 Totals include responses and burden associated with respondents and non-respondents.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	School Nutrition and Meal Cost Study (SNMCS) Supporting Statement Part A
Subject	OMB
Author	MATHEMATICA
File Modified	0000-00-00
File Created	2022-02-24