New OMB Supporting Statement Part B - final v2

Administration for Community Living

Fidelity Evaluation of ACL’s Evidence-Based Programs

OMB Supporting Statement – Part B

November 2021

1. Table of Contents

B. Statistical Methods 1

B.1. Respondent Universe and Sampling Methods 1

B.2. Information Collection Procedures 2

B.3. Methods to Maximize Response Rates 6

B.4. Tests of Procedures or Methods 7

B.5. Research Team 7

B. Statistical Methods

B.1. Respondent Universe and Sampling Methods

On behalf of the Administration for Community Living (ACL), within the U.S. Department of Health and Human Services (DHHS), Health Services Advisory Group, Inc. (HSAG) will collect information for the Fidelity Evaluation of ACL's Evidence-Based Programs. The data collection effort will involve surveys of representatives from three related but distinct populations of organizations:

• Discretionary Grantees that receive awards from ACL to provide evidence-based Falls Prevention or Chronic Disease Self-Management Education (CDSME) programs. New three-year grants are awarded annually and are competitive. All organizations that have been awarded discretionary grants between 2018 and 2020 are part of this population.

• State Units on Aging (SUAs) in all 50 states, Washington, D.C., and 5 U.S. territories, which receive mandatory Older Americans Act (OAA) Title III-D funding from ACL to deliver evidence-based health promotion programs. In addition to their mandatory funding, SUAs can compete for ACL discretionary grants.

• Implementation Organizations directly deliver evidence-based program (EBP) services to eligible members of the public under contracts or agreements with SUAs or discretionary grantees. Implementation organizations may in some instances be divisions of discretionary grantee organizations or SUAs rather than independent entities.

HSAG is responsible for sample design, sampling, survey instrument design, survey administration, communication with respondents, and data analysis activities. Two web survey instruments have been designed and included in this package: one for discretionary grantees and SUAs, and one for implementation organizations. This data collection effort requires survey responses from one respondent per organization. HSAG has worked with ACL to identify appropriate contacts at all discretionary grantee and SUA organizations.

The numbers of discretionary grantees and SUAs are shown in Exhibit 1 below. There will be no sampling of these organizations; this will be a census rather than a sample survey. All will be contacted for participation in the survey. The grantee population, including both discretionary grantees and SUAs, is small but diverse. Discretionary grantees may offer only Falls Prevention programs, only Chronic Disease Self-Management programs, or both, and may provide programs directly or through other organizations or both. SUAs from large coastal states serve populations that are orders of magnitude larger and typically more diverse than SUAs from small mountain and plains states. Collecting data from the full population is inherently representative of the population. Sampling would reduce the potential for reliably capturing variations in practices and experiences of grantees of different types and working in different circumstances, especially in the face of nonresponse.

In contrast to the grantees, the number of implementation organizations is unknown; while individual grantees know which organizations are implementing their ACL-funded programs, there is no central directory of implementation organizations from which to sample. Since the population size is unknown, the sampling fraction is also unknown. However, a sample can still be drawn with the assistance of the grantees. The upper limit on the size of the sample is presented in Exhibit 1. Sampling details, including an explanation of the sample size estimate, are provided in section B.2.

Exhibit 1: Populations and Samples
Population	Population Size	Maximum Sample Size	Sampling Fraction
State Units on Aging	56	56	100%
Discretionary Grantees*	47	47	100%
Implementation Organizations	Unknown	412**	Unknown
Total	Unknown	515	N/A
* SUAs with ACL discretionary grant awards are not counted in this category if the SUA and discretionary grant contacts are the same person. Organizations with two or more grant awards are counted only once. ** See section B.2 for details on how the sample size estimate was derived.

B.2. Information Collection Procedures

B.2.1. Statistical Methodology for Stratification and Sample Selection

Selecting Grantees

As described in Section B.1, all grantees will be included in the discretionary grantee and SUA survey sample: this will be a census. No sampling procedures are required. The high level of diversity among the discretionary grantees and State Units on Aging militates for as large a sample as possible, while the small size of the population makes collecting data from all grantees practical. A smaller sample would have reduced aggregate respondent burden, but at the cost of representativeness. With a census, the main stumbling block to making valid generalizations about the population is nonresponse, which ACL and its contractor HSAG will attempt to control and reduce.

Selecting Implementation Organizations

As noted in Section B.1, there is no known central directory of implementation organizations. Simple random sampling would assure representativeness with an optimal sample size, but lacking a central directory, that method is not possible. Therefore, implementation organizations will be selected in a two-stage process, with both stages requiring the cooperation of grantees. In technical terms, this is a multi-stage cluster design where the clusters are evidence-based programs. Organizations are sampled within programs and programs are sampled within grantees. Like grantees, implementation organizations are expected to be heterogenous. They are located in urban and rural areas, are associated with large and small SUAs or discretionary grantee organizations, serve communities with different cultural and linguistic backgrounds, and implement a wide variety of evidence-based programs. Each of these characteristics may have an impact on approaches to implementation and fidelity. The proposed sampling method will allow ACL to obtain information from as large and diverse a sample of organizations as is practical under the circumstances.

Selecting Programs

This stage takes advantage of a request for documentation ACL sent to its grantees asking them to provide HSAG with lists of their ACL-supported EBPs. From each provided program list, HSAG will select up to four programs using simple random sampling. If a grantee lists fewer than four EBPs, all will be selected. After selection, each program list will be reordered randomly to avoid bias in program selection.

HSAG will return program lists to grantees with instructions to select the first two programs that are currently being delivered and provide contact information for all of the organizations charged with implementing the program on a local level, including the name of each organization, the name of the program director—that is, the person in charge of overseeing the program’s implementation, the program director’s e-mail address, and the organization’s ZIP Code. Grantees will be asked to return implementation organization lists via e-mail. Due to the Coronavirus disease 2019 (COVID-19) Public Health Emergency, grantees have temporarily halted service delivery for many programs. If a grantee receives a program list that does not include two programs that are currently being delivered, they will be asked to select any program that is currently being delivered and then select from programs not being delivered to obtain a total of two program selections. If the grantee is implementing the program on a local level, they will be asked to include themselves in the program implementation organization list.

Sampling Implementation Organizations

An important objective of the sampling process is to ensure that organizations providing services in rural/non-metropolitan areas are represented. Therefore, for each of the implementation organization lists provided by the grantees, HSAG will determine the rural vs. non-rural classification of the organizations’ locations, using ZIP Code-level Rural-Urban Commuting Area Codes.¹ Each implementation organization list will be divided into rural and non-rural strata for sampling.

HSAG will sample two organizations from each list using stratified random sampling. From each list, one rural and one non-rural implementation organization will be selected. If one stratum is empty (e.g., there are no rural organizations), both organizations will be selected from the non-empty stratum.

Additional Notes on Selecting Implementation Organizations

The decision to select two programs per grantee assures a link between each grantee and their implementation organizations while capturing some of the internal heterogeneity within each grantee’s network. Selecting two implementation organizations per program and stratifying them by urban/rural location also captures internal heterogeneity. In the aggregate, collecting data from up to four organizations per grantee will allow HSAG to assess the characteristics of and processes used by a large group of implementation organizations providing services funded by ACL Title III-D and discretionary grants.

Note, however, that the representativeness of the sample depends on the cooperation of the grantees. If all 103 grantee organizations provide HSAG with program lists and subsequently provide HSAG with implementation organization lists for two programs with at least two organizations per list, the implementation organization sample will include 412 implementation organizations. Because the organizations can only be sampled when grantees provide program lists, and because program lists have only been received from roughly 60 percent of grantees to date, the final implementation organization sample is likely to be considerably smaller than 412.

B.2.2. Estimation Procedure

Response variables for this survey are all measured at the nominal or ordinal levels. Univariate tables with frequencies and percentages will be created for all survey items. Summary statistics, including minimum, maximum, range, median, and mode will be calculated for all ordinal survey items.

The survey includes seven items that focus on whether various aspects of evidence-based programs, such as program content and necessary materials, are provided to participants with fidelity to program guidelines. An index representing a rating of fidelity to program guidelines will be constructed from these seven survey items. The index will count the number of items where the respondent indicates that guidelines are “Always” adhered to.

Bivariate analyses will also be performed using methods appropriate to the research question and the level of measurement. The bivariate methods will be used in comparisons of results for discretionary grantees vs. formula grantees. Separate analyses will also be used to compare results for implementation organizations located in urban areas with organizations in rural areas. The methods listed in Exhibit 2 will be used to perform these analyses. Please note that the proposed analyses will only be possible where a sufficient number of responses are received in each of the comparison group categories.

Exhibit 2: Bivariate Tests
Level of Measurement	Analysis
Interval-ratio (index of fidelity to program guidelines only)	t test
Ordinal	Mann-Whitney U test
Nominal – dichotomous	One sample Z test
Nominal – multiple categories	Chi square test

Tests will be performed with alpha = .05. Where applicable, two-tailed tests will be performed (t test, one sample Z test).

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Justification

This is an exploratory study intended to assess grantee practices and experiences related to selection of evidence-based programs, program implementation, and the maintenance of program fidelity. The information collected will help ACL evaluate the degree to which grantees are conforming to its expectations in these areas and assess strengths and opportunities for improvement. The sample size was not defined according to specific criteria for statistical power or margins of error. For the grantee survey, which seeks a census of all members of the population, sample size considerations are moot. For the implementation organization survey, a census approach is not feasible because the population of implementation organizations is unknown, therefore collecting data from a large and diverse sample of the population was the objective, rather than being able to generalize results with a specific level of precision to the entire population.

As discussed in B.2.2 above, some bivariate statistical tests, including chi-square tests, t tests, Mann-Whitney U tests, and one sample Z tests, will be performed for descriptive purposes. Such tests will examine, for example, whether there is a difference between discretionary grantees and SUAs in the frequency with which they assess whether evidence-based programs are being implemented with fidelity. For any analysis, the limitations of the design and the effects of nonresponse will be noted and taken into account during data analysis and in reporting results.

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems requiring specialized sampling procedures.

B.2.5. Use of Periodic (Less Frequent Than Usual) Data Collection Cycles

There are no periodic data collection cycles associated with this study. This is a one-time data collection.

B.3. Methods to Maximize Response Rates

ACL has informed grantees about the objectives of this data collection effort and provided them with information about the importance and value of participation. ACL staff that communicate regularly with grantees will deliver further encouragement as the survey gets under way.

In addition, potential respondents to both the grantee and implementation organization surveys will be contacted multiple times by e-mail to encourage participation. These contacts will include:

• A survey announcement sent a week in advance of the opening of the survey. The announcement will include a brief description of the contents of the survey, the expected survey completion time, and a request to ensure that the most appropriate staff complete the survey.

• A survey invitation sent the day the survey opens. This will include key information from the announcement plus login information unique to the organization, the survey URL, instructions on how to login to the survey server and how to resume the survey after an interruption, as well as information on who to contact for technical assistance.

• Survey reminders sent one week and three weeks after the opening of the survey period, and one week before the closing of the survey period to any respondents who have not completed the survey. All reminders will include login information and instructions to minimize the effort of beginning or continuing participation in the survey.

No monetary or in-kind incentives will be offered to grantees or implementation organizations in advance of their participation or in exchange for their participation. Surveys will be offered via the Internet and will be designed for 508 compliance. If grantee or implementation organizations request assistance, reasonable efforts will be made to accommodate those requests.

B.4. Tests of Procedures or Methods

The survey instruments have been drafted and undergone two reviews: (1) an internal review within HSAG, and (2) a review by ACL staff and independent subject matter experts invited by ACL. Modifications to the content, structure, and length of the surveys have been made based on internal feedback and on feedback from ACL staff and subject matter experts. Feedback from ACL and subject matter experts were mainly concentrated on conforming language to that used by grantees and implementation organizations and on providing responses that best fit the practices of grantees and implementation organizations.

No formal testing of the survey instruments with potential respondents or informal reviews with potential respondents are planned. After the survey instruments are finalized and uploaded to internet survey software, they will be thoroughly tested to ensure that all portions of the survey are functioning as expected.

B.5. Research Team

The information for this study is being collected by HSAG, an independent research organization, on behalf of ACL. With ACL’s oversight, the contractor is responsible for the study design, instrument development, data collection, analysis, and report preparation.

Cindy Strickland is directing the research team in the HSAG Data Science and Advanced Analytics Division. The primary contacts regarding statistical methodology are Michael Lichter and Robert Fornango. Contact information for these individuals is provided below.

Name	Title	Phone Number	Email
Cindy Strickland, JD	Associate Director	602-801-6802	cstrickland@hsag.com
Michael Lichter, PhD	Senior Analyst	602-801-6511	mlichter@hsag.com
Robert Fornango, PhD	Executive Research Director	602-801-7167	rfornango@hsag.com

1 Rural-Urban Commuting Area (RUCA) codes utilize data on commuting between geographic areas in addition to population density information to characterize ZIP Code areas on a metropolitan/non-metropolitan continuum. RUCA codes are maintained by the United States Department of Agriculture Economic Research Service. For more information, please see https://www.ers.usda.gov/data-products/rural-urban-commuting-area-codes/.