Attachment C - Memo on FRN Responses

Response to the Public Comments on Federal Register Notice on Proposed Performance Measures for Healthy Marriage and Responsible Fatherhood Grantees

On November 6, 2014, in compliance with a legislative requirement, the Administration for Children and Families (ACF) published a Federal Register Notice (FRN) requesting public comment on proposed revised performance measures for Healthy Marriage and Responsible Fatherhood programs and additional data collection activities as part of the Fatherhood and Marriage Local Evaluation and Cross-site (FaMLE Cross-site) Project. The comment period closed January 6, 2015.

ACF received 57 requests for the measures and 28 emails with comments during the 60-day period, most containing multiple comments. Current grantees and researchers submitted the responses. There was some support expressed for the measures and ACF’s intent to increase our understanding of Healthy Marriage and Responsible Fatherhood programs. This document describes ACF’s responses to the emails in eight major areas in which comments were received:

1. Literacy levels

2. Length

3. Appropriateness of questions

4. Youth survey

5. Case management expectations

6. Mode of administration

7. Quarterly reporting

8. Miscellaneous

1. Comments regarding literacy levels

A number of commenters voiced concerns regarding low levels of literacy among participants in Healthy Marriage and Responsible Fatherhood programs and how low literacy levels would impact both comprehension of survey questions and how long it would take for participants to complete the surveys. Comments in this area also spoke to the large number of participants for whom English is not their first language and the need for surveys in languages other than English, particularly in Spanish.

ACF Response: We understand the concerns expressed about participant literacy levels and have plans in place to address these concerns. First, we are developing an electronic system called Information, Family Outcome, Reporting, and Management (nFORM) to collect data, which grantees will be required to use. Among other features, this system will include an option for participants to listen to the questions being read aloud through headphones and answer them directly using a computer or tablet (known as an audio computer-assisted self-interview -ACASI). The web-based nFORM system will be programmed to include all performance measures and will generate the proposed periodic reports. The system and associated grantee training and technical assistance will be provided to grantees at no cost. Grantees will use a portion of their grant funds to purchase and maintain computers or tablets for the electronic data collection, and headphones for the ACASI option. Spanish versions of all performance measures will also be available within the nFORM system, with the ACASI option, for those participants who prefer to respond in Spanish. The translated measures will be clear and specific and will use dialect-neutral terms as much as possible.

2. Comments regarding the length of the data collection instruments

Numerous comments suggested the data collection instruments were too long. Respondents expressed concerns that the length of the surveys would take away from time providing services, discourage potential participants from using services, and result in poor quality data due to “survey fatigue.”

ACF Response: As noted above, ACF will provide grantees with a management information system, nFORM, at no cost. This system will include capabilities for electronic administration of the performance measures (either via computer or tablet) and will have an ACASI option available. Using the electronic format with ACASI is likely to significantly reduce the time necessary to take the surveys for two key reasons. First, the system will automatically direct each person to the appropriate set of questions based on the person’s previous responses, so they will not have to navigate skip patterns. Second, ACASI will prevent the need to read the questions and responses, which could be arduous and time consuming for those with low literacy.

To further reduce the time required to complete the instruments, ACF is planning to shorten the length of the pretest and posttest by eliminating some questions.

3. Comments regarding the appropriateness of questions

Several comments questioned the sensitive nature of the questions. These comments expressed concerns that participants might find some questions offensive and be unwilling to respond to the questions or refuse services. Responders were also concerned about how minority race/ethnicity participants would react to the questions.

ACF Response: We understand some questions may be sensitive for certain populations. Participants are always free to skip any questions that they are not comfortable with and responses to questions on the pre- and post-tests will be kept private. The questions proposed are identical or similar to those that have been used satisfactorily in other efforts, including evaluations of Healthy Marriage and Responsible Fatherhood services. Further, the questions have been pre-tested with current Healthy Marriage and Responsible Fatherhood adult service recipients, including those from minority race/ethnicities, and no questions were refused or identified as being offensive or inappropriate.

4. Comments regarding the youth survey

Multiple comments were received regarding the appropriateness of the questions proposed for youth. Comments in this area suggested the questions could be better tailored for youth and should be more developmentally appropriate. There were also concerns that requiring this level of data collection from youth would necessitate school district and parental permission.

ACF Response: ACF appreciates the comments and helpful suggestions of alternative measures for youth. We will reexamine and revise the youth survey where appropriate and possible. The revised survey can be requested when ACF submits the instruments to the Office of Management and Budget (OMB) and publishes a 30-day FRN (notification of the new FRN will be made to the public in the same way it was for the past FRN). Techniques for obtaining school district and parental permission have been successfully used in other program areas, and will be shared with grantees.

5. Comments regarding case management expectations

Some commenters voiced concerns about the expectation to provide case management and follow-up with individual participants about referrals made to other community organizations or providers. Educators were particularly concerned that any requirements to provide case management would detract from their ability to provide classes.

ACF Response: Although we recognize that case management will require additional efforts on the part of grantees, ACF believes that case management is crucial to providing the support necessary to assist program participants. As such, reporting on case management will remain part of the performance management universe. However, ACF does recognize that in certain cases, such as youth relationship education programs being offered in schools, case management may not be appropriate or necessary. In those instances where OFA determines case management is necessary, nFORM will make case management easier to track for program providers.

6. Comments regarding the mode of administration

A small number of comments were received regarding administration of surveys online. Concerns were expressed about participants having difficulty completing surveys online as well as concerns about prisons and schools not having access to the internet.

ACF Response: Although online surveys will be required, paper and pencil surveys will still be available for those rare instances where internet access is unavailable (e.g. an outage) or a participant does not want to complete the surveys online. However, the online survey questions for participants will be presented in a clear, simple and straightforward manner and training and technical assistance will be provided to grantees so that they can assist participants as necessary.

7. Comments regarding the Quarterly Progress Report

A small number of comments were received concerning the proposed Quarterly Progress Report (QPR). Commenters questioned the usefulness of the QPR in relation to the time required to complete the reporting requirements. In addition, there was one comment about a three-month reporting period not being sufficient to get a clear picture of what is happening with a grantee.

ACF Response: ACF believes that quarterly reporting will allow for improved grant and program management as well as continuous quality improvement. nFORM will greatly reduce the reporting burden for grantees (for both the QPR and PPR) by automatically generating the data and inserting it into a draft QPR (which the grantee will need to review for accuracy, add narrative, and then submit to ACF).

8. Miscellaneous comments

Multiple miscellaneous comments were received, the majority recommending specific changes to questions, including clarifying language, or the addition of items to the performance measures. Due to the large number of suggestions on specific items (165), those will not be described here. We will adopt these suggestions as appropriate and possible.