Responses to Public Comments Received During the 60-Day Notice (1830-0027 November 2023)

National Reporting System (NRS) for Adult Education

Information Collection Request

OMB Control Number NRS 1830-0027

Responses to Public Comments Received During the 60-Day Notice

New Categories Available for Reporting Participant Sex on Tables 1, 2, and 2A

Comment

Many commenters supported adding “non-binary” and “no answer” options for reporting participant sex. Multiple commenters noted that too many barriers already exist for adult education participants and forcing them to choose a sex that does not reflect who they are may prove detrimental. Another commenter stated that the addition of these columns would “allow the collected data to more fully represent all participants in adult education and family literacy programs.” Other commenters supported adding the new options for reporting participant sex but suggested making the change to all relevant tables since it may be incongruous to report binary categories in some tables but not others. Other commenters welcomed the proposed categories, but wondered how they would align across other WIOA core programs and partner categories. For example, one commenter noted that these categories do not align with the “participant did not self-identify” value outlined in WIOA. As a result, this commenter felt that the proposed changes could make it difficult to align data when cross reporting with WIOA partners. The commenter suggested that OCTAE consider aligning these reporting changes to the “participant did not self-identify” value. Another commenter asked how follow-up outcomes would be attributed in the first two years of implementation when the categories are not identical and posited that the change could affect data match processes which use students' reported sex as a matching field. A few commenters supported adding the new options but encouraged exchanging the word gender for sex. These commenters stated that gender is the better label for including female, male, and non-binary categories.

Discussion

We appreciate the broad support for adding the new options for reporting participant sex and agree that they will yield a more accurate representation of the students we serve. We acknowledge that these new categories do not align with the present version of the Statewide Performance Report in the joint information collection request (ICR) approved under OMB Control No. 1205-0526. However, the joint ICR is currently undergoing a renewal process and includes proposed revisions that incorporate new participant sex categories which would align with those proposed in this ICR. After the renewal processes for the joint ICR (OMB Control No. 1205-0526) and this ICR (OMB Control No. 1830-0027) are concluded, OCTAE will provide technical assistance to states for the aligned collection and reporting of participant sex data on both ICRs.

Regarding the comment about state wage data matching that uses student reported sex as a matching field, states should determine procedures for data matching participant outcomes that best meet the specific requirements of the state's wage record system. OCTAE will continue to provide targeted technical assistance in these situations.

We understand the recommendation to replace the word “sex” with “gender” in this information collection. However, such a change would not align with the language in section 116 (d)(2)(B) of the Workforce Innovation and Opportunity Act (WIOA) which requires information specifying the levels of performance achieved with respect to the primary indicators of performance for each of the core programs with respect to individuals with barriers to employment, disaggregated by each subpopulation of such individuals, and by race, ethnicity, sex, and age. We also note that the joint ICR (OMB Control No. 1205-0526) uses the same language of the statute.

Change

No change. After the renewal processes for the joint ICR (OMB Control No. 1205-0526) and this ICR (OMB Control No. 1830-0027) are concluded, OCTAE will provide technical assistance to states for the aligned collection and reporting of participant sex on both ICRs.

New Option for Educational Functioning Level (EFL) Placement of Participants on Tables 1, 4, 4A, and 4C

Comment

Numerous commenters supported the idea of exempting participants from NRS testing for placement purposes if they are enrolled in programs designed to result in Measurable Skill Gain (MSG) types other than 1a, which consists of comparing the participant’s initial educational functioning level, as measured by a pre-test, with the participant’s educational functioning level, as measured by a post-test. One commenter appreciated that the implementation of assessment exemptions would be at the sole discretion of the state, thus allowing for logistical considerations. One commenter felt such changes would improve partnerships since pre-post testing requirements can result in attrition and difficulties with partner agencies and employers. Another commenter stated that the goal of most participants is to pass a high school equivalency assessment and the curriculum for that purpose does not help make pre-post-test gains. As a result, this commenter stated that “post-testing requirements may lead to student dropout by taking too much time with activities not relevant to their goals.” Some commenters stated that these changes would lessen the burden on both students and programs. One commenter called for expanding the pre-test exemption to more types of programs and students, while another commenter wanted all states to be required to implement alternative EFL placement, rather than leaving the decision at the sole discretion of the state. One recommendation advocated for removing participants with an alternative placement from the post-test denominator when calculating the state’s post-test rate. Several commenters were more cautiously optimistic about the proposed change and had questions about how it would work in practice. Shared concerns and questions among commenters included: (1) What alternative methods would be used to assess a student’s eligibility under the statute and EFL placement? (2) How would this affect compliance with AEFLA? (3) How would the proposed change affect curriculum design? (4) How would this change impact the pre-test requirement to gain entrance into Integrated Education and Training (IET)? (5) How would it be possible to compare performance across states without a pre/post testing requirement for specific program types? (6) If it is optional and only some states opt-in, can we reliably measure and compare progress across all states and territories? (7) What is required in the second year of implementation? Is it the inclusion of the columns into the tables? (8) How will this change impact title II's reputation of robust accountability? (9) How might these changes impact a state’s ability to estimate performance for annual performance negotiations? Might there be an impact be on overall performance and, therefore, potential sanctions? (10) How would high school equivalency preparation programs ensure appropriate placement of students without a pre-test? (11) Are there definitions of the program types mentioned in the Supporting Statement such as “postsecondary bridge programs” that would be covered by this flexibility? (12) How will the flexibilities around pre-tests impact compliance with the demonstrated effectiveness requirement as part of AEFLA competitions? Several commenters also recommended that a new row for alternative placement be added to tables such as Table 1 in order to ensure that all participants are counted. As an auxiliary to the proposed change, a few commenters suggested that programs be allowed to use more authentic or competency-based approaches to assessment in order to place students. Another commenter recommended expanding state assessment policy to include details on use of alternative assessment/placement and documenting all MSG types, while another suggested adding details on tools and methods for alternative EFL placement and associated changes to the narrative report. A few commenters favored using the MSG type numbering system and provided other suggestions to enhance clarity with regard to reporting.

Discussion

We welcome the many discerning questions raised by commenters about the operationalization of exempting participants from NRS testing for placement purposes if they are enrolled in programs designed to result in MSG types other than 1a. As articulated in the Supporting Statement, the decision to develop and implement alternative options for the EFL placement would be at the state's sole discretion. To determine program eligibility, a state may use assessments that are content-driven or performance-driven such as locator tests, criterion referenced tests, diagnostic assessments, authentic assessments, or a state may choose to develop and implement crosswalks between program benchmarks and NRS EFL levels.

The State, at its sole discretion, must assess the impact of exempting participants from NRS pre- and post-testing who are enrolled in programs designed to yield outcomes other than MSG type 1a. This includes the impact on performance negotiations, the State's overall MSG performance, the loss of potential gains under MSG type 1a, high school equivalency preparation programs, curriculum design, determining the readiness of participants for certain programs, and considerations for ensuring the AEFLA requirement for an applicant's demonstrated effectiveness in grant competitions. It is for these reasons that states will not be required to implement alternative EFL placement methods. Moreover, the State, at its sole discretion, may identify the programs that are designed to yield outcomes other than MSG type 1a. This includes postsecondary bridge programs, which are generally designed to support student transition to postsecondary education or training. The examples provided in the Supporting Statement of programs covered by this flexibility are not exhaustive and are for illustration purposes only. OCTAE cannot anticipate the array of possible names for programs designed to yield outcomes other than MSG type 1a. States may introduce alternative placement methods incrementally and at their own pace, conduct pilot studies, and apply them only to certain programs the State identifies. OCTAE will continue to provide targeted technical assistance in this regard.

The proposed change in EFL placement would not preclude compliance with AEFLA requirements, as states will continue to track and report performance under the MSG indicator as required in the statute and joint performance regulations. MSG outcomes will continue to be comparable across states as a discrete primary indicator of performance. States will continue to report only the most recent MSG type for a period of participation. Additionally, measuring and comparing MSG performance across states should follow existing joint accountability requirements which remain unchanged. We expect this proposed change to EFL placement to have no effect on the title II adult education program's consistent record of strong accountability. On the contrary, we anticipate that local providers may leverage the opportunity to focus resources more efficiently on key aspects of program accountability in lieu of activities that are no longer required.

The regulations at 34 CFR Part 462 do not permit the expansion of the state assessment policy to include details on the use of alternative placement and documenting all MSG types. In accordance with 34 CFR 462.40(c)(1) and 34 CFR 462.40(c)(2), a state’s assessment policy must require local eligible providers to measure educational gain with a test that the Secretary has determined is suitable for use in the NRS and identify the pre- and post-tests that the State requires eligible providers to use to measure the educational functioning level gain of ABE, ASE, and ESL students. Other placement alternatives or MSG types are not covered by 34 CFR Part 462.

We appreciate and agree with the recommendation from several commenters to add separate rows on tables 1, 4, 4A, and 4C to record participants who were placed using alternative methods. Adding these rows will help to track participant placement and outcomes with greater accuracy, without adding burden. In October 2024, states that elected to implement alternative EFL placements during program year 2023-24 would have the option to report partial-year data for alternative EFL placements done during the remainder of program year 2023-24. In October 2025, states that elect to implement alternative EFL placements during program year 2024-25 would be required to report full-year data for program year 2024-25.

We concur with commenters who favored using the MSG type numbering system on the NRS tables and provided other suggestions, such as the wording of table headings and instructions. We have made changes to incorporate those recommendations from commenters. We also agree that participants with an alternative placement should be removed from the post-test denominator when calculating the state’s post-test rate.

Change

New “Alternative Placement” rows have been added separately for the ABE and ESL sections on tables 1, 4, 4A, and 4C to record participants who were placed using alternative methods. Revisions have been made to the footnotes of table 1 to clarify instructions for alternative EFL placement and to specify that participants with an alternative placement should be removed from the post-test denominator when calculating the state’s post-test rate. The MSG type numbering system and clarifying language edits have been incorporated into the column headings and instructions for tables 4, 4A, 4B, 4C, and 11.

Consolidation of Rows on Tables 5 and 5A

Comment

Several commenters agreed that the proposed table 5 and 5A rows should be consolidated. Other commenters agreed but gave suggestions. One commenter observed that the data reported for tables 5 and 5A represent four of the six primary indicators of performance listed in WIOA, which are all the primary indicators except for MSG and effectiveness in serving employers. To reduce existing confusion about the difference between MSG and exit-based indicators of performance, the commenter suggested amending the titles for tables 5 and 5A by adding “Other Than Measurable Skill Gains.” Another commenter suggested including the label “Exit-Based Indicators of Performance.” One commenter asked, if the proposed change did occur, whether they would be able to disaggregate data to determine outcomes separately for those who enter postsecondary education vs. those who enter employment. The same commenter asked whether the change would be aligned with other core title programs that report secondary credentials (such as Title I Youth) and recommended technical assistance and time for states to reprogram their data systems to accommodate this change. Several commenters noted that the current proposal only mentions combining secondary school diploma outcome rows for tables 5 and 5A and not the other tables that include the same outcome. These reviewers suggested making the same change to parallel tables. Another commenter recommended that “OCTAE should clarify in Column A, last row that this includes ‘recognized’ secondary and post-secondary credentials, not just ‘all credentials’ as this seems ambiguous.”

Discussion

We share the enthusiasm expressed in support of the proposed consolidation of the two rows for reporting secondary school diploma outcomes and the expectation that the change would result in clearer reporting instructions, streamlined procedures, more reliable performance data.

State data systems should continue to collect disaggregated data and track how participants meet the requirements in section 116(b)(2)(A)(iii). That is, participants who obtain a secondary school diploma or its recognized equivalent are included in the percentage of those achieving the outcome under the credential attainment indicator only if such participants, in addition to obtaining a diploma or its recognized equivalent, have obtained or retained employment or are in an education or training program leading to a recognized postsecondary credential within one year after exit from the program. It will be essential for state data systems to continue to determine whether the criteria described in section 116(b)(2)(A)(iii) have been met, in order to report the attainment of a secondary school diploma under the credential attainment performance indicator. However, once those criteria have been met, it is not necessary for the state to disaggregate the outcomes on table 5 and 5A.

States currently consolidate the attainment of a secondary school diploma on NRS table 4 and the joint MSG report. The consolidation of the two rows on table 5 would align with the reporting practice that currently exists on table 4 and the joint MSG report. Thus, we do not anticipate any significant need for reprogramming state data systems. Additionally, this proposed change does not affect the reporting criteria on the joint Statewide Performance Report (SPR) or the specifications for the credential attainment performance indicator.

We appreciate the questions raised about NRS tables 8, 9, 10, and 11 that also have two separate rows for reporting the attainments of a secondary school diploma. We agree that this change should also be made on those tables to align with the changes on tables 5 and 5A. To support further alignment, we have identified the opportunity to revise the wording of the title on table 8 to match other tables.

We agree with the recommendation of inserting language to provide greater clarity with respect to the row header in column A on the last row of table 5 “Attained any credential (unduplicated).” Adding the words “recognized secondary or postsecondary” will underscore the statutory criterion that not all credentials may be reported for this primary indicator of performance. In keeping with this recommendation, we have identified the opportunity for clarification by adding the word “recognized” on the postsecondary credential row of tables 8, 9, 10, and 11.

Finally, we welcome the suggestion to clarify the difference between the MSG and exit-based indicators of performance on tables 5 and 5A, by adding the words “Exit-Based” to the title of both tables.

Change

The rows for reporting the attainment of a secondary school diploma have also been consolidated on NRS tables 8, 9, 10, and 11. The row header in column A on the last row of table 5 has been revised to “Attained a Recognized Secondary OR Postsecondary Credential (unduplicated).” The words “Exit-Based” have been added to the title of tables 5 and 5A.

Revised Column Headings and New Reporting Rows on Table 14

Comment

Several commenters agreed with the proposed changes to table 14, which would revise column headings by type of subrecipient funding and add new reporting rows. One commenter stated, adding “new columns to describe funding received on table 14 will add helpful clarity and specificity.” The same commenter noted that “adding the online list of providers may help our state in using statewide, aggregate data from other states to better understand the comparability of programming across AEFLA states and aligns with publicly available information already on AEI’s website.” Another commenter added that the proposed changes to table 14 will help them with “understanding the provider base of the Integrated English Literacy and Civics Education (IELCE) program.” However, some commenters suggested that OCTAE consider several points of clarification for table 14. These points include clarifying whether “state funding” refers only to adult education funds or other fund types; making it clear in Column C that instructional programs cannot operate solely on 243 funds; and not using placement level for program types in the online listing of provider instructions. Several commenters did not support requiring administrator names the state’s provider website due to maintenance burden and one commenter suggested to instead require a link to the local provider’s website where the information could be available. One commenter expressed concern that the required elements reflect the language of the federal program and are not the service terms used by the public to search for services, rendering the directory less helpful to the ultimate consumer.

Discussion

We concur with the recommendation for greater clarity regarding column H on table 14. The column header and instructions would benefit from inserting language that refers only to state funds used for matching the federal adult education grant. We appreciate the suggestions regarding the instructions for the state’s online directory of providers and its content. The program types shown in the instructions for table 14 regarding the state's online listing of providers do not represent the actual terms a state must use on its website. They are illustrative for the types of programs that the state’s online directory could contain. States may continue to use terms that are easily understood by the public and that make such a directory more helpful to the consumer. We agree that the local administrator's name and program levels are not necessary and that an optional local provider website address would be helpful if it is available.

Regarding the use of funds allocated under section 243, we note the possibility that a state, depending on the needs of a local area, may award a grant to a local provider that consists entirely of funds for IELCE programming.

Change

Table 14 has been revised to remove the local administrator's name from the instructions regarding the content of the state’s online directory of providers and an optional local provider website (if available) has been added to the instructions. The reference to program levels has been removed from the instructions with language indicating that the program type nomenclature is illustrative. The header for column H and the instructions have been revised to specify that only state matching funds should be reported.

Proposals for Uniform Distance Education Reporting Criteria for Tables 4C and 5A

Comment

Many commenters supported having uniform distance education reporting criteria. Due to the remote locations of their programs, two commenters stated that they could only support a definition of distance education that accommodates a rural setting and includes “no need to physically see students before registering them, and having no minimum number of contact hours before they start distance education.” The commenters added that the internet can be sparse in these remote locations and advocated for a flexible definition of distance education that would allow their programs “to utilize that flexibility in offering distance education in ways that may not involve the internet, such as with books, worksheets or by telephone.” To develop a definition of distance education, another commenter suggested that OCTAE review the National Center for Education Statistics’ (NCES) definition as a starting point. This commenter suggested that OCTAE work with state directors, researchers, and other stakeholders to formulate a distance education definition and reporting criteria that go beyond a yes/no classification. Other commenters suggested OCTAE consider including modes of distance education when developing reporting criteria, such as synchronous and asynchronous remote learning, hybrid classes, hyflex classes, and live virtual instruction. One commenter asked that guidance be provided to differentiate distance education and out of class or homework assignments for students using an online or digital product. Several commenters stated that public comment is not sufficient for developing a standardized definition and recommended obtaining dedicated and synchronous stakeholder engagement around creating definitions of distance education and related reporting criteria. Another commenter recommended deleting the distance education tables altogether stating that “students should have access to instruction through a variety of technology supported options and it is up to the state and local practitioners to monitor the effectiveness of those instructional modes and modify.” One commenter stated that it is also time for a new title for “distance learning.”

Discussion

We appreciate the many prudent and sensible recommendations from commenters who responded to the call to propose a standard definition of distance education and a uniform set of reporting criteria that could be applied across all States and outlying areas and enhance the usefulness and validity of the existing distance education tables 4C and 5A. Although one commenter proposed deleting tables 4C and 5A, public comment and stakeholder input to date do not appear to support such action.

Public comments submitted in this round express support for additional stakeholder engagement to develop a definition of distance education that can be operationalized in all states for nationally standardized NRS reporting. In response to comments received through this process, OCTAE will begin to conduct such consultations with a broad adult education audience upon conclusion of this ICR renewal. The discussions will include considerations for aligning the tables to best reflect the full array of distance education services delivered by adult education providers in every state.

Change

No change. OCTAE will engage in a broad consultative process to develop a definition of distance education that can be operationalized in all states and standardized NRS reporting requirements.

Revision of Narrative Report Instructions

Comment

Several commenters agreed with the proposed change regarding narrative report instructions. One commenter noted these changes “will clarify and streamline reporting.” Another commenter appreciated the inclusion of a separate question to discuss IET. Other commenters agreed with the proposal but offered suggestions. Several commenters suggested reducing NRS tables to free up time for the collection and reporting of innovative programs models, which could be shared through the narrative. This same commenter stated that clarification is needed for the term “used funds” in question 2 and whether that means exact expenditures for each listed activity. Clarification is also needed for the term “state-identified goals for the reporting period” and whether these are internal goals or goals shared by the state before the reporting period. Commenters stated that the addition of prompts related to the state’s goals would require additional text to share those goals and to provide a meaningful response. One of these commenters recommended raising the page limit to twelve. There were also two comments related to the last two IELCE questions of the report regarding unsubsidized employment and workforce development system. One of the commenters recommended combining these last two questions while the other asked for language in both prompts to be retained. Another commenter suggested removing the language “…and the number of IET programs available in the State by career pathways” from question 6. This is because requesting the specific number of IET programs in the state by career pathway would be too challenging to collect. One commenter stated that the prompt “Describe the State’s efforts in meeting the requirement to provide IELCE services in combination with integrated education and training activities as described in section 243(a)” could be strengthened by adding a clarifying request asking states to describe how they are ensuring programs access to IET per Section 243. The same commenter recommended reinstating prior language about "challenges" and "lessons learned" related to the final two prompts of the IELCE narrative section, which pertain to placing IELCE participants in unsubsidized employment and integrating with the local workforce development system, and inserting language in the first IELCE prompt paragraph on enabling participants to achieve the skills needed to function effectively as parents, workers, and citizens. Another commenter recommended adding the words “for all measurable skill gain types” to the prompt regarding the State’s progress toward meeting its negotiated levels of performance. Other suggestions include offering information on how to fund AEFLA without NRS paired post-testing; detailing tools and methods for alternative EFL placement; and detailing assessment documentation for all MSG types.

Discussion

We welcome the recommendations for adding clarifying language in the narrative report prompts to help elicit more substantive descriptions of state activities during the program year. We agree that the prompts could be refined and strengthened with some of the additional edits put forth by commenters.

We acknowledge the concerns expressed about some of the new language in the topic prompts. The term “used funds” in the prompt for the topic “State Leadership Funds” does not require the inclusion of expenditure amounts. It is intended to focus attention on the statutory requirements for the use of state leadership funds. The term “state-identified" in the prompt for the “Performance Data Analysis” topic refers to state activities that the state may have developed on its own to address a state-identified data issue, beyond the performance accountability requirements of WIOA. These types of state initiatives or activities are not required by the NRS and may be developed by the state before or during the reporting period.

We agree with the concerns raised regarding the use of the word “goals” throughout the prompts and have replaced the term with the word “effort” or “initiative” where appropriate. We expect the revised prompts will yield more useful information on specific action that was taken, rather than engaging in a discussion of goal setting and whether they were met. We also agree that including information about IET programs "by career pathways" could require new tracking procedures which a state may not have in place. Accordingly, we have removed that language. However, we hold that the number of IET programs is information which states currently track and, thus, would not remove that from the prompt. We concur with the recommendations to restore certain language to the IELCE prompts with respect to “challenges” or “lessons learned” as well as including language pertaining to the statutory requirement to provide IELCE services in combination with integrated education and training activities and enabling adults to acquire the skills needed to function effectively as parents, workers, and citizens. We expect the narrative report to benefit from a continued focus on these areas. We anticipate that these revisions, made in response to public comments, will enable the narrative report to maintain its current limit of 10 page.

The suggestion to remove or consolidate the statutory language in the narrative prompts did not appear to be supported by the recommendations made by other commenters seeking a narrative report more aligned with the purpose of the statute. Regarding the recommendation to add new language about MSG types and alternative EFL placement to the “Performance Data Analysis” prompt, we note that the prompt is intended to elicit a discussion about all primary indicators of performance, not just the MSG indicator. Adding the words “for all measurable skill gain types” could inadvertently direct the respondent’s focus on one indicator at the expense of others. Moreover, “Performance Data Analysis” theme allows for a range of topics which the state may wish to describe in its narrative report. The examples provided in the prompt are not prescriptive but instead are intended for illustration purposes only. We maintain that the narrative report is designed to encourage the sharing of innovative program models which states may include in the information they provide.

Change

Certain language has been restored or added to the IELCE prompt consistent with program provisions in the statute. The term "goals" has been removed or replaced with action-oriented words throughout the narrative report prompts. The phrase “by career pathways” has been removed from the IET prompt.

Additional Suggested Measurable Skill Gain Types

Comment

Several commenters supported the use of incremental gain level within an EFL as a measurable skill gain, though many of these commenters had suggestions. One commenter suggested measuring incremental gains by using posttest increases from pretest by a number of points equal to or greater than half the level, or having the learner move from one quartile within the level on the pretest to a higher quartile on the posttest. Another commenter recommended the consideration of model-based approaches such as multidimensional item response theory and classical approaches such as standard error of measurement criterion. Numerous commenters provided recommendations to allow for additional types of MSG—most commonly, for passing one or more subject tests of a high school equivalency assessment. Although one commenter cautioned that an MSG based on HSE single subject test passage would not be stringent enough. Many commenters also suggested adding a type of MSG related to digital literacy. Some commenters suggested additional career-related types of MSG, pre-defined learner goals, as well as a type of MSG associated with passing the citizenship/naturalization test. Another commenter supported incremental growth that uses a competency-based approach. This commenter noted that such an approach would involve developing a series of tasks at each level on the skills needed to perform in the workplace and in daily life, with a rubric for performance evaluation by a trained rater. These tasks would align with the content of existing standardized assessments, but learners would contribute to their identification and development. Another commenter stated that incorporating incremental growth would result in better reporting of learner gains. However, this commenter requested that states be allowed to develop and pilot innovative, alternative forms of performance measures designed to capture a truer and fuller picture of the progress learners make. One commenter was supportive of using incremental gains but does not want the measuring of incremental progress to result in additional testing. This commenter suggested leveraging existing digital assessment and learning tools, which make pre-and post-diagnostic assessments quicker and less overwhelming for the learner. The commenter also added “any measurement of incremental gains needs to recognize that many adults are seeking educational services for reasons other than diploma, credentials, and employment.”

Discussion

We are deeply appreciative of the panoply of innovative ideas that commenters shared in support of acknowledging additional measurable skill gain opportunities and celebrate the multiplicity of achievements that our students earn each program year. We share the excitement expressed by numerous commenters about the potential for measuring incremental education gain within an EFL and the opportunities for establishing additional MSG types. We acknowledge the many commenters who supported the formal recognition of passing one or more subject tests of a high school equivalency assessment and share their interest in discovering additional ways to demonstrate measurable skill gains. To that end, we have begun a deliberative process to examine research-based proposals for establishing criteria to measure incremental gain within an EFL, using NRS-approved assessments, and to consider other possible options for achieving a measurable skill gain, including the passing of one or more subject tests of a high school equivalency assessment.

Changing the performance accountability criteria for MSG types articulated in joint guidance is beyond the purview of this information collection request and requires a joint consultative process across the WIOA core programs. As we engage with our WIOA partners to consider possible changes to the joint guidance on WIOA performance accountability, we will confer with adult education stakeholders to weigh the potential effects of any changes on the AEFLA program.

Many commenters expressed the importance of acknowledging digital literacy skills in the NRS. The proposed new option for alternative EFL placement of participants in programs designed to yield MSG types other than MSG type 1a would provide states with the opportunity to measure and report the attainment of digital literacy skills in those types of programs.

In addition, the instructions on Table 1 have been revised to provide the definition of IET programs which encompass workforce preparation activities that help participants acquire a combination of skills, including digital literacy skills. We anticipate that the explicit naming of “digital literacy skills” in the context of the alternative EFL placement instructions will encourage states to consider how the attainment of such skills may be measured and reported. However, it is important to note that workforce preparation activities must be provided concurrently and contextually with adult education and literacy activities and workforce training in an IET program - not as stand-alone activities.

Under the proposed flexibilities for alternative EFL placement, states would have the discretion to use digital literacy assessments or other appropriate tools that the state determines as appropriate for measuring and reporting digital literacy skill gains and that are consistent with the purpose of the adult education service being delivered. For example, the use of digital literacy assessments and related certifications could be appropriate for reporting digital skill gains in the context of a workplace literacy or IET program.

Change

OCTAE will continue to engage in a consultative and deliberative process to examine research-based proposals for establishing criteria to measure incremental gain within an EFL, using NRS-approved assessments, and to consider other possible options for achieving a measurable skill gain. In response to recommendations to acknowledge the attainment of digital literacy skills in the NRS, the instructions for alternative EFL placement on Table 1 have been revised to include the definition of IET programs which encompass workforce preparation activities that help participants acquire a combination of skills, including digital literacy skills. OCTAE will provide technical assistance to states as they consider the opportunities for alternative EFL placement and the potential to measure and report digital literacy skills in the NRS.

Implementation Considerations

Comment

Multiple commenters noted how the proposed changes may require additional support to fully implement. One commenter was in favor of more flexibility in the ways Title II providers can document student progress. However, this commenter stated that more choices often result in more complexity. Because of this, states will need additional funding to upgrade their data collection systems. Similarly, another commenter argued that new options for EFL placement would place an undue burden on states to create a new system of determining EFL placement and placement in classes. To offset this burden, the commenter requested more funding to upgrade state data collection systems. Another commenter expressed concern that the inclusion of a requirement for the state to host a provider directory with identified specific elements would encompass a significant amount of ongoing work by state staff to collect and maintain such a directory. The same commenter noted that State systems that are used by local providers would need an overhaul to accommodate the proposed changes and state guidance would also require an overhaul to communicate the changes. The commenter also pointed out that States that use a commercially developed data system would still require time for the training of practitioners as well as data quality checks and validations as these changes are implemented. Regarding the proposed changes to the narrative report instructions, one commenter suggested that states be allowed a full year with “knowledge of these changes” before having to report on the 2024-25 year. In terms of creating a standardized definition of distance education, a commenter stated that defining terms for in-person, hybrid, blended or hyflex models will be too burdensome for systems enhancement and compliance.

Discussion

We anticipate that the proposed flexibilities for EFL placement would bring new opportunities for increased efficiencies but also new logistical or resource challenges that states must consider before deciding on a particular course of action. It is for this reason that states are not required to implement the proposed alternative EFL placement options. They would have a wide array of options from which to choose, including the option to continue their current EFL placement procedures.

The ongoing maintenance of state data systems is a routine administrative component of the state's established business procedures for program management. The software revisions potentially necessitated by the proposed table changes could require minimal coding adjustments. For example, the proposed consolidation of the two rows on table 5 do not require new calculations and may require the abbreviation of existing software code. Data systems already have the capability to extract a consolidated number, since they must identify the number of participants who attain a secondary school diploma before disaggregating those numbers across the two existing rows on table 5. The intent of establishing a standardized definition for distance education is to clarify the criteria for reporting participants on existing distance education tables. We would not expect the need for changes to state data systems, as states are already reporting participant outcomes on these existing distance education tables. However, these aspects will be considered in the consultations with adult education stakeholders.

The proposed changes in this renewal of the information collection are designed to facilitate streamlined state and local data collection methods, simplified program management and delivery of services, and other beneficial procedural efficiencies that result from the improved clarity of reporting requirements. States currently collect and maintain the pertinent data reported on the tables with proposed changes, such as provider information on the state’s website. States also conduct annual training activities which include an array of performance accountability topics covered by these proposed changes. These training events are already a component of the state's routine business activities and could be modified to include updated information to local providers.

The December 2024 implementation of the revised narrative report prompts will afford considerable flexibility to respondents. States will continue to have broad latitude to write a narrative report that reflects the state's adult education program implementation. States will not be required to address prompts for which the State did not have sufficient time to prepare. By December 2025, we anticipate that states will have sufficient time to become familiar with the revised prompts. OCTAE will continue to provide technical assistance in this regard.

Change

No change. OCTAE will provide technical assistance to states in support of their planning and implementation efforts.

Other Comments

Comment

In addition, several commenters called for reporting/counting all MSGs made by a student in a program year, not just the most recent, as a way of acknowledging their successes and possibly increasing students’ motivation to persist. Related to testing, one commenter suggested eliminating minimum hours before post-testing, as not all students need the same amount of time to demonstrate gains, while another commenter suggested tying learning outcomes to rigorous curricula rather than relying on standardized tests. Another commenter suggested removing the progress testing requirement after a student earns the first MSG in the fiscal year.

Discussion

These issues raised in these comments are determined by joint performance accountability requirements or by the instructions of commercially developed products and thus are beyond the purview of this ICR. The NRS does not currently require additional testing during a period of participation after the first MSG is achieved.

Change

No change.

Comment

Commenters suggested a variety of labeling and clarification or guidance changes including clarifying that transition to postsecondary does not require exit; providing guidance on MSG Type 4; and, providing guidance on allowable activities for corrections, such as special education and secondary school credit within Sec. 225 corrections education programs. Commenters also suggested a wide range of reporting changes. For example, a few commenters suggested having additional breakouts on existing tables, such as by program type and student background characteristics or teacher demographic data; adding a new table for reporting all MSG types to expanded groups of students; adding a workplace literacy table for gains; adding a new Table 1a to report participants who were exempted from pre- and post-testing; adding region/subregion categories to race options, or a "prefer not to answer" choice; adding digital literacy reporting for all participants; eliminating separate tables for distance education; making barriers to employment cumulative on Table 99; dropping reporting by period of participation; not having optional tables or rows; changing the order of items in the ICR instrument to benefit readers by positioning the narrative report and assessment policy at the beginning of the instrument; and making reporting more in line with the PIRL.

Discussion

We understand the continued need to provide clear guidance and timely technical assistance for a range of programmatic implementation issues. These topics are beyond the purview of this ICR renewal. However, we will continue to review existing guidance and technical assistance materials for opportunities to provide further clarifications or to create new materials that help to address program management needs.

Some of the proposals raised by these comments, such as adding new tables or collecting new data elements, could have the potential to add respondent burden that has not yet been evaluated to be commensurate with the value of the additional data to be collected. We will examine the proposals for these changes closely and continue to engage in a consultative process with adult education stakeholders to determine the need and utility of any new tables or data elements.

Other proposals raised by these comments fall outside the purview of this ICR renewal, such as adding new categories for participant race and ethnicity, eliminating reporting by periods of participation, changes to the data collected on table 99, or changing the exit requirement for the MSG type “transition to postsecondary education.” For those topics that fall within the purview of the joint ICR on performance accountability, such as changes to MSG criteria, we will engage with our WIOA partners to identify opportunities for further enhancements. In response to the recommendation of aligning reporting with the joint WIOA Participant Individual Record Layout (PIRL) under OMB Control Number 1205-0526, we note that the NRS collects all jointly required performance accountability data in accordance with the joint PIRL.

The recommendations to remove optional tables or rows from this ICR – or to require them – is not in keeping with consistent feedback from ICR respondents who have expressed that optional tables and rows are needed for reporting optional program activities. We also clarify that the order of items in the ICR instrument is not the order in which program information is conveyed to the public. For example, the public may access state narrative reports directly on the NRS website, without first reviewing the statistical tables. The sequence of reporting items in the instrument is intended to align with the chronological order in which they are submitted to the NRS. However, we have noted that the assessment policy item should appear immediately after the data quality checklist and have moved it to that position.

Change

The order of items in the ICR instrument has been changed to move the assessment policy immediately after the data quality checklist, to align with the chronological order in which components are submitted to the NRS.

Comment

Some commenters suggested policy-related or administrative changes. Several commenters recommended counting all students who seek services by eliminating the 12-contact hour requirement to be considered a participant. Commenters noted that some students only need a few hours to achieve their goals. One commenter recommended using the narrative report and table 99 to negotiate performance targets instead of table 4, which the commenter felt is too focused on paired standardized test gains. Another suggested that, rather than use the term “barriers to employment,” OCTAE can support more learner-centered language such as “individuals confronting barriers” to account for the fact that students may be confronting multiple barriers and therefore can be counted in more than one place. One commenter stated that the “IELCE programs delivered under WIOA Section 243 are yoked to a requirement to offer access to IET that limits programs’ flexibility to meet the needs of their participants” and “plagued by implementation challenges linked to its complex requirements since its inception.” Another commenter did not support the employment condition placed on the attainment of a secondary school diploma in section 116 of WIOA. The same commenter asserted that “completing secondary education is a great accomplishment in itself and should be acknowledged that way.” Commenters also suggested changes to the state data quality checklist such as requiring the state assessment policy to include protocols for measuring and documenting learning for all MSG types, differentiating credentials to include both recognized postsecondary credentials and other credentials of value, or requiring states to exempt participants pre- and post-testing.

Discussion

We recognize and appreciate the comments on a range of program administration issues, such as concerns about meeting employment criteria in order to count a secondary school diploma, eliminating the 12-contact hour rule to be considered a participant, the statutory requirements of the IELCE program, or using learner-centered language in place of terms such as “barriers to employment.” These topics are the subject of joint regulations or established in statute and thus are beyond the purview of this ICR renewal. The proposals for changes to the performance negotiations process are also not within the purview of this ICR renewal.

The recommendations for changes to the state data quality checklist, such as requiring the state assessment policy to include all MSG types, are not supported by existing regulation. In accordance with 34 CFR 462.41(a), a local eligible provider must measure the educational gains of students using only tests suitable for use in the NRS and that the State has identified in its assessment policy. The purpose of the assessment policy is to set forth the state’s requirements for assessment administration under 34 CFR 462.40, 34 CFR 462.41, and 34 CFR 462.42. Other MSG types are not covered by 34 CFR Part 462. The purpose of the Data Quality Checklist (DQC) is for the state to attest to the quality of the annual data submitted. The MSG indicator is treated as a whole indicator in the context of the DQC, like all other WIOA primary indicators of performance.

In regard to the recommendation to document learning for all MSG types, we note that the purpose of the existing joint MSG table is to report multiple MSGs achieved during a period of participation. Since the inception of the joint MSG table, we have identified the need for technical assistance in its use and provided widely available clarification to improve data collection and reporting accuracy. We will continue to provide targeted technical assistance when questions about this topic arise or when we identify issues through our data verification procedures.

Change

No change. OCTAE will continue to provide technical assistance for questions about the statute, regulations, and joint performance accountability requirements.

NRS Responses to Information Clearance Request 60-Day Notice

0