Student Achievement Responses to OMB 7-13-07

OMB Comments On

Postsecondary Student Achievement & Institutional Performance Pilot Program Grant Application

1. How will the results of this pilot program be used? Will there be an effort to encourage institutions outside of the consortium to adopt the strategies developed in the pilot program?

The consortium will be expected to develop an array of methodologies for evaluating student achievement and institutional performance that will serve a variety of purposes and stakeholders. The breadth of coverage will be great enough that institutions outside of the consortium will be able to adapt and implement the appropriate methodologies developed by the pilot program for use on their own campuses. In other words, the pilot will develop methodologies that will cover a large range of student majors, achievement outcomes, and institutional types. We anticipate significant involvement from the higher education associations, and they will be instrumental in encouraging other member institutions to adopt and implement the methodologies developed by the pilot.

In addition, FIPSE will require future awardees to identify which pilot program methodologies will be incorporated into their individual project evaluation plans and annual reports. This will enable FIPSE to begin collecting data on the impact of FIPSE funding on student achievement and institutional performance. It will also enable staff to conduct more comprehensive portfolio analysis by identifying all of the projects using particular pilot program measures and methodologies, and then comparing project level outcomes based on those methodologies.

2. Will ED be able to continue to monitor the extent to which this pilot led to institutional change and created a replicable model beyond the 18 month grant period? 

Absolutely! Because we will require future awardees to incorporate the appropriate methodologies and measures developed by the pilot project into their own project evaluation plans, we will be able to monitor the adaptation and implementation of the pilot project for years to come. FIPSE staff will also incorporate the long-term evaluation of the impact of this program into our internal strategic plan, and through our work with higher education associations, we will encourage and monitor the continued expansion of the pilot project outcomes. Because this pilot program is crucial to meeting the goals of the Commission and the ACC, it will receive a great deal of staff attention from Department personnel.

3. Could ED somehow collect multiple years of data from the pilot?

The pilot program is more focused on developing methods and measures for collecting data than it is on collecting the data itself. We anticipate small scale data collection efforts that serve as proof-of-concept exercises, but during the 18 months of the award, there will be limited data collection activities. Applicants must describe how they will implement the measures, methodologies and reporting strategies developed by the pilot in the years beyond the funded period, and we will give these plans careful consideration during the review process. But the main focus of the pilot is the development of new methods for assessing student achievement and institutional performance in areas where such assessment has never been done before (and where assessment is far more complex than administering a test or looking at endowment dollars!).

4. Has ED received any public comment on this ICR?

To my knowledge, the only questions or comments we have received have been requests for the actual application package. We have also gotten two calls asking if higher education associations can be involved, to which we have responded in the affirmative.

5. Will OPE consider a consortium’s experience in conducting an evaluation for the evaluation component of the grant?

Yes, this project is essentially an evaluation project. The whole point of this effort is to develop new methods and measures for evaluating student achievement and institutional performance. As such, we anticipate that outside experts will be part of the consortium team, and we will absolutely consider the experience of the evaluators in selecting the winning application. It must be clear in the project description that evaluation will be at the core of this work and not an after-thought at the end of the funded period.

6. Will IES be involved in this project?

We will be issuing this award as a cooperative agreement rather than a grant so that we can have greater involvement and oversight in this project. As such, IES will be asked to assist OPE program staff in evaluating the appropriateness and efficacy of the measures and methodologies proposed by the consortium. In the event that the consortium is interest in considering the development of a database, they will be encouraged to work with NCES to define the technical parameters of such a database.

7. To what extent will student privacy/confidentiality be taken into consideration during the collection of data?

Our selection criteria make it clear that we expect the applicant to include in the narrative a description of how they will meet student privacy/confidentiality requirements in their project design. We will ask IES for assistance in reviewing and validating these plans.

8. To what extent will ED look for variation in the institutions that it selects to participate in the pilot?

The selection criteria indicate that the consortium must include institutions with a diverse array of missions and that serve a variety of students, including those who transfer among institutions and programs, and non-traditional adult students. We believe that this will necessitate the inclusion of a wide variety of institutions in the consortium. At a minimum, to meet this requirement the consortium would likely include at least one research institution, one undergraduate institution, one liberal arts college, a public institution, a private institution, and one or more two-year colleges or technical schools.

9. How many institutions will be selected to participate in the pilot program?

We anticipate awarding one cooperative agreement to the consortium. It will be up to the consortium to determine how many institutions will be included in the consortium, but the project narrative will have to meet the requirements of including institutions with diverse missions and that serve a variety of students, including transfer students and non-traditional students.

10. Question 1 of Supporting Statement states that “This is a new FIPSE Special Focus Competition and was initiated in response to concerns raised by the Secretary of Education’s Commission on the Future of Higher Education and the Academic Competitiveness Council regarding the lack of long-term data on student outcomes and institutional performance.” When did the Commission raise these concerns? When did ED initiate the development of the competition?

Both the Commission report, which was released earlier in September 2006, and the ACC report, which was released in May 2007, makes multiple references to this problem. I believe that OUS started discussing the creation of this special focus competition upon the publication of the Commission report, and the final closing data notice and application package was developed during the months of May and June, 2007.