ED response to public comments

Responses to public comments on OMB package for the Evaluation of Comprehensive Technical Assistance Centers

1. The burden on the Comprehensive Assistance Centers (CACs) in terms of time and effort in responding to the national evaluation is high.  Several elements make it so.  One is how they define audience (below), another is the selection of 6 projects--this may be particularly difficult in CACs that represent one or two states--yet another is the National Evaluation interview schedule and site visits to CACs. Some Content Centers evaluate the utility or usefulness of their own products and services through interviews with Regional Center staff. This provides Content Centers with very important formative data for understanding regional needs, and for improving their products and services. A continuing cycle of improvement is not possible without such data. The collection of summative and formative data adds to the burden problem.

Response: We will be mindful of the burden on CACs. The determination of the number of projects sampled from each Center is based on the minimum number of projects that are required to represent the breadth of each Center’s work. The interview schedule consists of one day site visits to the Centers. Only one site visit will be conducted.

2. The burden on SEA participants is high for some of the same reasons listed above, and may have the unintended effect of depressing their participation in the CAC work. Motivating SEAs to engage with and maintain improvement work has been an ongoing challenge for some CACs.

Response: The data collection instruments in this OMB package do not include data collected from SEA participants. The data collection instruments to gather information from SEA staff are being developed and will be included in a July 2007 submission to OMB. As we move forward, we will be mindful of the burden and do all that we can to keep the surveys of SEA staff as brief as possible. However, because the national evaluation is responding to a legislative mandate to evaluate the extent to which the Centers are meeting needs of their clients, some response burden is warranted and necessary.

3. The inventory form specifies a "single audience" as one criterion for identifying "projects" and requires CACs to fill out the form for each audience separately.  But some CAC projects are intentionally aimed at reaching across multiple audience types. The goal in these instances is to foster common understanding around particular issues in order to increase the project’s usefulness for clients’, and to maximize the impact on implementation (for example, sustaining collective work in particular areas).  Reporting requirements conflict with this important purpose, complicate reporting procedures, and increase reporting time.

Response: We acknowledge that some Center projects deliberately reach across multiple audience types and therefore will drop “single audience” as a criterion for identifying projects.

4. Collaboration is defined as interaction with organizations within the federal system.  CACs spend time collaborating with other types of organizations thus yielding expertise that contributes to and adds value to the overall knowledge resources of the centers.

Response: The Collaboration column on the Project Inventory Form is designed simply to allow the evaluation team to identify those projects that are joint efforts of more than one Center. This is essential information for our sampling frame. Collaboration with other types of organizations is a topic that Center respondents will be welcome to address in as much detail as they choose in the onsite interviews.

5. In the Site Visit Interview Protocol, clarification on the difference between a “strand” and the topic areas outlined in the project inventory protocol is needed.

Response: There is no difference between a “strand” and the topic areas on the inventory protocol.

6. Topic areas are listed in the protocol, but it is unclear if Centers will be asked to discuss their work only in terms of the pre-defined topic areas, or whether they can use categories defined in their project design/plans?

Response: On the inventory, Centers must keep to the pre-defined topic areas. So, during the site visits it will be helpful for Centers to discuss their work in terms of the pre-defined topic areas, as it will facilitate our understanding of the projects on the inventory. However, Centers can also discuss their work during the site visits using categories in their project plans, but during the site visits, it will be necessary for us to reach agreement on assignment of the projects to the pre-defined topic areas. Our pilot analysis of several Centers’ management plans indicates that this will be feasible.

7. Clarification is needed on who will be interviewed at each Center. The cover letter states that the director and members of the staff will be interviewed. Will all staff be interviewed or only certain key staff? Will partner “staff” be interviewed or only staff of the prime grantee? “Members of the staff” should be clarified so that Centers can plan and make required staff available on the day of the site visit.

Response: It will depend on the Center. We expect that many Center directors, having seen the questions well in advance of the site visit, will delegate particular questions to other key staff, or will want other key staff to participate. We will interview all staff members identified by the Center director, as feasible within a one-day visit. We will not interview all staff. Partner staff or sub-contractors will be interviewed if the Center director feels it necessary.

8. The directions for the Site Visit Interview Protocol should provide some indication of the length of each interview and clarification of the questions that will be posed to the director and those that will be posed to staff. Otherwise, it is assumed that all interviewed staff will be asked all questions.

Response: The Center director should be available for the one-day visit. We expect that 2-3 hours will be spent covering questions on the interview protocol with the Center director. As needed to answer all questions fully other key staff may (at the Center Director’s discretion) be interviewed for up to one hour. In addition, we expect to spend 1 - 2 hours providing training on completing the Project Inventory Form.

9. For question #5 on the Site Visit Interview Protocol, how precise should Center staff be when discussing the amount of resources allocated? Is an estimate of time sufficient or is more detailed information needed?

Response: An estimate of time is sufficient. We will delete the first two prompts under Question 5, which ask for information about resource allocation. Instead, we will ask on the Inventory Form for a simple classification of the project as representing a major, moderate, or minor level of effort compared with other Center projects.

10. On the Instructions for Completing an Inventory of Projects, I propose that “Single audience” be redefined as “Intended audience”. The term “Single Audience” implies that the audience is entirely homogeneous in terms of their position. In reality many events may be composed of individuals for whom the event was not intended. Use of the term “Intended Audience” would clarify whether the event should be included under one or two or more separate projects.

Response: This criterion (“single audience”) will be dropped from the instructions accompanying the project inventory form.

11. May Centers include projects for which major activities were conducted by the end of June 2006 but which have potential continuation activities that have not yet occurred?

Response: Yes, all projects including ongoing projects should be listed on the Project Inventory Form.

12. In the second paragraph on page 10 of Part A and on the table in Exhibit 4 (page 12) the analysis utilizes a value for the “Proportion of level of effort.” However, neither the Project Inventory Form nor the Request for Materials for Expert Panel Review collects information on the level of effort. Providing this information would be very problematic because we (the Northwest Regional Comprehensive Center) do not track staff time on a project basis. Major changes would need to be made to the existing accounting system to accommodate a request of this nature.

Response: We will revise the Project Inventory Form to deal with this issue. The revised form will have an additional column that asks Centers to indicate whether they consider this to be a major, moderate, or minor project. This determination will be relative to the Center’s other projects and it is expected to reflect an estimate, not a detailed tracking of staff time.

3