National Evaluation of the Comprehensive Technical Assistance Centers
Statement for Paperwork Reduction Act Submission
Part B: Collection of Information Employing Statistical Methods
Contract ED-04-CO-0028
April 3, 2009
Prepared for
Institute of Education Sciences
U.S. Department of Education
Prepared by
Branch Associates, Inc.
Policy Studies Associates, Inc.
Decision Information Resources, Inc.
Contents
Part B: Collection of Information Employing Statistical Methods 1
B.1 Respondent Universe and Sampling Methods 3
B.2 Information Collection Procedures 4
B.3 Methods to Maximize Response Rates 4
B.5 Individuals Consulted on Statistical Aspects of Design 5
Part B: Collection of Information Employing Statistical Methods
The National Evaluation of the Comprehensive Technical Assistance Centers is Congressionally mandated under Title II of the Education Technical Assistance Act of 2002 (Section 204), Public Law U.S. 107-279. Title II requires that the National Center for Education Evaluation and Regional Assistance (NCEE), a division of the Institute of Education Sciences, U.S. Department of Education (ED), provides for ongoing independent evaluation of the Comprehensive Centers.
The 21 Comprehensive Centers provide support for the implementation of the No Child Left Behind Act (NCLB). The Educational Technical Assistance Act of 2002 authorized the Comprehensive Centers to provide technical assistance with NCLB implementation and the improvement of academic achievement. It gave discretion to ED to determine the priorities of the Centers (Sec. 207 of the Act). Using this authorization, ED designed the system of Centers now operating under cooperative agreements. First, ED charged the Centers with serving states as their primary focus because of the states’ pivotal role in supporting district and school implementation of NCLB. Second, ED established a two-tiered system of 16 Regional Comprehensive Centers (RCCs) and 5 Comprehensive Content Centers (CCCs) (Federal Register, June 3, 2005, p. 32583).
The evaluation of the current Comprehensive Centers began in September 2006. NCEE designed this evaluation both to meet the requirements of the legislative mandate (described below) and to support program improvement. The evaluation collects information about each Center’s annual work plans, the products and services they provide, and their collaboration with the States, each other, and other ED-supported technical assistance providers. Independent experts review the quality of each Center’s services and products, and clients complete surveys about their experience with the same set of services and products.
The statute established the following specific goals for the evaluation:
Analyze the services provided by the Centers
Determine the extent to which each of the Centers meets the objectives of its respective plan
Determine whether the services offered by each Center meet the educational needs of State educational agencies (SEAs)
Through implementation of the data collection efforts described in the evaluation’s previous OMB submissions, the evaluation has been able to collect information that will be used to describe the goals, structure and operations of the 21 Comprehensive Centers, assess the quality, relevance and usefulness of Centers’ work, and describe the extent to which this work addresses states’ priorities for technical assistance. The evaluation has also collected a limited amount of descriptive information related to the expansion of states’ capacity to implement key provisions of NCLB. The proposed data collection seeks systematically to gather more information regarding expansion of states’ capacity through the work of the Comprehensive Centers.
This is the third of three clearance requests submitted to OMB for the National Evaluation of the Comprehensive Technical Assistance Centers. The evaluation of the Comprehensive Technical Assistance Centers, as designed by NCEE, included an optional task to conduct Case Studies to identify lessons learned through the experiences of the Comprehensive Centers and to provide examples of expanded state capacity to meet SEA requirements under NCLB. The current submission is necessitated because NCEE exercised the Option within the Base Contract to conduct Case Studies of the Comprehensive Technical Assistance Centers. ED and the study team believe that conducting the work described herein will complement the ongoing evaluation objectives being carried out in the Base Contract. The data collected through the proposed case studies will provide in-depth descriptive information related to the nature of capacity-building initiatives undertaken by the Comprehensive Centers, how RCCs and CCCs work together to build the capacity of state staff, and specific examples of how capacity has or has not been expanded from the perspective of key SEA officials.
In addition, the case studies with selected SEAs present a timely opportunity for ED to learn about the issues that the passage of the American Reinvestment and Recovery Act (ARRA) may pose for states. To support the information needs of officials within ED, in light of the challenges and opportunities related to the recently adopted American Recovery and Reinvestment Act (ARRA), we have also added a protocol to learn about the potential needs for technical assistance that ARRA implementation poses for key SEA officials in states selected for the case study.
A sample of 10 cases will be identified for the case studies. We define a case as an SEA and its corresponding RCC as well as relevant CCCs. Through semi-structured interviews with SEA staff and staff at the relevant Comprehensive Centers (both RCCs and CCCs), the study team will collect information that describes the nature of technical assistance and capacity-building initiatives undertaken in the selected states and provides examples of how state capacity has been expanded, as perceived by SEA officials.
The overall goal of the case studies described in this submission is to address one of the key concerns of the national evaluation of the Comprehensive Technical Assistance Centers – the extent to which the Comprehensive Centers have expanded SEA capacity to meet the goals of NCLB. SEAs are charged with several responsibilities related to the implementation of NCLB, among them:
Building or managing a statewide system of support (SSOS) for districts and schools identified for improvement under NCLB;
Training or managing the state-level staff or school support teams (SST) who provide support to districts and schools identified for improvement under NCLB;
Supporting use of assessment data by schools and districts;
Disseminating information on scientifically-based research to districts and schools;
Providing training and other professional development to local educators in academic subjects (reading/language arts, mathematics, science);
Monitoring compliance with NCLB requirements in districts and schools;
Communicating with parents or the public;
Implementing policies and practices for English language learners;
Administering supplemental education services and choice provisions;
Formulating state policies that respond to NCLB requirements; and
Providing technical assistance to LEAs or other state-level staff.
This list of state responsibilities was identified through an analysis of the law’s key requirements for SEAs conducted by the study team. The first two responsibilities have been combined and, in the sections that follow, are referred to as State Systems of Support/ School Support Teams (SSOS/SST). For each state and its corresponding RCC identified as a case, our design calls for us to focus on two of these responsibilities.
Defining Capacity Building
In addressing the question of what role the Comprehensive Center network has played in enhancing the capacity of the states to meet NCLB requirements, we employ definitions of key terms identified in recent literature by LeFloch et al. (2008) and Linnell (2003). 1
We define SEA capacity to implement NCLB as the SEA's ability to achieve its mission related to NCLB over the long-term without direct assistance from external agencies. This definition reflects seven core elements, four of which we believe the CC system was designed by ED to influence and three of which are not part of ED's vision for SEA capacity building by the CC network:
The SEA’s financial resources;
Legislative and gubernatorial support;
Number of staff within the SEA dedicated to NCLB-related tasks;
The ability of the SEA to provide leadership in its work with schools and LEAs;
Technology, systems, and infrastructure in place in the SEA;
The knowledge, expertise, and skills of SEA staff; and
The SEA’s access to expertise in key areas related to its mission.
We will focus the case study on three (highlighted) of the last four elements – those factors that we believe the Comprehensive Center system can influence – while acknowledging the significant role that the first three factors can play in the enhancement of an SEA’s capacity. We will not focus on one of these last four elements – the SEA’s leadership in its work with schools and LEAs – because our data collection for this study does not include interviews at the LEA or schools levels.
B.1 Respondent Universe and Sampling Methods
In our design phase, and in light of the resource constraints we face, we have tried to develop a purposeful sampling strategy that will yield the most useful combination of 10 cases (defined as an RCC, an SEA, 2 NCLB topical areas and Content Centers, as relevant) to support the program improvement objectives of this data collection around SEA capacity building.
There are 3 initial considerations of which we were mindful in this selection process. First, we wanted to limit the cases we considered for selection to those where:
The underlying NCLB topical areas are priority areas for SEAs;
The Comprehensive Center network has provided substantial and recent technical assistance in these areas; and
The underlying activity included SEA capacity building as a significant component of the overall workplan.
To identify these cases, we relied on data provided by senior state managers2 in fall 2007 and fall 2008 based on their experience vis-à-vis the preceding Comprehensive Center program years.
Upon identifying possible cases for inclusion, we then analyzed the senior state manager data further and found the following preliminary themes emerging:
In terms of NCLB areas of responsibility, the most important area to SEAs for TA is SSOS/SST. For instance, 42 of the 51 SEAs (including the 50 states and the District of Columbia) reported receiving substantial TA in this area in 2007-08. No other NCLB area was identified by greater than 18 SEAs as having been a priority where substantial Center technical assistance had been provided in 2007-08. Based on this, we made a decision to examine this area in each of the 10 SEAs and their corresponding RCCs selected.
For each SEA, we developed a capacity building index (CBI) of reported capacity building experience by using senior state manager responses, again, in those areas that were a priority for technical assistance from the state’s perspective and in which substantial Center network technical assistance had been provided. We then developed for each RCC, based on all SEAs within its geographic area of responsibility, an average CBI, and ranked the Centers. Given the variation we observed here, and to provide as much coverage of the Center network as possible, we decided to select one SEA from each of 10 CC regions,
To capitalize within our case study research design on CBI variation, we chose to select SEAs from RCCs at both the high and low ends of the distribution, permitting us to look for common factors in terms of the capacity building experience among similarly rated Centers. To the extent that common themes emerge across similarly rated Centers and the three dimensions of capacity building on which we will gather data, our reporting will allow us to identify potentially useful lessons for program improvement.
Within each Regional Center, we will seek SEAs whose responses most closely reflect the overall perception of all SEAs served by that Region. Thus, among the highest ranking strata of RCCs, we will select SEAs that provided a relatively high ranking on capacity building, and vice versa.
We also identified substantial within-SEA variation in terms of capacity building across NCLB topical areas. Thus, our 2nd topical area within each SEA will be selected based on that area that offers the greatest contrast in reported success in capacity building with the 1st topical area (i.e., SSOS/SST, as explained above).
B.2 Information Collection Procedures
Prior to conducting the interviews with SEA and Center staff, respondents will be notified about the study via an introductory letter, sent by ED. The letter will describe the purposes of the case studies and urge cooperation. Evaluation team members will then contact staff at the SEAs and the relevant RCCs and CCCs to discuss who might best serve as respondents for each case, when visits to the Centers and the SEA sites will occur, and the agendas for each site visit. During the site visits, research staff will conduct one-on-one interviews with respondents using the interview protocols included in this submission. All interviews will be audio recorded and transcribed, provided the respondents agree.
Information regarding states’ responses to ARRA will be collected during site visits to the selected SEAs. Interviewees will be asked to discuss their states’ reactions to ARRA, plans for utilizing these funds, and ways ED and the Center network can support them in their efforts. Again, all interviews will be tape recorded, provided the respondents agree, and transcribed.
B.3 Methods to Maximize Response Rates
Case study site visits will take place at the convenience of SEA and Center staff. In addition, to ensure a 100 percent response rate from the Centers and SEAs, in the introductory letter ED will stress the importance of the evaluation. The results of the data collection will be analyzed and written in reports that will be useful to the Centers as they seek to improve their own operations and effectiveness. Thus, the value of the information garnered from the case studies should provide an incentive for selected states and Centers to participate in the information collection.
The data collection process and protocols will be pilot tested with a small number (less than 10) of respondents from one SEA, its corresponding RCC, and relevant CCCs.
B.5 Individuals Consulted on Statistical Aspects of Design
The case study data collection plans were developed by Branch Associates, Inc., Decision Information Resources, Inc. (DIR) and Policy Studies Associates (PSA). The research team is led by Alvia Branch, Project Director. Other members of the evaluation team who worked on the design of the case studies include: Cynthia Sipe (Branch Associates), Siobhan Cooney (Branch Associates), Brenda Turnbull (PSA), Russell Jackson (DIR), Nancy Dawson (DIR) and Carol Pistorino (DIR), with the consultation of Paul Strasberg (Department of Education). Contact information for these individuals is provided below.
Alvia Branch
Branch Associates, Inc.
215-731-9980
Cynthia Sipe
Branch Associates, Inc.
215-731-9980
Siobhan Cooney
Branch Associates, Inc.
215-731-9980
Brenda Turnbull
Policy Studies Associates, Inc.
(202) 939-5324
Russell Jackson
Decision Information Resources, Inc.
(713) 650-1425
Nancy Dawson
Decision Information
Resources, Inc.
(713) 650-1425
Carol Pistorino
Decision Information Resources, Inc.
(650) 473-1564
Paul J. Strasberg
Department of Education
(202) 219-3400
1 Our definitions largely draw on those of LeFloch, K.C., Boyle, A., & Therriault, S.B. (2008). Help Wanted: State Capacity for School Improvement. Washington, DC: American Institutes for Research and Linnell, D. (2003). Evaluation of Capacity Building: Lessons from the Field. Washington, DC: Alliance for Nonprofit Management.
2 Respondents to the senior state manager survey were asked: When your state requests technical assistance from outside sources, what are its priorities? To what extent is each of the following state responsibilities related to NCLB implementation a priority for the technical assistance the state requests? Respondents indicated whether each of 11 NCLB responsibilities was a major priority, moderate priority, minor priority or not at all a priority for their state.
| File Type | application/msword |
| File Title | Abt Double-Sided Body Template |
| Author | Abt Associates Inc |
| Last Modified By | #Administrator |
| File Modified | 2009-04-14 |
| File Created | 2009-04-14 |