Southeast AMSTI_OMB_Supp_Statemt_A_Rev_06.29.07

omb package

2.1.3: The Effectiveness of the Alabama Mathematics, Science and Technology Initiative (AMSTI)

Supporting Statement Part A

Date Submitted:

May 3, 2007

Contract Number:

ED-06-CO-0028

Submitted to:

Gil Garcia

Institute of Education Sciences

U.S. Department of Education

Submitted by:

Ludwig D. van Broekhuizen

REL-Southeast

SERVE Center

915 Northridge Street, Second Floor

Greensboro, NC 27403-2112

(800) 755-3277

(336) 315-7400

TABLE OF CONTENTS

OMB Package Supporting Statement A

1. Circumstances that Make Collection of Information Necessary 1

1.1 How Information is Necessary for Performance of REL-SE’s Function 1

1.2 How Information is Necessary for the State of Alabama 1

2. How the Information will be Used, by Whom, and for What Purpose 2

2.1 Utility to the State of Alabama 2

2.2 Utility to Educators, Policymakers and the General Public 3

2.3 Utility for Researchers 3

3. Use of Technical Collection Techniques and Information Technology 4

4. Identifying Duplication 4

5. Impact on Small Entities 5

6. Consequences if Study is not Conducted or if Conducted Less Frequently 5

6.1 Entire Study 5

6.2 Class Rosters, Student Demographics and Math, Reading and Science Achievement Test Scores 6

6.3 Teacher Web-based Surveys 6

6.4 AMSTI Study Teacher Classroom Observations and Teacher Interviews 6

6.5 Principal Web-based Surveys 7

6.6 Principal Interviews 7

6.7 Professional Development Trainer Logs 7

6.8. Professional Development Participant Surveys 7

7. Special Circumstances 8

8. Compliance with 5 CFR 1320.8(d) 8

9. Payment of Gifts 9

10. Assurances of Confidentiality 9

11. Questions of a Sensitive Nature 10

12. Estimate of Annual Hour Burden 11

A12.1: Number of Respondents and Frequency of Response 12

A12.2: Hour Burden by Each Form 19

A12.3: Annualized Cost for Each Instrument 19

13. Estimate of Total Cost Burden 21

14. Estimate Cost to the Federal Government 21

15. Reason for Program Changes or Adjustments 21

16. Publication Results of Data Collection 21

17. OMB Approval for Not Displaying OMB Date 23

18. Exceptions to the Certificate Statement 23

Supporting Statement for Request for OMB Approval of Data Collection/Needs Assessment for the REL-SE

Part A. Justification

1. Explain the circumstances that make data collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

This request by the Regional Educational Laboratory for the Southeast (REL-SE) seeks a three- year clearance to collect data as part of a study investigating the effectiveness of the Alabama Mathematics, Science, and Technology Initiative (AMSTI). This study will consist of a group randomized control trial with quantitative and qualitative data collection to enable the Alabama State Department of Education (ALSDE) to make decisions about the initiative.

1.1 How Information is Necessary for Performance of REL-SE’s Function

The REL-SE is one of ten regional laboratories funded by the U.S. Department of Education's Institute of Education Sciences for the purpose of providing research-based information and services to all 50 states and territories. These Laboratories form a nationwide education knowledge network, building a bank of information and resources shared and disseminated nationally and regionally to improve student achievement.

Since the passage of the No Child Left Behind Act (NCLB), REL-SE has been charged with state and district responses to NCLB. REL-SE is responsible for conducting studies that reflect NCLB's emphasis on evidence-based education and NCLB's requirements to improve student outcomes.

1.2 How Information is Necessary for the State of Alabama

This work is in response to a request by the Alabama State Department of Education (ALSDE) for a scientific study of the effectiveness of AMSTI. This study is needed so that ALSDE, following the requirements of NCLB, can make decisions about this initiative based on scientific data regarding the program’s effectiveness at improving student achievement.

The AMSTI program was developed by the ALSDE to improve the quality of mathematics and science instruction in grades Kindergarten through 12 (K-12) using technology. State staff have posted detailed information about the program on the AMSTI website (www.amsti.org). In addition, there are two documents, created by the State Department, which highlight key components of AMSTI. These include the Report on the Review of the Literature: Alabama Mathematics, Science, and Technology Initiative Committee, and the Executive Summary of the Annual Report on the Alabama Science in Motion Program.

2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

Under a contract with the U.S. Department of Education, REL-SE and its subcontractors will use the information as part of a randomized control trial. The evidence from this experiment will be used by the ALSDE and the Alabama legislature in consideration of whether to extend funding to the program being evaluated.

2.1 Utility to the State of Alabama

The information that will be collected, analyzed, and reported by means of the AMSTI study will be used by the state of Alabama to make decisions about program continuation, expansion, and improvement. In order to provide the precise information required by the state, the study evaluates the program’s theory of action. AMSTI was designed to improve mathematics and science instruction (using technology) in Alabama. The developers of AMSTI posited that the key to improving student test scores in mathematics and science lay in the quality and effectiveness of teachers. The most direct way to increase teacher quality, in the AMSTI view, was to develop an in-depth, comprehensive professional development program reflective of the national standards in mathematics, science, and technology, and to provide teachers with a variety of resources to support what they learned in that program.

Specifically, the study seeks to answer the following questions, key to evaluating AMSTI’s theory of action:

a. On the Impact of AMSTI

1. What is the impact of AMSTI on student achievement during the first and second years of implementation?

2. What is the impact of AMSTI on instructional practices of teachers during the first and second years of implementation?

3. Does two years of AMSTI have a greater impact on student achievement than one year, due to delayed impact?

4. Does AMSTI have a lasting effect on a school’s success in increasing student achievement after the level of support for the intervention has been diminished?

5. How does the impact of AMSTI vary with characteristics of teachers and students and with the school technology environment?

b. On the Implementation of AMSTI

6. Is the delivery of the regional professional training consistent with the stated design of the AMSTI program? Are the technology materials that AMSTI uses made available to classrooms on a timely basis? Are all the follow-up supports in the program’s design delivered as intended?

7. What is the relationship between the training provided by each of the regional sites and classroom instructional practice observed in those regions?

8. How does classroom implementation of AMSTI vary with teacher characteristics, student characteristics, and the technology environment of the school?

The AMSTI program began with a small number of school districts and has expanded yearly in order to provide materials and services to a greater proportion of the schools in the state. The state is currently considering offering the program to all schools in the state. In order to commit the necessary resources to this endeavor, the legislature requires solid evidence of program effectiveness. In addition, it requires information on the benefit of AMSTI within specific regions, among specific populations, and data on the specific aspects of the program that are related to improvement in student achievement.

2.2 Utility to Educators, Policymakers and the General Public

This study will also provide needed information for educators, policymakers and the general public who have a stake in improving math and science programs. Within Alabama, stakeholders need to know whether their state program is effective, and if so, under what conditions it is effective. For stakeholders considering applying to become an AMSTI school, this information is critical to that decision. Stakeholders who are already part of the AMSTI program require this information to determine whether to continue the program and how to implement the program. Parties outside of Alabama also have an interest in learning about the effectiveness of AMSTI, either to decide whether to adopt AMSTI in their own states or to gain information that will allow them to compare other programs to the AMSTI program, in order to estimate effectiveness and to inform implementation.

We did not use a probability-based sample design when selecting schools for the study. Instead, using best practice in education research, we used a convenience sample consisting of schools that applied to AMSTI within three geographic regions of the state. Within the pool of applicants from those regions we selected the schools such that the mean of the sample closely matched their regions in terms of demographics and test scores.

In this case we were constrained from using a probability-based sample design given the design of the intervention (80% of the teachers in any school had to commit to using the AMSTI program prior to randomization). The education research community recognizes that probability-based sample designs provide the best evidence for generalizable results. However, the results of this study will be informative to schools and districts within Alabama that have similar demographics, and they may use the results to inform their decisions regarding participating in the AMSTI program. These practices are broadly consistent with practices that scientists have used to generalize across many areas over many decades (Shadish, Cook and Campbell, 2002, pg.349). We will work with OMB to ensure that we use appropriate language to caution that the results are not broadly generalizable.

2.3 Utility for Researchers

There is a paucity of scientifically based evidence regarding the effectiveness of math and science programs. This study is intended to help fill this gap. What is learned will add to the scientific literature for reference in future studies. In addition, the data will be kept in the data warehouse of Empirical Education (with all identifying information removed) for use in future studies.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.

Web-based surveys will be administered to teachers and principals. This use of technology reduces the burden to the schools in comparison to the added time and effort needed for paper surveys (e.g., paper surveys entail distribution to the correct individuals and follow-up to ascertain receipt by individuals, making a paper copy after the completion of a survey and then mailing it back to the researchers, responding to queries from researchers when paper copies are delayed or lost in the mail and when researchers require clarification due to illegible responses, etc.). The data from web-based surveys are immediately uploaded into the researchers’ database and immediately connected to the respondent’s identifying information, allowing researchers to quickly seek clarification if questions arise. However, all study participants will be informed that paper copies of the surveys are also available.

Participants will receive an e-mailed invitation containing a link to the survey on the web. The necessary e-mail addresses will be obtained from the participants when they sign consent forms to participate in the study. Researchers will then test the addresses to ensure that each participant receives messages. In order to further verify the correct contact for the correct teacher participants, researchers will follow-up with an e-mail request for teacher information on the grade and class they teach. These processes will help to ensure that the actual surveys will reach the appropriate participants in a timely manner.

Investigators request class rosters from school systems and demographic and assessment data from the state. Researchers e-mail a query to each school system which school system staff enter into their student data systems and immediately export the rosters. The school systems then only need to return these results back to the researchers by one of two methods at the convenience of the school system: either US Mail or secure ftp. Once researchers have cleaned the roster data, we send a spreadsheet by mail with the students’ state identifiers to the state. The state then runs a query to retrieve the needed data for each student and returns the data to the researchers by mail or secure ftp.

4. Describe efforts to identify duplication. Show specifically why any similar information available cannot be used or modified for use for the purposes described in item 2 above.

This study is not duplicating other work and is indeed filling a gap. The study is necessary because there is a lack of scientific evidence concerning the effectiveness of educational programs in math and science. ALSDE has, however, commissioned evaluations of AMSTI in the past. Three external evaluations of AMSTI found that the students in the earliest AMSTI schools consistently outperformed their counterparts in non-AMSTI schools (Institute for Communication Research [ICR], 2004, 2005, 2006). Reported findings for middle school students in AMSTI schools were particularly dramatic, showing that their Stanford 10 math scores were up to 8 percentile points higher than those of non-AMSTI students, and their Stanford 10 science scores were up to 5 percentile points higher. The evaluators also found that AMSTI had “spillover” effects with respect to reading outcomes (ICR, 2006).

With regard to the past research, however, it is important to note that these evaluations were quasi-experimental in nature, relying upon a comparison of the AMSTI schools against demographically matched schools. As with so many quasi-experimental studies, however, results may have been subject to substantial selection bias. AMSTI schools had been interested enough early in the program’s history to know what it was and to volunteer to have it implemented in their schools. Since 80% of the teachers in each school were required to participate in the program, it is likely that teachers in the volunteering schools possessed a more positive disposition towards technology use in the classroom as well as a greater willingness to invest substantial time in their own professional development as compared to non-AMSTI schools. Because the quasi-experimental criterion for comparability was only that the two schools were in the same region, it may not have been sufficient to protect them from selection bias. (ICR, University of Alabama, 2006).

Thus, while the AMSTI schools may very well have outperformed non-AMSTI schools, it is difficult to determine whether this was due to other factors or to the AMSTI intervention absent a more rigorous experimental research design. A group-randomized control trial (RCT) of AMSTI, by removing all potential bases of selection bias, represents an advance over all previous research work on the intervention. This study will provide ALSDE with critical information on program effectiveness that is not currently available from other research.

5. If the collection of information impacts small businesses or other small entities (item 5 of OMB Form 83-1), describe any methods used to minimize burden.

Researchers will collect the absolute minimum amount of data that is needed in order to meet the study objectives. We will collect demographic and assessment data centrally. The only information collected at the classroom level will be observation/ interview data and short teacher surveys.

6. Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

6.1 Entire Study

The No Child Left Behind Act insists that education providers base their decisions on scientifically based research. However, there is very little research that carefully measures the effectiveness of different pedagogical strategies. Furthermore, there is a gap between the theories on which the published materials are based and the realities in the classroom. Even with the most rigorously designed academic research, a persistent gap remains between what educational scientists know about what works in general and what works in their particular school district. In the case of AMSTI, the state of Alabama is seeking scientific evidence that is not otherwise available in order to make critical policy decisions regarding this program by measuring directly the impact of instructional practices in its classrooms on student achievement.

6.2 Class Rosters, Student Demographics and Math, Reading and Science Achievement Test Scores

Data are collected from school systems and existing state records. For each replication, these data requests are conducted once for the baseline information and then annually (three times in total) in order to capture any changes in demographics and to gather each new year’s assessment scores. If we did not collect assessment data annually, we would not be able to provide ALSDE with scientific analyses on whether AMSTI is effective at improving achievement. If we did not collect demographic data annually, we would not be able to determine whether results differed among subgroups.

6.3 Teacher Web-based Surveys

The survey burden for teachers (both treatment and control) will consist of up to 80 minutes annually, as 20 minute surveys will be administered four times during the first year and four times during the second year of the original study and of the replication study. The surveys collect regular quantitative data (e.g., the number of minutes teaching math and science) over the course of several months so that researchers can run statistical analyses of program impact on instruction across all study regions as well as by groups of teachers (e.g., by level of teacher experience). Without quantitative teacher data, it would be difficult to make meaningful connections between program implementation and achievement. Because instruction varies daily, researchers attempt to estimate averages of instructional activities by sampling teachers’ responses monthly. Only four monthly collections will be possible based on the timeline for OMB approval. Each survey asks the teachers to recall their instruction over the course of the prior two weeks. If data were not collected each of the four months, researchers would not be able to track teacher implementation over a sufficiently long period of time to estimate average implementation. Copies of the four surveys are provided in Appendices A through D.

6.4 AMSTI Study Teacher Classroom Observations and Teacher Interviews

A subset of the teachers (42) will be selected for classroom observation and interview annually for each replication. Teachers (both treatment and control) will be chosen at random from a stratified sample (seven strata: grades 4, 5, 6, 7, and 8 math, grades 5 and 7 science) so that data correspond to student achievement, trainer log and training participant data. These teachers will have an added burden of about 60 minutes. The observation will in no way interfere with the class time. Observations allow an external view of implementation in order to triangulate with other implementation data. Teacher interviews give context to the observation data.

A different subset of 42 teachers (both treatment and control) will be interviewed only. Teachers who are interviewed will have an additional burden of 15-20 minutes at one time only. Interviews, in general, allow for much richer information than can be gathered by surveys, which, again better informs the analysis of program implementation. The stand-alone interviews allow for a second data point at each school (at two schools there will be two observations and two interviews in order to provide three observations and three interviews at each grade level in each region) for classroom level qualitative data. Without observations followed by interviews the analysis on the impact of AMSTI on classroom instruction would be informed only by teacher self-reports. If stand-alone interviews were eliminated, the amount of qualitative data collected through site visits would be very small. Observation and Interview Protocols are provided in Appendices E through G.

6.5 Principal Web-based Surveys

Principal surveys are only collected once during the first year and once during the second year of the study. The burden is about 30 minutes annually. The surveys are collected in the beginning of the year to gain baseline information on school climate, technology, curriculum, professional development, and instruction in the schools during the previous school year. Principal level data are critical to providing information from the perspective of the school rather than only at the classroom level. Without principal information, the study would rely purely on classroom level data regarding implementation. In addition, we would not be able to learn about integral aspects of the program, such as community involvement. The Principal Web-based Survey is provided in Appendix H.

6.6 Principal Interviews

Principal interviews are only conducted once during the first year and once during the second year of the original and of the replication studies for both treatment and control principals. The burden is only 15-20 minutes each year. Interviews happen in the spring in order to learn about the implementation from a school level (as opposed to classroom level) perspective after the school has had the majority of the year to incorporate the program into their classrooms. Interviews allow for much richer information than is provided by the survey.. Without the interviews, the only information on program implementation from the school level would be supplied by the annual principal survey. The Principal Interview Protocols are provided in Appendices I and J.

6.7 Professional Development Trainer Logs

Trainers of grade 5 and grade 7 teachers complete logs during the Summer Institutes the first and second years of the original and of the replication studies. Trainers will have a burden of up to 105 minutes over the course of the training (10 minutes after each session to complete a brief log plus less than five minutes to complete a background form only once). Without the logs we would not have detailed information on a daily basis regarding the extent to which trainers cover all of the training materials or which materials are covered. The Professional Development Trainer Logs are provided in Appendices K through N.

6.8. Professional Development Participant Surveys

Professional Development Participant surveys are completed only by grade 5 and grade 7 math and science teachers participating in the corresponding training sessions. Trainees will have a burden of up to 20 minutes only once for the completing of questionnaires. Without the participant surveys, information on training would rely solely on self-reporting by the trainers. The Professional Development Participant surveys are provided in Appendices O through R.

7. Explain any special circumstances that would cause an information collection to be conducted in a manner: *requiring respondents to report information to the agency more often that quarterly; *requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it; *requiring respondents to submit more than an original and two copies of any document; *requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records, for more than three years; *in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study; *requiring the use of a statistical data classification that has not been reviewed and approved by OMB; *that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; *or requiring respondent to submit proprietary trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.

The survey burden for teachers will consist of up to 80 minutes annually, as 20 minute web-based surveys will be administered 4 times over the course of the first and second years of the study. The surveys collect regular quantitative data (e.g., the number of minutes teaching math and science) over the course of several months so that researchers can run statistical analyses of program impact on instruction across all study regions as well as by groups of teachers (e.g., by level of teacher experience). Without quantitative teacher data, it would be difficult to make meaningful connections between program implementation and achievement. Because instruction varies daily, researchers attempt to estimate averages of instructional activities by sampling teachers’ responses monthly. Each web-based survey asks the teachers to recall their instruction over the course of the prior 2 weeks. If data were not collected each of the four months, researchers would not be able to track teacher implementation over a sufficiently long period of time to estimate average implementation.

8. If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency’s notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken buy the agency in response to these comments. Specifically address comments received on cost and hour burden. Consultation with representatives of those from who compile records should occur at least once every 3 years – even if the collection of information activity is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.

Below is wording to be used for documentation of the Federal Register notice:

In accordance with the Paperwork Reduction Act of 1995, the public was given 60 days to review and comment on the Federal Register Notice (Month Day, 2007, Vol. xx, No. xxx, Pgs. xxxxx - xxxxx).

A copy of the Federal Register Notice is attached in Appendix S.

9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

Teachers in the Original Study are offered a $100 stipend as appreciation for their time and effort to complete the four web-based surveys. At the request of the OMB we have reduced the stipend to $50 for the Replication Study. The stipend is considered appropriate given that the surveys require 20 minutes of the teacher’s time for four consecutive months. Teachers’ time is very limited – they often receive 45-minutes or fewer per day for lesson planning and all other tasks outside of classroom instruction. We believe it is important to demonstrate to the teachers that we appreciate this sacrifice on their part. We attribute our excellent response rates – over 90% - in part to the fact that we do honor teachers in this way. The validity of the results is greatly dependent on the level of teacher response.

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

We provide respondents the following assurance of confidentiality:

The collection of information in this study is authorized by Public Law 107-279 Education Sciences Reform Act of 2002, Title I, Part C, Sec. 151(b) and Sec. 153(a). Participation is voluntary. You may skip questions you do not wish to answer; however, we hope that you will answer as many questions as you can. Your responses are protected from disclosure by federal statute (PL 107-279 Title I, Part C, Sec. 183). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law. Data will be combined to produce statistical reports. No individual data that links your name, school name, address, telephone number, or identification number with your responses will be included in the statistical reports.

Teachers, trainers and administrators within treatment and control schools are asked to sign consent forms as follows:

• AMSTI Teacher Consent Form

• AMSTI Principal Consent Form

• Professional Development Participant Consent Form

• AMSTI Principal Interview Consent Form

• Control Principal Interview Consent Form

• Stand-Alone Teacher Interview Consent Form

• Teacher Observation Interview Consent Form

• Professional Development Trainer Log Consent Form

The research and the consent forms were approved by the University of North Carolina at Greensboro Institutional Review Board, which insures that research involving people follows federal regulations. Consent forms assure participants of the following:

By signing this consent form, you agree that you understand the procedures and any risks and benefits involved in this research, and that you agree to participate in the project during the 2006-2007 school year. You are free to refuse to participate or to withdraw your consent to participate in this research at any time without penalty or prejudice; your participation is entirely voluntary. Your confidentiality will be protected because you and your school will not be identified by name as a participant in this project.

Documentation of UNCG IRB approval of the AMSTI Study (#067033; 8/25/06) is provided as Appendix T. Consent forms are provided in Appendices U through AA.

In the management and storage of data, we will strictly adhere to all NCES Statistical Standards (see http://nces.ed.gov/statprog/2002/std4_2.asp) to ensure that strict confidentiality is maintained under all circumstances.

All collected data will be kept in a secure location where only the principal investigators and researchers with certificates of confidentiality will have access. Hardcopies of all data will be destroyed at the conclusion of the study; electronic data will be kept as long as needed, but the data records will be fully anonymized, i.e., all personally identifying information will be removed. Empirical Education will retain the data in a secure location without personally identifying information for use in re-analysis and follow-on research. All publications and other publicly available documents will not contain information that will make it possible to individually identify a student, teacher, trainer, principal, or school.

11. Provide additional justification for any questions of a sensitive nature such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

No questions of a sensitive nature will be asked.

12. Provide estimates of the hour burden of the collection of information. The statement should: *Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desirable. If the hour burden on respondents is expected to vary widely because of differences in activity, size or complexity, show the range of estimated hour burden, and explain the reasons for the variance. Generally, estimates should not include burden hours for customary and usual business practices. *If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-1. *Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriated wage rate categories. The cost of contracting out or paying outside parties for information collection activities should not be included here. Instead this cost should be included in Item 13.

Introduction

The AMSTI study consists of an original study in three regions of Alabama during the 2006-2007 and the 2007-2008 school years, and a replication study in two new regions of Alabama with new participants during the 2007-2008 and the 2008-2009 school years. Both studies will use identical data collection instruments to collect survey information, observe classroom instruction, and interview teachers and principals. The hour burden and cost burden estimates for each instrument are the same for each year and are the same for the original and the replication studies. The numbers of participants should also be similar for the original and the replication studies, with the following exception: in the summer of 2006 researchers conducted the Principal Web-based Surveys, the Professional Development Participant Surveys, and the Professional Development Trainer Logs under separate funding, so those items are not part of this request nor are they included in the burden table for 2006-2007.

Table 1

Instruments by Type of Respondent and Year of Data Collection

Type of Respondent	Instruments	2006-2007	2007-2008	2008-2009
A. Principals	Principal Surveys		Y	Y
	Principal Interview Protocols	Y1	Y	Y
B. Teachers	Web-Based Teacher Survey #1	Y2	Y	Y
	Web-Based Teacher Survey #2	Y3	Y	Y
	Web-Based Teacher Survey #3	Y4	Y	Y
	Web-Based Teacher Survey #4	Y5	Y	Y
	AMSTI Study Teacher Classroom Observation Protocols	Y6	Y	Y
	Teacher Interview Protocol	Y7	Y	Y
	Professional Development Participant Survey		Y	Y
C. Trainers	Professional Development Trainer Background Sheet		Y	Y
	Professional Development Trainer Logs (completed after each of the 10 training sessions)		Y	Y
D. School Systems	Query to Collect Class Rosters from STI Data System	Y	Y	Y
E. State	Request to Collect Demographic and Assessment Data	Y	Y	Y

A12.1: Number of Respondents and Frequency of Response

Following are four tables for the four years of data collection plus one table that averages the annual burden for the four years. The calculations in these tables are responding to all the questions asked by OMB. The tables contain four years of data collection including both the original and the replication studies. The final number of annual respondents and annual responses on the 83-I are calculated by averaging the numbers for the four years.

Table 2 contains the burden estimates for the 2006-2007 school year. Only the original study occurs during this year.

Table 2

2006-2007 Original Study Only

Type of Respondent	Number of Respondents	Data Collection Instrument	Number of Responses	Hours per Respondent	Total Time Burden Hours	Estimated Hourly Cost to Each Respondent	Estimated Total Cost
A. Principals	408	Principal Interview Protocols	40	0.33	13.33	$35.16	$468.80
Principal Total	40		40		13.33		$468.80
B. Teachers	3249	Web-Based Teacher Survey #1	324	0.33	108.00	$27.30	$2,948.40
	32410	Web-Based Teacher Survey #2	324	0.33	108.00	$27.30	$2,948.40
	32411	Web-Based Teacher Survey #3	324	0.33	108.00	$27.30	$2,948.40
	32412	Web-Based Teacher Survey #4	324	0.33	108.00	$27.30	$2,948.40
	4213	AMSTI Study Teacher Classroom Observation Protocols	42	1	42.00	$27.30	$1,146.60
	8414	Teacher Interview Protocol	84	0.33	28.00	$27.30	$764.40
Teacher Total	324		1422		502.00		$13,704.60
C. School System	40	Query to Collect Class Rosters from STI Data System	40	0.25	10.00	10.05	$100.50
School System Total	40		40		10.00		$100.50
D. State	1	Request to Collect Demographic and Assessment Data	1	3	3.00	19.1	$57.30
State Total	1		1		1.00		$57.30
All Participants	405		1503		528.33		$14,331.20

Table 3 contains the burden estimates for the 2007-2008 school year. Both the original and the replication studies occur during this year.

Table 3

2007-2008 Original and Replication Studies

Type of Respondent	Number of Respondents	Data Collection Instrument	Number of Responses	Hours per Respondent	Total Time Burden Hours	Estimated Hourly Cost to Each Respondent	Estimated Total Cost
A. Principals	80	AMSTI Principal Web-Based Survey	80	0.5	40	$35.16	$1,406.40
	80	Principal Interview Protocols	80	0.33	26.67	$35.16	$937.60
Principal Total	80		160		66.7		$2,344.00
B. Teachers	648	Web-Based Teacher Survey #1	648	0.33	216	$27.30	$5,896.80
	648	Web-Based Teacher Survey #2	648	0.33	216	$27.30	$5,896.80
	648	Web-Based Teacher Survey #3	648	0.33	216	$27.30	$5,896.80
	648	Web-Based Teacher Survey #4	648	0.33	216	$27.30	$5,896.80
	84	AMSTI Study Teacher Classroom Observation Protocols	84	1	84	$27.30	$2,293.20
	168	Teacher Interview Protocol	168	0.33	56	$27.30	$1,528.80
	140	Professional Development Participant Survey Grade 5 Math	140	0.17	23.33	$27.30	$637.00
	120	Professional Development Participant Survey Grade 5 Science	120	0.17	20.00	$27.30	$546.00
	80	Professional Development Participant Survey Grade 7 Math	80	0.17	13.33	$27.30	$364.00
	80	Professional Development Participant Survey Grade 7 Science	80	0.17	13.33	$27.30	$364.00
Teacher Total	648		3264		1074.0		$29,320.20
C. Trainers	8	Professional Development Trainer Logs Grade 5 Math	8	0.08	0.67	31.25	$20.83
	8	Professional Development Trainer Logs Grade 5 Science	8	0.08	0.67	31.25	$20.83
	6	Professional Development Trainer Logs Grade 7 Math	6	0.08	0.50	31.25	$15.63
	12	Professional Development Trainer Logs Grade 7 Science	12	0.08	1.00	31.25	$31.25
	34	Professional Development Trainer Background Sheet	34	0.17	5.67	31.25	$177.08
Trainer Total	34		68		8.50		$265.63
D. School Systems	80	Query to Collect Class Rosters from STI Data System	80	0.25	20.00	10.05	$201.00
School System Total	80		80		20.00		$201.00
E. State	1	Request to Collect Demographic and Assessment Data	1	3	3.00	19.1	$57.30
State Total	1		1		3.00		$57.30
All Participants	843		3573		1172.17		$32,188.13

Table 4 contains the burden estimates for the 2008-2009 school year. Researchers collect data from school systems and the state for both original and replication studies, but only collect school site data from replication schools.

Table 4

2008-2009 Original and Replication Studies

Type of Respondent	Number of Respondents	Data Collection Instrument	Number of Responses	Hours per Respondent	Total Time Burden Hours	Estimated Hourly Cost to Each Respondent	Estimated Total Cost
A. Principals	40	AMSTI Principal Web-Based Survey	40	0.5	20	$35.16	$703.20
	40	Principal Interview Protocols	40	0.33	13.33	$35.16	$468.80
Principal Total	40		80		33.33		$1,172.00
B. Teachers	324	Web-Based Teacher Survey #1	324	0.33	108	$27.30	$2,948.40
	324	Web-Based Teacher Survey #2	324	0.33	108	$27.30	$2,948.40
	324	Web-Based Teacher Survey #3	324	0.33	108	$27.30	$2,948.40
	324	Web-Based Teacher Survey #4	324	0.33	108	$27.30	$2,948.40
	42	AMSTI Study Teacher Classroom Observation Protocols	42	1	42	$27.30	$1,146.60
	84	Teacher Interview Protocol	84	0.33	28	$27.30	$764.40
	70	Professional Development Participant Survey Grade 5 Math	70	0.17	11.67	$27.30	$318.50
	60	Professional Development Participant Survey Grade 5 Science	60	0.17	10.00	$27.30	$273.00
	40	Professional Development Participant Survey Grade 7 Math	40	0.17	6.67	$27.30	$182.00
	40	Professional Development Participant Survey Grade 7 Science	40	0.17	6.67	$27.30	$182.00
Teacher Total	324		1632		537.00		$14,660.10
C. Trainers	4	Professional Development Trainer Logs Grade 5 Math	4	0.08	0.33	$31.25	$10.42
	4	Professional Development Trainer Logs Grade 5 Science	4	0.08	0.33	$31.25	$10.42
	3	Professional Development Trainer Logs Grade 7 Math	3	0.08	0.25	$31.25	$7.81
	6	Professional Development Trainer Logs Grade 7 Science	6	0.08	0.50	$31.25	$15.63
	17	Professional Development Trainer Background Sheet	17	0.17	2.83	$31.25	$88.54
Trainer Total	17		34		4.25		$132.81
D. School Systems	80	Query to Collect Class Rosters from STI Data System	80	0.25	20.00	10.05	$201.00
School System Total	80		80		20.00		$201.00
E. State	1	Request to Collect Demographic and Assessment Data	1	3	3.00	19.1	$57.30
State Total	1		1		3.00		$57.30
All Participants	462		1827		597.58		$16,223.21

Table 5 contains the burden estimates for the 2009-2010 school year. Only the replication study occurs during this year.

Table 5

2009-2010 Replication Study Only

Type of Respondent	Number of Respondents	Data Collection Instrument	Number of Responses	Hours per Respondent	Total Time Burden Hours	Estimated Hourly Cost to Each Respondent	Estimated Total Cost
A. School Systems	40	Query to Collect Class Rosters from STI Data System	40	0.25	10.00	10.05	$100.50
School System Total	40		40		10.00		$100.50
B. State	1	Request to Collect Demographic and Assessment Data	1	3	3.00	19.1	$57.30
State Total	1		1		3.00		$57.30
All Participants	41		41		13.00		$157.80

Table 6

Average Burden

School Years	Total Respondents	Total Responses	Total Time Burden Hours	Estimated Total Cost
2006-2007	405	1503	528.33	$14,331.20
2007-2008	843	3573	1172.17	$32,188.13
2008-2009	462	1827	597.58	$16,223.21
2009-2010	41	41	13.00	$157.80
Average	438	1736	578	$15,725.08

A12.2: Hour Burden by Each Form

The AMSTI Principal Web-Based Survey (control and treatment) should take approximately 30 minutes to complete. This estimate is based on the time required by principals to complete the survey when the survey was piloted during August of 2006.

The Principal Interview Protocols (control and treatment) should take approximately 20 minutes to complete. Burden was determined based on the study team’s experience in conducting similar data collections.

The four AMSTI Teacher Web-Based surveys (control and treatment) should each take approximately 20 minutes to complete. This estimate is based on experience from similar web-based teacher surveys previously conducted by Empirical Education Inc.

The AMSTI Study Teacher Classroom Observation Protocols (control and treatment) should take approximately 1 hour to complete. Burden was determined based on the study team’s experience in conducting similar data collections.

The Teacher Interview Protocols (control and treatment) should take approximately 20 minutes to complete. Burden was determined based on the study team’s experience in conducting similar data collections.

The Professional Development Participant Surveys should take approximately 10 minutes to complete. The participant surveys were piloted with all grade 5 and grade 7 math and science teacher training participants in the 2006 training Institutes.

The Professional Development Trainer Logs should take about 5 minutes and the Background Sheets should take about 10 minutes. The trainer checklist and background sheets were piloted with all grade 5 and grade 7 math and science trainers in the 2006 training institutes.

The query to collect class rosters from the STI data system should take about 15 minutes in each school system. The collection of demographic and assessment data from the state should take about three hours. These estimates are based on telephone conversations with school system staff and with state level staff.

A12.3: Annualized Cost for Each Instrument

The cost for each instrument was estimated by multiplying the hour burden by the number of participants by the estimated hourly wage for that participant type. Then the total costs for each year were averaged over the four years.

Principals’ hourly wages were estimated using figures from two sources. The following list of median principal salaries from around the state was taken from Salaries.Com at http://swz.salary.com/salarywizard/layouthtmls/AL/swzl_compresult_state_AL_ED03000012.html on October 20, 2006. This table indicates that there is very little regional variation in principal salaries. The median from this table, $67,514, was used to estimate principal salary. Then the salary was divided by the number of hours worked annually or 8 hours per day on 240 days per year. This number was provided in a telephone conversation by the Alabama Education Association on October 13, 2006.

Table 7

Median Principal Salary by City in Alabama

City	Median Salary
Dothan	$66,031
Florence	$66,623
Top of Form Montgomery	$67,106
Top of Form Mobile	$67,514
Tuscaloosa	$67,840
Top of Form Huntsville	$69,411
Top of Form Birmingham	$69,506

Teachers’ hourly wages were estimated using figures from two sources. According to a press release from the American Federation of Teachers titled, Alabama Ranks 43^rd in the Nation for Teacher Pay, dated October 5, 2005, Alabama teachers’ average salary was $38,282 during the 2003-2004 school year. This annual salary was then divided by the number of hours worked annually or 7.5 hours per day on 187 days per year. This number was provided in a telephone conversation by the Alabama Education Association on October 13, 2006.

Trainer salary information was provided by the Alabama State Department of Education by telephone on October 12, 2006. Salaries for school system Clerical – Administrative Support staff and for state level Data Processing IT Operations Specialists were provided by the state by e-mail on April 18, 2007.

13. Provide an estimate for the total annual cost burden to respondents or record keepers resulting from the collection of information. (Do not include the cost of any hour burden shown in Items 12 and 14.) The cost estimate should be split into two components: (a) a total capital and start-up cost component (annualized over its expected useful life) and (b) a total operation and maintenance and purchase of services components. The estimates should take into account costs associated with generating, maintaining, and disclosing or providing the information. Include descriptions of methods used to estimate major cost factors including system and technology acquisition, expected useful life or capital equipment, the discount rate(s), and the time period over which costs will be incurred. Capital and start-up costs include, among other items, preparations for collecting information such as purchasing computers and software; monitoring, sampling, drilling, and testing equipment; and record storage facilities. *If cost estimates are expected to vary widely, agencies should present ranges of cost burdens and explain the reasons for the variance. The cost of purchasing or contracting out information collections services should be a part of this cost burden estimate. In developing cost burden estimates, agencies may consult with a sample of respondents (fewer than 10). utilize the 60-day pre-OMB submission public comment process and use existing economic and regulatory impact analysis associated with the rulemaking containing the information collection, as appropriate. Generally, estimates should not include purchases of equipment or services, or portions thereof, made: (1) prior to October 1, 1995, (2) to achieve regulatory compliance with requirements not associated with the information collection, (3) for reasons other than to provide information or keep records for the government, or (4) as part of customary and usual business or private practices.

There is no estimated respondent cost burden for this project other than the time spent on responding to the surveys/interviews, participating in classroom observations, or e-mailing data.

14. Provide estimates of annualized costs to the Federal government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operation expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information. Agencies may also aggregate cost estimates from Items 12, 13, and 14 in a single table.

The data collection will be conducted by two subcontractors: the Academy for Educational Development (AED) and Empirical Education, Inc. (EEI). We estimate that AED’s effort for data collection purposes equals approximately 20% of the budget allocated for AMSTI; EEI’s effort for data collection purposes equals approximately 35% of the budget allocated. The costs include staff time for developing the data collection instruments (includes salary and fringe); communicating with school, district and state staff; conducting observations and interviews; travel costs; other direct costs (postage, equipment, duplication); and the prorated amount of the approved indirect cost rate for each organization. Total cost: $722,323 over the course of the study.

15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-1.

Not applicable.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

Given the size and scope of the AMSTI study, it will be important to disseminate study findings through different methods and with products suited to a variety of interested audiences. Methods and products will include articles in peer-reviewed, research- and practitioner-oriented, journals, brief fact sheets appropriate for researchers, practitioners, and parents, PowerPoint conference presentations, and additions to the AMSTI website established by ALSDE. If time and resources permit, the development of a book that highlights AMSTI and other IES-funded Task 2 studies at the REL-SE could be considered. Audiences to target will include: (a) ALSDE staff, (b) the AL university training programs providing the PD events and preparing AL teachers in math and science education, (c) school principals and teachers in AL, and (d) nationwide state boards of education and other policymakers, including those at the federal level within IES and elsewhere.

Journal articles can be developed within three categories: (a) those that focus on student academic achievement outcomes, including variation of impacts among different types of students and based on variations in exposure to different levels of AMSTI implementation practices; (b) those that describe in detail the classroom implementation practices of teachers, including comparison between experimental and control teachers, and factors such as teacher experience levels or availability of technology that might have mediated implementation practices; and (c) those that describe the process of establishing and sustaining the state-level infrastructure for AMSTI, including the in-depth, comprehensive professional development program.

Research articles concerning student achievement outcomes and classroom implementation practices can be developed primarily targeting the research community and other interested groups and individuals (e.g., policymakers; university training program staff). Practitioner-oriented articles on the same topics can target school principals, teachers, school board members, policymakers, and training staff.

Brief, colorful and engaging fact sheets that describe the study, methodology, key findings and implications for practice will be developed for broad dissemination within Alabama, among other Labs, and at appropriate conferences. Different fact sheets can be developed to meet the information needs of different audiences.

A set of conference presentations that describe AMSTI, the study design and methods, and outcomes can be made at the American Educational Research Association, the American Evaluation Association, and other similar research-oriented conference events. Conference presentations that are more focused on describing the AMSTI intervention and how it was implemented in school settings will be conducted at conferences of school-based professionals (e.g., state-level program and curriculum specialists and managers, principals, and classroom teachers). In addition, the research team will work closely with ALSDE staff to identify conferences, meetings and other venues within Alabama at which presentations, tailored to the relevant audience, can be made. Web casts can be considered as an efficient, lower cost method to provide study information to large audiences.

The research team will also work with ALSDE to develop and post information about the study and its findings on the state’s AMSTI website, www.AMSTI.org., as it will in parallel with the IES website for the Regional Lab System and its implementation contractor.

Table 8

Project Timeline

Month/Year	Activity
February 2006	Randomization of AMSTI Study Schools
June-July 2006	Summer Institutes to Train New AMSTI Teachers and Principals
July 2006	Web-Based Principal Surveys
August 2006	AMSTI Implemented in AMSTI First Classrooms
January-April 2007	Web-Based Teacher Surveys
March-April 2007	Classroom Observations and Teacher Interviews, Year One
July 2007	Submit Interim Report, Year One
June-July 2007	Year Two Summer Institutes for AMSTI First Group and Year One Summer Institutes for AMSTI Second Group
July 2007	Web-Based Principal Surveys, Year Two
August 2007	AMSTI Implemented in AMSTI First and AMSTI Second Group Classrooms
September 2007	Submit Final Reports, Year One
January-April 2008	Web-Based Teacher Surveys, Year Two
March - April 2008	Classroom Observations and Teacher Interviews, Year Two
July 2008	Submit Interim Report, Year Two
September 2008	Submit Technical and Non-Technical Final Reports, Year Two
September 2009	Submit Technical and Non-Technical Final Reports, Year Three

Note. The replication study timeline will be identical to that of the original study, except that all replication study activities will be one year later than the activities for the original study.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

The OMB approval expiration date will be given to all survey respondents for all surveys.

18. Explain each exception to the certification statement identified in Item 19, “certification for Paperwork Reduction Act Submissions,” of OMB Form 83-1.

There are no exceptions to the certification statement

1 As of May 1, 2007, we do not yet have OMB clearance, so researchers will not be able to conduct these interviews during the 2006-2007 school year.

2 The teacher web-based surveys were administered under different funding during the 2006-2007 school year, due to the fact that OMB clearance had not yet been received.

3 The teacher web-based surveys were administered under different funding during the 2006-2007 school year, due to the fact that OMB clearance had not yet been received.

4 The teacher web-based surveys were administered under different funding during the 2006-2007 school year, due to the fact that OMB clearance had not yet been received.

5 The teacher web-based surveys were administered under different funding during the 2006-2007 school year, due to the fact that OMB clearance had not yet been received.

6 As of May 1, 2007, we do not yet have OMB clearance, so researchers will not be able to conduct these observations during the 2006-2007 school year.

7 As of May 1, 2007, we do not yet have OMB clearance, so researchers will not be able to conduct these interviews during the 2006-2007 school year.

8 As of May 1, 2007, we do not yet have OMB clearance, so researchers will not be able to conduct these interviews during the 2006-2007 school year.

9 The teacher web-based surveys were administered under different funding during the 2006-2007 school year, due to the fact that OMB clearance had not yet been received.

10 The teacher web-based surveys were administered under different funding during the 2006-2007 school year, due to the fact that OMB clearance had not yet been received.

11 The teacher web-based surveys were administered under different funding during the 2006-2007 school year, due to the fact that OMB clearance had not yet been received.

12 The teacher web-based surveys were administered under different funding during the 2006-2007 school year, due to the fact that OMB clearance had not yet been received.

13 As of May 1, 2007, we do not yet have OMB clearance, so researchers will not be able to conduct these observations during the 2006-2007 school year.

14 As of May 1, 2007, we do not yet have OMB clearance, so researchers will not be able to conduct these interviews during the 2006-2007 school year.