CLSD_OMB Part A_FINAL (002)

CLSD_OMB Part A_FINAL (002).docx

Comprehensive Literacy Program Evaluation: Comprehensive Literacy State Development (CLSD) Program Evaluation

OMB: 1850-0945

Document [docx]
Download: docx | pdf



Contract Number 91990018C0020

Comprehensive Literacy Program Evaluation: Comprehensive Literacy State Development Grant Program

Part a: supporting statement for paperwork reduction act submission

December 11, 2020

Submitted to:

U.S. Department of Education

Institute of Education Sciences

550 12th Street, SW

Washington, DC 20202

Project Officer: Tracy Rimdzius

Contract Number: 91990018C0020

Submitted by:

American Institutes for Research

1000 Thomas Jefferson Street, NW

Washington, DC 20007

Phone: (202) 403-5000

Facsimile: (202) 403-5001

Project Director: Jessica Heppen





Notice of Trademark: “American Institutes for Research” and “AIR” are registered trademarks. All other brand, product, or company names are trademarks or registered trademarks of their respective owners.

Copyright © 2020 American Institutes for Research®. All rights reserved. No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, website display, or other electronic or mechanical methods, without the prior written permission of the American Institutes for Research. For permission requests, please use the Contact Us form on www.air.org.


Contents



Part A. Supporting Statement for Paperwork Reduction Act Submission

This package requests clearance from the Office of Management and Budget (OMB) to conduct data collection activities for the legislatively mandated evaluation of the Comprehensive Literacy State Development (CLSD) Program. The evaluation will provide information on the implementation of the program and its impact on student reading achievement. The Institute of Education Sciences (IES), within the U.S. Department of Education, contracted with American Institutes for Research (AIR) and its partners—Abt Associates, National Opinion Research Center (NORC), and Instructional Research Group (IRG)—to conduct the study.

The CLSD evaluation is the second component of an evaluation of the U.S. Department of Education’s comprehensive literacy programs. An earlier clearance package (OMB control number 1850-0945) covered the first component of the evaluation—an implementation study of the Striving Readers Comprehensive Literacy Program (the precursor program to CLSD).

The CLSD evaluation will examine program implementation, estimate the impact of CLSD funding on student achievement, and compare trends in achievement in CLSD and similar non-CLSD schools. The Department awarded CLSD grants to 13 state education agencies (SEAs) in fiscal year (FY) 2019 and awarded 11 CLSD grants to 10 new SEAs in FY2020, for a total of 24 grants to 23 SEAs.1 We will include all FY2019 and all FY2020 grantees and subgrantees in implementation data collection activities and achievement trend analyses; we will include a subset of FY2019 subgrantees (cohort 1) and a subset of FY2020 subgrantees (cohort 2) in impact data collection activities. This package requests clearance for all of the data collection activities through the 2022-23 school year. A separate package will be submitted at a later date for the last two years of data collection needed for the achievement trend analyses, which will take place following the 2024-25 school year.

Justification

A1. Circumstances Necessitating the Collection of Information

This evaluation of CLSD is necessary because many U.S. students do not acquire even basic literacy skills, despite recent educational policy focused on college and career readiness. Students in rural communities, with disabilities, who are English learners (ELs), or who are in foster care are especially at risk. To narrow the gap in literacy between disadvantaged students and other students, in 2011 the federal government launched the Striving Readers Comprehensive Literacy (SRCL) program; its successor, the CLSD program, followed in 2019.2 Given the scope of the challenge and the federal investment in CLSD, and SRCL before that, it is critically important that policymakers, administrators, and educators have access to information on the implementation and impact of these grants. In recognition of this, Congress mandated that IES conduct an independent evaluation of the SRCL and CLSD programs.3 This evaluation will provide an up-to-date national picture of how CLSD state grantees, districts, and schools are advancing the literacy skills of children from birth through Grade 12, with a special focus on improving outcomes for disadvantaged children.

SEA grantees are to use the CLSD grants to enhance literacy instruction to ultimately improve student reading and writing. Under the CLSD program, SEAs in turn use their funds to support subgrants to local education agencies (LEAs) or nonprofit early learning providers to implement high-quality literacy instruction in schools and early childhood education programs.

States are required to collaborate with early childhood programs, conduct or revise a needs assessment, develop or revise a state comprehensive literacy plan, and develop an implementation plan explaining how the state will carry out state-level activities.

States are asked to do the following activities, targeted to disadvantaged populations.

  • Develop or revise a plan to support implementation of comprehensive literacy instruction;

  • Promote evidence-based activities;

  • Provide technical assistance to LEAs on designing and implementing literacy programs;

  • Coordinate with institutes of higher education to enhance pre-service courses for teacher candidates in explicit, systematic, and intensive instruction in evidence-based literacy methods;

  • Review and update with teachers and institutes of higher education the state licensure or certification standards in literacy instruction;

  • Make information about promising instructional practices to improve literacy achievement available on state websites; and

  • Administer and monitor subgrants to LEAs.

Aligned to SEAs’ activities and the state literacy plan, LEAs use CLSD funds to implement comprehensive literacy instruction for all children from birth through Grade 12, with a focus on serving disadvantaged children. Specifically, LEAs are asked to do the following activities.

  • Provide comprehensive literacy instruction to all children during the school day, augmented by after-school or out-of-school time;

  • Provide intensive, supplemental, accelerated, and explicit intervention and support to children who are below grade level;

  • Train providers and personnel to develop and administer evidence-based early childhood and K–12 family literacy initiatives supported by promising, moderate, or strong evidence;

  • Coordinate involvement of early childhood education staff in literacy development of the children they serve;

  • Develop and engage early childhood education and K–12 education staff in high-quality professional development that is sustained and coherent; and

  • Engage families in evidence-based family literacy strategies.

As a result of the prioritization, support, planning, and implementation of these activities by the SEAs and LEAs participating in the CLSD program, children attending participating schools and early learning programs are expected to have improved reading and literacy. Exhibit A1 depicts the program logic model for CLSD.

Exhibit A1. CLSD Program Logic Model

The CLSD evaluation will address four research questions:

  1. How do subgrantees target CLSD awards to schools and early learning programs, and for what types of literacy interventions and approaches are CLSD funds used?

  2. What is the impact of CLSD funding on student reading/English language arts achievement in Grades 3–5, after 2 years?

  3. What is the impact of CLSD funding on teachers’ literacy instruction?

  4. How do trends in reading and math achievement differ for CLSD-funded schools and similar non-CLSD schools?

The evaluation will rely on interviews of all state grantees and a survey of all district subgrantees to answer research question 1. To address questions 2 and 3, the evaluation will draw on extant data and school-level survey data in a sample of approximately 130 elementary schools that agree to be randomly assigned to an earlier or later CLSD funding group. The evaluation will draw on statewide administrative data to address question 4.

A2. Purpose and Use of Data

We will collect extant data, interview and survey data, and achievement data to address the research questions. In particular, we will collect the following extant data:

  • Policy documents: To help address question 1, we will collect and review grantee applications, SEAs’ requests for proposals, subgrantee applications, and state comprehensive literacy plans.

  • Subgrantee contact information: To obtain contact information for the subgrantee survey administration, we will request complete lists of all awarded subgrantees in each state, including districts and consortia, as well as the schools funded by each subgrantee.

  • Teacher rosters: To determine the sample of teachers to survey, we will request the names and contact information for the teachers of Grades 3 – 5 in the elementary schools participating in the randomized trial, during the winter of the 2021-22 school year (cohort 1) and winter of the 2022-23 school year (cohort 2).

To obtain information on CLSD-supported activities and the implementation of CLSD, we will interview state grantees and administer a subgrantee survey, a school leader survey, and a teacher survey.

  • Grantee interviews: To complement the analysis of documents for research question 1, we will interview SEA representatives from each of the 13 SEAs that received a FY2019 CLSD grant and each of the 11 SEAs that received a FY2020 CLSD grant. The primary purpose of the grantee interviews is to understand the ways SEAs allocate subgrant awards and support subgrantees’ implementation. The grantee interview protocol will enable the study team to verify or clarify elements of the applications, explore the rationale for grantee approaches, and collect additional information about subgrantee award procedures and evidence supporting selected interventions.

  • Subgrantee surveys: We will administer this survey to all subgrantees from both FY2019 and FY2020 grants, both consortia of districts and individual districts, to address research question 1. The survey instrument includes questions about selection of CLSD schools, use of CLSD funds, and district-level CLSD activities and supports.

  • School leader surveys: We will administer the school leader survey to principals or identified literacy leads in the approximately 130 elementary schools participating in the randomized trial. This survey will be used to measure literacy practices and interventions, the school leaders’ own literacy-related professional development activities, the use of data for continuous improvement, and changes associated with the CLSD grant to address research question 3. This survey will be administered through an online platform in spring 2021 and 2022 for schools in cohort 1, and in spring 2022 and 2023 for schools in cohort 2.

  • Teacher surveys: To further address research question 3, we will use the teacher survey to measure literacy instruction and supports for teachers in Grades 3–5 in the approximately 130 elementary schools participating in the randomized trial. The survey will include items about teachers’ instructional practices, use of literacy materials, literacy-related professional development, support from instructional coaches, use of data, and changes in literacy instruction from the prior year. Teacher surveys will be administered online in spring 2022 for schools in cohort 1 and in spring 2023 for schools in cohort 2. For teachers who do not respond to the online survey, we will mail a paper survey to their schools. 

To obtain outcomes data for the randomized trial and the longitudinal trends in outcomes, we will request achievement data from each of the 13 SEAs that received FY2019 CLSD grants and each of the 10 new SEAs that received FY2020 CLSD grants.

  • Request for student achievement data: To address research question 2, about the impact of CLSD on student literacy outcomes, and research question 4, about achievement trends, we will collect student-level data from each SEA. To permit the team to compare outcomes for students in CLSD and non-CLSD schools (question 4), we will request data covering all schools in each state. For all students in Grades 3–8 and one high school grade, we will request deidentified data on student state math and reading/English language arts assessments, along with the name of the school attended and the following student background characteristics and program participation data: gender, ethnicity, race, English learner status, special education status, and eligibility for free or reduced-price lunch. These data will allow us to study student subgroups in addition to all students in a school. We will request data on students with a common identifier across years, starting with the 2016–17 school year (4 years before CLSD funding is received by districts) and ending with the 2021–22 school year for cohort 1 and 2.4,5 To address research question 2 based on elementary schools in the impact study, we will use achievement and demographic data from 2018–19 through 2021–22 (cohort 1) and from 2020–21 through 2022–23 (cohort 2). We will use a longer panel of data (from 2016–17 through 2024–25) to address research question 4.

Exhibit A2 presents the data collection activities, research questions, respondents, modes, and schedule for the extant, survey, and interview data that the team will collect for the evaluation. The schedule for FY2019 grantees and cohort 1 subgrantees is presented first, followed by the schedule for FY2020 grantees and cohort 2 subgrantees.











Exhibit A2. Data Collection

Data Type

Research Questions (RQ)

Respondent

Mode

Schedule

Extant data

Policy documents: Grantee applications, state RFAs, subgrantee applications, and comprehensive literacy plans

RQ 1

NA

Retrieved from SEA websites

Fall 2020;

Fall 2021

Subgrantee contact information: List of all subgrantees and CLSD schools in a given state, including grades included in CLSD

RQ 1

SEAs, only if not available through websites

Retrieved from SEA websites, email request if necessary

Fall 2020;

Fall 2021

Teacher rosters: List of teachers in each elementary school participating in the RCT

RQs 2-3

Principal or school liaison in elementary schools participating in the RCT

Electronic communication

Winter 2022;

Winter 2023

Interviews and surveys

Survey data

Grantee interviews: Information on subgrant award decisions, technical assistance, and support for continuous improvement

RQ 1

SEA CLSD project directors

Telephone interviews using semi-structured protocol

Spring 2021;

Spring 2022

Subgrantee surveys: Information on use of subgrant funds, targeted schools, activities in CLSD schools, continuous improvement

RQ 1

CLSD subgrantee (consortia or district) project directors

Online questionnaire

Spring 2021;

Spring 2022

School leader surveys: Information on activities in elementary schools participating in the RCT, including curricula, coaching, professional development, evidence for reading interventions, vertical alignment, use of data

RQs 2-3

Principals or literacy leads of elementary schools participating in the RCT

Online questionnaire

Spring 2021 &
Spring 2022;

Spring 2022 &

Spring 2023

Teacher surveys: Information on literacy instruction, differentiation, topics covered, use of engaging materials, professional development, coaching, use of data

RQs 2-3

Teachers of grades 3 – 5 in elementary schools participating in the RCT

Online questionnaire

Spring 2022;

Spring 2023

Achievement data

Student achievement data: Reading/English language arts and math standardized test score data and student demographic information

RQs 2 and 4

SEAs or LEAs

Electronic communication request

Fall 2023

A3. Use of Improved Technology to Reduce Burden

The data collection plans reflect sensitivity to efficiency and respondent burden. We will use a variety of information technologies to maximize the efficiency and completeness of the information gathered for this study and to minimize the burden on respondents at the state, district, and school levels:

  • Use of extant data. When possible, data will be collected through the Department and SEA websites and through sources such as EDFacts and other web‑based sources. For example, before undertaking data collection activities that impose any burden on respondents, we will review grantee applications, subgrantee applications, and any additional information available on SEA or LEA websites to avoid asking questions that otherwise could be addressed through extant sources.

  • Online surveys. We will administer the subgrantee, school leader, and teacher surveys through a web‑based platform to facilitate and streamline the response process.

  • Electronic submission of certain data. Grantees will be asked to electronically submit extant data, including student achievement and demographic data.

  • Support for respondents. A toll‑free number and an email address will be available during the data collection process to permit respondents to contact us with questions or requests for assistance. The toll‑free number and email address will be included in all communication with respondents.

A4. Efforts to Avoid Duplication of Effort

Whenever possible, we will use existing data, including EDFacts, CLSD grantee and subgrantee applications, and federal monitoring reports. This will reduce the number of questions asked in the surveys and interviews, thus limiting respondent burden and minimizing duplication of previous data collection efforts and information.

A5. Efforts to Minimize Burden on Small Businesses and Other Small Entities

The primary entities for the evaluation are state, district, and school staff. We will minimize burden for all respondents by requesting only the minimum data required to meet evaluation objectives. Burden on respondents will be further minimized through the careful specification of information needs. We will keep our data collection instruments short and focused on the data of most interest.

A6. Consequences of Not Collecting the Data

The data collection plan described in this submission is necessary for the Department to respond to the legislative mandate to evaluate the CLSD program. The CLSD grant program represents a substantial federal investment, and failure to collect the data proposed through this evaluation would limit the Department’s understanding of how the program is implemented and how it supports literacy needs at the local level through comprehensive reading interventions. Understanding the strategies and approaches that the subgrantees and schools implement and how they use CLSD funds will enable federal policy makers and program managers to monitor the program and provide useful, ongoing guidance to states and districts. Moreover, this evaluation will yield rigorous evidence about the impact of CLSD funding on student achievement after 2 years. Estimates of the impact of CLSD funding on student achievement will inform program improvement, allocations, and priorities going forward. The consequences of not collecting the proposed extant, survey, and interview data for this evaluation would result in a lost opportunity to learn about the implementation and effectiveness of the CLSD program to inform continuous program improvement.

A7. Special Circumstances Justifying Inconsistencies With Guidelines in 5 CFR 1320.6

There are no special circumstances associated with this data collection.

A8. Federal Register Announcement and Consultation

  1. Federal Register Announcement

The 60-day Federal Register Notice to solicit public comments was published by the Department on June 10, 2020, Vol. 85, page 35419. No substantive public comments have been received to date. The 30-day Federal Register Notice will be published to solicit additional public comments.

  1. Consultations Outside the Agency

The experts who formulated the study design and contributed to the content of the instruments include Drs. Jessica Heppen, Michael Garet, Eleanor Fulbeck, and Kerstin LeFloch from AIR. In addition, our study team is relying on a technical working group (TWG) of researchers and practitioners to provide input on the data collection instruments developed for this evaluation as well as other methodological design issues. The TWG consists of researchers with expertise in issues related to literacy, instruction, grant implementation, and evaluation methods. We will consult the TWG throughout the evaluation. TWG members are listed below.

  • Kymyona Burk, Mississippi Department of Education

  • Cynthia Coburn, Northwestern University

  • Thomas Cook, George Washington University

  • Barbara Foorman, Florida State University

  • Pam Grossman, University of Pennsylvania

  • Carolyn Hill, MDRC

  • James Kim, Harvard University

  • Julie Morrill, Georgia Department of Education

  • Timothy Shanahan, Center for Literacy, University of Illinois at Chicago

  • Elizabeth Tipton, Northwestern University

  • Sharon Vaughn, University of Texas at Austin

A9. Payment or Gift to Respondents

School leaders and teachers are the targets of numerous requests to complete data collection instruments on a wide variety of topics from state and district offices as well as independent researchers. The Department and several decades of survey research support the benefits of offering incentives to achieve high response rates (Dillman, 2007; American Statistical Association and American Association for Public Opinion Research, 2016; Jacob & Jacob; 2012). Accordingly, we propose incentives for the school leader and teacher surveys to partially offset respondents’ time and effort in completing the surveys. Specifically, we propose to offer a $25 incentive to school leaders and teachers for completion of a survey, to acknowledge the 25 minutes required to complete it. This proposed amount is within the incentive guidelines outlined in the March 22, 2005, “Guidelines for Incentives for NCEE Evaluation Studies,” memo prepared for OMB. Incentives are proposed because high response rates are needed to ensure that the survey findings are reliable, and data from the surveys are essential to understand literacy instruction and professional supports in CLSD-funded schools.

A10. Assurance of Confidentiality

We are vitally concerned with maintaining the anonymity and security of the data. We have extensive experience in collecting information and maintaining the confidentiality, security, and integrity of extant, survey, and interview data. All members of our study team have obtained their certification on the use of human subjects in research. This training addresses the importance of the confidentiality assurances given to respondents and the sensitive nature of handling data. Our team also has worked with the Institutional Review Board at AIR to secure approval for this evaluation, thereby ensuring that the data collection complies with professional standards and government regulations designed to safeguard research participants.

We will conduct all data collection activities for this evaluation in accordance with relevant regulations and requirements. These include the Education Sciences Institute Reform Act of 2002, Title I, Part C, Section 183, which requires that the director of IES “develop and enforce standards designed to protect the confidentiality of persons in the collection, reporting, and publication of data.” The evaluation also will adhere to the requirements of Part D of Section 183, which prohibit disclosure of individually identifiable information, as well as make the publishing or inappropriate communication of individually identifiable information by employees or staff a felony. Finally, the evaluation will adhere to the requirements of Part E of Section 183, which require “[all] collection, maintenance, use, and wide dissemination of data by the Institute … to conform with the requirements of section 552 of Title 5, United States Code, the confidentiality standards of subsections (c) of this section, and sections 444 and 445 of the General Education Provisions Act (20 U.S.C. 1232 g, 1232h).” These citations refer to the Privacy Act, the Family Educational Rights and Privacy Act, and the Protection of Pupil Rights Amendment.

We will assure respondents that confidentiality will be maintained, except as required by law. We will include the following statement in the Notice of Confidentiality in all voluntary requests for data:

Information collected for this evaluation comes under the confidentiality and data protection requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for research purposes. The reports prepared for this evaluation will summarize findings across the sample and will not associate responses with a specific district, school, or individual. We will not provide information that identifies you, your school, or district to anyone outside the study team, except as required by law. In addition, no one at your school or in your district will see your responses.

Please note that data on state policies and resources/supports may be reported by state. Thus, while personally identifiable information about individual respondents will not be released, data displayed by state could be attributed to the state agency or possibly an individual respondent.

This evaluation does not include the collection of sensitive information. All respondents will receive information regarding survey topics, how the data will be used and stored, and how their confidentiality will be maintained. Individual participants will be informed that they may stop participating at any time. The goals of the evaluation, the data collection activities, the risks and benefits of participation, and the uses for the data are detailed in an informed consent form that all participants will read and sign before they begin any data collection activities. We will store the signed consent forms in secure file cabinets at the contractors’ offices.

The following safeguards are routinely required of contractors for IES to carry out confidentiality assurance, and they will be consistently applied to this evaluation:

  • All data collection employees sign confidentiality agreements that emphasize the importance of confidentiality and specify employees’ obligations to maintain it.

  • Personally identifiable information is maintained on separate forms and files, which are linked only by sample identification numbers.

  • Access to a crosswalk file linking sample identification numbers to personally identifiable information and contact information is limited to a small number of individuals who have a need to know this information.

  • Access to hard copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.

  • Access to electronic files is protected by secure usernames and passwords, which are only available to approved users. Access to identifying information for sample members is limited to those who have direct responsibility for providing and maintaining sample crosswalk and contact information. At the conclusion of the study, these data are destroyed.

  • The plan for maintaining confidentiality includes staff training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses. It also includes built-in safeguards concerning status monitoring and receipt control systems.

In addition, all electronic data will be protected using several methods. The contractors’ internal networks are protected from unauthorized access, including through firewalls and intrusion detection and prevention systems. Access to computer systems is password protected, and network passwords must be changed on a regular basis and must conform to the contractors’ strong password policies. The networks also are configured such that each user has a tailored, limited set of rights, granted by the network administrator, only to the files approved for access (which are stored in a secure cloud environment). Access to all electronic data files associated with this study is limited to researchers on the data collection and analysis team.

The data will be used to summarize findings in an aggregate manner or will be used to provide examples of program implementation and impact in a way that does not associate responses with a specific site or individual. The circumstances of state‑level respondents are somewhat different: The state‑level interviews, by their nature, focus on policy topics that are in the public domain. Moreover, it would not be difficult to identify CLSD directors in each state and thus determine the identity of the state‑level respondents. Acknowledging this, we will endeavor to protect the privacy of the state‑level interviewees and will avoid using their names in reports and attributing any quotes to specific individuals.

NORC is the subcontractor with primary responsibility for survey data collection and management. NORC maintains a long-standing adherence to protecting respondent confidentiality and has instituted stringent data security controls. All staff also must read and sign a legally binding pledge to uphold the confidentiality provisions established under the Privacy Act of 1974. Furthermore, all personally identifiable information will be removed from respondent data, and unique identification numbers will be assigned. To ensure computer and data security, NORC follows the NIST 800-53 R4 framework and complies with federal regulations as follows:

  • The web survey application runs as a two- or three-tier model: web server, application (app) server, and database. The app server and database servers are located on the NORC internal network. The web servers are separated by firewalls from the internet and the internal network.

  • All firewall rules are customized for each web server. Only the required ports are allowed through the firewall.

  • All web applications use HTTPS/TLS encryption.

  • All NORC servers follow the Center for Internet Security configuration standard.

  • All NORC servers run the McAfee® Antivirus Software. Updates are pushed out daily or when there is a critical update.

  • All NORC servers are physically located in a secured data center with card key access. The data center has its own cooling and environmental controls from the rest of the building.

  • NORC uses an intelligent log management system for all servers. The log management system monitors all servers in real time for errors.

  • NORC internally and externally monitors all web servers.

After the evaluation concludes, the data will be transmitted to the National Center for Education Statistics (NCES) for safekeeping as a restricted-use file. Before transmittal, the data will be stripped of any individual identifiers. Researchers wishing to access the data for secondary analysis must apply for an NCES license and agree to the rules and procedures guiding use of restricted-use files.

A11. Sensitive Questions

No questions of a sensitive nature are included in this study.

A12. Estimated Response Burden

Exhibit A3 presents an estimate of time burden for the data collections, broken down by data collection task. These estimates are based on our experience collecting extant and survey data from SEAs, LEAs, schools, leaders, and teachers. We will conduct data collections separately for FY2019 and FY2020 grantees. For the one SEA that received both a FY2019 and a FY2020 grant, we will collect student achievement data only once.

The total sample size is 2,075, and the expected number of respondents is 1,836. The total burden is an estimated 994 hours.

The total estimate of 994 hours includes the following efforts: 4 hours for SEAs to gather and submit policy documents for each of the 24 CLSD grants; 1 hour for SEAs to gather and submit subgrantee information for each of the 24 CLSD grants; 4 hours to collect student achievement data for each of the 23 SEAs that received grants; 30 minutes (0.50 hours) for the subgrantee survey, which will be administered once to representatives from each LEA that receives a subgrant; 25 minutes (0.42 hours) for the school leader survey, which will be administered once annually to one school leader in each of the approximately 130 schools that participate in the randomized trial; 25 minutes (0.42 hours) for the teacher survey, which will be administered once to Grade 3, 4, and 5 reading/English language arts teachers in the approximately 130 schools that participate in the randomized trial; 20 minutes (0.33 hours) for each of the approximately 130 schools that participate in the randomized trial to provide a roster of reading/English language arts teachers who teach Grades 3, 4, and 5; and 1 hour for an SEA representative for each of the 24 CLSD grants to participate in an interview. The estimated cost to respondents is $40,889.

Exhibit A3. Summary of Estimated Response Burden

Data Collection

Total Sample Size

Expected Response Rate

Expected N of Respondents



Annual Response Rate

Unit Response Time

(minutes)

Total Burden (hours)

Annual Burden (hours)

Hourly Rate

Estimated Burden Cost

Extant data

Policy documents

24

100%

24

8

240

96

32

$65

$6,240

Subgrantee contact information

24

100%

24

8

60

24

8

$65

$1,560

Teacher rosters

130

100%

130

43

20

44

15

$27

$1,188

Interviews and Surveys

Grantee interviews

24

100%

24

8

60

24

8

$65

$1,560

Subgrantee surveys

550

92%

506

169

30

253

84

$44

$11,132

School leader surveys (2021-22)

130

90%

110

37

25

46

15

$44

$2,024

Teacher surveys

1,170

85%

995

332

25

415

138

$27

$11,205

Achievement

Student achievement data

23

100%

23

7

240

92

31

$65

$5,980


TOTAL

2,075


1,836

612


994

331


$40,889

Averaged over the 3-year clearance period, the data collection’s annual sample size is 692 and the annual expected number of respondents is 612. The annual burden for this collection is 331 hours.6

A13. Estimate of Annualized Cost for Data Collection Activities

No additional annualized costs for data collection activities are associated with this data collection beyond the hour burden estimated in section A12.

A14. Estimate of Annualized Cost to Federal Government

The estimated cost to the federal government for the CLSD evaluation component of this contract (the subject of this clearance package), including development of the data collection plan and data collection instruments as well as data collection, analysis, and report preparation, is $6,133,571. Averaged over the 3-year clearance period, the annual cost to the federal government for the CLSD evaluation component of this contract is $2,044,524.7

A15. Reasons for Changes in Estimated Burden

This is a revision of the collection approved for the first evaluation component—the SRCL implementation study (OMB control number 1850-0945). A newly awarded cohort of grantees in the data collection requiring changed burden across instruments requiring revisions. This results in a decrease in burden and responses of -4,212 responses and -1,751 hours. The total burden and responses are 612 responses and 331 hours.

A16. Plans for Tabulation and Publication

The evaluation will produce a final report that describes SEAs, LEAs, and schools that participate in the CLSD program. It will also provide descriptive information about school selection, use of CLSD funds, and district-level CLSD activities and supports. In addition, the final report will present the impact of CLSD funding in LEAs that participate in the randomized trial and compare statewide longitudinal trends in reading outcomes for students in CLSD schools relative to those in similar non-CLSD schools.

  1. Analysis Plan

  • Research question 1: How do subgrantees target CLSD awards to schools and early learning programs, and for what types of literacy interventions and approaches are CLSD funds used? We will address research question 1 through descriptive analyses that draw on policy documents, subgrantee information, subgrantee survey data, and grantee interview data. Specifically, we will describe participants in the CLSD program, including how they were selected to participate. We also will describe the amount and distribution of funding received by participating SEAs, LEAs, and schools, and the primary activities, strategies, programs, and supports participants implemented to improve the literacy of the children they serve.

  • Research question 2: What is the impact of CLSD funding on student reading/English language arts achievement in Grades 3–5, after 2 years? To address research question 2, we will use a school-level randomized controlled trial (RCT). We will conduct random assignment separately in cohort 1 and cohort 2. Within each participating district in cohort 1, we will randomly assign about half of the participating elementary schools to receive CLSD funding and begin implementing CLSD activities in the 2020–21 school year (first funding group) and the other half to wait to begin implementing until the 2022–23 school year (second funding group). Within each participating district in cohort 2, we will randomly assign about half of the participating elementary schools to receive CLSD funding and begin implementing CLSD activities in the 2021–22 school year (first funding group) and the other half to wait to begin implementing until the 2023–24 school year (second funding group).

The RCT will focus on Grades 3–5, because state accountability testing programs include these grades and thus achievement outcome data will be available without requiring the burden of additional testing for the study. For cohort 1, the analyses will focus on the average impact over 2 years, for students who are in Grade 3 in the 2020–21 school year and Grade 4 in the 2021–22 school year; and students who are in Grade 4 in 2020–21 and Grade 5 in 2021–22. For cohort 2, the analyses also will focus on the average impact over 2 years, for students who are in Grade 3 in the 2021–22 school year and Grade 4 in the 2022–23 school year; and students who are in Grade 4 in 2021–22 and Grade 5 in 2022–23.

As a secondary analysis, we will examine the 1 year effects for students who are in Grade 5 in the 2020–21 school year (cohort 1) and in Grade 5 in the 2021–22 school year (cohort 2). The impact will be estimated by comparing outcomes for students in schools assigned to the first funding group relative to students in those schools assigned to the second funding group, pooled across the two cohorts. We designed the study to be able to detect an impact of 0.10 standard deviations, which is a reasonable effect given prior studies of literacy interventions (Basma & Savage, 2018; Gersten, Newman-Gonchar, Haymond, & Dimino, 2017; Wanzek et al., 2015; Wanzek et al., 2018). According to the study’s power calculations, the study must include 128 elementary schools to detect an effect of this size.

  • Research question 3: What is the impact of CLSD funding on teachers’ literacy instruction? In addition to estimating the difference in reading achievement between CLSD funded and non-CLSD funded elementary schools, we will also examine differences in reported literacy practices by teachers. The practices of interest are elements of comprehensive literacy instruction emphasized by the CLSD program, measured through the teacher survey.

To provide context for the impact estimates, we will draw on data from the school-level survey in schools participating in the RCT to describe how CLSD-funded activities and practices are implemented in schools in the first funding group, and describe the “business as usual” condition in schools in the second funding group.

  • Research question 4: How do trends in reading and math scores of students in CLSD-funded schools compare with those for students in similar non-CLSD schools? To address research question 4, we will conduct a comparative interrupted time series analysis (CITS), separately in each CLSD state. The CITS will focus on Grades 3–8 and one tested grade in high school (the grades for which testing is required under ESEA). For the CITS, we will first match CLSD-funded schools to comparison schools that exhibit a similar trend in student achievement during the years before CLSD. As a second step, we will measure the degree to which student achievement improves in CLSD-funded schools relative to comparison schools. Conducting a CITS analysis in conjunction with a school-level RCT will allow us to compare the impact estimates obtained using the RCT and a CITS based on the set of subgrantees included in the RCT. To the extent that results are similar, we will have greater confidence in the results of a CITS analysis that represents all the schools and districts that received CLSD funds.

  1. Publication of Results

We will use the data collected to prepare a report that clearly describes how the data address the key evaluation questions, highlights key findings of interest to policymakers and educators, and includes charts and tables to illustrate key findings. The report will be written in a manner suitable for distribution to a broad audience of policymakers and educators. The Department and AIR will publicly disseminate the report through their respective websites.

The data collected will be of immediate interest to policymakers and practitioners because it will provide timely, detailed, and policy-relevant information on a major federal grant program. The evaluation will offer unique insight into how CLSD grant funds are used, and the impact of those funds on student literacy outcomes.

A17. Display of Expiration Date for OMB Approval

All data collection instruments will display the OMB approval number and expiration date.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions to the certification statement are requested or required.

References

American Statistical Association and American Association for Public Opinion Research. (2016). Joint American Statistical Association/AAPOR statement on use of incentives in survey participation. Retrieved from https://www.aapor.org/Publications-Media/Public-Statements/AAPOR-Statement-on-Use-of-Incentives-in-Survey-Par.aspx.

Basma, B., & Savage, R. (2018). Teacher professional development and student literacy growth: A systematic review and meta-analysis. Educational Psychology Review, 30, 457–481.

Dillman, D. A. (2007). Mail and internet surveys: The tailored design, second edition, 2007 update. Hoboken, NJ: John Wiley. ISBN: 0-470-03856-x.

Jacob, R. T., & Jacob, B. (2012). Prenotification, incentives, and survey modality: An experimental test of methods to increase survey response rates of school principals. Journal of Research on Educational Effectiveness, 5(4), 401–418.

Gamse, B. C., Jacob, R. T., Horst, M., Boulay, B., & Unlu, F. (2008). Reading First Impact Study final report (NCEE 2009-4038). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Gersten, R., Newman-Gonchar, R. A., Haymond, K. S., & Dimino, J. (2017). What is the evidence base to support reading interventions for improving student outcomes in grades 1–3? (REL 2017–271). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southeast. Retrieved from http://ies.ed.gov/ncee/edlabs.

Rodgers, W. L. (2011). Effects of increasing the incentive size in a longitudinal study. Journal of Official Statistics, 2(27), 279–299.

Wanzek, J., Stevens, E. A., Williams, K. J., Scammacca, N., Vaughn, S., & Sargent, K. (2018). Current evidence on the effects of intensive early reading interventions. Journal of Learning Disability. doi:10.1177/0022219418775110

Wanzek, J., Vaughn, S., Scammacca, N., Gatlin, B., Waker, M. A., & Capin, P. (2015). Meta-analyses on the effects of tier 2 type reading interventions in grades K–3. Education Psychology Review, 28(3), 551–576.

Established in 1946, with headquarters in Washington, D.C., the American Institutes for Research® (AIR®) is a nonpartisan, not-for-profit organization that conducts behavioral and social science research and delivers technical assis­tance, both domestically and internationally, in the areas of education, health, and the workforce. For more information, visit www.air.org.

MAKING RESEARCH RELEVANT

AMERICAN INSTITUTES FOR RESEARCH

1000 Thomas Jefferson Street NW

Washington, DC 20007-3835 | 202.403.5000

www.air.org


1 One SEA received both a FY2019 and a FY2020 CLSD grant.

2 CLSD is a discretionary federal grant program authorized by Title II, Part B, Subpart 2, Section 2222 of the Elementary and Secondary Education Act (ESEA).

3 Grantees must cooperate with a national evaluation of the CLSD program (34 CFR 75.591) (Federal Registrar, v.84, n. 86, p. 19063).

4 States did not administer standardized achievement assessments in the 2019–20 school year due to the COVID-19 public health crisis. Our approach to dealing with the missing 2019-20 data for research questions 2 and 4 differs for FY2019 and FY2020 grantee states. To address research question 2 for FY2019 grantee states, we will use data from 2018–19 as the final pre-intervention year to establish baseline equivalence. To address research question 4 for FY2019 grantee states, we will use data from 2016–17, 2017–18, and 2018–19 to assess trends for 3 years prior to the time of the CLSD grant awards to establish the pre-intervention trend. To address research question 2 for FY2020 grantee states, we will use data from 2020–21 as the final pre-intervention year to establish baseline equivalence. To address research question 4 for FY2020 grantee states, we will use data from 2017–18 and 2018–19, and 2020–21 to assess trends for 3 years prior to the time of the CLSD grant awards, to establish the pre-intervention trend.

5 As noted previously, to address research question 4 we will need to request data from SEAs for the 2023–24 and 2024–25 school years. We will seek approval for this request in a subsequent OMB package submitted at a later date.

6 The total number of annual hours is slightly different from the sum of the annual hours for each data collection item due to rounding.

7 The annual cost to the federal government of the CLSD evaluation component of this contract is $1,226,714, averaged over the full 5 year period of performance. Also note that the total contract value (including the SRCL evaluation component of the contract) is $10,860,008.





File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleAIR Proposal
SubjectAIR Proposal
AuthorKolegas, Tracy (Bennett)
File Modified0000-00-00
File Created2021-01-13
© 2024 OMB.report | Privacy Policy