National Study to Inform 21CCLC_OMB Package2_Part A

National Study to Inform the 21st Century Community Learning Centers Program

Part A: Supporting Statement for Paperwork Reduction Act Submission

October 5, 2021

Submitted to:

U.S. Department of Education

Institute of Education Sciences

National Center for Education Evaluation and Regional Assistance

550 12th Street, S.W.

Washington, DC 20202

Project Officer: Erica Johnson
Contract Number: 91990019C0056

Submitted by:

Mathematica

P.O. Box 2393
Princeton, NJ 08543-2393
Telephone: (609) 799-3535
Fax: (609) 799-0005

Project Director: Susanne James-Burdumy
Reference Number: 50846

CONTENTS

A. Justification 1

Introduction 1

A1. Circumstances making the collection of information necessary 2

A2. Purposes and use of the information collection 2

A3. Use of information technology to reduce burden 6

A4. Efforts to identify duplication 7

A5. Efforts to minimize burden in small businesses 7

A6. Consequences of not collecting the information 8

A7. Special circumstances justifying inconsistencies with guidelines in 5 CFR 1320.6 8

A8. Federal register announcement and consultation 8

A9. Payments or gifts 9

A10. Assurances of confidentiality 10

A11. Questions of a sensitive nature 11

A12. Estimates of response burden 11

A13. Estimate of total capital and startup costs/operation and maintenance costs to respondents or record-keepers 14

A14. Annualized cost to the federal government 14

A15. Reasons for program changes or adjustments 14

A16. Plans for tabulation and publication of results 14

A17. Approval not to display the expiration date for OMB approval 16

A18. Exception to the certification statement 16

REFERENCES 17

Appendix A: Confidentiality Agreement

appendix b: afterschool center director survey

Appendix C: Afterschool Center Coaching Log

appendix D: afterschool center director interview

Appendix E: Request for Student Afterschool Attendance Records

appendix F: request for district administrative records

appendix G: afterschool center staff survey

Appendix H: student survey

Appendix I: school-day teacher survey

EXHIBITS

A.1. Research questions 2

A.2. Data sources for the study 4

A.3. Major data collection activities, by year 6

A.4. Participants in the June 2020 Technical Working Group meeting 8

A.5. Proposed incentives and comparison to NCEE guidelines 9

A.6. Estimated respondent time burden and cost 12

A.7. Estimation methods for each study research question 15

1. Justification

Introduction

The U.S. Department of Education (ED)’s Institute of Education Services (IES) requests clearance for data collection activities to support a study of afterschool strategies in the Nita M. Lowey 21st Century Community Learning Centers (21st CCLC) program. The 21st CCLC program funds services during nonschool hours, primarily during the school year.¹ The services aim to help students meet state academic standards, particularly for students in low-performing schools that serve high concentrations of low-income families (U.S. Department of Education 2018a).² Afterschool centers supported by program funds provide a broad range of activities and services, such as academic enrichment, physical activity, service learning, and activities to engage families. Program activities and services may play a crucial role in addressing the substantial learning loss and other challenges that have occurred as a result of the COVID-19 pandemic (Kuhfeld et al. 2020).

This study will have two components. The first is a national snapshot of strategies that afterschool centers in the 21st CCLC program use to serve their students and families. The national snapshot will complement and extend information from the program’s annual performance measures by providing an in-depth understanding of the outcomes centers aim to promote and the diverse ways their activities and services for students and families, supports for staff, and improvement strategies are designed to promote these outcomes. Describing these strategies can provide insights into ways that centers seek to address longer-term challenges, such as learning loss and trauma, stemming from the pandemic. The second component is an evaluation of a continuous quality improvement system implemented in the program’s afterschool centers (referred to as the “National Study of Continuous Quality Improvement to Inform the 21st CCLC Program”). The evaluation will examine the implementation and effectiveness of a system focused on improving staff practices that promote students’ social and emotional skills. Promoting these skills may be particularly important to compensate for the effects of the pandemic, in light of evidence that remote learning has negatively affected students’ social and emotional well-being (Duckworth et al. 2021).

This package requests clearance for data collection activities that will occur in March 2022 or later and impose burden on respondents. A previously submitted package (ICR Reference No. 202102-1850-003) requested clearance for data collection activities that will occur before March 2022.

A1. Circumstances making the collection of information necessary

The 21st CCLC program faces a need for two types of information this study is designed to collect. First, the program needs information on afterschool centers’ strategies for meeting students’ needs, which can help program stakeholders identify areas of strength and priorities for further support. Prior national studies of the program were conducted more than a decade ago (Dynarski et al. 2004; Black et al. 2009; Penuel and McGhee 2010). Since that time, the program has changed significantly. For example, the Every Student Succeeds Act, which reauthorized the program in 2015, established new requirements for how states monitor grantee performance that emphasized tracking student progress and also stressed the use of evidence-based practices. While the program’s annual performance measures provide data on a variety of important outcomes, they do not provide detailed information on the specific ways in which centers promote key student and family outcomes that contribute to student success. Afterschool centers in the 21st CCLC program may offer activities and services focused on various outcomes such as academic skills, social and emotional development, physical well-being, and family engagement. This study will fill a critical gap by describing the avenues through which afterschool centers strive to promote student success, and therefore, work to advance the program’s goals.

Improving program quality has long been a focus of afterschool program developers and researchers (Vandell 2013; Naftzger et al. 2018; Allen et al. 2019), and evidence on the effectiveness of promising program improvement strategies is the second type of information that would benefit the program. A continuous quality improvement system consisting of tools, procedures, and supports for improving program quality is one such strategy that has strong appeal to the 21st CCLC program. This strategy is applicable to a wide range of afterschool centers and can be tailored to each center’s circumstances and needs. Despite several states adopting such systems, there is limited rigorous evidence of this strategy’s effectiveness.

A2. Purposes and use of the information collection

IES contracted with Mathematica and its partners—the Forum for Youth Investment’s Weikart Center for Youth Program Quality, Pemberton Research, Research for Action, and Synergy Enterprises (together, “the study team”)—to conduct this study. The study’s two components will address a variety of research questions (Exhibit A.1).

Exhibit A.1. Research questions

National snapshot of afterschool centers’ strategies

1. What key outcomes for students and families do afterschool centers in the 21st CCLC program aim to promote, and what strategies do they use to promote them?

Evaluation of a continuous quality improvement system

2. To what extent did centers implement key aspects of the continuous quality improvement system in the first year of the system? What challenges did they encounter in implementing the system, and how did they address these challenges?

3. Across two years, what improvement strategies were used by centers that implemented the continuous quality improvement system? How did these differ from centers that did not implement the system?

4. How did the system affect the practices of center staff, including practices to promote students’ social and emotional skills? How did the system affect students’ social and emotional skills?

5. What is the cost-effectiveness of the continuous quality improvement system?

The national snapshot of afterschool centers’ strategies will be based on information from a nationally representative sample of centers in the 21st CCLC program in spring 2022. The study will examine the outcomes these centers aim to promote and the strategies they use to promote them.

The evaluation of a continuous quality improvement system will examine a system based on a repeating cycle of activities whereby afterschool centers assess their own quality, develop improvement plans, and implement these plans through trainings and other staff supports. Prior research found this type of system improved the quality of staff practices (Smith et al. 2012) but did not examine effects on student outcomes. The system to be tested here is designed to promote students’ social and emotional skills, which prior evidence suggests can be influenced by afterschool practices (Durlak et al. 2010) and relate to academic outcomes emphasized by the 21st CCLC program (Cunha et al. 2010; Kautz et al. 2014; U.S. Government Accountability Office 2017).

The evaluation will provide evidence on whether a continuous quality improvement system focused on students’ social and emotional skills implemented over two school years (2021-22 and 2022-23) can lead to better student outcomes. Implementing the system over a two-year period will allow sufficient time for study-provided trainings and supports to influence the practices of afterschool center staff and for those practices to influence student outcomes. The evaluation will be based on an experimental design, in which an anticipated 100 participating afterschool centers will be randomly assigned to either a treatment group that will implement the continuous quality improvement system or a control group that will not.

This package discusses all data collection activities proposed for the two components of the study. However, the package only requests clearance for data collection activities that will occur in March 2022 or later and impose burden on respondents. A previously submitted package requested clearance for data collection activities that will occur before March 2022.

The study’s analysis will rely on administrative data, observations, interviews, and surveys to address the study’s research questions. Exhibit A.2 shows the mode of data collection and purpose of each data source.

Exhibit A.2. Data sources for the study

Data	Type of data (primary or administrative)	Source	Mode of collection	Use(s) in study
National snapshot of afterschool centers’ strategies
Afterschool center director survey	Primary	Afterschool center directors	Web survey	To document the student and family outcomes that afterschool centers in the 21st CCLC program aim to promote and the strategies they use to promote them
Evaluation of a continuous quality improvement system
Afterschool center observations (no burden)	Primary	Independent observers’ ratings	In-person center observation	To assess the quality of afterschool center staff’s practices that promote social and emotional skills
Afterschool center coaching log	Primary	Afterschool center directors and co-leads in centers assigned to the treatment group	Web form	To monitor on an ongoing basis whether afterschool center directors and co-leads are coaching staff with the intended frequency in centers in the treatment group To document the amount and focus of this coaching
Afterschool center director interview	Primary	Afterschool center directors	In-person interview	To document: Center activities that assess center quality and identify program improvement goals Training and coaching that center directors received Focus of center practices for promoting social and emotional skills; challenges with improving those practices and strategies to address them Challenges in implementing a continuous quality improvement system and strategies to address them (among treatment centers only) Coordination between afterschool centers and schools Center directors’ background characteristics
Student afterschool attendance records	Administrative	Afterschool centers	Electronic submission or paper	To identify a sample of students who will be surveyed and whose school-day teachers will be surveyed To explore whether the effect of the continuous quality improvement system on students’ social and emotional skills differs according to how regularly students attend their afterschool center
District administrative records	Administrative	School districts	Electronic submission	To explore the effect of the continuous quality improvement system on secondary outcomes (those that could be affected but not necessarily within the study time frame), including students’ school attendance, disciplinary incidents, and academic achievement To describe the characteristics of students participating in the study
Afterschool center staff survey	Primary	Afterschool center staff	Paper survey	To document: Staff awareness, involvement, and support for improvement goals Training and coaching that staff received Focus of staff practices for promoting social and emotional skills Staff perceptions of challenges in implementing a continuous quality improvement system Staff members’ background characteristics
Student survey	Primary	Selected students in afterschool centers	Paper survey	To estimate effects of the continuous quality improvement system on the social and emotional skills that students demonstrated after school
School-day teacher survey	Primary	Selected school-day teachers of students in the sample	Paper survey	To estimate effects of the continuous quality improvement system on the social and emotional skills that students demonstrated in school, and on their course grades (a secondary outcome) To assess teachers’ connection to the afterschool center
Parent/guardian questionnaire and permission form	Primary	Parents/guardians of all students who attend the study afterschool centers in the first month of the 2021-2022 school year	Paper survey	To measure students’ social and emotional skills at baseline and students’ household characteristics—factors that the study will account for when estimating effects of the continuous quality improvement system To obtain parental permission for students’ participation in the study

Notes: Shaded rows indicate the data sources for which clearance is being requested in this submission. Clearance was requested in a previous submission for the parent/guardian questionnaire and permission form.

The study will last approximately six years (July 2019 to June 2025), with data collection occurring from fall 2021 through fall 2023. Exhibit A.3 shows the schedule of data collection activities.

Exhibit A.3. Major data collection activities, by year

	Data collection activities for
Timing	National snapshot of afterschool centers’ strategies	Evaluation of a continuous quality improvement system
Fall 2021		Afterschool center observations Parent/guardian questionnaire and permission form Afterschool center coaching log (through spring 2023)
Winter 2021		Student afterschool attendance records
Spring 2022	Afterschool center director survey	Afterschool center observations Afterschool center director interview Afterschool center staff survey Student survey School-day teacher survey
Spring 2023		Afterschool center observations Afterschool center director interview Afterschool center staff survey Student survey School-day teacher survey Student afterschool attendance records (end of school year)
Fall 2023		District administrative records

A3. Use of information technology to reduce burden

The data collection plan is designed to obtain information in an efficient way that minimizes respondent burden. When feasible, the study team will gather information from existing data sources, using the most efficient methods available. The study team will use technology to reduce burden for the data sources described below.

National snapshot of afterschool centers’ strategies

• Afterschool center director survey. The study team will ask 21st CCLC center directors to complete a web-based survey. Respondents will be able to complete the data collection instrument at a location and time of their choice, and the survey’s built-in checks and programmed skips will reduce errors.

Evaluation of a continuous quality improvement system

• Afterschool center coaching log. The study team will ask afterschool center directors and co-leads to complete a web-based coaching log after every coaching session they conduct with afterschool center staff. Respondents will be able to complete the log at a location and time of their choice, and the log’s built-in checks and programmed skips will reduce errors.

• Afterschool center director interview. Trained interviewers from the study team will interview afterschool center directors in person. Interviewers will take notes with laptop computers and, with respondents’ permission, audio-record the interviews. This approach will not reduce the burden on respondents but will facilitate accurate analysis of the data.

• Student afterschool attendance records. To minimize burden on afterschool centers when collecting these data, afterschool centers will be able to submit these data either electronically or in paper form, depending on which approach is most convenient to them. If they choose to submit the data electronically, the study team will ask them to upload the data to a secure file transfer site. If they submit the data in paper form, Mathematica field staff will collect these documents and enter the data into an electronic form for analysis.

• District administrative records. The study team will ask districts to submit administrative records in an electronic form using a secure file transfer site. Although the study team will specify the required data elements, the study team will reduce burden on the district by accepting any format in which the data are provided. Regardless of the form in which it is received, the data will be converted to a consistent format so that it can be combined with data submitted by other districts and will be suitable for analysis.

A4. Efforts to identify duplication

No similar studies are being conducted, and there is no equivalent source for the information to be collected. Moreover, the data collection plan reflects careful attention to the potential sources of information for this study, particularly to the reliability of the information and the efficiency in gathering it. The data collection plan avoids unnecessary collection of information from multiple sources. For example, student achievement will be measured using scores from state-administered student assessments, instead of administering an assessment as part of this study. Information obtained from the afterschool center staff survey, student survey, school-day teacher survey, and parent/guardian questionnaire and permission form is not available elsewhere.

A5. Efforts to minimize burden in small businesses

No small businesses will be involved in this study. The primary small entities for this study are afterschool centers and school districts. The data collection procedures have been designed to minimize burden on these entities:

• Afterschool centers. To minimize burden on afterschool centers when collecting student afterschool attendance records, the study team will communicate with afterschool center administrative staff in advance of the request to ensure they understand what is being asked of them. As discussed in Section A3, afterschool centers will be able to provide the data in whatever format—electronic or paper—is most convenient for them. If they submit the records in paper form, the study team will pick up the records at a time that is convenient for them. The only other data collection activity involving afterschool centers, afterschool center observations, imposes no burden on the centers.

• School districts. To minimize burden on school districts when collecting administrative records, the study team will request only the variables and records that are essential to addressing the study’s research questions. In addition, the study team will collect all requested records from each district at one time (Fall 2023) to avoid repeated requests.

A6. Consequences of not collecting the information

The data collection plan described in this submission is necessary for ED to characterize the strategies of afterschool centers in the 21st CCLC program and generate evidence about the effects of a continuous quality improvement system on afterschool centers’ staff practices and students’ outcomes. Without these data, the 21st CCLC program would not have up-to-date descriptive information on the student outcomes that centers in the program aim to promote or the strategies they use to promote these outcomes. Program stakeholders would lack recent, national, and large-scale evidence on the effectiveness of promising strategies.

A7. Special circumstances justifying inconsistencies with guidelines in 5 CFR 1320.6

There are no special circumstances associated with this data collection.

A8. Federal register announcement and consultation

a. Federal register announcement

A 60-day notice to solicit public comments was published in the Federal Register, Volume 86, No. 111, pages 31300–31301 on June 11, 2021. Two public comments were received. In response to one of those comments, minor changes were made to the afterschool center director interview and afterschool center staff survey. These changes refined (1) the wording of questions about gender and (2) the questions about practices on which staff received training and coaching.

The 30-day notice will be published to solicit additional public comments.

b. Consultations outside the agency

To inform the design of both components of the study, the study team convened an external Technical Working Group (TWG) meeting in June 2020 that included individuals with expertise in the 21st CCLC program, afterschool strategies, social and emotional skills, evaluation design, and statistical analyses. Input from the meeting will help ensure the study is of the highest quality and that findings are relevant to policymakers and program stakeholders at the state, grantee, and center levels. Exhibit A.4 provides the names, titles, and affiliations of the eight individuals who participated in the TWG meeting.

Exhibit A.4. Participants in the June 2020 Technical Working Group meeting

Name	Title and affiliation
Tom Akiva	Associate Professor, University of Pittsburgh
Jean Grossman	Senior Fellow, MDRC; Lecturer in Economics and Public Affairs, Princeton University
Jan Handa	Manager, 21st CCLC Grant Program, Nebraska Department of Education
Gwynn Hughes	Senior Program Officer, Charles Stewart Mott Foundation
Stephanie Jones	Gerald S. Lessor Professor in Early Childhood Development, Harvard University
Chris Rhoads	Associate Professor, University of Connecticut
Elizabeth Sanders	Associate Professor, University of Washington
Lorraine Thoreson	Manager, 21st CCLC Grant Program, Michigan Department of Education

c. Unresolved issues

There are no unresolved issues.

A9. Payments or gifts

Burden payments have been proposed for afterschool center staff, school-day teachers, and afterschool centers in the study. Compensating study participants for their time and effort to complete the study’s surveys and forms will help the study achieve high response rates on these data collection activities. High response rates will, in turn, allow the study team to more accurately address the study’s research questions.

Exhibit A.5 details the proposed payments and their justification. Payments will be made in the form of a check or gift card. Afterschool center staff and school-day teachers will receive payments for completing the surveys administered to them. Afterschool centers will be asked to assist with collecting the student survey and following up with absent students, as well as distributing and collecting the parent/guardian questionnaire and permission form. The afterschool centers will receive payments if the response rate at their center exceeds 90 percent for the student survey or the parent/guardian questionnaire and permission form. The proposed amounts for all payments are consistent with the incentive guidelines outlined in the March 22, 2005 memo, “Guidelines for Incentives for NCEE Evaluation Studies,” prepared for OMB.

Exhibit A.5. Proposed incentives and comparison to NCEE guidelines

Data collection activity	Recipient of payment	Amount of payment	Justification for payment
Afterschool center staff survey	Afterschool center staff	$30 for completing each survey	$30 is consistent with NCEE’s guidelines for a high-burden survey of instructional staff. The survey is lengthy (30 minutes) and requires staff to answer detailed questions about the amount and focus of the supports they have received and the challenges they have encountered in improving their practices.
Student survey	Afterschool center	$100 if 90 percent of the student surveys are completed for the afterschool center	This payment represents approximately $8 per student, which is less than NCEE’s guideline of $15 for a low-burden student survey. Ensuring high response rates on this survey is critical because it is a main data source for students’ social and emotional skills, a key study outcome. To support high response rates, afterschool centers must make substantial effort to arrange the time and venue for the group administration of this survey and help locate the students who need to take it.
School-day teacher survey	School-day teacher	$10 per student if completing the survey for a single student; $15 per student if completing a survey for more than one student	The $10 per student is consistent with NCEE’s guidelines for a high-burden teacher rating of students. Ensuring high response rates on this survey is critical because it is the other main data source for students’ social and emotional skills, a key study outcome. To complete this survey (which will take at least 10 minutes, depending on the number of students the teacher is asked about), teachers must answer a detailed set of questions about their students’ grades and behavior. The proposed incentive will be especially important to encourage responses from school-day teachers with little connection to the afterschool center who may feel little responsibility to cooperate with the study’s data collection. In addition, since answering the survey for multiple students poses a substantial burden, the study team has proposed a larger incentive for teachers with more than one student.
Parent/guardian questionnaire and permission form	Afterschool center	$100 if 90 percent of the parent/guardian questionnaires and permission forms are completed for the afterschool center	This payment represents approximately $2.50 per parent, which is less than NCEE’s guideline of $15 for a low-burden parent survey. Completed permission forms are necessary for the study to collect all other student-level data. To obtain completed permission forms and parent/guardian questionnaires, afterschool centers must make arrangements to distribute these forms and questionnaires to families who seek to enroll their children in the center. Although this requires significant effort by centers, the study team anticipates working with the centers to reduce the burden of collecting these forms and questionnaires, such as by attaching them to the centers’ normal enrollment form for families. Therefore, the proposed payment to each center for this effort ($100) is the same as the proposed payment for the student survey, even though the sample for this data collection is much larger (see Exhibit A.6).

A10. Assurances of confidentiality

Mathematica and its research partners will conduct all data collection activities for this study in accordance with relevant regulations and requirements, which are:

• The Privacy Act of 1974, P.L. 93-579 (5 U.S.C. 552a)

• The “Buckley Amendment,” Family Educational Rights and Privacy Act (FERPA) of 1974 (20 U.S.C. 1232g; 34 CFR Part 99)

• The Protection of Pupil Rights Amendment (PPRA) (20 U.S.C. 1232h; 34 CFR Part 98)

• The Education Sciences Reform Act of 2002, Title I, Part E, Section 183

The study team will protect the confidentiality of all data collected for the study and will use it for research purposes only. The Mathematica project director will ensure that all individually identifiable information about respondents remains confidential. All data will be kept in secured locations, and identifiers will be destroyed as soon as they are no longer required. All members of the study team having access to the data will be trained and certified on the importance of confidentiality and data security. When reporting the results, data will be presented only in aggregate form, such that individuals, schools, afterschool centers, grantees, and districts are not identified. Included in all voluntary requests for data will be the following or similar statement:

“Mathematica and its subcontractors—Pemberton Research, Research for Action, Synergy Enterprises, and the Forum for Youth Investment—follow the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for research purposes. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific regional or district afterschool program, afterschool center, district, school, or individual. We will not provide information that identifies respondents to anyone outside the study team, except if required by law.”

The following safeguards are routinely used by Mathematica to maintain data confidentiality, and they will be consistently applied to this study:

• All Mathematica employees are required to sign a confidentiality pledge (Appendix A) that emphasizes the importance of confidentiality and describes employees’ obligations to maintain it.

• Personally identifiable information (PII) is maintained on separate forms and files, which are linked only by random, study-specific identification numbers.

• Access to hard copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.

• Access to computer data files is protected by secure usernames and passwords, which are only available to specific users who have a need to access the data and who have the appropriate security clearances.

Mathematica’s standard for maintaining confidentiality includes training staff regarding the meaning of confidentiality, particularly as it relates to handling requests for information, and providing assurance to respondents about the protection of their responses. It also includes built-in safeguards concerning status monitoring and receipt control systems. In addition, all study staff who have access to confidential data must obtain security clearance from ED, which requires completing personnel security forms, providing fingerprints, and undergoing a background check.

A11. Questions of a sensitive nature

No questions of a sensitive nature will be included in this study.

A12. Estimates of response burden

For the data collection activities occurring in March 2022 or later—the focus of this clearance request—the total response burden is 2,120 hours.

Exhibit A.6 provides an estimate of time burden for the data collection activities, broken down by instrument and respondent. In addition, the exhibit presents estimates of indirect costs to all respondents for each data collection instrument.

Exhibit A.6. Estimated respondent time burden and cost

Respondent type and data collection activity	Time per response (hours)	Maximum number of responses per respondent	Number of respondents	Total time burden (hours)		Average hourly wage	Cost per response	Total cost burden
National snapshot of afterschool centers’ strategies, school year 2021-2022 (March 1, 2022 – June 30, 2022)
Afterschool center directors
Afterschool center director survey (spring 2022)	0.5	1	213	107		$25.81a	$12.91	$2,749
Evaluation of a continuous quality improvement system, school year 2021-2022 (March 1, 2022 – June 30, 2022)
Afterschool center directors
Afterschool center coaching log (spring 2022)	0.08	13	50	52		$25.81a	$2.06	$1,342
Afterschool center director interview (spring 2022)	1	1	85	85		$25.81a	$25.81	$2,194
Afterschool center staff
Afterschool center coaching log (spring 2022)	0.08	13	50	52		$15.93c	$1.27	$828
Afterschool center staff survey (spring 2022)	0.33	1	255	84		$15.93c	$5.26	$1,341
Students
Student survey (spring 2022)	0.2	1	1,105	221		$0.00	$0.00	$0
School-day teachers
School-day teacher survey (spring 2022)	0.17	1 for each student on whom a teacher is reporting	1,105 students on whom teachers are reporting	188		$28.69d	$4.88	$5,389
2021-2022 hours and cost totals				789				$13,843
Evaluation of a continuous quality improvement system, school year 2022-2023 (July 1, 2022 – June 30, 2023)
Afterschool center directors
Afterschool center coaching log (fall 2022-spring 2023)	0.08	36	50	144		$25.81a	$2.06	$3,717
Afterschool center director interview (spring 2023)	1	1	85	85		$25.81a	$25.81	$2,194
Afterschool center staff
Afterschool center coaching log (fall 2022- spring 2023)	0.08	36	50	144		$15.93c	$1.27	$2,294
Afterschool center staff survey (spring 2023)	0.33	1	255	84		$15.93c	$5.26	$1,341
Afterschool centers
Student afterschool attendance records (end of school year 2022-2023)	2	1	100	200		$18.84b	$37.68	$3,768
Students
Student survey (spring 2023)	0.2	1	850	170		$0.00	$0.00	$0
School-day teachers
School-day teacher survey (spring 2023)	0.17	1 for each student on whom a teacher is reporting	850 students on whom teachers are reporting	145		$28.69d	$4.88	$4,146
2022-2023 hours and cost totals				972				$17,459
Evaluation of a continuous quality improvement system, school year 2023-2024 (July 1, 2023 – June 30, 2024)
School districts
District administrative records (fall 2023)	20	1	18	360		$46.21e	$924.20	$16,636
2023-2024 hours and cost totals				360				$16,636
Total hours and costs across all years				2,120				$47,937
Average burden per year within the 3-year data collection period				707				$15,979

^a Average hourly wage for center directors is the wage for “Preschool and daycare education and childcare administrators” from the U.S. Bureau of Labor Statistics (2019).

^b Average hourly wage for afterschool center front office staff is the wage for “Secretaries and administrative assistants, except legal, medical, and executive” from the U.S. Bureau of Labor Statistics (2019).

^c Average hourly wage for afterschool center staff is the wage for “Other teachers and instructors” from the U.S. Bureau of Labor Statistics (2019).

^d Average hourly wage for school-day teachers is the wage for “Elementary school teachers, except special education” from the U.S. Bureau of Labor Statistics (2019).

^e Average hourly wage for school district staff is the wage for “Database administrators and architects” from the U.S. Bureau of Labor Statistics (2019).

A13. Estimate of total capital and startup costs/operation and maintenance costs to respondents or record-keepers

There are no direct, start-up, or maintenance costs to respondents or record keepers associated with this data collection.

A14. Annualized cost to the federal government

The total cost to the federal government for this study is $13,059,008. The estimated average annual cost over the six-year evaluation is $2,176,501.

A15. Reasons for program changes or adjustments

This is a new collection.

A16. Plans for tabulation and publication of results

a. Analysis plan

The study team will address the research questions for each component of the study using descriptive, comparative, and regression analyses (Exhibit A.7):

National snapshot of afterschool centers’ strategies

• Descriptive analyses: The study team will calculate the average values of continuous variables. For categorical variables, the study team will calculate the percentage of sample members in each category. The study team will also report measures of the precision of these estimates, such as confidence intervals.

• Comparative analyses: To compare groups, such as comparing characteristics of different types of centers, the study team will report the magnitudes of differences between groups and assess their statistical significance.

Evaluation of a continuous quality improvement system

• Descriptive analyses: The study team will apply the same approach for descriptive analyses as in the national snapshot of afterschool centers’ strategies.

• Comparative analyses: The study team will apply the same approach for comparative analyses as in the national snapshot of afterschool centers’ strategies.

• Regression analyses: The study team will use regression analyses in two ways. First, the study team will use them to estimate the effect of the continuous quality improvement system on afterschool center staff practices and student outcomes. The study team plans to control for covariates that represent baseline characteristics of students and their families, including students’ demographic characteristics and baseline social and emotional skills. Second, the study team will use regression analyses to understand the circumstances under which the continuous quality improvement system may be most successful. To do so, the study team will examine how effects on student outcomes and staff practices relate to key factors, such as characteristics of the centers and the ways they implemented the continuous quality improvement system.

Exhibit A.7. Estimation methods for each study research question

Research question	Descriptive analyses	Comparative analyses	Regression analyses
National snapshot of afterschool centers’ strategies
RQ 1. What key outcomes for students and families do afterschool centers in the 21st CCLC program aim to promote, and what strategies do they use to promote them?	X	X
How do centers use activities and services for students and families, supports for staff, and improvement strategies to promote key outcomes?	X	X
What challenges do centers face when trying to promote key outcomes, and what are their approaches to addressing those challenges?	X	X
Evaluation of a continuous quality improvement system
RQ2. To what extent did centers implement key aspects of the continuous quality improvement system in the first year of the system? What challenges did they encounter in implementing the system, and how did they address these challenges?	X
RQ 3. Across two years, what improvement strategies were used by centers that implemented the continuous quality improvement system? How did these differ from centers that did not implement the system?	X	X
In centers that implemented the system, what strategies did center leaders and staff use to improve the quality of their center? How did these strategies differ from centers that did not implement the system?	X	X
In centers that implemented the system, what training and coaching did center staff receive? How did these differ from centers that did not implement the system?	X	X
RQ 4. How did the system affect the practices of center staff, including practices to promote students’ social and emotional skills? How did the system affect students’ social and emotional skills?			X
How did the effects of the system differ according to students’ attendance at their afterschool center?			X
Did the effects of the system differ according to other characteristics of students or their centers, or according to the ways in which centers implemented the system?			X
RQ 5. What is the cost-effectiveness of the continuous quality improvement system?	X

b. Publication plan

The study team will prepare publications for both components of the study, including two reports and one brief.

National snapshot of afterschool centers’ strategies

The first report, to be released in 2023, will provide findings from the national snapshot of afterschool centers’ strategies (research question 1). Using a nationally representative sample of 21st CCLC afterschool centers, it will describe the outcomes for students and families that centers aim to promote; the diverse ways that centers use activities and services for students and families, supports for staff, and improvement strategies to promote these outcomes; and challenges the centers face as well as the strategies they use to address those challenges.

Evaluation of a continuous quality improvement system

The brief, to be released in 2023, will describe treatment centers’ early experiences implementing the continuous quality improvement system (research question 2). It will document the extent to which the centers put into place key aspects of the system in the first year of its implementation. It will also describe centers’ challenges in implementing the system and their strategies to address them.

The final report, to be released in 2025, will examine both the implementation and effectiveness of the continuous quality improvement system over two years (research questions 3 through 5). It will describe the extent to which treatment centers implemented the system as intended; the resulting contrast between treatment and control centers in their approaches to quality improvement; the effects of the system on staff practices and students’ social and emotional skills; how the effects of the system were related to key factors (such as students’ attendance, centers’ characteristics, and the ways in which centers implemented the system); and the system’s cost-effectiveness.

A17. Approval not to display the expiration date for OMB approval

The Institute of Education Sciences is not requesting a waiver for the display of the OMB approval number and expiration date. The study will display the OMB expiration date.

A18. Exception to the certification statement

No exceptions to the certification statement are requested or required.

REFERENCES

Allen, P.J., R. Chang, B.K. Gorrall, L. Waggenspack, E. Fukuda, T.D. Little, and G.G. Noam. “From Quality to Outcomes: A National Study of Afterschool STEM Programming.” International Journal of STEM Education, vol. 6, 2019.

Black, A., M. Somers, F. Doolittle, R. Unterman, and J. Grossman. “The Evaluation of Enhanced Academic Instruction in After-School Programs: Final Report.” NCEE 2009-4077. Washington, DC: U.S. Department of Education, Institute of Education Sciences National Center for Education Evaluation and Regional Assistance, September 2009.

Cunha, F., J. J. Heckman, and S. M. Schennach. “Estimating the Technology of Cognitive and Noncognitive Skill Formation.” Econometrica, vol. 78, no. 3, 2010, pp. 883–931.

Duckworth, A., T. Kautz, A. Defnet, E. Satlof-Bedrick, S. Talamas, and L. Steinberg. “The Thriving Gap: Students Attending School Remotely Suffer Socially, Emotionally, and Academically.” Unpublished manuscript, 2021.

Durlak, J. A., R. P. Weissberg, and M. Pachan. “A Meta-Analysis of After-School Programs That Seek to Promote Personal and Social Skills in Children and Adolescents.” American Journal of Community Psychology, vol. 45, nos. 3–4, 2010, pp. 294–309.

Dynarski, M., S. James-Burdumy, M. Moore, L. Rosenberg, J. Deke, and W. Mansfield. “When Schools Stay Open Late: The National Evaluation of the 21st-Century Community Learning Centers Program: New Findings.” Washington, DC: U.S. Department of Education, Institution of Education Sciences, National Center for Education Evaluation and Regional Assistance, 2004.

Elliott, S. N., C. J. Anthony, J. C. DiPerna, P. Lei, and F. M. Gresham. “SSIS Brief Scales Series.” Scottsdale, AZ: SAIL CoLab, 2020.

Kautz, T., J. J. Heckman, R. Diris, B. Ter Weel, and L. Borghans. “Fostering and Measuring Skills: Improving Cognitive and Non-Cognitive Skills to Promote Lifetime Success.” Working Paper 20749. Cambridge, MA: National Bureau of Economic Research, 2014.

Kuhfeld, M., J. Soland, B. Tarasawa, A. Johnson, E. Ruzek, and J Liu. Borghans. “Projecting the Potential Impact of COVID-19 School Closures on Academic Achievement.” Educational Researcher, vol. 49, no. 8, pp. 549-565, 2020.Naftzger, N. “A Summary of Three Studies Exploring the Relationships Between Afterschool Program Quality and Youth Outcomes.” Conference paper presented at 2014 Ready by 21 National Meeting. Washington, DC: American Institutes for Research, 2014.

Naftzger, N., S. Sniegowski, C. Smith, and A. Riley. “Exploring the Relationship between Afterschool Program Quality and Youth Development Outcomes: Findings from the Washington Quality to Youth Outcomes Study.” Naperville, IL: American Institutes for Research, 2018.

Penuel, W. R., and R. McGhee, Jr. “21st Century Community Learning Centers Descriptive Study of Program Practices.” Washington, DC: U.S. Department of Education, Office of Planning, Evaluation and Policy Development, 2010.

Smith, C., T. Akiva, S. Sugar, Y. Lo, K. Frank, S. Peck, K. Cortina, and T. Devaney. “Continuous Quality Improvement in Afterschool Settings: Impact Findings from the Youth Program Quality Intervention Study.” Washington, DC: Forum for Youth Investment, 2012.

U.S. Department of Education. “21st Century Community Learning Centers.” March 2018a. Available at https://www2.ed.gov/programs/21stcclc/index.html. Accessed November 11, 2020.

U.S. Department of Education. “21st Century Community Learning Centers (21st CCLC) Analytic Support for Evaluation and Program Monitoring: An Overview of the 21st CCLC Performance Data: 2016–17.” 13th report. Washington, DC: U.S. Department of Education, 2018b.

U.S. Government Accountability Office. “Education Needs to Improve Oversight of its 21st Century Program.” GAO-17-400. Washington, DC: U.S. Government Accountability Office, 2017.

U.S. Bureau of Labor Statistics. “May 2019 Occupation Profiles.” May 2019. Available at https://www.bls.gov/oes/current/oes_stru.htm.

Vandell, D. L. “Afterschool Program Quality and Student Outcomes: Reflections on Positive Key Findings on Learning and Development from Recent Research.” In W. S. White & T. K. Peterson (Eds.), Expanding minds and opportunities: Leveraging the power of afterschool and summer learning for student success, 2013. Retrieved 12/8/20 from https://www.expandinglearning.org/expandingminds/article/afterschool-program-quality-and-student-outcomes-reflections-positive-key.

www.mathematica-mpr.com

Improving public well-being by conducting high quality,
objective research and data collection

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ TUCSON, AZ ■ Washington, DC ■ WOODLAWN, MD

1 Most participants (71 percent) are students attending afterschool centers during the school year, with the remainder being family members (14 percent) or summer attendees (15 percent; U.S. Department of Education 2018b).

2 For the 2020-2021 school year, states could apply for a waiver that would allow 21st CCLC programs to provide supplemental activities during the school day when students are not receiving in-person instruction, such as activities that would support students’ remote learning.