Supporting_Statement_Part_A 060515

The U.S. Department of Education (ED) will conduct a demonstration to test the effectiveness of promising strategies to improve college-related outcomes in the federal college access program Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP). The demonstration will use a randomized controlled trial (RCT) design to test the effectiveness of sending customized messaging to students, during the summer after high school graduation and in the fall and spring of their expected first year of college.

This study is being sponsored by ED’s Institute of Education Sciences (IES), in collaboration with the Office of Postsecondary Education, and implemented by Abt Associates Inc. and its partners, Survey Research Management (SRM), Program and Policy Insight (PPI), Digital Design Group (DDG), and consultant Ben Castleman (together, the “study team”).

Overview of the GEAR UP Program

Chapter 2 of Subpart 2 of Part A of Title IV of the Higher Education Act of 1965, as amended (HEA) authorizes the GEAR UP program. The program provides funding – over $300 million in FY 2015 – to state agencies and local partnerships to support disadvantaged students in obtaining a high school diploma and to prepare them for success in postsecondary education. Grantees are expected to provide a comprehensive set of intervention services such as academic support, counseling, mentoring, and outreach to disadvantaged middle and high school students, as well as information about college financial aid options and the benefits of postsecondary education to students and their families. State grantees may, and partnership grantees must, focus on a cohort of students (typically beginning in 7^th grade) and provide services to them through high school, though funds can be used to serve other students as well. State grantees are required to, and local partnership can, also provide scholarships to help participating students pay for college attendance.

Importantly, starting with the 2008 Higher Education Opportunity Act, GEAR UP grantees may now apply for a seventh year of funding to support students into their first year of college in order to improve college enrollment and persistence (referred to as the 7^th year). The first group of grantees with this funding, those awarded grants in FY 2011, will begin providing extended services to 2016-2017 seniors transitioning to postsecondary education the following year.

Overview of the Student Messaging in GEAR UP Demonstration

This demonstration is designed to address a new need among GEAR UP grantees for strategies to serve students during the 7^th year. According to recent focus groups conducted by Abt with GEAR UP grantees, grantees find it difficult to reach every GEAR UP student on a one-on-one basis while in high school, and to track and support students as they leave school and transition to college, particularly since students may be scattered across a variety of postsecondary institutions both in and out of state. Nationally, as many as 40 percent of first-generation, low-income high school seniors intending to go to college fail to enroll the following fall, with information gaps, logistical complexities, and concerns about fitting in identified as primary explanations of this “summer melt” (Castleman and Paige, 2014). Of those who matriculate, many students drop out during their first year or fail to re-enroll the second year (NSC, 2014b), and persistence is lower among students from low-income and high minority high schools (NSC, 2014a).

The demonstration will investigate promising approaches emerging from recent rigorous research that has investigated low-cost student messaging to improve college matriculation (Castleman & Page 2013) and college persistence (Castleman & Page, 2014), and targeted interventions to develop growth mindset or individuals’ beliefs that abilities, particularly academic abilities, can be developed through effort (Yeager, Walton & Cohen 2013). Outcomes for this study will include college enrollment, FAFSA completion, and persistence, which will be drawn from administrative data.

To test whether these approaches enhance the existing GEAR UP services, customized messages, delivered via a digital format (phone texting), will be sent to students identified as seniors, who intend to go to college, in high schools that are part of the GEAR UP program. Two independent treatments of student messaging will be tested; both treatments will consist of text messages sent to students during the summer that follows their senior year in high school and extending into their freshman year in college. Treatment 1 (milestone messages) will consist of text messages to remind students of key milestones related to college matriculation and college persistence, and direct students to where they can receive additional support, including from GEAR UP advisors. Treatment 2 (enhanced messages) will include the milestone messages plus messages designed to reinforce a growth mindset or an understanding that intelligence can be developed. In both treatments, students will be able to respond to the message to receive additional support from a counselor. All students will continue to be eligible for any regular GEAR UP services provided.

Students randomly assigned to Treatment 1 (milestone messages) will receive a minimum of 10 messages over the summer designed to relay information that students may not be aware of or may not have easy access to. The messages will inform students about key enrollment-related tasks they need to complete such as submitting a deposit, filling out the housing form, submitting their proof of health insurance form, registering for classes, etc. Students will receive a minimum of 12 messages during their freshman year designed to support students’ completion of their first semester and persistence into the second semester. These messages will focus on key tasks to complete such as paying tuition and registering for the next semester as well as common challenges faced in the first semester such as academic unpreparedness and lack of knowledge about campus supports.

Treatment 2 (enhanced messages) will consist of the milestone messages, plus additional messages that involve growth mindset messaging. These messages will consist of an activity that engages students with the idea that intelligence is malleable. Students randomly assigned to Treatment 2 will receive the milestone messages, plus approximately three additional growth mindset messages over the summer and an additional five during their freshman year.

All messages will be grounded in both social psychological and behavioral economic research suggesting that the messages be aligned to key time points such as housing deadlines, registration deadlines, tuition payment due dates, the add/drop period, and mid-term exams and that the messages be action oriented such as submit your housing form and go to this webpage to register for classes. Students will also be able to respond to the messages to receive support from a GEAR UP advisor.

To the extent possible, messages will be customized for each student based on the information they provide on the baseline survey about college intentions. For example, deadlines for submitting particular forms and links to find more information will be specific to the college a student indicated as the planned college of enrollment on their baseline survey.

Exhibit A-1 contains a preliminary outline of key prompts that are timed to key milestones or pressure points in enrolling and persisting in college.

Exhibit A-1. Draft Outline of Student Messages

Topic	Personalization
Treatment 1
Summer messages
Log into college's web portal to access important paperwork and deadlines	Link to specific website
Register for orientation and placement tests	Institution-specific registration deadlines
Complete housing forms	Link to specific forms, include deadlines
Sign up or waive health insurance (if relevant)	Link to specific forms, include deadlines
Submit deposit for fall enrollment	Institution-specific deposit deadlines
Register for fall classes	Institution-specific registration deadlines
Reminder of orientation and/or welcome activities dates	Institution-specific dates/activities
College messages
Provide links to campus resources for student support	Links to specific resources
Reminder about upcoming midterms	Link to campus specific academic support center
Mindset booster on overcoming challenges in college	Link to campus counseling center
Check financial aid eligibility for second year	Link to requirements
Register for spring classes	Institution-specific registration deadlines
Treatment 2 (all messages above, plus the enhancements below)
Summer message to introduce intelligence as something that can grow
Links to activities that promote growth mindset
Pre-midterm exams reminder that intelligence can grow through dedicated study
Post-midterm check on how exams went
Pre-final exams link to video about growing intelligence

Overview of the Evaluation

The study will use a rigorous random assignment design to test the effectiveness of sending customized messages to GEAR UP students that provide students with key information and allow students to respond for additional support. GEAR UP projects that were awarded awards in Federal Fiscal Year 2011 will be invited to volunteer for the demonstration (see Appendix A for invitation), because they are the first cohort of grantees to have funds to support a 7^th year of services. To test the effectiveness of the customized messaging, the study will assign students within participating high schools selected by volunteer GEAR UP projects to a control condition (eligible for all GEAR UP services), or one of two treatment conditions (eligible for all GEAR UP services, plus receiving customized text messages). Students will be randomly assigned to a condition at the end of their senior year of high school. This two-treatment design maximizes our ability to investigate the effects of messaging associated with key college milestones, as well as messaging that includes a social-psychological enhancement.

At the end of the 2016-2017 school year, students in high schools served by GEAR UP projects who declare their intention to enroll in college will be randomly assigned to receive only regular GEAR UP 7^th year services (Control), milestone messages in the summer after high school and freshman year of college (Treatment 1), or milestone messages plus growth mindset messages in the summer after high school and freshman year of college (Treatment 2). The comparison of the Control group to each Treatment group will test the impacts of messaging over and above the regular 7^th year GEAR UP activities. The comparison of Treatment 1 to Treatment 2 will test whether the growth mindset addition to the messaging has an effect above that of the milestone messages alone. Exhibit A-2 displays the groups into which students will be randomly assigned to test the impacts of different messaging on college outcomes.

Exhibit A-2: Randomization into Treatment Groups

Randomization	College Intending Seniors
End of 2016-17 school year	C1	T1	T2
	- Regular GEAR UP services	- Regular GEAR UP services	- Regular GEAR UP services
		- Milestone messages	- Milestone messages
			- Growth mindset messages

The Annual Performance Reports (APR) that GEAR UP grantees submit to ED do not contain student-level data. Thus, baseline data will be collected from all students via a survey, and college-related outcome data will be extracted from national datasets (National Student Clearinghouse Data (NSC) and the Federal Student Aid (FSA) database).

The study will address the three research questions:

To what extent do the messages—above and beyond the services GEAR UP may already provide—improve student college-related outcomes? Is the effect of milestone messaging different with the addition of growth mindset messages?
What types of college advising are typically received by GEAR UP students?
Is there variation in the impact of the messages, and to what extent is the variation associated with student characteristics or features of the GEAR UP project or school?

To answer these questions we will conduct both impact and descriptive analyses. The first report, which will address each research question with a focus on college enrollment-related outcomes, will be published in 2018. The final report, which will also include college persistence, will be available in 2020.

Exhibit A-3 presents the research questions along with the data sources for each question, the analytic approach and outcomes of interest.

Exhibit A-3. Evaluation Questions, Data Sources, Analytic Approach, and Outcomes of Interest

Research Question	Data Sources	Analytic Approach	Outcomes of Interest
To what extent do the messages—above and beyond the services GEAR UP may already provide—improve student college-related outcomes? Is the effect of milestone messaging different with the addition of growth mindset messages?	Student rosters Student baseline survey data National Student Clearinghouse Data Federal Student Aid (FSA) database	Impact analysis of RCT (HLM) Baseline survey sample: 16,000 students in 80 schools. Analytic sample size: approximately 5,600 students in 80 high schools that intend to enroll in college, equally divided across treatment and control groups.	College enrollment College persistence into sophomore year FAFSA completion
What types of college advising are typically received by GEAR UP students?	Student baseline survey data	Descriptive analysis Survey sample: 16,000 students in 80 schools.	Description of GEAR UP program components provided and received
Is there variation in the impact of the messages, and to what extent is the variation associated with student characteristics or features of the GEAR UP project or school?	Student baseline survey data National Student Clearinghouse Data Federal Student Aid (FSA) database	Descriptive (subgroup) analysis Sample size: 5,600 students in 80 high schools that intend to enroll in college	College enrollment College persistence into sophomore year FAFSA completion

Altogether, we expect that 80 GEAR UP high schools will participate in the research project, with an estimated total of 16,000 seniors.¹ All seniors will be surveyed, with the consent process giving them (or the parents) an option to opt out of completing the baseline survey or other data collection activities. The survey will gather information from students for participation in the student messaging intervention (see Appendix C for the consent letter and Appendix D for the student survey).

It is expected that 70 percent of seniors will express intent to attend college² and 50 percent of these seniors would provide assent and information for texting, resulting in 5,600 students overall, with 1,866 students in each group.

Circumstances Making the Collection of Information Necessary

The study’s data collection and design will allow for a rigorous assessment of a promising practice within GEAR UP that focuses on program improvement targeting college-related outcomes for participating disadvantaged youth. Nationally, children from disadvantaged families hold lower educational expectations (Goyette, 2008; Kao & Tienda, 1998) and are less likely to attend, persist in, and complete college than their more economically advantaged peers (e.g. Haskins, 2008; Bailey & Dynarski, 2011). For those who do enroll, the transition to and first year in college can bring particular challenges (Liu, Sharkness, & Pryor, 2008; Darling & Smith, 2008). While factors such as students’ academic preparation and financial circumstances are cause for concern, a substantial number of low-income students fall off track in the complex processes of obtaining financial aid, and applying to, enrolling in and staying in college (Bowen, Chingos, & McPherson, 2009; Roderick, Nagaoka, Coca, & Moeller, 2008).

Experimental evidence suggests that relatively low-cost interventions may help low-income students remain on track. Text messaging interventions have had positive effects on college outcomes. For example, an intervention that involved ten text messages with customized messages to college-intending high school graduates increased college enrollment among students with less access to college-planning supports and who were not as far along with their college planning at the completion of high school (Castleman & Page, 2013). Also, two to three hours of transition counseling during the summer after high school slightly increased college enrollment among students in some sites and improved persistence into the sophomore year even more (Castleman, Page & Schooley, 2014). Finally, messaging during students’ freshman year providing information about financial aid deadlines and FAFSA completion improved college persistence into the sophomore year among community college students (Castelman & Page, 2014).

Experimental evidence also suggests that brief interventions focused on the notion that intelligence and personal attributes are malleable can have a positive effect on academic outcomes. For example, a brief exercise in which college students wrote letters to middle school students, encouraging them to work hard and describing intelligence as expandable, resulted in higher perception of the malleability of intelligence, academic engagement, and academic performance among the college students (Aronson, Fried & Good, 2002). An experimental study found that high school freshmen who engaged in a light-touch activity that included reading a summary of neuroscience research, next reading testimonials from upperclassmen who had read the summary and applied it to their own peer conflicts, and then writing their own testimonials for future freshmen reported lower stress, less frequent physical illness and better academic performance than students who had not participated in the brief activity (Yeager et al. 2014). Another study found that students exposed to a two pronged intervention—first reading about other students who felt they did not belong in college, but that this sense of not belonging dissipates over time, and then writing about how their own worries had changed over time—earned higher grades, and reported being happier and healthier than the control groups in the study (Walton & Cohen 2011).

These emerging practices can be tested in the context of GEAR UP, and fill a need to identify effective (or ineffective) strategies for providing 7^th year program services in low-cost ways. The first cohort of GEAR UP students eligible for those services will be high school seniors in school year 2016-2017. The demonstration is being introduced at a key time, before grantees have solidified their plans for the 7^th year and when they are most open to implementing strategies that will be assessed using random assignment.

Purposes and Use of the Information Collection

This ICR requests approval for data collection that involves burden to the public, which will occur primarily when study participants are seniors in high school. Much of the data collection will draw upon administrative data, so the study’s burden on projects and students will mainly be limited to the student roster collection and baseline survey collection.

All information will be collected by the Abt study team. Exhibit A-4 outlines the combination of administrative and survey data that will be collected to examine the implementation and impacts of the customized messaging intervention.

Exhibit A-4. Data Collection Plan

Schedule	Data Collection	Activity	Respondent	Mode
Winter 2015-2016	List of participating schools	GEAR UP grantees will identify schools that serve seniors and are willing to participate in the study	GEAR UP project directors/ site coordinators	E-mail
Fall 2016	List of GEAR UP seniors	Gather student rosters of high school seniors with access to GEAR UP services from participating schools	GEAR UP project directors/ site coordinators	Electronic
Spring 2017	Administer baseline survey	Administer a brief baseline survey to capture student assent, intention to enroll in college, and a small number of covariates for the analysis (e.g., demographics)	Students (high school)	Paper/Electronic
Fall 2017	Obtain data from National Student Clearinghouse (NSC) and Federal Student Aid Center (FSA)	NSC will provide matriculation and persistence outcome data; FSA data will indicate whether student completed the FAFSA for the first and second years of college	None (administrative record)	Electronic
Fall 2018	Obtain data from National Student Clearinghouse (NSC) and Federal Student Aid Center (FSA)	NSC will provide matriculation and persistence outcome data; FSA data will indicate whether student renewed the FAFSA	None (administrative record)	Electronic

Data to Be Collected

This ICR requests clearance for all collection activities of the demonstration.

Student Rosters. The study team will collect rosters of high school seniors who have access to GEAR UP services in the high schools in the demonstration. In order to obtain this information, the study team will request student rosters from the GEAR UP site coordinator from participating schools (see Appendix B for roster request). In requesting student rosters, the study team will request only directory information as designated under the Family Educational Rights and Privacy Act (FERPA) and its implementing regulations (20 U.S.C 1232g and 34 CFR Part 99) such as the student’s name, email, date of birth, telephone number, zip code, parent contact information, and district or school identification numbers.³ These data will help in locating students and parents for the study’s consent processes and baseline data collection effort.

The study team will reach out to each site coordinator via a telephone call during the middle of the 2016-2017 academic year to request student rosters of 12^th grade GEAR UP students. Site coordinators will be asked to submit student rosters through a password protected secure file transfer portal (SFTP).

Baseline Student Survey. The baseline student survey will be administered in the spring of the 2016-2017 school year to high school seniors who have not opted-out or whose parents have not opted-out (see Appendix C for consent letter). The survey is voluntary and student assent will be collected at the beginning of the baseline student survey. The baseline survey will collect data that will: be used to determine eligibility for the intervention (students must be intending to enroll in college); be used as covariates to improve the precision of impact estimates; and allow for checks on the equivalence of the treatment and control group individuals who are included in the final analytic sample (see Appendix D for student survey). The baseline survey will gather information about student demographics and other characteristics that are likely correlated with college-going outcomes. These characteristics will include college-going expectations and plans,⁴ such as the student’s college choice for fall 2017; educational expectations; academic preparation; and college advising and supports received. As such the student survey does not contain any questions of a sensitive nature. In developing the survey we will consider scales with established validity and reliability whenever possible.

The baseline student surveys will be administered in paper or electronic format. To facilitate data collection, the site liaison will work with each GEAR UP school to: facilitate the logistics for the baseline student survey administration; track response rates and communicate with project directors/site coordinators, as needed; remind absent students about the survey (see Appendix E for survey reminder); and distribute individualized thank you or reminder letters to students near the end of the survey fielding period.

Postsecondary Enrollment and Financial Aid Data. In November 2017, postsecondary enrollment data will be extracted from the National Student Clearinghouse (NSC) for students who were seniors in 2016-2017. The NSC data cover all 50 states and both public and private institutions. The NSC provides student-level data on college enrollment and completion for over 3,600 member institutions; together, these institutions represent 91 percent of enrolment in higher education in the U.S. Although the NSC has lower coverage of two-year than four-year institutions, fewer private than public institutions, and gaps in data in some states, no other national-level datasets contain individual student-level data on postsecondary enrollment, full-time versus part-time status and completion.

In November 2017, data on FAFSA completion and receipt of Federal student aid (such as Pell grants, Federal Work-Study, etc.) will be obtained from the FSA data center. To obtain these data, we will prepare a data file containing identifying information for students in the study sample for submission to the FSA.

NSC and FSA data will be obtained again in November 2018 to assess persistence into a second year of college for the treatment and control groups.

The collection of all NSC and FSA data will be completely non-intrusive to students since their directory information from rosters and baseline surveys will be used for all subsequent follow-ups.

Use of Information Technology and Burden Reduction

The data collection plan reflects sensitivity to issues of efficiency, accuracy, and respondent burden.

The study will use a combination of mechanical and electronic technology to collect data. For each data collection task, the study team has selected the form of technology that enables the collection of valid and reliable information in an efficient way while minimizing respondent burden.

The Abt study team will collect student rosters from site coordinators. To minimize burden, the evaluator’s electronic mail address and toll-free telephone number will be included on the request for student rosters should site coordinators have questions, and site coordinators will be provided with an SFTP link to upload rosters with minimal effort and time. Taken together, these procedures are all designed to minimize the burden on respondents.

Additionally, we will explore with individual schools whether the student baseline survey can be administered electronically, or with a paper-and-pencil option if computer access on a classroom-level scale is not feasible.

Further, the study will rely on existing sources of data for much of the data, which will add no respondent burden.

Efforts to Identify Duplication

To date, there has been no rigorous study of strategies that could improve the effectiveness of enhancements to GEAR UP related to students’ matriculation and persistence in college. Our survey instruments will be designed to reflect the efforts of a consortium of state GEAR UP grantees to develop common program service definitions. To the extent possible, the study team will use existing data for the study rather than duplicate data collection efforts. The study team will utilize all publically available data on high school characteristics and about GEAR UP programming. The information collected in the student rosters and in the student survey is not available elsewhere.

Efforts to Minimize Burden in Small Businesses

No small businesses will be involved as respondents. The primary small entities for this study are GEAR UP high schools. The study team will minimize burden by training data collection staff to make their contacts with GEAR UP site coordinators as straightforward and concise as possible. Further, the study team will administer the student survey, minimizing the burden placed on site coordinators. All notification mailings, conversations, and presentations are designed to be clear, brief, and informative.

Consequences of Not Collecting the Information

The consequences of not collecting specific data are described below:

Without collecting the student rosters, the study team could not identify the survey sample or define treatment and control groups, rendering it impossible to conduct the study.
Without collecting data through the student baseline survey, the study team would not be able to identify which GEAR UP students intend to matriculate in college, and, consequently, which students are eligible to receive the messages. In addition, not collecting the student baseline survey data would limit the study’s ability to make conclusions about the intervention impact on matriculation and persistence.
Without collecting the NSC and FSA data, the study team could not measure the key outcomes and complete the impact analyses.

Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6

There are no special circumstances concerning the collection of information in this study.

Consultation Outside the Agency
1. Federal Registrar Announcement

A 60-day notice to solicit public comments was published in the Federal Register on March 13, 2105 (vol. 80, no. 49, p. 13360). One public comment that addressed this data collection was received. IES responded to this comment by adding additional detail about the intervention.

Consultations Outside the Agency

Over the course of the study, the study team will assemble a Technical Working Group (in consultation with ED) composed of consultants with various types of expertise in the areas relevant to this study. The Technical Working Group will convene in the spring of 2015 to discuss the study design, instrumentation, and data collection plans.

Unresolved Issues

There are no unresolved issues.

Payments or Gifts to Respondents

There will be no payments or gifts to respondents.

Assurance of Confidentiality

The study team will conduct all activities in accordance with all relevant regulations and requirements. These include the Education Sciences Institute Reform Act of 2002, Title I, Part E, Section 183, that requires “[all] collection, maintenance, use, and wide dissemination of data by the Institute … to conform with the requirements of section 552 of Title 5, United States Code, the confidentiality standards of subsections (c) of this section, and sections 444 and 445 of the General Education Provisions Act (20 U.S.C. 1232 g, 1232h).” These citations refer to the Privacy Act, the Family Education Rights and Privacy Act, and the Protection of Pupil Rights Amendment.

In addition, all data collected for the study shall remain confidential in accordance with Section 552a of Title 5, United States Code, the confidentiality standards subsection (c) and sections 444 and 445 of the General Educations Provision Act. Subsection (c) of Section 183, referenced above, requires the director of IES to “develop and enforce standards designed to protect the confidentiality of persons in the collection, reporting, and publication of data.” The study will also adhere to requirements of subsection (d) of Section 183 prohibiting disclosure of individually identifiable information as well as making the publishing or inappropriate communication of individually identifiable information by employees or staff a felony.

In addition, the following verbatim language will appear on all letters, fact sheets, and other study materials:

Per the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific program, district, or individual. Any willful disclosure of such information for nonstatistical purposes, except as required by law, is a class E felony.

Data will be presented in aggregate statistical form only. All study staff involved in collecting, reviewing, or analyzing individual-level data will be knowledgeable about data security procedures and will sign nondisclosure agreements (see Appendix F). Respondents will be assured that all information identifying them or their school will be kept private to the extent allowed by law. The confidentiality procedures adopted for this study during all rounds of data collection, data processing, and analysis consist of the following:

All paper files will be converted to an electronic format and the paper files will be shredded immediately after they have been converted.
Electronic data files with sensitive data will be removed from computers and working servers in a manner that ensures that the information cannot be recovered.
At end of contract with the ED, the Abt study team will prepare a restricted use data file of de-identified data that will be submitted to ED. The Abt study team will destroy all student identifiers but will retain de-identified student data for three years.
All electronic copies of de-identified student data maintained by the Abt study team will be destroyed within three years of the final contract payment from ED, unless otherwise directed by ED at the request of the Abt study team or by ED’s request.

Questions of a Sensitive Nature

There are no questions of a sensitive nature included in the information requested.

Estimate of Response Burden

The total annual respondent burden for the data collection effort covered by this clearance request is 1,387 hours. Exhibit A-5 presents time estimates of respondent burden for the data collection activities requested for approval in this submission. The burden estimates are based on the following assumptions:

One site coordinator is expected at each of the anticipated 80 high schools. It is assumed that collecting and submitting the GEAR UP student rosters in each school will take approximately two hours.
The salary estimate for School and Career Counselors is based on an average hourly wage of $25.77 in 2012.⁵
An average of 200 students in the 80 schools. It is estimated that the survey will take 15 minutes to complete.

Exhibit A-5. Estimate of Respondent Burden

Informant/Data Collection Activity	# of Respondents	# of Responses	Hours/ Response	Total Burden Hours/ respondent	Total Burden Hours	Cost per Respondent (hourly wage)	Total Costs per Respondent	Total Costs
GEAR UP Site Coordinators
Collect and Submit Student Rosters	80	1	2.0	2.0	160	$25.77	$51.54	$4,123.20
Annual	27				53.3			$1,374.4
Students
Student Baseline Data Collection	16,000	1	0.25	0.25	4,000	NA	NA	NA
Annual	5,333				1333.3			NA
Total					4,160			$4,123.20
Annual Totals	5,360				1386.6			$1,374.4

Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record-Keepers

There are no annualized capital/startup or ongoing operation and maintenance costs involved in random assignment or the collection of student rosters.

Estimates of Costs to the Federal Government

The estimated cost to the federal government of the data collection activities for the study as described above is $1,769,371. The data collection activities will be carried out from fall 2015 to fall 2018. Thus, the average annual cost to the federal government is $442,342.

Changes in Burden

This is a request for a new collection of information.

Plans for Analysis, Publication and Schedule
1. Analysis Plans

Impact and descriptive analyses will be conducted to answer the study research questions, as described below. Part B of this Supporting Statement provides additional methodological detail.

To what extent do the messages—above and beyond the services GEAR UP may already provide—improve student outcomes? Is the effect of milestone messaging different with the addition of growth mindset messages?

Given that participating schools are volunteering to participate in the study (rather than being selected at random) and inferences will be focused on students in these or similar schools, the study will use a model with fixed treatment effects to assess the effect of messaging on student outcomes (Schochet, 2008).⁶The study team will control for student demographic characteristics, including gender, race/ethnicity, and first generation college student status. Reflecting the randomization of students within schools, the study team will include a set of dummy variables for schools and a treatment indicator to provide an estimate of the average impact of the messaging demonstration on students across all schools.

The study team will conduct a test of the null hypothesis of no treatment impact for each treatment on each outcome variable.⁷ The study team will interpret a positive and statistically significant impact at the 5-percent level (using a two-tailed test) as compelling scientific evidence that the intervention improves the targeted outcome (e.g., college enrollment, college persistence, FAFSA completion). For continuous outcome variables, the study team will estimate multilevel linear models; for dichotomous outcomes (e.g., college enrollment, FAFSA completion), the study team will estimate multilevel logistic regression models.

Multilevel Logistic Regression

For dichotomous outcomes (e.g., enrollment, persistence, and FAFSA completion), a logistic regression will be used. This model is:

Where is the covariate-adjusted log-odds of the outcome occurring (e.g., enrolled in college, completed FAFSA) versus not occurring (e.g., did not enroll in college, did not complete FAFSA) for students in school j and is a regression coefficient indicating the difference in log-odds of the outcome between the treatment and control group, controlling for all other variables in the model..

To convert estimates produced by logistic regression to an impact measured in percentage point increases in probability of the binary outcome, we calculate the individual level probability of the outcome given treatment implied by individual level covariates and model estimates, we repeat this calculation for individuals assigned to the control group, and we find the mean difference between these two probabilities.

Hierarchical Linear Model

The following model will be used to estimate program impacts on continuous outcome variables. Given that our core outcome measures (i.e., confirmatory contrasts) are all dichotomous (i.e., college enrollment, FAFSA completion, and persistence into the second year of college), this HLM describes our methodology for assessing exploratory contrasts.

Y_ij= ₀ + ₁(T_ij) + ₂(Y^*_ij) + _2+j _j + _2+(J-1)+m _mij+

where Y_ijis the value of the outcome (e.g., amount of Federal financial aid received) for the i^th student in the j^th GEAR UP school; T_ijis 1 if student i is randomized to the treatment condition (the impact of each treatment will be modeled) in schools j and 0 if assigned to the control condition in school j; Y^*_ijis the baseline score for the outcome of interest for student i in school j; Sch_j = 1 if student is in school j, =0 else; _mijare additional covariates representing demographic characteristics of student i at school j (e.g., race/ethnicity, English language learner status); ₀is the conditional mean follow-up score for the outcome of interest for control students; is the treatment effect i.e. the mean difference of the outcome between intervention and control students. The overall treatment estimate ( ₁) will be a precision-weighted estimate, where the weights are inversely proportional to the variance of the treatment effect in each school. is the residual error term assumed to be normally distributed with mean 0 and variance ².

What types of college advising are typically received by GEAR UP students?

Descriptive analyses will be used to provide information on the types of college advising GEAR UP students typically receive. It is important to understand the existing college advising to which the messaging will be added. Further, this collection and analysis has the added benefit of filling a gap in program information about the intensity of and emphasis on college advising. The study team will describe the types and intensity of college advising regularly received by GEAR UP students at the high school level as reported on the student survey. One example of the type of college advising that might be received by students is one-on-one counseling or advising sessions about applying to college; the study team would describe the percent of students attending such sessions. These results will provide another important source of contextual information for interpreting impact findings.

Is there variation in the impact of the messages, and to what extent is the variation associated with student characteristics or features of the GEAR UP project or school?

To examine variation in program impacts by student characteristics or features of the GEAR UP project or school, the study team will include an interaction of the treatment indicator with the subgroup variable of interest in the impact models described above.⁸ The study team will include interaction terms in the model for our core dichotomous outcomes of college enrollment, FAFSA completion, and college persistence (i.e., equation 1 above), allowing the treatment impact to vary by student characteristics, such as knowledge of the college application process, understanding of financial aid options and college costs, or amount of college advising received (e.g., one-on-one counseling or advising sessions about applying to college ) – or by program characteristics such as whether the GEAR UP program was funded under a state grant or a partnerships grant. If significant interaction effects are found, the sample will be broken into subgroups, the treatment effect will be estimated for each subgroup, and the means and variances of the subgroup will be compared.

Timeline and Publication plans
1. Timeline

The study is expected to be conducted across five years. Exhibit A-6 (below) displays the full study timeline. The student baseline survey will be administered in the spring of 2017, the random assignment will take place in the late spring of 2017. NSC and FSA data will be collected in the fall of 2017 and 2018.

Exhibit A-6 Study Timeline

	Timeframe	Activity
Planning	Fall 2014	Refine the study design and develop data collection forms
	Winter 2015	Revise data collection forms; obtain approval from Abt’s Institutional Review Board (IRB); intervention design
	Spring 2015	Submit Office of Management and Budget (OMB) package (for approval by October); refine intervention design
	Summer 2015	Refine intervention design; revise OMB package as needed Recruitment
Recruitment	Fall 2015	Develop recruitment materials; Finalize intervention design
	Winter 2016	Recruit GEAR UP grantees and schools; Setup intervention message delivery system.
	Spring 2016	Ensure schools and districts have necessary approval from their review boards
	Summer 2016	Finalize commitments from grantees and schools
Roster and Baseline Data Collection	Fall 2016	Collect 12th grade rosters from participating schools
	Winter 2016	Obtain student and parent consent
	Spring 2017	Administer baseline surveys to all seniors with consent. Conduct random assignment of students intending to enroll in college.
Intervention Implementation	Summer 2017	Deliver college enrollment reminders.
Intervention Implementation	Fall & Spring 2017	Send college persistence messages through students’ first year of college
Outcomes Data Collection	Fall 2017	Obtain extant data from National Student Clearinghouse (NSC) and Federal Student Aid data center (FSA)
Outcomes Data Collection	Fall 2018	Obtain extant NSC and FSA data
Reporting	Spring 2018	Draft Report #1
	Summer 2018	Final Report #1
	Summer 2019	Draft Persistence Report
	Winter 2020	Final Persistence Report

Publication plans

The evaluation plans call for two reports. The first, published in summer 2018, will be based on data collected through 2017 that addresses college enrollment and FAFSA completion. The second report will be available in early 2020 and will cover additional outcomes including college persistence and FAFSA renewal.

Approval to Not Display Expiration Date

No exemption is requested. The data collection instruments will display the expiration date.

Exceptions to Item 19 of OMB Form 83-1

The submission describing data collection requires no exemptions to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).

References

Aronson, J., Fried, C., & Good, C. (2002). Reducing the effects of stereotype threat on African American college students by shaping theories of intelligence. Journal of Experimental Social Psychology, 38, 113-125.

Aud, S., Hussar, W., Johnson, F., Kena, G., Roth, E., Manning, E., Wang, X., & Zhang, J. (2012). The condition of education 2012 (NCES 2012-045). U.S. Department of Education, National Center for Education Statistics. Washington, DC. Retrieved June 27, 2012 from http://nces.ed.gov/pubsearch.

Bailey, M. J., & Dynarski, S. M. (2011). Inequality in postsecondary education. In G. J. Duncan and R. J. Murnane (Eds.). Whither opportunity? Rising inequality and the uncertain life chances of disadvantaged children. New York, NY: Russell Sage Foundation.

Bowen, W. G., Chingos, M.M., & McPherson, M.S. 2009. Crossing the Finish Line: Completing College at America’s Public Universities. Princeton, NJ: Princeton University Press.

Castleman, B. L. & Page, L. C. (2013). The not-so-lazy days of summer: Experimental interventions to increase college entry among low-income high school students. New Directions for Youth Development, 140, 77-97.

Castleman, B.L., & Page, L.C. (2014). Freshman year financial aid nudges: An experiment to increase FAFSA renewal and college persistence. EdPolicy Works Working Paper. Available http://curry.virginia.edu/uploads/resourceLibrary/29_Freshman_Year_Financial_Aid_Nudges.pdf

Castleman, B.L., Page, L.C., & Schooley, K. (2014). The forgotten summer: Mitigating summer attrition among college-intending low-income high school graduates. The Journal of Policy Analysis and Management. DOI: 10.1002/pam.21743

Darling, R. & Smith, M. (2008). First-generation college students: First-year challenges. In M. S. Hunter, B McCalla-Wriggins, & E. White (Eds) Academic Advising: New Insights for Teaching and Learning in the First Year. National Resource Center for the First-Year Experience and Students in Transition, Columbia SC.

Goyette, K. A. (2008). College for some to college for all: Social background, occupational expectations, and educational expectations over time. Social Science Research, 37(2), 461-484.

Haskins, R. (2008). Education and economic mobility. In J. B. Isaacs, I. V. Sawhill, and R. Haskins (Eds.). Getting ahead or losing ground: Economic mobility in America. Washington, DC: The Brookings Institution. Retrieved June 27, 2012 from http://www.brookings.edu/research/reports/2008/02/economic-mobility-sawhill.

Kao, G., & Tienda, M. (1998). Educational aspirations of minority youth. American Journal of Education, 106(3), 349-384.

Liu, A., Sharkness, J., & Pryor, J. (2008). Findings from the 2007 Administration of Your First College Year (YFCY): National Aggregates. Higher Education Research Institute, University of California, Los Angeles. Available http://www.heri.ucla.edu/PDFS/YFCY_2007_Report05-07-08.pdf

National Center for Education Statistics (2012). Digest of Education Statistics, 2011. Washington, DC: US Department of Education.

National Center for Education Statistics (2013). Common Core of Data (CCD), "Public Elementary/Secondary School Universe Survey," 2011-12. http://nces.ed.gov/programs/digest/d13/tables/dt13_216.80.asp

National Student Clearinghouse (2014a). Report: High School Benchmarks 2014, National College Progression Rates for high schools participating in the National Student Clearinghouse Student Tracker Service. Available http://nscresearchcenter.org/hsbenchmarks2014

National Student Clearinghouse (2014b). Snapshot Report: First-Year Persistence and Retention Rates by Starting Enrollment Intensity: 2009-2012. Available: http://nscresearchcenter.org/snapshotreport-persistenceretention14/

Roderick, M., Nagaoka, J., Coca, V., & Moeller, E. (2009). From High School to the Future: Making Hard Work Pay Off. Chicago: Consortium on Chicago School Research.

Schochet, P. (2008). Statistical Power for Random Assignment Evaluations of Education Programs. Journal of Educational and Behavioral Statistics, 33(1), 62-87.

U.S. Department of Education. (2014). Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) Funding Status. Retrieved on January 13, 2015 from: http://www2.ed.gov/programs/gearup/funding.html

Walton, G. & Cohen, G. (2011). A brief social-belonging intervention improves academic and health outcomes of minority students. Science, 331, 1447-1451.

Yeager, G., Walton, G., & Cohen, G. (2013). Addressing achievement gaps with psychological interventions. Kappa, 94, 62-65.

Yeager, D.S., Johnson, R., Spitzer, B., Trzesniewski, K., Powers, J., & Dweck, C.S. (2014). The far-reaching effects of believing people can change: Implicit theories of personality shape stress, health, and achievement during adolescence. Journal of Personality and Social Psychology. 106(6), Jun 2014, 867-884.

1 An estimate of the number of seniors in each school is 200, given that the average secondary school enrollment in 2011-2012 was 788 students (NCES, 2013).

2 Among 2010 seniors, 82 percent of graduating students planned to attend a postsecondary institution and the percent among parents with no college education was 71 percent (Aud et al. 2012).

3 Only districts and schools that have directory information policies in place prior to July 1, 2016 that allow for student information designated as “directory information” to be shared in compliance with Federal law with the evaluator will be included in the participating GEAR UP high schools. In accordance with FERPA, if any parents or guardians of a student opt out of the disclosure of “directory information,” the school or district will not provide the “directory information” for that student to the evaluator unless the parent or guardian of that student provides consent under 34 CFR 99.30.

4 Students who indicate some intent to attend college will be eligible for the intervention and will comprise the pool of students who are randomly assigned to the treatment and control conditions.

5 Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, 2014-15 Edition, School and Career Counselors, on the Internet at http://www.bls.gov/ooh/community-and-social-service/school-and-career-counselors.htm (January 8, 2015).

6 If we suspect that treatment effects differ by school, we will interact schools with the treatment indicator.

7The study team will also test the null hypothesis of no difference between Treatment 1 (milestone messages) versus Treatment 2 (enhanced messages).

8 This approach requires the assumption that each characteristic is not correlated with the marginal effect of the other model covariates. We will evaluate whether this assumption is tenable during preliminary data exploration.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Allan Porowski
File Modified	0000-00-00
File Created	2021-01-25

Supporting_Statement_Part_A 060515

Supporting_Statement_Part_A 060515.docx

Student Messaging in GEAR UP Demonstration

OMB: 1850-0920

Part A: Supporting Statement for Paperwork Reduction Act Submission

Justification

Introduction

Circumstances Making the Collection of Information Necessary

Purposes and Use of the Information Collection

Data to Be Collected

Use of Information Technology and Burden Reduction

Efforts to Identify Duplication

Efforts to Minimize Burden in Small Businesses

Consequences of Not Collecting the Information

Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6

Consultation Outside the Agency

Federal Registrar Announcement

Consultations Outside the Agency

Unresolved Issues

Payments or Gifts to Respondents

Assurance of Confidentiality

Questions of a Sensitive Nature

Estimate of Response Burden

Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record-Keepers

Estimates of Costs to the Federal Government

Changes in Burden

Plans for Analysis, Publication and Schedule

Analysis Plans

Multilevel Logistic Regression

Hierarchical Linear Model

Timeline and Publication plans

Timeline

Publication plans

Approval to Not Display Expiration Date

Exceptions to Item 19 of OMB Form 83-1

References