1850-NEW Text Ed SS Part A rev 6-14-17

This Office of Management and Budget (OMB) package is a new clearance request for data collection activities to support “Text Ed: A Study of Text Messaging to Improve College Enrollment Rates Among Disadvantaged Adults” (referred to as the “Text Ed Study”). The study provides a unique opportunity to rigorously evaluate whether the use of a promising, low cost messaging strategy can help Education Opportunity Center (EOC) grantees meet the program’s goal of increasing college enrollment among disadvantaged adults. EOCs — representing one of the eight U.S. Department of Education TRIO programs and hosted primarily by postsecondary institutions —provide informational services related to the college-going process and financial aid options.

The Institute of Education Sciences (IES) at the U.S. Department of Education (ED) has contracted with MDRC and Dr. Lindsay Page at the University of Pittsburgh to conduct this evaluation. The study team will recruit up to 20 EOCs to participate in the study. Eligible clients within each EOC study site will be randomly assigned to receive either the services typically provided by the grantee (control), or the text messaging enhancement in addition to the grantee’s typical services (treatment). Random assignment will occur on a rolling basis from spring 2018 through spring 2020 and will include approximately 6,000 EOC clients. The study will rely primarily on administrative data. Among the administrative data obtained for the study will be EOC intake records, which include background data for individuals who contact the EOC for services, are study-eligible, and consent to participate.

PART A. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION

Circumstances making collection of information necessary

Statement of need for a rigorous evaluation

Educational Opportunity Centers (EOCs) generally serve low-income individuals who are 19 years and older who want to enter or continue to pursue postsecondary education.¹ Many individuals served (known as clients) are potential first generation college-goers. An important objective of the program is to counsel clients on financial aid options and to assist in the college and financial aid application process (U.S. Department of Education, 2009). EOCs serve a large number of clients—a minimum of 1,000 per year--who are spread out over a wide geographic catchment area. Services and supports are typically provided through phone conversations and in-person drop-in meetings, resulting in a relatively “light touch” program, averaging $246 per client in 2015.² Automated, personalized text messaging is a low-cost strategy to strengthen the support that EOC staff provide to clients.

A growing body of research already shows that well-crafted text messages can improve outcomes for low-income adults. For example, behavioral interventions have been shown to increase on-time child care subsidy renewal (Mayer et al., 2015), produce greater engagement in welfare-to-work programs (Farrell et al., 2016), increase child support payments (Baird et al., 2016), spur greater participation in anti-poverty programs (Dechausay, Anzelone, & Reardon, 2015), increase take-up of the Earned Income Tax Credit (Bhargava & Manoli, 2015); influence school selection (Hastings & Weinstein, 2008), and improve parental engagement (Mayer, Kalil, Oreopoulos, & Gallegos, 2015). Based on this research, a hypothesis is that messages that include timely, personalized information concerning college enrollment activities and deadlines, tips for completing forms and other procedures, and a link back to the EOC to answer questions and provide further assistance, could serve as a low-cost method for the EOCs to better serve clients in their large catchment areas, help manage the high caseloads of EOC staff, and effectively encourage clients toward college enrollment. Existing research suggests that this kind of behavioral intervention – designed to increase postsecondary enrollment among low-income adults – could be effective and merits rigorous evaluation.

Legislative authorization for this evaluation is found in Section 402(H) of the Higher Education Opportunity Act of 2008 (20 U.S.C. 1070a–18), which authorizes the Secretary to make grants or enter into contracts to rigorously evaluate the effectiveness of strategies and practices supported under TRIO programs.

Study design and research questions

The Text Ed Study will be an individual-level random assignment study taking place at up to 20 EOCs with a total sample size of approximately 6,000 clients. The goal is to have a sample of EOCs that reflects the diversity of EOCs nationally across two key dimensions – geographic region and host institution type – such that the intervention effect estimates from the evaluation will be informative to EOCs from different parts of the country with diverse contexts and at different types of institutions.

Study-eligible clients will be those who (1) are age 18 and over; (2) go through the EOC’s intake process during the study’s rolling random assignment period; (3) have their high school diploma or equivalent at the time of intake and are not currently enrolled in a postsecondary institution; (4) intend to enroll in postsecondary education by the fall of 2020; and (5) are willing and able to communicate in English with their EOC via text message. Eligible clients who agree to participate in the study will be randomly assigned to receive the EOC’s typical services (control group) or to receive the EOC’s typical services plus the text messaging enhancement (treatment group).

Impacts: The study will address the following research questions regarding impacts:

Does providing personalized messages to EOC clients increase Free Application for Federal Student Aid (FAFSA) completion rates and enrollment rates in postsecondary education?
To what extent does the effectiveness of messaging vary across EOC grantees and client subgroups?

Implementation: The study will address the following research questions regarding implementation:

What are EOCs’ and clients’ experiences with the text messaging? For instance, how often do clients receive automated text messages? How often do clients respond to these messages by sending text messages back to their EOC? To what extent do EOC staff provide individualized text message responses to incoming client messages?

Study intake (the period during which the random assignment of clients will occur) will begin in February 2018 and continue through April 2020. FAFSA completion and college enrollment will be tracked through December 2020.

The text messaging intervention

The study’s intervention is an automated, two-way text messaging intervention between EOC clients and staff. The goal of this intervention is to enhance existing EOC services by providing behaviorally informed text messages to EOC clients in their process of exploring, applying, and transitioning to postsecondary education. Messages will be simple and action-oriented to guide and encourage clients through barriers to accessing higher education. A central focus of the text messages will be to keep clients on track with time-sensitive processes while encouraging them through psychological barriers. Other messages will address the logistical barriers faced by EOC clients, such as child care and transportation.

Core components of the intervention

Automated text messages. Under the premise that many EOC clients could benefit from the same information (i.e. FAFSA deadlines, information about resources, and event dates), and that there are known cycles for when clients will need reminders, the intervention will be structured around a series of automatically distributed text messages. Messages will be automatically sent to clients at the appropriate time, based on where clients are in the process of selecting and applying to a postsecondary institution. Some messages, such as those pertaining to FAFSA submission, will be reminders that are relevant to clients at all EOCs. Other messages, such as events that provide information about local postsecondary options and placement testing, will be customized to individual EOCs.

The content of each automated message will fall into one of six domains:

Introduction to the messaging intervention and conclusion. These messages will formally introduce and close out the messaging campaign for each participating student.
College-going processes. These messages will serve to identify and keep clients on track with each discrete step in the college-going process, such as:
- College search and application
- FAFSA / financial aid application and income verification (if required)
- Review of financial aid package(s) and college selection
- College transition processes (e.g., orientation, placement tests, paying tuition bill)
Supporting logistics. These messages will address logistical and life issues that students may need to manage in order to successfully matriculate to college. These issues many include:
- Child / dependent care
- Transportation
- Housing
- Balancing school with other aspects of life
Specific EOC events. These messages will serve to remind EOC clients about specific EOC programming events (e.g., FAFSA completion events, job fairs, campus visits) and to encourage client participation in the events.
Psychosocial barriers to accessing college. These messages will serve to highlight and address various psychosocial barriers to postsecondary attainment that EOC clients may experience. These barriers may include lack of social belonging on the college campus; lack of congruence between sense of self and postsecondary goals; and lack of support for postsecondary goals from friends and family.
General “check-in” messages. Other content will include general “check-in” messages to help establish the EOC as a place of caring and support (Wanless, 2016).

The messaging intervention will be managed via a text messaging partner that will be identified as part of a competitive bid process. The expectation is that clients are sent text messages once every one to two weeks, for an average of three automated messages per month.

Automated Response Messages. A second component of the intervention is the automated response feature. The expectation is that when clients respond to the automatically generated text messages, when appropriate, the system will generate an automated response to provide the client with additional information, feedback and/or encouragement.

Staff Initiated Response Messages. In other cases, if clients respond to the automatically generated text messages and their responses require individualized follow-up, EOC staff will respond to the text messages to provide support, guidance, or next steps. EOC staff will do this using their computer, a tablet, or a cell phone via the text messaging partner’s web-based messaging platform.

Staff Training. The project team will provide training to staff in the form of webinars and phone conversations. This training will include a detailed introduction to the text messaging software system that they will use to engage with clients, an overview of the messaging content itself, and a discussion of best practices for communicating via text.

Purposes and uses of data

Existing administrative data will be used for most of the key data sources. All data will be obtained by MDRC under contract with ED (contract number ED-IES-16-C-0016). The various data sources, purposes of collecting these data, and timing of the data collection are summarized in Table A.1 and also described below:

EOC staff interview protocol. The EOC staff interview will serve as a data source for collecting information about study sites’ business as usual programming and informing the site-level customization of the messages. It includes information on what first interactions look like between clients and EOCs; what the follow-up services look like after the initial meeting; the duration of clients’ engagement with an EOC; and timeframes between when a client first begins engaging with an EOC and when they plan on enrolling in college. EOC staff will also be asked for a list of the postsecondary institutions that their clients attend, which the study team will use to develop one-page resource guides. Staff will also be asked about how they typically help clients overcome stumbling blocks in the college-going and financial aid process so that the content of the messages can complement the typical practices of EOC staff. The interviews will be conducted by the study team in summer/fall 2017. Prior to the interview, the study team will collect as much of the needed information as possible from existing resources (e.g., EOC websites) in order to reduce staff burden and will only ask the EOC staff themselves for information that is not already publicly available. The EOC staff interview protocol is provided in Appendix A.

Program intake forms. Client-level data that the EOCs already collect as part of their intake process will be obtained from EOCs by the study team. These program intake forms contain individual identifiers (i.e. name, date of birth, Social Security Number), background data about participants, and contact information. A list of the study-relevant data that will be obtained from the program intake forms is provided in Appendix B. These data will be used by EOC staff to determine eligibility for the study and by the study team to describe the study sample and obtain outcome data from Federal Student Aid (FSA) and the National Student Clearinghouse (NSC) records. Most EOCs use paper-based intake forms, which will be scanned and sent securely through a secure FTP site, or through pre-paid FedEx envelopes. While client intake and study enrollment will occur on a rolling basis, staff will batch and send program intake forms to the study team on a weekly basis for clients who are eligible for the study and consent to participate. A small number of EOCs use online intake forms, which will be sent securely to the study team on a rolling basis electronically. Intake data will be sent to the study team beginning in February 2018 through April 2020.

EOC Client Baseline Information Form. Eligible clients who consent to participate in the study will also be asked to complete a Baseline Information Form during the program’s intake process. Data from this form will be used to conduct random assignment of clients and to customize and send the text messages to clients in the treatment group. The Baseline Information Forms will be sent electronically (i.e. through a secure online platform) daily (or as quickly as EOC staff are able) to the study team, in order that random assignment can occur as quickly as possible and clients in the treatment group can begin receiving the messages. The forms will be sent by EOC staff to the study team on a rolling basis beginning in February 2018 through April 2020. The EOC Client Baseline Information Form is provided in Appendix C.

Biweekly updates from EOC staff. For each study EOC, the study team will extract the list of treatment clients from the text messaging technology platform, and will provide these to EOC staff. The lists will include client names; cell phone numbers; known information about clients’ current status in the college application, enrollment, and financial aid process; whether or not the client is planning on working and going to school at the same time; whether or not the client has children they care for; the name of the EOC staff member assigned to work with the client; and the date when that information was last updated (time stamped by the technology platform). EOC staff will be asked to review the lists and provide updates to clients’ status via the Text Ed technology platform if they have more recent information than what is presented in the list. A copy of the list of fields that will be provided to EOC staff is provided in Appendix D. These updates from EOC staff will be used to provide the study team with up-to-date client contact information and information on where clients are in the financial aid and college exploration/application process – information that will be important for ensuring that the text messages reach EOC clients and are appropriately customized. These lists will be produced by the study team and shared with EOC staff biweekly (i.e. once every two weeks) from February 2018 through summer 2020.

Text messaging records. Text messaging data will be extracted from the text messaging partner, who will be a subcontractor to the study team. These data are byproducts of operating the texting system and will be used for the purposes of describing implementation and understanding how EOC clients engage with the intervention. The text messaging data for each text message includes: (1) the associated study participant ID; (2) the direction of the message (sent to client / received by client); (3) the message sender; (4) the verbatim message content; and (5) the date and time the message was sent. The text messaging partner will also provide aggregate information, by site, including the number of messages that were sent and received, the number of EOC clients who actively engaged in text communication, and the number of clients who opted out of the messaging service. These data will be extracted in April 2018, July 2018, October 2018, October 2019, and October 2020.

Federal financial aid administrative data. Using the data included on the program’s intake forms, the study team will work with IES and ED’s Office of Federal Student Aid (FSA) so that impacts on financial aid outcomes are computed. MDRC will work with IES to prepare and submit the first data and analysis request by December 2018 and the second by December 2020.

Postsecondary education enrollment data. Semester-level data obtained from the National Student Clearinghouse (NSC) will provide the study with a national picture of the higher education enrollment activities of each of the clients in the study. NSC data include the name of the college and the type of institution that a participant is enrolled in. Data will be requested annually in December 2018, December 2019, and December 2020.

Table A.1: Description, Purpose, and Timing of Data Collection Activities

Data Source	Description of data collection activity	Purpose of data collection	Timing of data collection
Site-level Background Information
Interviews with EOC staff	Interviews with EOC staff (protocol provided in Appendix A) will be conducted to learn about EOC study sites’ business as usual practices (for example, information about what types of topics and activities are covered in the first meeting and what the typical follow-up practices are); and to inform the site-level customization of the intervention (for example, what types of “life circumstance” supports are typically provided).	These site-level data will be used to describe EOCs' business as usual practices and inform the site-level customization of the text messaging intervention.	Summer/Fall 2017
Client Baseline Information (treatment and control groups)
EOC Program Intake Forms	Program intake forms containing information about EOC clients that are collected as part of the standard EOC intake process will be sent to the study team. The data fields that are critical to the study – demographic data, alternate contact information, name, date of birth and Social Security Number -- are listed in Appendix B.	These data will be used to describe the sample, conduct subgroup analyses; obtain outcome data from FSA and NSC; and contact clients in the treatment group in case cell phone numbers become change or become inactive.	On a weekly basis beginning in February 2018 through April 2020
EOC Client Baseline Information Form	A Baseline Information Form will be administered to consenting clients (study participants) during tEOCs client intake. The form (see Appendix C) will collect client-level background and contact information, along with other information that will be used to inform the customization of the text messages.	These data will be used to randomly assign clients to the treatment or control group and send customized text messages to treatment clients.	On a daily basis beginning in February 2018 through April 2020
Updating Client-level Information to Inform the Intervention (treatment group)
Biweekly updates from EOC staff	The study team will electronically provide EOC staff with lists of treatment clients and pre-populated information about those clients (see Appendix D). Staff will be requested to update clients’ information (e.g., cell phone number; current status in the college application, enrollment, and financial aid process), if known, on a biweekly basis through the Text Ed technology platform.	This information will be used to make sure the team has the most up-to-date cell phone number possible, and will help ensure that the text messages are appropriately customized and relevant to clients’ life circumstances and progress in the college-going and financial aid process.	Biweekly beginning in February 2018 through Summer 2020
Implementation Data (treatment group)
Text messaging records from text messaging partner	Electronic data to be collected from the text messaging partner, which will include study participants’ ID and details of the text message (i.e. direction, sender, verbatim message content, and date/time message was sent).	These data will be used to describe implementation.	April 2018, July 2018, October 2018, October 2019, and October 2020
Client Outcomes (treatment and control groups)
Federal financial aid data	Electronic data to be obtained through IES from ED’s Office of Federal Student Aid on study participants’ FAFSA completion and financial aid awards.	These data will be used for measuring the impact of the text messaging intervention on study participants’ financial aid outcomes.	December 2018 and December 2020
National Student Clearinghouse	Electronic data obtained from the National Student Clearinghouse on study participants’ postsecondary enrollment status.	These data will be used for measuring the impact of the text messaging intervention on postsecondary enrollment.	December 2018, December 2019, and December 2020

Use of technology, including to reduce burden

The data collection plan is designed to obtain information as efficiently as possible in order to reduce burden on study participants. At EOC sites where there are electronic intake procedures in place, we will embed our study intake and random assignment procedures within the existing electronic process to reduce burden on EOC sites and clients. This includes providing an online version the Client Baseline Information Form and providing clients with an opportunity to provide their informed consent electronically. EOCs that do not use electronic intake procedures will have the option of scanning the Baseline Information Forms and consent forms and sending them electronically via a secure FTP site, or mailing them directly to MDRC using a pre-paid, self-addressed FedEx envelope. MDRC staff will use these forms to enter the data electronically. In addition, all implementation data used to describe the implementation of the intervention will be collected electronically from the text messaging partner as previously described. EOC staff will also submit the biweekly status updates electronically directly through the Text Ed technology platform. Finally, the two data sources that will be used to address the impact questions related to FAFSA completion and postsecondary enrollment (administrative databases of the National Student Clearinghouse and the Department’s Office of Federal Student Aid) are centralized databases where data will be obtained electronically.

Efforts to avoid duplication

Whenever possible, the study team will use existing data to avoid duplication of effort. While there a few questions (specifically, name, date of birth, and cell phone number) collected on both the program’s intake form and the Baseline Information Form, including those items on the Baseline Information Form is critical to ensuring that the time between when an EOC client enters the study and is randomly assigned to condition is as short as possible so that clients who are assigned to the treatment group can begin receiving the study’s messages right away. As discussed above, the study team will also be collecting site-specific information by first reviewing publicly available data. The study team will only ask EOC staff the questions in Appendix A when the information is not publicly available. Text messages will ask clients about their status in the college application and enrollment process, but not all clients are likely to reply. The biweekly updates from EOC staff (reported via the technology platform) are the only other data source for obtaining updates about clients to ensure that the intervention is appropriately customized. The other data collection activities (text messaging data from text messaging partner, financial aid award and application data from FSA, and postsecondary enrollment data from the National Student Clearinghouse) are already designed to be as efficient as possible and to avoid duplication of efforts by collecting information from reliable and centralized data sources.

Methods to minimize burden on small entities

The primary entities for the study are Education Opportunity Centers, most of which are hosted by postsecondary institutions; some are hosted by non-profit organizations. We will minimize burden for all respondents by requesting only the minimum data required to meet study objectives. For example, our outcome data are based entirely on administrative data collected directly from the Federal Student Aid Office and the National Student Clearinghouse. In addition, burden on EOC grantees will be further minimized by embedding the Baseline Information Form into the program’s existing intake process so that the collection of these data is as streamlined as possible. This will include, for example, creating an online version of the Baseline Information Form if the EOCs have an online intake process. Similarly, when collecting information that will inform the customization of the intervention, the study team will only collect data that are not publicly available and critical to ensuring that the intervention is responsive to the needs of the EOC client population. Finally, the biweekly status update forms are designed so that EOC staff will update the pre-populated information in the technology platform only if there is new information about the client. Also, all fields in the biweekly update form and the Baseline Information Form are designed to obtain structured, rather than open-ended, responses to minimize the response burden.

Consequences of not collecting data

The data collection activities described in this package are necessary to customize the intervention, deliver the intervention, and/or evaluate the intervention. Failing to collect these data would miss an opportunity to inform the field with rigorous evidence on the effectiveness of the use of behavioral messaging to increase postsecondary enrollment among low-income adults interested in college and specifically those in ED’s TRIO programs.

The interviews with EOC Staff are necessary to understand the center’s business as usual practices and to customize the text messaging intervention at the site level, which is an integral part of the theory of how the intervention can best impact EOC client outcomes (Sanders & Kirkman, 2014). These data include information that is specific to the individual EOCs so that the intervention messages can communicate detailed information that is specific to the client and their EOC, such as the exact due dates for colleges where the client applies and reminders about local EOC events. Although obtaining this information will impose some burden on program staff, without it the study will not be able to customize the text messages or have a systematic understanding of how EOC staff support individual clients in practice.

The program intake forms contain individual-level identifiers, demographic information, and alternate contact information. The individual-level identifiers (name, date of birth, and Social Security Number) are necessary for obtaining outcome data from the Federal Student Aid Office and the National Student Clearinghouse. Without the demographic information collected by EOCs on the intake forms, the study team’s ability to systematically describe the study sample, understand how the sample compares to the overall EOC population, or conduct subgroup analyses would be very limited. In addition, not having alternate contact information would limit the study team’s ability to follow-up with clients if the cell phone number provided on the Baseline Information Form becomes inactive.

Without information from the EOC Client Baseline Information Form, the client-level customization and delivery of the intervention would not be possible. Collecting these data is critical to conducting individual-level random assignment and launching a customized text messaging intervention in a timely way.

The biweekly updates from EOC staff are also important to customizing the intervention for individual clients. These data will be used to provide the study team with up-to-date information on where clients are in the financial aid and college exploration/application process, and other life circumstances that the intervention is providing customized messages on. The biweekly updates are also important for providing a mechanism for updating cell phone numbers if/when they change. Not collecting these data would cause the study team to miss changes in clients’ statuses that are important to ensuring that the messages are delivered, and remain relevant and timely for individual clients.

Text messaging records are needed to describe implementation of the text messaging intervention. Without these data the study will not be able to determine whether the intervention was implemented with fidelity and interpret the impact estimates.

The financial aid data (collected from the Department’s Office of Federal Student Aid via IES) and postsecondary enrollment data (collected from the National Student Clearinghouse) are needed to evaluate the impact of behavioral messaging on financial aid outcomes and postsecondary enrollment rates for low-income adults. As discussed in item 1, little is known about how well-crafted behavioral messaging strategies improve education and financial aid outcomes for older prospective students. Without these data the study would not be able to determine the impact of the intervention on client outcomes.

Special circumstances

None apply to the data collection activities described in this package.

Federal register comments and persons consulted outside the agency

The 60-day notice to solicit public comments was published on 3/21/17, VOL. 82, page 14507. No public comments requiring response were received.

This project will assemble a Technical working Group (TWG) twice during the study period to obtain external perspectives on the intervention design, evaluation design, data collection activities, and data analyses. The first TWG meeting was convened on May 1, 2017. A second TWG meeting will be convened in summer 2021. In addition to these two formal meetings, TWG members will also be consulted periodically or asked to review elements of the intervention design, research design, analysis, and other materials. The following individuals are members of the TWG:

Name	Title and Affiliation
Michael Collins	Vice President, Building Educational Pathways at Jobs for the Future
Lucy Jones	TRIO Consultant
Cait Lamberton	Ben L. Fryrear Chair of Marketing and Associate Professor of Marketing; University of Pittsburgh School of Business
Rebecca Maynard	University Trustee Professor of Education Policy and Social Policy, Graduate School of Education, University of Pennsylvania
Nancy Nelson	Educational Opportunity Center Director at Indian Hills Community College
Awilda Rodriguez	Assistant Professor in the Center for the Study of Higher and Postsecondary Education, University of Michigan
Sally Sadoff	Assistant Professor of Economics and Strategic Management, Raday School of Management, University of California, San Diego
Beth Tipton	Assistant Professor of Applied Statistics in the Human Development Department at Teachers College, Columbia University

Payment or gifts to respondents

We do not plan to use incentives on this project.

Assurances of confidentiality provided to respondents

All data collection activities will be conducted in full compliance with the Department of Education regulations to maintain the confidentiality of data obtained on private persons and to protect the rights and welfare of human research subjects as contained in the Department of Education regulations. These activities will also be conducted in compliance with other Federal regulations including the Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the Family Educational Rights and Privacy Act of 1974, 20 USC 1232g, 34 CFR Part 99; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal Common Rule for the Protection of Human Subjects in Research (34 CFR 97).

Study participant intake and consent with EOC clients will be managed by EOC staff members at participating EOC study sites. An explicit statement describing the project, the data collection, and confidentiality will be provided to study participants. EOC staff will be trained on these activities via webinars. These trainings will be delivered prior to the study launch date and will include step-by-step written instructions about: data transfers between the EOCs, MDRC, and the selected text messaging partner. Eligible clients will be informed of the evaluation, data collection, and given the opportunity to consent to join. Clients will also have the opportunity to ask questions about the study. Consent will be documented on paper with a client signature or digitally with a consent check-box. The consent form is provided in Appendix E.

Confidentiality assurances during data collection:

All data collection employees at MDRC, and any data collection sub-contractors thereof, sign confidentiality agreements that emphasize the importance of confidentiality and specify employees’ obligations to maintain it.
Personally identifiable information (PII) is maintained on separate forms and files, which are linked only by sample identification numbers.
Access to a crosswalk file linking sample identification numbers to personally identifiable information and contact information is limited to a small number of individuals who have a need to know this information or manage the data.
Access to hard copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.
Access to electronic files is protected by secure usernames and passwords, which are only available to approved users. Access to identifying information for sample members is limited to those who have direct responsibility for providing and maintaining sample crosswalk and contact information, and managing the data.
The plan for maintaining confidentiality includes staff training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses. It also includes built-in safeguards concerning status monitoring and receipt control systems.

Confidentiality assurance during analysis: The data collected for this study will be used only for broadly descriptive and statistical purposes. The report prepared for the study will summarize findings across the sample and will not associate responses with a specific EOC grantee site or individual. In no instances will the study team provide information that identifies specific EOC sites, EOC staff, or EOC clients to anyone outside the study team, except as required by law.

Data storage: MDRC and subcontractors will store all data in compliance with its federally approved data security plan with state-of-the-art data security protections that have passed reviews by the Social Security Administration and other federal agencies. All quantitative data will be stored on a secure IES server. MDRC access will be restricted to specific project staff identified by the data manager. Data that is used for analysis is stored with research ID numbers rather than actual client or staff identification information. All information from these research activities will be reported in the aggregate, and no disaggregated information will be shared with EOCs or the general public. In addition, a restricted use file with project data will be created, but it will not contain any actual individual identifiers.

Method of data destruction: All data containing individually identifiable records will be destroyed by an appropriate fail-safe method, including physical destruction of the media itself or deletion of the contents on our servers. After the study is completed, the study team will create a restricted use file of the data collected and submit that file to IES. This file will have been stripped of all individual client and EOC site identifiers.

Justification of sensitive questions

Not applicable.

Estimates of annualized burden hours and costs

Table A.2: Estimates of Total Burden Hours and Costs

Data Collection Activity	Number of respondents	Frequency of response	Hour burden	Cost to respondents
EOC clients (individuals) completing the consent form	6,000	Once	500	$8665
EOC clients (individuals) completing the Baseline Information Form	6,000	Once	500	$8,665
EOC staff sending client-level study data (Baseline Information Forms, consent forms, program intake forms)	20	Rolling or in batches over a period of 26 months (2.17 years)	333	$10,859.13
EOC staff interview	20-40	Once	50	$1,630.50
Biweekly updates from EOC staff	20	Biweekly over a period of 30 months (2.5 years)	400	$13,044

Table A.2 provides an estimate of the time burden for the data collection activities involving EOC staff and clients. These estimates are based on the instruments included in the appendices and our experience collecting intake and interview data from individuals. A summary of the burden hour calculations by respondent/data request is provided below:

EOC Client Consent Form. The study sample target is 6,000 individual EOC clients. We estimate that reviewing and completing the consent form will take approximately 5 minutes, for a total burden of 500 hours and an annual burden of 167 hours. The cost to respondents was calculated using mean hourly earnings provided by the Bureau of Labor Statistics National Compensation Survey³: $17.33 x 500 burden hours = $8,665, at an annualized cost to respondents of $2,888.

EOC Client Baseline Information Form. The study sample target is 6,000 individual EOC clients. We estimate that completing the Baseline Information Form will take approximately 5 minutes, for a total burden of 500 hours and an annual burden of 167 hours. The cost to respondents was calculated using mean hourly earnings provided by the Bureau of Labor Statistics National Compensation Survey⁴: $17.33 x 500 hour burden = $8,665, at an annualized cost to respondents of $2,888.

EOC Staffing Sending Client Forms to Study Team. We estimate that for each site, EOC staff will spend an average of 20 minutes weekly to send the study-related forms (Baseline Information Forms, consent forms, program intake forms) over the course of a little over two years (26 months). The hour burden was calculated based on the assumption that approximately 20 EOC staff members (one per site) would spend 20 minutes of their time sending data electronically or by mail every week for 26 months (assume 50 rounds⁵) : 20 staff members X 20 minutes x 50 rounds = 333 hours of total burden, and an annualized burden of 111 hours. The cost to respondents was calculated using mean hourly earnings provided by the Bureau of Labor Statistics National Compensation Survey: $32.61 x 333 hours = $10,859.13, with an annualized cost to respondents of $3,619.71.

EOC staff interview. For each EOC in the study (goal is 20), one to two staff members (20-40 total)⁶ will be interviewed about population services and needs; modes of client engagement; and the EOC’s calendar of events to help inform the customization of the intervention. Each interview will take on average about 75 minutes and will take place once prior to the implementation of the intervention. The number of respondents assumes the highest numbers in the range: 40 EOC staff. The annual hour burden was calculated based on the assumption that all 40 EOC staff would participate in a 75 minute interview: 40 EOC staff x 75 minutes = 50 hours, and an annualized burden of 17 hours. The cost to respondents was calculated using mean hourly earnings provided by the Bureau of Labor Statistics National Compensation Survey: $32.61 x 50 hour burden = $1,630.50 and an annualized cost to respondents of $543.50.

Biweekly updates from staff. For each EOC study site, one staff member per site (20 total) will review a list of treatment clients and associated information such as clients’ current status in the college application and enrollment process on a biweekly basis. EOC staff will update the information as needed via the Text Ed technology platform. We estimate that the process of reviewing and updating the information on the list will take, on average, 20 minutes biweekly: 20 minutes x 20 sites = 400 minutes approximately 24 times per year for 2.5 years, equaling 400 hours and an annualized burden of 133 hours. The cost to respondents was calculated using mean hourly earnings provided by the Bureau of Labor Statistics National Compensation Survey: $32.61 x 400 hours = $13,044 and an annualized cost to respondents of $4,348.

The total number of respondents is 6,040 (6,000 individual EOC clients, and 40 EOC staff members maximum) with a total burden estimate of 1,783 burden hours, and a total cost of $42,863.63. The annual number of respondents is 2,013 and 594.33 annual burden hours, at an annual cost of $14,287.87.

Estimates of other total annual cost burden to respondents

There are no additional respondent costs associated with this data collection beyond the burden estimated in item A12.

Annualized cost to the federal government

The total cost for the base period of the study is $1,500,000 for two years (2016-2018), with an annualized cost of $750,000. There is also an optional four-year period (2018-2022) with a total cost of $2,704,605, and an annualized cost of $676,151.25.

Explanation for program changes or adjustments

This submission to OMB is a new information collection request for approval.

Time, schedule, publication and analysis plan

Schedule & Publication Plan

The study team will prepare two reports. The first will describe data analyses and findings in response to the key implementation and impact evaluation questions. The report will be consistent with IES peer review standards and will be approximately 50 pages long, excluding the executive summary and technical appendices. The second report will be a short report (approximately 20 pages) that describes the intervention, including the steps to implement it in the field. The purpose of this report is to provide information to EOC grantees and others who may be interested in adopting the intervention should the study reveal evidence of effectiveness. Both reports are expected to be released in 2022. The reports will follow guidance provided by the IES Style Guide (Institute for Education Sciences, 2005).

Analysis Plan

Implementation: To investigate the implementation of this text messaging effort, we will rely on analysis of the text message data itself. From these data, we will be able to generate descriptive evidence to answer the following question:

What are EOCs’ and clients’ experiences with the text messaging? For instance, how often do clients receive automated text messages? How often do clients respond to these messages by sending text messages back to their EOC? To what extent do EOC staff provide individualized text message responses to incoming client messages?

To answer this question, the study team will calculate (1) the average number of messages each client receives for each of the six primary message domains described in the intervention description⁷; (2) the average number of messages that clients send in response to the automated messages for each domain; and (3) the average number of follow-up messages EOC staff members send as a follow-up to client responses.

Impacts: Our main focus is to estimate the cross-EOC mean effect ( ) of the information strategy on FAFSA completion and college enrollment. To estimate for this multisite trial, we will use the following two-level model discussed by Raudenbush (2015) with fixed EOC-specific intercepts ( ) for each site, random EOC-specific impact coefficients ( ), and fixed coefficients ( ) for individual-level baseline covariates ( ):

Level One: Individuals

where: (1)

Level Two: EOCs

(2)

where (3)

Here, is the observed value of the outcome for individual from EOC ; is 1 if individual from EOC was randomized to treatment and 0 otherwise; is the value of baseline covariate for individual from EOC ; is the coefficient for covariate ; are around 20 fixed EOC-specific intercepts; is the population mean treatment effect for EOC ; is the cross-site mean treatment effect for the population of EOCs; is a random error that varies independently and identically across individuals within EOC and experimental conditions, with a mean of zero and a variance of ; and is a random error that varies independently and identically across EOCs in the population with a mean of 0 and a variance of ( ).

This model allows to be estimated. is the standard deviation of the EOC-level treatment effect distribution, and , quantifies how much the effectiveness of the messaging strategy varies across EOCs, information relevant to the generalizability of findings. If is large, then the effectiveness of the information strategy varies a lot across EOCs. This would imply that, while provides a useful summary of the overall average effectiveness of the information strategy, it masks that the information strategy is more/less effective at particular EOCs. In contrast, if is small, the effectiveness of the information strategy is fairly consistent across EOCs.

Approval to not display OMB expiration date

All data collection instruments will include the OMB expiration date.

Explanation of exceptions to the Paperwork Reduction Act

No exceptions are being requested.

References

Baird, P., Cullinan, D., Landers. P., and Reardon, L. (2016). Nudges for child support: Applying behavioral insights to increase collections. OPRE Report 2016-01. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Bhargava, S and Dayanand M. (2015). "Psychological Frictions and the Incomplete Take-Up of Social Benefits: Evidence from an IRS Field Experiment." American Economic Review, 105(11): 3489-3529.

Dechausay, N., Anzelone, C., and Reardon, L. (2015). The power of prompts: Using behavioral insights to encourage people to participate. OPRE Report 2015-75. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Farrell, M., Smith, J., Reardon, L., and Obara, E. (2016). Framing the message: Using behavioral economics to engage TANF recipients. OPRE Report 2016-02. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Hastings, J., and Weinstein, J. (2008). “Information, School Choice and Academic Achievement: Evidence from Two Experiments.” The Quarterly Journal of Economics 123(4), 1373-1414.

Higher Education Opportunity Act of 2011, 20 U.S.C. §1070a–11 et seq. (2008)

Institute of Education Sciences. (2005). IES Style Guide. U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved November 30, 2016, from https://nces.ed.gov/statprog/styleguide/pdf/styleguide.pdf.

Mayer, A., Calmeyer, E., Cullinan, D., and Patterson, K. (2015). Engaging providers and clients: Using behavioral economics to increase on-time child care subsidy renewals. Washington, DC: Office of Policy Research, Administration for Children and Families, U.S. Department of Health and Human Services.

Mayer, S. E., Kalil, A., Oreopoulos, P. and Gallegos, S. (2015). Using behavioral insights to increase parental engagement: The parents and children together (PACT) intervention. NBER Working Paper No. 21602. Retrieved November 30, 2016 from http://www.nber.org/papers/w201602.

Raudenbush, S. W., (2015). Estimation of Means and Covariance Components in Multisite Randomized Trails. Learning About and from Variation in Program Impacts (MDRC). New York, New York.

Sanders, M. and Kirkman, E. (2014). “I’ve booked you a place. Good luck: a field experiment applying behavioural science to improve attendance at high-impact recruitment events”. Center for Market and Public Organization, University of Bristol. Working Paper No. 14/334. Accessed on December 13, 2016 from http://www.bristol.ac.uk/media-library/sites/cmpo/documents/WebVersion.pdf.

U.S. Department of Education, Office of Postsecondary Education, A Report on the Educational Opportunity Centers Program: 2007–08, With Select Comparative Data, 2002–07, Washington, D.C., 2009.

Wanless, S. B. (2016). The role of psychological safety in human development. Research in Human Development, 13(1), 6-14.

1 EOCs also may provide services to individuals who are under 19 years provided that there is not a Talent Search program in the area served by the EOC.

2 https://www2.ed.gov/programs/trioeoc/funding.html

3 https://www.bls.gov/ncs/ocs/sp/nctb1344.pdf

4 https://www.bls.gov/ncs/ocs/sp/nctb1344.pdf

5 Based on 52 weeks in a year, subtracting two weeks for holidays.

6 A range is provided because some EOCs may operate at multiple sites and have variations in programs, events, and client needs by site. In those cases, the study team will interview one EOC staff member per site.

7 (1) Introduction to the messaging intervention and conclusion; (2) college-going processes; (3) supporting logistics; (4) specific EOC events; (5) psychosocial barriers to accessing college; and (6) general “check-in” messages.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Jacklyn Willard
File Modified	0000-00-00
File Created	2021-01-22