1850-NEW AMP SS Part A OMB_Package_Attendance_Messaging_ptA_30day

1850-NEW AMP SS Part A OMB_Package_Attendance_Messaging_ptA_30day.docx

Impact Evaluation of Parent Messaging Strategies on Student Attendance

OMB: 1850-0940

Document [docx]
Download: docx | pdf

Shape1


June 2017


Impact Evaluation of Parent Messaging Strategies on Student Attendance

OMB Clearance Request: Supporting Statements, Part A


Data Collection





Impact Evaluation of Parent Messaging Strategies on Student Attendance

OMB Clearance Request: Supporting Statements Part A

Data Collection

June 2017




1000 Thomas Jefferson Street, NW
Washington, DC 20007-3835
202.403.5000

www.air.org

Copyright © 2017 American Institutes for Research. All rights reserved.




Contents

Page


Exhibits

Page


Introduction

Attendance in school is critically important for students’ short- and long-term academic and lifelong success. A number of studies show a link between attendance and outcomes such as academic performance, high school graduation, drug and alcohol use, and crime.1 While some of the research on the consequences of poor attendance focuses on middle and high school students, chronic absence in early grades is also linked with negative outcomes, including lower reading and mathematics achievement, and higher absenteeism in middle and high school.2 Analyses in multiple states and school districts indicate that students who are chronically absent in the early grades are significantly less likely than their peers to read on grade level by grade 3, which, in turn, puts them at greater risk of dropping out of high school.3

According to recently released data by the U.S. Department of Education’s Office for Civil Rights, 11 percent of elementary grade students, or 3.5 million children, were chronically absent―defined as missing 15 days or more―during the 2013-14 school year. Preventing absenteeism early on may be more effective and less costly than intervening with older students, potentially helping to avoid the consequences of chronic absenteeism over multiple years.

This study will develop and rigorously test an innovative, low-cost, parent-focused text messaging intervention, meant to reduce elementary school absenteeism. The study will use a sequential multiarmed design in which, in the fall semester, families will be randomly assigned to one of two “first-stage” intervention conditions or to a business-as-usual (BAU) control group. In the spring semester, families in the control group will continue with business-as-usual. Families assigned to one of the first-stage interventions will continue to receive messages consistent with their first-stage assignments if their children are not chronically absent in the fall. However, families with children who are chronically absent in the fall, despite first-stage messaging, will be randomly assigned to one of two amplified “second-stage” intervention conditions. The study will examine the relative impact of the first- and second-stage messaging strategies separately and in combination. The study will also document the implementation of the text messaging intervention and its costs.

Description of the Text Messaging Intervention

The text messaging intervention is multiarmed and adaptive in that families that are not “responsive” to the initial messages in the first stage receive amplified messaging strategies in the second stage. The intervention in this study is designed to be adaptive because (1) different text messaging strategies may be more or less effective for different families, and (2) the costs of different strategies for increasing attendance vary; more costly second-stage strategies are saved for those families who are not responsive to less costly first-stage approaches.

In the next section, we first describe the logic model on which the intervention is based, followed by a description of the first- and second-stage messaging strategies.

Logic Model

The following logic model is grounded in the information-motivation-behavioral skills (IMB) model,4 which is regularly applied when developing interventions in the health fields.5 As shown in Exhibit 1, the study uses the IMB model as a basis for identifying malleable factors related to student attendance that could be changed through a text messaging intervention: (1) information deficit (e.g., a lack of adequate information about the consequences of chronic absenteeism or not knowing how many days a child has missed school), (2) lack of personal or social motivation (e.g., not believing school will make a difference in their child’s future), and (3) lack of adequate behavior skills (e.g., not knowing effective strategies to help their children get to school on time, such as making sure to set the alarm). Changes in parents’ or guardians’ beliefs, motivations, and behaviors should improve student attendance and reduce chronic absenteeism.6 Increased time in the classroom due to regular attendance should improve students’ academic (e.g., achievement) and noncognitive (e.g., engagement) outcomes.

Exhibit 1. Logic Model for the Text Messaging Intervention



Description of First- and Second-Stage Messaging Strategies

Text messages will be sent to parents assigned to the study’s intervention conditions during the 2017-18 school year. Parents assigned to the control group, also known as the business-as-usual (BAU) condition, will receive only information the district would typically share related to attendance and will not receive any of the study’s text messages.

As noted above, the text messaging intervention has two stages. The first stage will take place in fall 2017 when parents will be randomly assigned to the BAU condition or to one of two treatment conditions: Basic Informational Messaging – Positive (BIM-pos) or Basic Information Messaging – Negative (BIM-neg). The second stage will take place in spring 2018. During this stage, parents whose children missed more than 10 percent of instructional days during the fall, despite first-stage messages, will be randomly reassigned to one of two “amplified” treatment conditions: School Staff Outreach or Goal Commitment Messaging. Parents whose children missed fewer than 10 percent of instructional days will continue to receive the same kind of messages as in the fall.

Below we describe each treatment arm in more detail. Subsequently we describe the unique research questions that the multiarmed, sequential nature of the intervention allows us to answer.

First-Stage Messaging

The two first-stage messaging conditions (October–December) are: (1) Basic Informational Messaging – Positive (BIM-pos) and (2) Basic Informational Messaging – Negative (BIM-neg).

  1. BIM-pos includes preventive weekly messages, same-day notifications to parents when their child is absent, and positive feedback messages when their child returns to school.

  • Weekly preventive messaging. All parents in the BIM-pos condition will receive a preset automated message on Sunday nights on the importance and positive benefits of consistent attendance, and tips for avoiding common reasons for absences with links to additional resources on external websites.

  • Same-day notification when a child is absent. When a child is recorded as absent by the school in the student information system (SIS; that is, a school district-maintained database), a text message will be sent to his or her parents notifying them that their child is absent. These same-day notifications are personalized with the child’s name and include the number of days absent so far and a positively stated benefit of regular attendance. If a child misses four consecutive days of school, parents will receive a text informing them that the same-day notifications will end and asking them to contact the child’s school about their child’s absence, if they have not done so already.

  • Same-day positive feedback messaging when the child returns to school. If the child has been marked as absent the day before and is marked present on the current day, a message will be sent to parents to acknowledge their child’s attendance, with a positively stated benefit of regular attendance.

  1. BIM-neg also includes weekly preventive messages, same-day notifications to parents when their child is absent, and same-day feedback when the child returns to school.

  • Weekly preventive messaging. All parents in the BIM-neg condition will receive a preset automated message on Sunday nights on the importance of regular attendance and consequences of chronic absenteeism (negatively framed), tips for avoiding common reasons for absences, and an option to receive more information (through a link to a web-page).



  • Same-day notification when a child is absent. As with BIM-pos, these messages are personalized with the child’s name and include the number of days absent so far, but instead of a positively stated benefit of regular attendance, the message will include a briefly stated negative consequence of missing school.


  • Same-day positive feedback messaging when the child returns to school. As with BIM-pos, but with a negatively stated consequence of missing school.

Second-Stage Messaging

The two second-stage messaging conditions (January–June) are (1) School Staff Outreach and (2) Goal Commitment Messaging. Families will receive a second-stage strategy if they are determined to be “nonresponsive” to their first-stage intervention (BIM-pos or BIM-neg). The spring messaging conditions are both intended to offer more tailored information and resources to parents and to increase parent motivation to make sure their child attends school regularly.

  1. School Staff Outreach adds to BIM-pos or BIM-neg school staff-initiated, two-way communication through text messaging, in which school staff conduct additional outreach to parents whose children have ongoing attendance issues. Parents in this group also continue to receive the messaging they received in the fall. In late December 2017 (following completion of the first stage), the study team will alert the school staff member tasked with implementing the systematic outreach, during the month of January, to contact all parents who have been assigned to the Outreach strategy. If parents are not responsive to the school staff-initiated text messages in January, the school staff will reach out to parents using other modes of communication (e.g., phone, e-mail, backpack mail, postal mail). The main goal of the School Staff Outreach strategy is to establish two-way communication that allows the school staff person to identify possible reasons for absences and to suggest resources or supports that may help. The School Staff Outreach strategy aims to (1) increase parents’ motivation to make sure their child attends school each day through person-to-person contact with the school, and (2) improve their behavioral skills related to actually getting their child to school each day, via tailored support and resources that address their specific needs and circumstances.

  2. Goal Commitment Messaging adds to BIM-pos or BIM-neg options for obtaining more tailored tips and resources in the weekly preventive messages, and parent goal-setting messages. Parents in this group will receive a text message at the beginning of the week asking them to commit to goal of perfect attendance for that week. This message is interactive, requesting parents to reply whether they will commit (reply YES). At the end of the month, parents will receive a message reporting back the number of weeks that month their child had perfect attendance. When the Goal Commitment Messaging is added to BIM-pos, the feedback is positively framed (e.g. your child met the perfect attendance goal during 3 weeks this month!). When added to BIM-neg, the feedback is negatively framed (e.g. your child missed the perfect attendance goal during 1 week this month). The Goal Commitment Messaging strategy aims to (1) increase parents’ motivation to ensure their child’s attendance though weekly goal-setting with monthly feedback, and (2) improve their behavioral skills related to their child’s attendance with the addition of options to receive more tips or resources on topics of relevance to them.

The spring messaging strategies are costlier than the fall messaging strategies. The School Staff Outreach condition adds direct outreach from school staff to parents and thus takes more staff time and effort than automated text messages. The Goal Commitment Messaging, though automated, adds two additional components that ask for more parent time and effort: parent goal setting and options to obtain tailored tips and resources. In addition to cost, the fall and spring messaging strategies differ in emphasis with respect to the information-motivation-behavioral skills model. The fall messaging strategies are primarily informational, while both spring messaging strategies, in different ways as described, aim to increase parent motivation and behavioral skills related to their elementary grade children’s attendance. For these reasons, the School Staff Outreach and Goal Commitment Messaging strategies are reserved for the parents of children for whom first-stage messaging is not successful.

The study design, described in the next section, will allow us to examine the relative effects of the intervention arms and stages for all students and particularly for parents of students at-risk for chronic absence.

Study Description

Evaluation Design

The evaluation uses a sequential multiple assignment randomized trial (SMART) design7 (see Exhibit 2). In late September 2017, we will randomly assign families within schools to one of the two first-stage messaging conditions, Basic Informational Messaging – Positive (BIM-pos) (BIM-pos) or Basic Information Messaging – Negative (BIM-neg), or to the business-as-usual (BAU) control condition. Between October and mid-December, families will receive messages consistent with their first-stage intervention condition. Parents assigned to the intervention conditions whose children were not chronically absent between October and mid-December (i.e., did not miss 10 percent or more of the school days during this period) will be considered “responsive,” and will continue with their first-stage messaging from January 2018 through the end of the school year. Parents with children who were chronically absent, despite the first-stage messaging, will be considered “nonresponsive.” The nonresponsive families will be rerandomized to one of the two “amplified” second-stage messaging conditions, School Staff Outreach or Goal Commitment Messaging, in January. Parents in the School Staff Outreach or Goal Commitment Messaging conditions will receive messages consistent with their treatment group assignment from January through the end of the year.

Exhibit 2. SMART Design to Evaluate the Adaptive Text Messaging Intervention

Research Questions

The study design is guided by four research questions (RQs) on the impact of the text messaging strategies and by two research questions related to implementation. RQ1 focuses on comparing the two first-stage messaging strategies for informing parents about the importance of regular attendance (BIM-pos or BIM-neg) to each other, and then to the BAU control group. RQ2 examines the relative effect of the two second-stage messaging conditions: School Staff Outreach and Goal Commitment Messaging. RQ3 focuses on the effects of the four different combinations of first- and second-stage messaging strategies (i.e., the four embedded adaptive interventions). RQ4 examines whether particular combinations of the first- and second-stage messaging strategies are more effective for students with different characteristics. RQ5 focuses on implementation fidelity, and RQ6 focuses on the costs of implementation and the cost effectiveness of the adaptive interventions.

RQ1. What is the average impact on student attendance of informing parents through two messaging strategies (1) preventive messages and same-day attendance notifications using positive framing (BIM-pos) or (2) preventive messages and same-day notifications using negative framing (BIM-neg), as compared to BAU and to each other?

RQ2. For those parents who do not respond to the initial first-stage messaging (BIM-pos or BIM-neg), can we improve attendance with school staff-initiated parent outreach and interpersonal support or by providing goal commitment messaging with additional tips and resources?

RQ3. Do the four combinations of first- and second-stage messaging strategies (i.e., the four adaptive interventions) have effects on end-of-year attendance and achievement when compared to each other and to BAU?

RQ4. Do the first- and second-stage messaging strategies work better for some families than for others?

RQ5. How is the text messaging intervention implemented?

RQ6. What are the costs of implementing the intervention and what is its cost effectiveness?

Sample and Setting

The evaluation will take place in 4 school districts, each with at least 10 participating elementary schools that are considered low performing in terms of student achievement, and have high levels of absenteeism (e.g., 20 percent or more of students missing 10 percent or more of instructional days per year). We anticipate that approximately 26,000 families and 30,000 students in 60 schools will participate in the study.

Decisions regarding the number of schools needed for the study were based on desired minimum detectable effect sizes (MDES) for each outcome. The study team and the Institute of Education Sciences set the target MDES based on reasonable and policy-relevant expected impacts on various types of outcomes for this kind of intervention. We then calculated the required sample size for the target MDES, using assumptions based on a review of existing literature.8

Data Collection Strategy

Exhibit 3 provides an overview of data collection activities for the study. Data sources that present a burden to study participants, parents, or school and district staff are indicated by an asterisk. Justification for the burden is provided in the paragraphs below the table.

Exhibit 3. Overview of Data Collection

Data Collection Activity

# of Respondents

Timeline

Fall

2017

Spring 2018

Summer 2018

District Records Data Collection*

4

X

X

X

Text Message Provider Back-End Data

1

X

X

X

Parent Survey*

2,000


X


American Institutes for Research Accounting System

1



X

District Accounting System*

4

X


X

District Staff Interview*

4

X

X


School Attendance Counselor Log*

60


X


Districts Records Data Collection

  • Enrollment, student characteristics, attendance, and achievement data. We will request extant district data at three time points. The extant data request will be clearly specified in each district’s memorandum of understanding, which they will sign to participate in the study. The extant district data will be a direct export from the district’s student information system with the relevant fields, and will be uploaded to a cloud-based system that meets all of the U.S. Department of Education’s security requirements. These data will be used to conduct random assignment (i.e., establishing the intervention groups), describe the study sample, create covariates for impact analyses, calculate attrition and mobility rates, and create outcome measures.

    • September 2017. We will collect a first round of extant data including enrollment, attendance, and student characteristics. The data request will include student attendance information from the 2016-17 school year and from the first month of the 2017-18 school year. We will use these data to identify which students are at risk of chronic absenteeism (i.e., students who were absent 10 percent or more of the school days during the 2016-17 school year or in the first month of the 2017-18 school year), and for the random assignment of students into the first-stage groups (BIM-pos, BIM-neg, or BAU). The student characteristics will include students’ grade, age, race/ethnicity, individualized education program status, free or reduced-price lunch status (if available), and English language learner status. These data will be used to describe the baseline sample and create covariates for impact analyses.

    • December 2017. A second round of enrollment and attendance records will include student data from the beginning of October to the middle of December. We will use these data to determine which families in the BIM-pos and BIM-neg first-stage messaging groups will be randomly assigned to the School Staff Outreach or Goal Commitment Messaging strategies in the second-stage.

    • June 2018. The third round of extant data will include enrollment, attendance, and achievement data at the end of the 2017-18 school year. We will use these data to create our main attendance and achievement outcome measures. The end-of-year enrollment data will be used to calculate attrition and mobility.

Parent Survey. We will administer a parent survey in April and May 2018 to a sample of 2,000 families—this sample will be a randomly selected sub-sample of the families that participated in the study. The brief survey will be used to capture valid and reliable measures of parent attitudes and beliefs about school attendance and their perceptions of the text messaging intervention, which are not available from extant data for the participating districts and schools.

District Accounting System. We will request information from participating districts regarding costs associated with setting up the text messaging intervention (e.g., new infrastructure such as servers to boost district server capacity). We will request these records at the beginning and end of the 2017-18 school year.

District Staff Interviews. At the beginning and end of the 2017-18 school year, the study team will collect additional information on the costs of setting up and implementing the text messaging intervention through short, structured interviews with the heads of districts’ Information Technology (IT) and Student Information System (SIS) departments. In these interviews, we will gather information on (1) the hardware and software necessary to set up and support the system, (2) the effort school and district IT/SIS staff spend on the necessary technical expertise to integrate the system, and (3) the time it takes IT/SIS staff to monitor the attendance data and text messages throughout the year.

School Attendance Counselor Logs. We will ask school attendance counselors to fill out a short log three times during the 2017-18 spring semester, in order to estimate the costs of the School Staff Outreach strategy that are not captured in the text messaging provider’s back-end data (e.g., attendance counselors following up with phone calls and parent meetings after initial outreach by text messaging). The logs ask the counselors to indicate how often they contacted or were in communication with parents of students in each of the intervention conditions. The data from the logs will be used to document the level of effort schools spent on students and their families assigned to the School Staff Outreach strategy, relative to other treatment arms.



Part A. Supporting Statement for Paperwork Reduction Act Submission

  1. Circumstances Making Collection of Information Necessary

Most school districts have policies in place to systematically address student absenteeism. Typical district attendance practices include parent notification by letters, calls, parent meetings, home visitation, and, for students with significant numbers of unexcused absences, referrals to truancy programs and family courts. District policies are changing from reactive to preventive due to an increasing understanding of the importance of attendance and that negative “punishments” such as withholding grades, cutting benefits, or requiring suspensions, citations, and fees may contribute to the problem, not solve it.9

Text messaging interventions are becoming increasingly popular due to their low cost, scalability, and evidence of impact. In other fields, such as public health and prevention, text messaging using mobile technology has been widely tested, showing positive impacts on people’s health behaviors in a variety of contexts, including HIV prevention, medication adherence, pregnancy education, substance abuse, smoking cessation, weight loss, diabetes management, and depression management. Many of these campaigns include a series of automated and interactive text messages that are designed to guide individuals through a process of behavioral change.10

School districts have increasing capacity to use technology to implement messaging interventions. Many districts currently use push notifications from mobile apps, multimedia messaging service, and short message service messages to contact parents directly on their mobile phones for multiple purposes, including mass messages regarding school closings, events, and requirements; notifications about emergencies; requests and reminders for parents; and increasingly often same-day notifications when students are not in school.

An evaluation to determine whether a text messaging intervention on attendance is effective and reduces district costs for addressing chronic absenteeism is thus warranted. To our knowledge, this project is the first multidistrict random assignment study of the impact of a text messaging intervention for parents on student attendance and achievement.

Consistent with the Consolidated and Further Continuing Appropriations Act, 2015 (PL 113-235), which provides the legislative authority to conduct this study, this study will focus on low-performing schools with high levels of poverty and student absenteeism.11



  1. Purpose and Use of the Data: How the Data Will Be Collected, by Whom, and for What Purpose

The information gathered through this data collection will be analyzed by American Institutes for Research (AIR)—an evaluation contractor of the Institute of Education Sciences (IES)—and its subcontractors to study the implementation and impacts of the attendance text messaging intervention in elementary schools. The new data collection required for the study includes a parent survey for a sub-sample of participating parents, district staff interviews (two per district), and School Attendance Counselor Logs (three per school). In addition to new data collection, the study also requires extant data provided by participating districts. Exhibit 3 (above) shows a timeline of these data collection activities.

The data collected for this study will be used by IES to report on the feasibility and effectiveness of a low-cost but technically sophisticated, adaptive messaging interventions designed to reduce absenteeism in elementary schools. If this evaluation were not completed, the U.S. Department of Education, states, districts, schools, and the public would not have an accurate understanding of the impact of an increasingly popular method (text messaging) for informing parents of their children’s attendance. Moreover, these stakeholders would not understand how basic text messaging might be enhanced by taking an adaptive approach.



  1. Use of Technology to Reduce Burden

Parents will receive web-based surveys accessible through a smartphone, tablet, or a computer. Administration of web-based surveys enables easy access and reduces burden through complex skip patterns that are invisible to respondents, as well as prefilled information based on responses to previous items when appropriate. Web-based surveys are also less expensive than paper and phone surveys, due to their increased processing and data collection speeds. Phone and paper survey options will be offered to respondents as part of the follow-up effort with individuals who do not respond. The School Attendance Counselor Logs will also be web based. The interviews with district Information Technology (IT) and Student Information System (SIS) department personnel in fall 2017 and spring 2018 will be administered by phone to reduce burden on respondents and reduce travel costs to the evaluation.



  1. Efforts to Avoid Duplication

The primary data collection effort planned for this project will produce data that are unique and target the research questions identified for this project. The primary data collection includes parent surveys, district staff interviews, and school staff logs.

Parent surveys. The main purpose of these data is to capture valid and reliable measures of parent attitudes toward and knowledge of absenteeism, which are not available from extant data for the participating districts and schools.

District staff interviews and school staff logs. The main purposes of these data are to describe implementation of the text messaging intervention and to conduct cost-effectiveness analyses. These data are not available from district records.


  1. Methods to Minimize Burden on Small Entities

The data will be collected from district and school staff and parents. No small businesses or entities will be involved in the data collection.


  1. Consequences of Less Frequent Data Collection

If the proposed data collection is not done, it will not be possible for IES to report findings from a large-scale, rigorous evaluation of an attendance text messaging intervention in low-performing, high-poverty, and high-absenteeism schools to Congress, other policymakers, and practitioners seeking effective ways to support student learning.



  1. Special Circumstances Relating to the Guidelines of
    5 CFR 1320.5

No special circumstances apply to this study.



  1. Federal Register Comments and Persons Consulted Outside the Agency

The 60-day Federal Register notice was published on March 31, 2017, Vol. 82, page 16,029. No published comments have been received to date.

The individuals listed in Exhibit 4 serve on the Technical Working Group for the Evaluation.

Exhibit 4. Technical Working Group Members

Expert

Organization

Hedy Nai-Lin Chang

Executive Director, Attendance Works

Peter Bergman

Assistant Professor of Economics and Education, Teachers College, Columbia University

Daniel Almirall

Research Assistant Professor, Survey Research Center, Institute for Social Research, University of Michigan

Fiona Hollands

Adjunct Associate Professor of Education, Columbia University

Lorien Abroms

Associate Professor of Prevention and Community Health, School of Public Health and Health Services, George Washington University

Lindsay Page

Assistant Professor of Research Methodology, School of Education, University of Pittsburgh

Susanna Loeb

Barnett Family Professor of Education, Stanford University

Mel Atkins

Executive Director of Community and Student Affairs, Grand Rapids Public Schools



  1. Payment or Gifts to Respondents

High response rates are needed to ensure the study measures are valid and reliable. Offering honoraria for parents, who have no obligation to participate or respond, will help achieve high response rates on the end-of-year survey. Specifically, a sample of 2,000 parents will be asked to complete a brief 10-minute survey in spring 2018. The planned incentive value is $15, consistent with the 2005 National Center for Education Evaluation recommendation for a “low burden” data collection effort.12

The importance of providing data collection incentives in federal studies has been described by other researchers, given the recognized burden and need for high response rates.13 The use of incentives has been shown to be effective in improving response rates and reducing the level of effort required to obtain completions, particularly with low-income populations.14 Incentives in educational settings have also been shown to be effective; for example, in the Reading First Impact Study commissioned by IES, monetary incentives had significant effects on response rates among teachers. A sub-study requested by the Office of Management and Budget (OMB) on the effect of incentives on survey response rates for teachers showed significant increases when an incentive of $15 or $30 was offered to teachers as opposed to no incentive.15



  1. Assurances of Confidentiality Provided to Respondents

All data collection activities will be conducted in full compliance with the Department of Education regulations to maintain the confidentiality of data obtained on private persons and to protect the rights and welfare of human research subjects as contained in the Department of Education regulations. These activities will also be conducted in compliance with other Federal regulations including the Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the  Family Educational Rights and Privacy Act of 1974, 20 USC 1232g, 34 CFR Part 99; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b. Information collected for this study comes under the confidentiality and data protection requirements of IES (Education Science Reform Act of 2002, Title 1, Part E, Section 183).

An explicit verbal or written statement describing the project, the data collection, and confidentiality will be provided to study participants. These participants will include parents as well as school staff providing school attendance logs and district IT and SIS department staff participating in interviews.

Information collected for this study comes under the confidentiality and data protection requirements of IES. All information from this study will be kept confidential as required by the Education Sciences Reform Act of 2002 (Title I, Part E, Section 183). Responses to this data collection will be used only for statistical purposes. Personally identifiable information about individual respondents will not be reported. We will not provide information that identifies an individual, school, or district to anyone outside the study team, except as required by law.

AIR’s Institutional Review Board has approved “alteration of consent” for parents’ participation in the study and district record request.16 The parent consent form can be found in Appendix F.


Confidentiality assurances during data collection:

  • All data collection staff at AIR and any data collection subcontractors will go through any required background clearances (i.e., e-QIP) and sign confidentiality agreements that emphasize the importance of confidentiality and specify employees’ obligations to maintain it.

  • Personally identifiable information (PII) will be maintained on separate forms and files, which will be linked only by study-specific identification numbers. All data containing PII will be stored in a cloud-based server system that meets ED’s security requirements.17

  • Access to a crosswalk file linking study-specific identification numbers to PII and contact information will be limited to a small number of individuals who have a need to know this information. All staff with access to these data will go through required background clearances (i.e., e-QIP).

  • Access to hard copy documents will be strictly limited. Documents will be stored in locked files and cabinets. Discarded materials will be shredded.

  • Access to electronic files will be protected by secure usernames and passwords, which will be available only to approved users. Access to identifying information for sample members will be limited to those who have direct responsibility for providing and maintaining sample crosswalk and contact information. At the conclusion of the study, these data will be destroyed.

  • The plan for maintaining confidentiality includes staff training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses. It also includes built-in safeguards concerning status monitoring and receipt control systems.

Confidentiality assurance during analysis: The data collected for this study will be used only for broadly descriptive and statistical purposes. In no instances will the study team provide information that identifies districts, schools, principals, teachers, or students to anyone outside the study team, except as required by law. More detail on AIR’s procedures to ensure data security is provided below.

Data storage: AIR and subcontractors will store all data in compliance with a federally approved data security plan. All quantitative data will be stored on a secure network, accessible only to specific project staff identified by the data manager. Data that will be used for analysis will be stored with study-specific ID numbers rather than actual student or staff identification information. Notes/digital recordings from interviews and all survey data will be reviewed only by the research team and stored in a secure area accessible only to the research team. All data containing PII will be stored in secure cloud-based servers that comply with ED’s security requirements.

Once the study is completed, original data collected for the project will be destroyed and only the restricted access file, without any actual identifiers for the districts, schools, or individual respondents, will remain. The restricted-access file will be reviewed by IES’s Disclosure Review Board to ensure that it fully complies with all applicable data privacy and confidentiality laws. Once the restricted file has been approved, it will be submitted to IES at the conclusion of the project.

Method of data destruction: All data containing individually identifiable records will be destroyed by an appropriate fail-safe method, including physical destruction of the media itself or deletion of the contents on AIR’s servers.


  1. Justification of Sensitive Questions

There are no sensitive questions in any of the instruments used to collect data for the study, including the parent survey, School Attendance Counselor Logs, and District Staff Interview Protocols.



  1. Estimates of Annualized Burden Hours and Costs

The total estimated hour burden for the data collections for the study is 597 hours. Exhibit 5 summarizes reporting burden on respondents for each data source. The respondent pool will include parents and district and school staff. For each data collection activity, there will be a corresponding number of respondents assumed and a total number of respondents estimated. Exhibit 5 also provides the number of administrations for each data collection activity over the course of the study, average estimates for the amount of time required for each activity in minutes, and the total burden hours calculated for each activity to be completed by the required respondents. This proposed information collection does not impose a financial burden on any respondents, and respondents will not incur any expenses. Total annual respondents, responses, and burden hours can be found in Exhibit 5.

The introduction to supporting Statement A provides more detail about the timeline of data collection activities listed in Exhibit 5 and describes why these activities are essential to the study.

Exhibit 5. Estimated Annual Burden and Costs for Data Collection

Assumed Number of Respondents

Estimated Number of Respondents per Administration‡

Number of Administrations

Average Burden Hours per Response (Hours)

Total Annual Burden (Hours)

Total Number of Responses

Total Number of Respondents‡

District Records Data Collection

4

4

3

16.00

192

12

4

Parent Survey

2,000

1,700

1

0.17

289

1,700

1,700

District Accounting System

4

4

2

0.50

4

8

4

District Staff Interview

4

4

2

0.50

4

8

4

School Attendance Counselor Log

60

54

3

0.67

108

162

54

Total Annual Burden

2,072

1,766

10

0.22

597

1,890

1,766

‡An 85 percent response rate is assumed for parent surveys and a 90 percent response rate is assumed for School Attendance Counselor Logs.




  1. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

Not applicable. The information collection activities do not place any capital cost or cost of maintaining capital requirements on respondents.



  1. Annualized Cost to the Federal Government

The total cost for the study is $2,997,286 over 3 years, for an annualized cost of $1,401,430 in year 1; $1,104,200 in year 2; and $491,656 in year 3. The annual cost to the federal government is $999,095 per year.



  1. Explanation for Program Changes or Adjustments

This submission to OMB is a new request for approval.



  1. Time, Schedule, Publication, and Analysis Plan

Schedule and Publication Plan

The project schedule is as follows:


  • Random assignment of families within schools in October 2017;

  • Text message intervention implementation from October 2017 to June 2018;

  • Completion of data collection and creation of analysis files for final report by September 2018;

  • Analysis of findings for final report in fall 2018 through winter 2019;

  • Preparation of a project final report and restricted-use data file for review by IES and release in October 2019.

AIR anticipates a report that includes an introductory chapter, a chapter on study design and data collection, a chapter discussing the nature and implementation of the attendance text messaging intervention as well as “business-as-usual” services provided to control students and families, and a chapter presenting impact findings. The report will have an appendix that details the cost of the intervention. The report will follow guidance provided in the National Center for Education Statistics (NCES) Statistical Standards18 and the IES Style Guide.19



Analysis Plan

Implementation. To assess implementation of the attendance text messaging intervention, the study team will use data obtained from the text message vendor (back-end data including information on the number and type of messages, when they were sent, opening rates) and School Attendance Counselor Logs. Analysis of these data will allow the team to describe the implementation of the attendance text messaging intervention overall, and to describe variation over time (i.e., fall and spring) and within and between participating school districts. The study team will also analyze responses to parent survey items about the text messaging intervention to describe parents’ reactions and perceptions of it.

Impacts. To estimate the impacts of the text messaging intervention on student attendance and student achievement, the study team will rely on district records. The study team will create two main outcome measures of student attendance using district records: (1) a binary indicator reflecting whether or not the student missed 10 percent of instructional days over a given time period (e.g., October–December), and (2) a continuous measure of percentage of enrolled days over a given time period (e.g., the full school year). For the student attendance outcomes, we will estimate program impacts after implementation of first-stage approaches (research question (RQ1)), the added effect of second-stage strategies (RQ2), and the effects of the four embedded adaptive interventions (RQ3). The main student achievement outcome measures will be scores on state standardized assessments in reading/English language arts and mathematics. For the student achievement outcomes, the focus will be on the effects of the four embedded adaptive interventions (RQ3) at the end of the school year. All of the main analyses will be conducted for the full sample and for the subgroup of students identified as being at-risk for chromic absenteeism at baseline, according to the attendance data from the previous year or from the first 20 to 30 days of school.

Cost effectiveness. To estimate the cost effectiveness of the text messaging intervention, the study team will use the Resource Cost Model (RCM). The RCM is a methodological approach to cost analysis that involves explicitly organizing the data gathering effort around the specific activities related to provision of a service, strategy, or intervention. The method has its roots in the “ingredients” approach to cost analysis,20 by modeling the structure and “ingredients” of services as they are actually provided under the intervention. Importantly, the RCM database will contain sufficient information to permit analysis of variation in the intervention implementation costs across different districts. After the RCM is populated, the costs associated with the adaptive interventions will be combined with the results of the impact analysis to provide a variety of cost-effectiveness estimates. Specifically, the estimates will measure the effectiveness of the various interventions in terms of the cost per additional unit improvement of attendance (e.g., cost per additional student day of attendance). In addition, the study team will conduct analyses breaking down costs under each of the intervention components by the main services provided. The main results of this analysis will be written with practitioner and policymaker audiences in mind, with easy-to-understand graphical and tabular presentations in the main text and a technical appendix for readers seeking a higher level of detail. Ultimately, the findings will provide unique insights into the resource investments required to implement similar interventions.


  1. Approval to Not Display OMB Expiration Date

All data collection instruments will include the OMB expiration date.



  1. Explanation of Exceptions to the Paperwork Reduction Act

No exceptions are needed for this data collection.













References for Supporting Statements, Part A

Abroms, L.C., Whittaker, R., Free, C., Van Alstyne, J.M., and Schinderler-Ruwisch, J.M. (2015). Developing and Pretesting a Text Messaging Program for Health Behavior Change: Recommended Steps. JMIR mHealth and uHealth, 3(4): e107. Retrieved December 8, 2016 from https://mhealth.jmir.org/2015/4/e107/.

Allensworth, E.M., and Easton, J.Q. (2005). The On-Track Indicator as a Predictor of High School Graduation. University of Chicago, Consortium on Chicago School Research. Retrieved December 8, 2016 from http://consortium.uchicago.edu/publications/track-indicator-predictor-high-school-graduation.

Allensworth, E.M., and Easton, J.Q. (2007). What Matters for Staying On-Track and Graduating in Chicago Public High Schools: A Close Look at Course Grades, Failures, and Attendance in the Freshman Year. University of Chicago, Consortium on Chicago School Research. Retrieved December 8, 2016 from https://consortium.uchicago.edu/publications/what-matters-staying-track-and-graduating-chicago-public-schools.

Almirall, D., Nahum-Shani, I., Sherwood, N.E., and Murphy, S.A. (2014). Introduction to SMART Designs for the Development of Adaptive Interventions: With Application to Weight Loss Research. Translational Behavioral Medicine, 4(3): 260–274. Retrieved December 8, 2016 from http://link.springer.com/article/10.1007/s13142-014-0265-0.

Applied Survey Research. (2011). Attendance in Early Elementary Grades: Associations with Student Characteristics, School Readiness, and Third Grade Outcomes. San Francisco: Attendance Works. Retrieved December 8, 2016 from http://www.attendanceworks.org/wordpress/wp-content/uploads/2010/04/ASR-Mini-Report-Attendance-Readiness-and-Third-Grade-Outcomes-7-8-11.pdf.

Baker, M.L., Sigmon, J.N., and Nugent, M.E. (2001). Truancy Reduction Keeping Students in School. U.S. Department of Justice, Office of Justice Programs. Washington, DC: National Criminal Justice Reference Service. Retrieved December 8, 2016 from https://www.ncjrs.gov/pdffiles1/ojjdp/188947.pdf.

Balfanz, R., and Byrnes, V. (2013). Meeting the challenge of combating chronic absenteeism: Impact of the NYC Mayor’s Interagency Task Force on chronic absenteeism and school attendance and its implications for other cities. Baltimore, MD: Everyone Graduates Center, Johns Hopkins University School of Education.

Berry, S.H., Pevar, J., and Zander-Cotugno, M. 2008. The use of incentives in surveys supported by federal grants. Santa Monica, CA: RAND Corporation.

Chang, S. J., Choi, S., Kim, S., and Son, M. (2014). Intervention strategies based on Information-Motivation-Behavioral Skills model for health behavior change: A systematic review. Asian Nursing Research, 8, 172–181.

Chang, H.N., and Romero, M. (2008). Present, Engaged, and Accounted For: The Critical Importance of Addressing Chronic Absence in the Early Grades. New York, NY: National Center for Children in Poverty. Retrieved December 8, 2016 from http://www.nccp.org/publications/pdf/text_837.pdf.

Dillman, D.A. 2007. Mail and Internet Surveys: The Tailored Design Method (2nd ed.). New Jersey: John Wiley & Sons, Inc.

Ehrlich, S. B., Gwynne, J. A., Pareja, A. S., Allensworth, E. M., Moore, P., Jagesic, S., and Sorice, E. (2014). Preschool attendance in Chicago public schools: Relationship with learning outcomes and reasons for absences: Research Summary. Chicago: University of Chicago Consortium on Chicago School Research.

Fisher, J.D., and Fisher, W.A. (1992). Changing AIDS-risk behavior. Psychological Bulletin, 111(3), 455–474.

Ginsburg, A., Jordan, P., and Chang, H. (2014). Absences Add Up: How School Attendance Influences Student Success. San Francisco: Attendance Works. Retrieved December 8, 2016 from http://www.attendanceworks.org/wordpress/wp-content/uploads/2014/09/Absenses-Add-Up_090114-1-1.pdf

Gottfried, M. (in press). Chronic Absenteeism in the Classroom Context: Effects on Achievement. Urban Education. Retrieved from: http://www.attendanceworks.org/wordpress/wp-content/uploads/2013/01/AW-gottfried_chronic_peers-2.pdf

Gunderson, J. (2010). Getting teenagers back to school: Rethinking New York State’s response to chronic absence. New York, NY: Vera Institute of Justice. Retrieved from: https://www.vera.org/publications/getting-teenagers-back-to-school-rethinking-new-york-states-response-to-chronic-absence

Harris, R., Jones, L. and Finnegan, D. (2001). Using TANF sanctions to increase high school graduation. Journal of Sociology & Social Welfare, 28(3), 211–222.

Henderson, T., Hill, C., and Norton, K. (2014). The connection between missing school and health: A review of chronic absenteeism. Retrieved from: https://www.upstreampublichealth.org/sites/default/files/Chronic%20Absence%20and%20Health%20Review%20FINAL%2010.10.2014.pdf

Henry, K.L., and Thomberry, T.P. (2010). Truancy and escalation of substance use during adolescence. Journal of Studies on Alcohol and Drugs, 71(1), 115–124.

Hernandez, D.J. (2011). Double jeopardy: How third-grade reading skills and poverty influence high school graduation. Annie E. Casey Foundation. Retrieved from: http://gradelevelreading.net/wp-content/uploads/2012/01/Double-Jeopardy-Report-030812-for-web1.pdf

James, Tracy. 1997. Results of the Wave I incentive experiment in the 1996 survey of income and program participation. In Proceedings of the Survey Methods Section (pp. 834–839), American Statistical Association.

Jung, H., Therriault, S., and Prencipe, L. (2012). Massachusetts Early Warning Indicator System risk model documentation: Early and late elementary. Boston, MA: MA Executive Office of Education, Department of Early Education and Care, Elementary and Secondary Education, and Department of Higher Education.

Levin, H. (1983). Cost-Effectiveness: A Primer. Newbury Park, CA: Sage Publications.

Levin, H., and McEwan, P. (2001). Cost Effectiveness Analysis: Methods and Applications. Thousand Oaks, CA: Sage Publications

Morgan, E., Salomon, N., Plotkin, M., and Cohen, R. (2014). The school discipline consensus report: Strategies from the field to keep students engaged in school and out of the juvenile justice system. The Council of State Governments Justice Center, 4, 112.

Murphy, S.A., and Almirall, D. (2009). Dynamic treatment regimens. Encyclopedia of Medical Decision Making. Thousand Oaks: Sage Publications

Nahum-Shani, I., Qian, M., Almirall, D., Pelham, W.E., Gnagy, B., Fabiano, G.A., and Murphy, S.A. (2012). Experimental design and primary data analysis methods for comparing adaptive interventions. Psychological Methods, 17(4); 457–477.

National Center for Education Evaluation. 2005. Guidelines for incentives for NCEE impact evaluations. March 22. National Center for Education Evaluation: Washington, DC.

National Center for Education Statistics. 2002. Statistical Standards (NCES 2003601). Washington, DC: National Center for Education Statistics.

Neild, R. C., and Balfanz, R. (2006). Unfulfilled promise: The dimensions and characteristics of Philadelphia’s dropout crisis, 2000–2005. Philadelphia Youth Network. Retrieved from: http://files.eric.ed.gov/fulltext/ED538341.pdf

Ready, D.D. (2010). Socioeconomic disadvantage, school attendance, and early cognitive development the differential effects of school exposure. Sociology of Education, 83(4), 271–286.

Singer, E., and Kulka, R.A. 2002. Paying respondents for survey participation. In M. Vander Ploeg, R.R. Moffitt, and C.F. Citro (Eds.), Studies of welfare populations: Data collection and research issues (pp. 105–128). Washington, DC: National Academy Press.

U.S. Department of Education, Institute for Education Sciences. 2005. IES Style Guide. Retrieved from www.nces.ed.gov.

ABOUT AMERICAN INSTITUTES FOR RESEARCH

Established in 1946, with headquarters in Washington, D.C., American Institutes for Research (AIR) is an independent, nonpartisan, not-for-profit organization that conducts behavioral and social science research and delivers technical assistance both domestically and internationally. As one of the largest behavioral and social science research organizations in the world, AIR is committed to empowering communities and institutions with innovative solutions to the most critical challenges in education, health, workforce, and international development.

1000 Thomas Jefferson Street NW
Washington, DC 20007-3835
202.403.5000

www.air.org

LOCATIONS

Domestic

Washington, D.C.

Atlanta, GA

Austin, TX

Baltimore, MD

Cayce, SC

Chapel Hill, NC

Chicago, IL

Columbus, OH

Frederick, MD

Honolulu, HI

Indianapolis, IN

Metairie, LA

Naperville, IL

New York, NY

Rockville, MD

Sacramento, CA

San Mateo, CA

Waltham, MA

International

Egypt

Honduras

Ivory Coast

Kyrgyzstan

Liberia

Tajikistan

Zambia

1 Chang and Romero 2008; Ehrlich et al. 2014; Ginsburg, Jordan, and Chang 2014; Gottfried in press; Allensworth and Easton 2005, 2007; Baker, Sigmon, and Nugent 2001; Henderson, Hill, and Norton 2014; Henry and Thornberry 2010; Neild and Balfanz 2006; Balfanz and Byrnes 2013.

2 Applied Survey Research 2011; Ehrlich et al. 2014; Ready 2010.

3 Hernandez 2011; Jung, Therriault, and Prencipe 2013.

4 Fisher and Fisher 1992.

5 Abroms et al. 2015; Chang et al. 2014.

6 From here forward, we refer to parents and guardians as “parents,” for simplicity.

7 For more information about SMART, see Almirall et al. 2014; Murphy and Almirall 2009; Nahum-Shani et al. 2012.

8 For RQ 1, the study is powered to detect a 5.0 percentage point difference in rates of chronic absenteeism among the first-stage intervention conditions (BIM-pos and BIM-neg) and BAU. For RQ 3, the study can detect a 6.8 percentage point difference in chronic absenteeism among the four embedded adaptive interventions and the BAU control condition, and has an MDES of 0.24 for student achievement outcomes. Power for RQ 2 can be estimated once it is determined whether interactions between the first- and second-stage messaging strategies exist.

9 Gunderson 2010; Harris, Jones, and Finnegan 2001; Morgan et al. 2014.

10 Abroms, et al., 2015.

11 The Consolidated and Further Continuing Appropriations Act, 2015 (PL 113-235) is available at https://www.gpo.gov/fdsys/pkg/PLAW-113publ235/html/PLAW-113publ235.htm

12 In 2005, the National Center for Education Evaluation (NCEE) submitted a memorandum to OMB outlining guidelines for incentives for NCEE Evaluation Studies and tying recommended incentive levels to the level of burden (represented by the length of the survey).

13 Berry, Pevar, and Zander-Contugno 2008; Singer and Kulka 2002.

14 Dillman 2007; James 1997.

15 National Center for Education Evaluation (March 22, 2005).

16 Administration of informed consent would likely result in a biased sample; therefore, AIR’s Institutional Review Board approved “alteration to consent” allowing an opt-out process.

17 The PII will be stored either on ED’s cloud-based system or AIR’s cloud-based system, Secure Analytics Workbench, which will have secured Authorization to Operate status before data collection.

18 National Center for Education Statistics 2002.

19 U.S. Department of Education, Institute of Education Sciences 2005.

20 Levin 1983; Levin and McEwan 2001.

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleThe Impact Evaluation of Parent Messaging Strategies
SubjectOMB Clearance Request
AuthorAIR
File Modified0000-00-00
File Created2021-01-22
© 2024 OMB.report | Privacy Policy