NM Assessing the Role of Noncognative and school environmental factors REL study OMB SUPPORTING STATEMENT A clean

SUPPORTING STATEMENT A:

REQUEST FOR CLEARANCE OF INFORMATION COLLECTION FOR

“Assessing the Role of Noncognitive and School Environmental Factors in Students’ Transitions to High School in New Mexico”

February 2015

Submitted to: Submitted by:

U.S. Department of Education SEDL

Institute of Education Sciences 4700 Mueller Blvd.
555 New Jersey Ave. NW, Rm. 308 Austin, TX 78723

Washington, DC 20208 Phone: (800) 476-6861

4700 Mueller Blvd. Austin, TX 78723

800-476-6861

www.relsouthwest.org

This publication was prepared for the Institute of Education Sciences (IES) under contract ED-IES-12-C-00012 by Regional Educational Laboratory Southwest, administered by SEDL. The content of the publication does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government. The publication is in the public domain. Authorization to reproduce in whole or in part for educational purposes is granted.

Contents

A. Justification 4

Introduction 4

1. Circumstances Necessitating the Data Collection 6

2. How, by Whom, and for What Purpose Information Is to Be Used 8

3. Technological Collection Techniques 10

4. Efforts to Avoid Duplication of Effort 11

5. Sensitivity to Burden on Small Entities 11

6. Consequences to Federal Program or Policy Activities if the Collection Is Not Conducted or Is Conducted Less Frequently Than Proposed 12

7. Special Circumstances 12

8. Federal Register Announcement and Consultation 12

c. Unresolved Issues 14

9. Payment or Gift to Respondents 14

10. Data Confidentiality 14

11. Additional Justification for Sensitive Questions 16

12. Estimates of Hour Burden 16

13. Estimate of Total Annual Cost Burden to Respondents or Record-Keepers 23

14. Estimates of Annualized Cost to the Federal Government 23

15. Reasons for Program Changes or Adjustments 23

16. Plan for Tabulation and Publication and Schedule for Project 23

17. Approval Not to Display the Expiration Date for OMB Approval 25

18. Exception to the Certification Statement 25

References 26

Attachment A-1. Parent Consent Form 28

Attachment A-2. Student Survey Administration Instructions 29

Attachment A-3. Student Survey 30

Attachment A-4. Superintendent Email 31

Attachment A-5. Superintendent Follow-up Email 32

Attachment A-6. Superintendent Telephone Script 33

Attachment A-7. Principal Telephone Script 34

Attachment A-8. Student Records Data 35

Attachment A-9. IRB Approval 36

Attachment A-10. Educational Sciences Reform Act (ESRA) 37

Attachment A-11. Federal Register Notices 38

Attachment A-12. Confidentiality Form and Affidavits 39

Attachment A-13. Technical Working Group Suggestions 40

Attachment A-14. High School Website and eblast 41

A. Justification

Introduction

The U.S. Department of Education (ED) requests clearance under the Office of Management and Budget (OMB) clearance agreement (OMB number [IES to complete]) for activities related to the Regional Educational Laboratory (REL) Program. ED, in consultation with SEDL, intends to examine relationships between measures of students’ noncognitive factors and school environments and grade 9 outcomes in New Mexico.

Although graduation rates have been on the rise nationally, New Mexico consistently achieves one of the lowest graduation rates in the United States. Only about 70 percent of students in the 2012-13 graduating cohort earned a high school diploma, with graduation rates varying significantly by race/ethnicity (New Mexico Public Education Department, 2014). While 77 percent of White students in this cohort graduated in four years, only 68 percent of Hispanic and 64 percent of Native American students did the same (New Mexico Public Education Department, 2014). This study will use data from a minimum of twenty schools in New Mexico to examine how noncognitive factors (e.g., growth mindset, learning strategies, and self-efficacy) and school environmental factors (e.g., perceptions of school safety, supportive teachers and counselors, usefulness of academic work) relate to three measures that Allensworth and Easton (2007) identified as mattering most for a successful transition (i.e., a successful freshman year)—overall freshman year GPA, number of course failures in all courses during freshman year, and freshman-year absences.

OMB approval is being requested for the use of the recruitment materials, survey data collection and extant data collection in participating New Mexico high schools, and extant data collection in associated New Mexico school districts. The survey is designed to collect data on students’ perceptions of their noncognitive skills and school climates. The extant student data will provide information about student demographics and achievement.

The results of this study will contribute to the body of knowledge used to inform practitioners and policymakers in New Mexico about the role of noncognitive and school environmental factors in students’ transitions to high school. It may also provide additional avenues to explore in future research that uses more rigorous designs or that explores new areas suggested by study results. These results can also help schools or districts to determine where they might want to focus their finite resources with regard to helping students to make successful transitions to high school.

This study will address the following research questions:

1. How do grade 9 students in New Mexico perform on average on measures of noncognitive factors?

   1. Do students’ scores on measures of noncognitive factors differ significantly by race/ethnicity?

   2. Do students’ scores on measures of noncognitive factors differ significantly by student achievement level?

   3. Do students’ scores on measures of noncognitive factors differ significantly by whether they are in the racial/ethnic minority/majority of the student body?

2. How supportive do grade 9 students in New Mexico perceive their high school environments to be?

   1. Do students’ scores on measures of the supportiveness of their high school environments differ significantly by race/ethnicity?

   2. Do students’ scores on measures of the supportiveness of their high school environments differ significantly by student achievement level?

   3. Do students’ scores on measures of the supportiveness of their high school environments differ significantly by whether they are in the racial/ethnic minority/majority of the student body?

3. Are there relationships between measures of grade 9 students’ noncognitive factor and the supportiveness of their high school climates and successful transitions to high school¹ controlling for student background characteristics?

   1. Do students’ scores on measures of their noncognitive factors and supportiveness of their high school environments differ significantly by race/ethnicity?

Data collection for this project consists of surveys to be administered to grade 9 students in participating high schools in New Mexico and extant data file collection from associated school districts and the New Mexico Public Education Department (NM PED). Specifically, in this OMB clearance package, ED is requesting clearance for the following data collection approach and approval of recruitment materials:

• Recruitment materials for participating districts and schools

• A paper-and-pencil survey of grade 9 students in participating high schools in New Mexico

• Extant data collection consisting of:

  • Student records data (specifically, student ID and grade 9 GPA – information not collected by NM PED) to be obtained from the district offices associated with the high schools participating in the study

  • Grade 9 classroom rosters containing student names and IDs from the high schools participating in the study

  • Data on student achievement and background characteristics from NM PED

Each research question will be answered using a combination of survey and extant data. ED believes that the data collections for which clearance is being requested represent the bare minimum necessary to examine relationships between measures of students’ noncognitive factors and school environments and grade 9 outcomes in New Mexico.

1. Circumstances Necessitating the Data Collection

Recent studies have linked a range of factors to increased dropout rates, including high absenteeism, low levels of school engagement, increased academic expectations, reduced support from teachers, problematic or deviant behavior, work or family responsibilities, moving to a new school in grade 9, and attending a school with lower achievement scores (Allensworth & Easton, 2007; Christie, Jolivette, & Nelson, 2007; Dweck, Walton, & Cohen, 2011; Rumberger, 2004; Suh & Suh, 2007). Studies such as these have led researchers to conclude that students need more than just strong academic preparation to succeed in high school—they need strong nonacademic preparation as well. That is, students need skills—noncognitive skills—to help them regulate their academic behaviors, learn and study effectively, persist in school, and achieve a sense of belonging. The term noncognitive factors refers to the student attitudes, beliefs, skills and dispositions about school and learning that are associated with positive academic outcomes and school success². Following Farrington et al (2012), this study will focus on five different types of noncognitive factors: academic mindsets (i.e., students’ beliefs about their ability to perform well in school), academic behaviors (i.e., behaviors that are commonly associated with being a good student), academic perseverance (i.e., self-control and grit), learning strategies (i.e., a person's approach to learning and using information) , and a sense of belonging (i.e., feeling that one belongs or is a part of a group). School climate can also play a role in supporting students’ transitions to high school. Freiberg and Stein (1999) described school climate as “the heart and soul of the school and the essence of the school that draws teachers and students to love the school and to want to be a part of it”. Positive school climate has been associated with better student academic achievement, graduation, and behavioral outcomes, and has been the focus of several recent initiatives for school reform, including the federal Safe and Supportive Schools program (Voight, Austin, & Hanson, 2013).

New Mexico consistently achieves one of the lowest graduation rates in the United States, with only about 70 percent of students in the 2012-13 graduating cohort earning a high school diploma (New Mexico Public Education Department, 2014). While 77 percent of White students in this cohort graduated in four years, only 68 percent of Hispanic and 64 percent of Native American students did the same (New Mexico Public Education Department, 2014). As such, practitioners and policymakers in New Mexico are interested in closing achievement gaps by improving high school transitions for American Indian and Hispanic students. These individuals, which includes representatives from several of the New Mexico Regional Education Cooperatives, the New Mexico Legislative Education Study Committee, three New Mexico universities, and NM PED are particularly interested in issues around academic preparation for high school and the relationships between noncognitive factors and student success in grade 9. In response to this interest, ED’s contractor, REL Southwest, is proposing an examination of grade 9 students’ perceptions of their noncognitive skills and high school environments and how those perceptions are related to academic success in grade 9.

This study will investigate students’ perceptions of their noncognitive skills and school climates, as well as examine whether these perceptions are related to students’ grade 9 absences, number of course failures, and GPA. This study will also investigate whether the results differ by students’ race/ethnicity. Regression analyses will be used to examine differences in students’ perceptions of their noncognitive skills and school climates by race/ethnicity, while multilevel structural equation models will be used to investigate relationships between students’ noncognitive and school environmental factors and their grade 9 academic outcomes. The study will use extant data from school, district and state databases, as well as survey data collected as part of this study. Data from all of these sources will be used to answer all research questions.

The timeline for data collection is shown in Table 2.

Table 1. Data Collection Timeline

Data Collection	Purpose	Requesting OMB Clearance?	Sept/Oct 2015	June 2016
Student names and State ID numbers for grade 9 students from participating schools	Distribute survey forms Reliably link survey data with extant student data files	Yes	X
Student survey administration	Collect noncognitive factors and school environmental factors data	Yes	X
Student grade 9 grade point averages (GPAs) from participating districts	Outcome data for study analyses	Yes		X
Extant student data files from NMPED	Student achievement data Student background characteristics	Yes		X

See attachment A-3 for the student survey and attachments A-4 through A-7 for the recruitment materials. Attachment A-8 contains the extant data elements to be provided by NM PED, and participating schools and districts to ED contractors.

2. How, by Whom, and for What Purpose Information Is to Be Used

ED’s contractor for REL Southwest will analyze the data to be collected through this study using statistical models and procedures that are preapproved by the Institute of Education Sciences (IES). The contractor will then summarize the findings in a report that will undergo review for quality and relevance by the National Center for Education Evaluation and Regional Assistance’s (NCEE’s) external review contractor. After the report has undergone IES review, findings will be presented to participating schools and districts, alliance members, and the NM PED (primary audiences). It will be published through IES for educators and education researchers (secondary audiences).

One of the primary audiences for this study are the schools and districts participating in the study. The results of this study will inform policies, programs, and practices for students at the school- and district-level in New Mexico, as well as provide insight for schools and districts outside of New Mexico with similar demographics. First, if the noncognitive skills measured by the survey are predictive of grade 9 outcomes, students’ aggregated responses to the survey could be used to identify gaps in the types of skills most associated with positive grade 9 outcomes. Those gaps could then be addressed through targeted programs or other forms of assistance. For example, some students may benefit from help with study skills, while others may benefit from interventions geared to change their mindsets or academic behaviors. Second, if the school environmental factors measured by the survey are predictive of grade 9 outcomes, schools and districts could make changes to address school climate issues. Addressing problems early could help stem the state’s high dropout rates.

Another of the primary audiences for this study is NM PED. Presently, a representative sample of New Mexico high schools participate in the biennial administration of the Youth Risk and Resiliency Survey funded by the New Mexico Department of Health and the New Mexico Public Education Department (PED) and administered by the University of New Mexico. This survey is long (over 135 items) and focuses primarily on student health and behaviors. NM PED is interested in supplementing the Youth Risk and Resiliency Survey in off years with another survey that is shorter and focuses specifically on noncognitive factors and school climate. They are particularly interested in assessing these factors early, as part of their ongoing efforts to reduce New Mexico’s high student dropout rate. PED is interested in adopting the survey as a whole, or the scales that are shown to be strong predictors of grade 9 transitions, if scales included on the survey are shown to be highly predictive of student outcomes with regard to attendance, course failure and GPA at the end of grade 9.

Secondary audiences for this report include schools and districts outside of New Mexico and other researchers who are interested in understanding the role of noncognitive and school environmental factors in student transitions to high school, including those interested in implementing a similar survey.

To provide this information to the primary audience and the secondary audience, ED’s contractor is requesting OMB clearance to perform the data collection activities listed in Table 1, which includes:

• Collection of student names and State ID numbers from participating high schools for ED’s contractor to use to distribute survey forms to students in school classrooms and to link student survey responses to extant student achievement and demographic data to be provided to ED’s contractor by NM PED. Collecting this information at the start of the study period will allow ED’s contractor to pre-populate student information into the blank survey forms and minimize the risk of survey results and extant data not being correctly linked. This data collection is likely to be necessary at the school level because districts typically do not maintain classroom roster information; if districts do have this information, we will request it from them to reduce the burden on participating schools. We estimate that it will take a total of 25 hours (1 hour per school) for administrative staff to print lists of student names and State ID numbers for grade 9 students by classroom, if they are unable to send this electronically.

• Collection of student grade 9 GPA data from districts. Since NM PED does not collect student GPAs from districts, ED’s contractor will obtain this data for the grade 9 students from study schools in each of the participating districts. We estimate that no more than 6 districts will participate in the study. We also estimate that it will take one district data manager approximately 1 hour to pull and transmit this data.

• Distribution of information about the study to parents via the school website and email. Administrative staff from each of the schools will post information about the study, including contact information for REL Southwest, on the school website. They will also send email messages to parents of eligible children informing them about the study, and inviting them to contact REL Southwest with any questions about the study or if they do not want their child to participate. Parents will be able to notify REL Southwest if they do not wish for their student to participate in the study via email or telephone.

• Administration of the student survey, including coordination activities with individual schools to determine the best dates to administer the survey and how to facilitate the shipping of survey materials, distribution of student surveys in classrooms, and collection of completed student surveys. We estimate that it will take a total of 24.8 hours to coordinate dates for survey administration and to facilitate survey shipping activities with school administrative staff. The survey has been designed to be completed in approximately 20 minutes, and we assume instructions and survey distribution will take about 10 minutes (a total of 30 minutes). About 3750 students will complete the survey once during fall 2015. As such, we estimate that survey data collection in the schools will take a total of 1875 hours.

• Collection of extant student achievement and demographic data from NM PED. We estimate that pulling and transmitting student achievement and demographic data should take approximately 8 hours.

3. Technological Collection Techniques

Data obtained from NM PED and the school districts will be transferred to REL Southwest using a secure file sharing network. Files will be uploaded to the file sharing workspace using a secure login, and data sent to SEDL will be encrypted and password protected during transmission. Once the files have been transmitted, they will be downloaded to a secure server and deleted from the shared workspace.

The data collection plan reflects sensitivity to issues of efficiency, accuracy, and respondent burden. To address the study’s research questions, the contractor will collect data using electronic data collection tools. The electronic tools include the following:

• A secure electronic file transfer protocol site that allows schools and districts to transfer administrative records to ED’s contractor in an efficient and secure way.

We are unable to administer the student survey electronically due to issues of computer access. Most high schools in New Mexico do not participate in a 1:1 program in which each student has access to his or her own computer or technology device. Asking teachers to bring students to a computer lab or to schedule time with mobile computer carts is beyond the scope of this study. The use of paper-and-pencil surveys reduces burden on teachers and schools. However, parents will be notified about the study via email, and information about the study will be included school websites. Parents will be invited to contact REL Southwest about the study via email or telephone.

4. Efforts to Avoid Duplication of Effort

To the extent possible, this project will rely on extant administrative data that are available for students, rather than asking individuals to provide the data for study purposes. Presently, no other systematic effort has been made or is currently underway to collect the same information that will be collected by the student survey for these student groups; as such there is no alternative source of the information collected by the student survey. The only data collected that will be unique to this study are the student survey data.

5. Sensitivity to Burden on Small Entities

The use of extant student administrative records from NM PED will reduce the burden on schools and districts by ensuring that only the minimum amount of original data is requested in order to meet the objectives of this study. As noted earlier, if districts do collect classroom rosters at the start of the school year, schools will not be requested to provide any extant data. Only the minimum amount of extant data necessary to complete the project will be requested from schools, districts, and NM PED. The student survey has been designed to be administered and completed in about 30 minutes—the amount of time necessary to collect the data needed to answer study questions.

6. Consequences to Federal Program or Policy Activities if the Collection Is Not Conducted or Is Conducted Less Frequently Than Proposed

The Education Science Reform Act of 2002 states that the central mission and primary function of the regional education laboratories is to support applied research and provide technical assistance to state and local education agencies within their region (ESRA, Part D, section 174[f]). If the proposed data were not collected, REL Southwest would not be fulfilling its central mission to serve the states in the region and provide support for evidence-based research. The research questions addressed in this study respond to questions raised by practitioners and policymakers in New Mexico, including representatives from several of the New Mexico Regional Education Cooperatives, the New Mexico Legislative Education Study Committee, three New Mexico universities, and the New Mexico Public Education Department (NM PED). If the proposed data were not collected, practitioners and policymakers would not have any data on students’ perceptions of their noncognitive skills and/or school environments to help them determine where they might want to focus their resources with regard to helping students to make successful transitions to high school. Additionally, as discussed above the results of this study will help NM PED decide whether or not to adopt the survey as a whole and/or to include scales that are strongly predictive of successful grade 9 transitions in their ongoing survey work.

This is a one-time study (i.e., not recurring) and therefore periodicity is not addressed.

7. Special Circumstances

There are no special circumstances.

8. Federal Register Announcement and Consultation

1. Federal Register Announcement

A 60-day Federal Register Notice was published on May 29, 2015. A 30-day notice was published on [DATE TO BE COMPLETED BY IES]. No public comments have been received to date.

2. Consultations Outside the Agency

ED and/or the REL Southwest contractor have consulted with individuals regarding the availability of data, the soundness of the evaluation design for addressing evaluation questions, and the clarity of measures. Specifically, a technical working group (TWG) comprising experts in research methodology and REL Southwest’s core areas of emphasis, which was assembled by the REL Southwest contractor to review studies. The TWG met twice, April 29, 2014 and August 4, 2014, to discuss the changes to the graduation requirements being implemented as a result of HB 5, the study methodology, and measures. The contractor was required to submit to ED the TWG comments and the contractors’ plan for addressing those comments (see appendix A-12).

Members of the TWG include:

Dan Goldhaber, Ph.D.

Director, CALDER (National Center for Analysis of Longitudinal Data in Education Research)

Vice President, American Institutes for Research (AIR)

Director, Center for Education Data & Research (CEDR), University of Washington Bothell

Co-Editor, Education Finance and Policy

3876 Bridge Way N, Suite 201

Seattle, WA 98103

Ph: 206-547-1562

Fax: 206-547-1641

E-mail: dgoldhab@uw.edu

Geoffrey Borman, Ph.D.

Professor of Education, University of Wisconsin—Madison

Deputy Director of the University of Wisconsin's Predoctoral Interdisciplinary Research Training Program

Senior Researcher, Consortium for Policy Research in Education.

348 Education Building
1000 Bascom Mall
Madison, WI 53706-1326

Ph: 608-263-3688

Fax: 608-265-3135

E-mail: gborman@education.wisc.edu

Johannes M. (Hans) Bos, Ph.D.

Vice President and Program Director, International Development, Evaluation, and Research (IDER) Program

American Institutes for Research

2800 Campus Drive, Suite 200
San Mateo, CA 94403

Ph: 650-843-8100

Fax: 650-843-8200

E-mail: jbos@air.org

W. Steven Barnett, Ph.D.

Board of Governors Professor and Director of the National Institute for Early Education Research

Rutgers University

73 Easton Avenue
New Brunswick, NJ  08901

Ph: 848-932-4350 x23132

Fax: 732-932-4360

E-mail: sbarnett@nieer.org

c. Unresolved Issues

There are no unresolved issues.

9. Payment or Gift to Respondents

This request for OMB clearance includes small payments via gift cards for classroom teachers and school administrative staff who assist with the study.

Teachers and school administrative staff will be provided with small payments for assisting with preparation for survey administration. Both classroom teachers, who assist with survey distribution, and school administrative staff, who assist with collection classroom rosters containing student names and state IDs, will receive $25 Amazon.com gift cards. The gift cards will be distributed by REL Southwest researchers on the date of survey administration for each individual school.

Participating schools and districts will also be offered copies of individual school reports displaying school-specific findings from the survey as well as copies of the final study report investigating relationships between noncognitive and school environment factors and grade 9 outcomes. The individual school reports will include a summary of the data collected from students at each particular schools, as well as aggregate data for all schools in the sample. No other incentives or payments are planned.

10. Data Confidentiality

The data collection efforts that are the focus of this clearance package will be conducted in accordance with all relevant federal regulations and requirements. The Southwest REL will be following the new policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183 requires “All collection, maintenance, use, and wide dissemination of data by the Institute” to “conform with the requirements of section 552 of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act (20 U.S.C. 1232g, 1232h).” These citations refer to the Privacy Act, the Family Educational Rights and Privacy Act, and the Protection of Pupil Rights Amendment.

Every measure will be taken to protect the confidentiality of the data collected and the data will be used for the purpose of the study. All survey responses will be kept confidential, and will only be used for the purpose of the study. No one at the school, district, or the state will have access to survey responses that include respondents’ names or other information that could potentially be used to identify individuals. REL staff will administer and collect all surveys. The project has been approved by E&I Review Services, which serves as SEDL’s Institutional Review Board to review research involving human subjects. E&I is registered with the Office of Human Research Protection (OHRP). E&I’s IRB Organization number is 000065. E&I’s IRB Registration number, effective until November 28, 2015, is IRB00007807³.

In addition, for student information, the data collection efforts will ensure that all individually identifiable information about students, their academic achievements and information with respect to individual schools, shall remain confidential in accordance with section 552a of Title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act. The study will also adhere to requirements of subsection (d) of section 183 prohibiting disclosure of individually identifiable information as well as making the publishing or inappropriate communication of individually identifiable information by employees or staff a felony. All administrative records will be sent to ED’s contractor by districts using a file transfer protocol (FTP). Access to the FTP site will be password protected, and all data will be immediately deleted from the FTP site upon successful download by ED’s contractor. All data files will be stored on secure server administered by ED’s contractor.

ED’s contractor will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released publicly. Information from participating institutions and respondents will be presented at aggregate levels in reports. Information on respondents will be linked to their institution but not to any individually identifiable information. No individually identifiable information will be maintained by the study team upon study completion.

To protect confidential data, only the contractor’s data management staff, investigators, and research staff will have access to the data files on a “need-to-know” basis. Any identifiable variables, raw data, or derived variables will be stored in encrypted files on a secure data management site. Access to this site will be limited to staff assigned to the project. Any data obtained for this study will be used only for statistical and descriptive analyses. All identifiers will be destroyed as soon as they are no longer required. Public study reports will not identify the name of any specific analysis unit (e.g., students, school staff members, or schools). In no case will information be reported when the total number for a quantity represents fewer than four cases, per IES guidance. Moreover, any data that permit identity disclosure, when used in combination with other known data, will not be published or made available in restricted-use files.

All members of the study team have obtained their certification on the protection of human subjects in research, and REL Southwest staff members will also have obtained federal security clearances. The REL study team will submit to the NCEE security officer a list of the names of all people who will have access to respondents and data. All staff members working on the project who have access to the data or to respondents will be required to sign a confidentiality pledge and affidavits of non-disclosure (see copies of the forms in Attachment A-12; we will obtain the appropriate signatures). The project team will track new staff and staff who have left the study and ensure that additional signatures will be obtained or clearances will be revoked.

Respondents to the surveys will be informed that project staff are committed to keeping data confidential and that participation in the data collection activities is voluntary.

11. Additional Justification for Sensitive Questions

No questions of a highly sensitive nature appear in any instrument, including the student survey. In addition, participants will be informed that their responses are voluntary, and they may decline to answer any question.

12. Estimates of Hour Burden

There are three components for which ED’s contractor has calculated hours of burden for this clearance package: (1) school recruitment activities, (2) student survey administration, and (3) extant student records data collection. Table 2 shows the hourly burden overall and for each component. The total burden associated with this study is 2319.6 hours. For each of the three years of the study, the annualized burden is estimated to be 773.2 hours. This burden estimate includes the total time required to recruit district superintendents and principals to participate in the study—47.8 hours—the time to coordinate administration and administer the student survey—2257.8 hours—and the burden estimate for extant data collection—14 hours. For this data collection, the burden was estimated based on the contractor’s performance of recruitment activities, as well as the contractor’s previous experience administering student surveys and obtaining extant data from districts. The annualized number of responses is 2590.7 (a total of 7772 across all three years of the study).

Table 2. Time Burden Estimates for the REL Southwest Noncognitive and School Environmental Factors Study

Instrument	Person Incurring Burden	Number of Respondents	Frequency of Response	Hours per Response	Total Burden (Hours)
Recruitment
Recruitment emails to district superintendents	District superintendent	40	1	0.17	6.8
Follow-up recruitment email to district superintendents	District superintendent	30	1	0.17	5.1
Second follow-up recruitment email to district superintendents	District superintendent	25	1	0.17	4.3
Recruitment telephone calls with district superintendents	District superintendent	20	1	.33	6.6
Recruitment and coordination calls with principals	School principal	25	2	.50	25
Subtotal	--	140	--	--	47.8
Survey Data Collection
Coordination calls with school administrative staff	School administrative staff	25	3	.33	24.8
Collection of student names and State IDs	School data manager	25	1	1	25
Distribution of study information to parents	School administrative staff	25	1	1	25
Parental consent forms	Parent/guardian	3750	1	.08	300
Teacher assistance with survey distribution	Teachers	50	2	.08	8
Student survey administration	Students	3750	1	.50	1875
Subtotal	--	7625	--	--	2257.8
Extant Student Data Collection
Extant student background and achievement data	State data manager	1	1	8	8
Extant student grade 9 grade point averages (GPAs)	District data manager	6	1	1	6
Subtotal	--	7	--	--	14
Total	--	7772	--	--	2319.6
We assume an average of 20 minutes per respondent to complete the survey, plus an additional 10 minutes for survey distribution and instructions (a total of 30 minutes). Teachers will assist with survey distribution (a total of 5 minutes per class).

Burden for Recruitment Activities

The total estimated burden for recruiting districts and schools to participate in the study is 47.8 hours, including 16.2 hours for email messages to district superintendents, 6.6 hours for telephone calls to district superintendents, and 25 hours for telephone calls with high school principals. Recruitment activities will take place during year 1 of the study.

The student survey will be administered in a minimum of 20-25 public high schools, within five or six districts. ED’s contractor will partner with the New Mexico Public Education Department (PED) for recruitment activities. ED’s contractor plans to recruit the largest school districts in New Mexico—Albuquerque, Santa Fe, Gallup, and Las Cruces – with the goal of two of these districts agreeing to participate. Because New Mexico school districts tend to be small, it is expected the remaining school districts will be relatively small—approximately one to two high schools. The goal for recruitment outside of the largest school districts is to include at least one predominantly Hispanic district; at least one predominantly Native American district; and at least one district with a mixed population of Hispanic, Native American, and White students in the project. The rationale for selecting these types of districts is to ensure that we have an adequate representation of American Indian and Hispanic students—the groups of students for which alliance members are interested in closing achievement gaps.

Starting in summer 2015, ED’s contractor will begin recruiting high schools to participate in the study. For recruiting purposes, with the exception of the four largest, school districts statewide will be stratified based on racial/ethnic composition—predominantly Hispanic, predominantly Native American, mixed—and recruited in groups. Within each strata, school districts will be stratified by size—from largest to smallest—and the five largest school districts in each strata will be targeted for recruiting. Additional rounds of recruitment activities will occur until at least one school district from each strata has agreed to participate. In all, there will be five or at most six districts that participate in this project.

Recruitment will be conducted via email and telephone. Once OMB clearance has been granted, ED’s contractor will send an email message to district superintendents introducing the study and inviting them to participate (see attachment A-4). The emails will outline the goals of the study, the content of the survey, and schools’ in the district’s role in survey dissemination and collection. A follow-up email will be sent to all non-responding superintendents one week after the initial email (see attachment A-5). One week after distribution of the follow-up email, another identical follow-up email will be sent to superintendents. Each e-mail will take up to 10 minutes to read and respond to. If enough districts have not agreed to participate in the study, REL Southwest will begin conducting telephone recruiting calls (see attachment A-6). The REL Southwest researcher will ensure that district superintendents understand the nature of the study and the responsibilities of participating schools. Once superintendents have agreed to participate in the study, they will be asked to sign a memorandum of understanding/consent form indicating that their district agrees to participate in the study. The goal is to include 5 or 6 school districts for a total of 20-25 high schools⁴.

After district superintendents have agreed to participate in the study. REL Southwest researchers will conduct telephone calls with principals from all of the high schools in each of the participating districts. The telephone calls will introduce principals to the study and familiarize them with the survey. At this time, schools will be invited to participate in the study (see attachment A-7). REL Southwest expects the process of gaining principal consent may require two phone calls. Principals will then designate a staff member to work with REL Southwest to further coordinate receipt of survey materials and dates for survey administration. Each telephone call is estimated to last approximately 30 minutes.

Burden for Student Survey Administration

The total estimated burden for student survey administration is 2232.8 hours. This estimate includes 24.8 hours for coordination activities with high school administrative staff, 25 hours for collection of student names and state IDs from high schools, as well as 1875 hours for administration and completion of the student survey (10 minutes for survey distribution and instructions plus 20 minutes for survey completion).⁵ It also includes 5 minutes per classroom for assistance with survey distribution from teachers, totaling 8 hours. This estimate assumes an average of 2 teachers per school who each teach two grade 9 English classes. Finally, the estimate includes 5 minutes per parent/guardian to review/sign the parental consent form.

With regard to parental consent, prior to administration of the student survey, teachers will be asked to distribute parent/guardian consent forms to students a few weeks prior to the scheduled survey administration. The consent forms inform parents/guardians about the nature of the study, as well as the contents of the student survey. Additionally, school administrative staff will post information about the study on the school website and send email messages detailing the study to parents. This study uses passive consent. If parents/guardians do not want their child to participate in the study, they can check the box on the parental consent form and have their student return it to their English teacher. They can also contact REL Southwest research staff using the email address or telephone number included on the school website and email messages. Copies of the parent consent form, in Spanish and English, are shown in appendix A-1.

Burden for Extant Data Collection

The total estimated burden for participating districts and NM PED to compile and transmit secondary data to ED’s contractor is 14 hours. This calculation assumes one data manager at NM PED works a collective total of 8 hours (1 total day) on compiling the data request for student background and achievement data. We are requesting uncomplicated files with a small number of variables. The data manager will only need to pull records that match the State student IDs for grade 9 students in participating schools. We have already obtained a student data MOU from NM PED.

The total estimated burden for each of the districts to compile and transmit extant grade 9 student grade point averages is 6 hours (approximately 1 hour per district). The district data manager will only need to pull grade point average records that match the State student IDs for grade 9 students in participating schools.

Annualized Costs for Respondents

The total cost to respondents for the components of this study that require burden from outside sources—the recruitment activities, survey coordination, and the extant student records data collection—is provided in Table 4. The total respondent cost associated with this study is approximately $4213. The annualized cost for this three-year study is $1404.33. The total cost of the recruitment activities is $2029; the total cost of survey coordination and distribution is $1554; and the total cost of extant student records data collection is $630.

Table 4. Estimates of Annualized Costs for Respondents

Instrument	Type of Respondent	Total Burden Hours	Hourly Wage Rate1	Monetary Cost of Burden
Recruitment
Recruitment emails to district superintendents	District superintendent	6.8	$41	$279
Follow-up recruitment email to district superintendents	District superintendent	5.1	$41	$209
Second follow-up recruitment email to district superintendents	District superintendent	4.3	$41	$176
Recruitment telephone calls with district superintendents	District superintendent	8.3	$41	$340
Recruitment and coordination calls with principals	High school principal	25	$41	$1025
Subtotal	--	35	--	$2029
Survey Coordination and Distribution
Coordination calls with school administrative staff	School administrative staff	24.8	$18	$446
Collection of student names and State IDs	School data manager	25	$18	$450
Distribution of study information to parents	School administrative staff	25	$18	$450
Teacher assistance with survey distribution	Teachers	8	$26	$208
Subtotal	--	57.8	--	$1554
Extant Data Collection
Extant student background and achievement data	State data manager	8	$45	$360
Extant student grade 9 GPA data	District data manager	6	$45	$270
Subtotal		14	--	$630
Total	--	106.8	--	$4213

NOTE: The hourly wage rates for district administrative staff are based on mean wage rates in New Mexico reported by the Bureau of Labor Statistics (2013). For district superintendents and high school principals the mean wage for education administrators of elementary and secondary schools is used ($40.50). For school administrative staff the mean wage for office and administrative support workers is used ($17.98). For teachers the mean wage for secondary teachers is used ($26). For state data managers the mean wage for database administrators is used ($45.00).

13. Estimate of Total Annual Cost Burden to Respondents or Record-Keepers

There are no start-up costs associated with this collection.

14. Estimates of Annualized Cost to the Federal Government

The annualized cost to the federal government for all project activities is $152,666.67. The estimated total cost for the three-year project is $458,000.

15. Reasons for Program Changes or Adjustments

This is a new study.

16. Plan for Tabulation and Publication and Schedule for Project

Tabulation plans

All results for REL studies are made available to the public through peer-reviewed reports that are published by IES. Individual school reports containing school-specific data will be created and disseminated to each participating school. The data contained in the individual school reports will be publically available through ED’s contractor by request.

The extant student records data will not be turned over to the REL’s IES project officer.

Publication plans

All results for REL studies are made available to the public through peer-reviewed reports that are published by IES. There will be three reports associated with this study. A Making Connections report will present the results of the survey analysis as well as the results of analyses investigating the relationships between the measures of students’ noncognitive factors and perceptions of school environments and successful grade 9 transitions. A Stated Briefly: Survey Development and Results report will describe the process used to assess survey responses and finalize the scales included in the structural equation models. In particular, it will describe the process used to assess and finalize the survey scales. Final Rasch-produced item measures and scale reliabilities will be included in the report. This report will also provide descriptive statistics for survey responses. A Stated Briefly: Report Summary report will serve as a companion product to the Making Connections report. The goal of this brief is to summarize the findings of the Making Connections report for a nontechnical audience, including alliance members, educators, school administrators, and education policymakers.

Tabulation Plans

We will begin our analyses by assessing the measurement properties of the survey scales included on the noncognitive and school environmental factors survey. Although most of the scales on the survey have been validated in other surveys (see appendix C), we will conduct analyses to ensure that all of the items on a scale hold together and that each scale achieves a reliability rating of 0.70 or greater. Since we are conducting a field test of our newly developed survey, reliability ratings of 0.70 should be sufficient at this early stage of analysis (Nunnally, 1978). First, we will conduct our analyses using the entire sample. Then, we will separate the sample by racial/ethnic group—Hispanic, Native American, and White—and run the analyses separately by group. That is, we will run separate calibrations by subgroup and compare item functioning from these separate calibrations by subgroup, and we will examine DIF table in Winsteps. We think these steps will be particularly valuable, because our sample will contain a large number of Native American and Hispanic students—students who have been underrepresented in studies investigating noncognitive factors. Rasch modeling techniques will be employed to assess whether the items hold together as a scale, to estimate scale reliability, and to create individual summary scores for each scale.

Next, we will use results from the survey analyses to answer our research questions. To answer research questions 1 and 2, we will present descriptive statistics for students’ reports of their noncognitive skills and perceptions of their high school environments. We will also use inferential statistics to test differences in means on the survey scales between Hispanic, Native American, White and other students.

To answer research question 3, we will use multilevel structural equation modeling⁶ to investigate relationships between measures of New Mexico grade 9 students’ perceptions of their noncognitive skills, school environments, and successful transitions to high school (i.e., overall freshman GPA, number of grade 9 course failures, and freshman year attendance). The analyses will be conducted using two-level models in which students comprise level 1 and schools comprise level 2. All models will be analyzed using MPlus. A series of models will be estimated and compared in order to find the model that best fits the data. The series of models will be analyzed using weighted least squares estimation, due to facilitate the inclusion of binary variables in the models. Cross-validation of the models will be conducted by randomly splitting the overall sample into two subsamples of approximately equal size.

The project schedule is presented in Table 5.

Table 5. Schedule of Activities

Product/Activity	Date
Draft Proposal to REL Southwest COR	July, 2014
Revised Draft Proposal	October, 2014
Draft OMB Package to REL Southwest COR	February, 2015
Final Proposal	February, 2015
Notice of IRB Clearance	February, 2015
Project Start	August, 2015
Draft OMB Package Submission	March, 2015
Final OMB Package Approval	August, 2015
Noncognitive and School Environmental Factors Survey Administration	Late September/ Early October 2015
Individual School Survey Results	February, 2016
Draft Stated Briefly: Survey Development	May, 2016
Final Stated Briefly: Survey Development	August, 2016
Draft Making Connections Report to REL Southwest COR	September, 2016
Draft Stated Briefly: Report Summary to REL Southwest COR	October, 2016
Final Making Connections Report	May, 2017
Final Stated Briefly: Report Summary	September, 2017

17. Approval Not to Display the Expiration Date for OMB Approval

Approval not to display the expiration date for OMB approval is not requested.

18. Exception to the Certification Statement

No exceptions to the certification statement are being sought.

References

Allensworth, E., & Easton, J. (2007). What matters for staying on-track and graduating in Chicago Public High Schools: A close look at course grades, failure, and attendance in the freshman year. Chicago, IL: Consortium on Chicago School Research.

Balfanz, R., & Neild, N. (2006). An extreme degree of difficulty: The educational demographics of urban neighborhood high schools. Journal of Education for Students Placed at Risk, 11(2), 123-141.

Child Trends. (2013). High school dropout rates: Indicators on children and youth. Washington, DC: Child Trends.

Christie, C. A., Jolivette, K., & Nelson, C. M. (2007). School characteristics related to high school dropout rates. Remedial & Special Education, 28(6), 325–339.

Dweck, C., Walton, G., & Cohen, G. (2011). Academic tenacity: Mindsets and skills that promote long-term learning. White paper. Seattle, WA: Gates Foundation.

EPE Research Center. (2006). An essential guide to graduation policy and rates. Bethesda, MD: Education Week.

Farrington, C., Roderick, M., Allensworth, E., & Nagaoka, J. (2012). Teaching adolescents to become learners: The role of noncognitive factors in shaping school performance. Chicago, IL: Consortium on Chicago School Research.

Freiberg, H. & Stein, T. (1999). Measuring, improving, and sustaining healthy learning environments. In H. J. Freiberg (ed.) School Climate: Measuring, Improving, and Sustaining Healthy Learning Environments. Philadelphia, PA: Falmer Press.

New Mexico Public Education Department, (2014). Data Dashboards. http://ped.state.nm.us/ped/DDashIndex.html

Rumberger, R. (2004). Why students drop out of school. In Orfield, G. (Ed.), Dropouts in America: Confronting the Graduation Crisis (pp. 131-156). Cambridge, MA: Harvard Education Press.

Suh, S., & Suh, J. (2007). Risk factors and levels of risk for high school dropouts. Professional School Counseling, 10(3), 297–306.

Voight, A., Austin, G. & Hanson, T. (2013). A climate for academic success: How school climate distinguishes schools that are beating the achievement odds. San Francisco, CA: WestEd.

Attachment A-1. Parent Consent Form

See attachment

Attachment A-2. Student Survey Administration Instructions

See attachment

Attachment A-3. Student Survey

See attachment

Attachment A-4. Superintendent Email

See attachment

Attachment A-5. Superintendent Follow-up Email

See attachment

Attachment A-6. Superintendent Telephone Script

See attachment

Attachment A-7. Principal Telephone Script

See attachment

Attachment A-8. Student Records Data

See attachment

Attachment A-9. IRB Approval

See attachment

Attachment A-10. Educational Sciences Reform Act (ESRA)

See attachment

Attachment A-11. Federal Register Notices

See attachment

Attachment A-12. Confidentiality Form and Affidavits

See attachment

Attachment A-13. Technical Working Group Suggestions

See attachment

Attachment A-14. High School Website and eblast

See attachment

1 In this study, a successful transition to high school is measured by students’ grade 9 GPAs, number of absences, and number of course failures.

2 Economists generally refer to these factors or skills as noncognitive because they are not measured by commonly adminis­tered cognitive tests, such as IQ tests or academic examinations (Farrington et al., 2012); psychologists and sociologists often refer to these attitudes and behaviors as social-emotional factors.

3 When renewed, E&I’s new registration number will be provided.

4 All estimates contained in Supporting Statement A assume recruitment of no more than 6 districts and 25 high schools.

5 Collection of student names and IDs are listed under survey data collection because this data will be used to pre-populate surveys with student information, ensuring a more accurate linking between student survey and extant data.

6 A multilevel structural equation model will be used if the interclass correlations are reasonably high (above 0.05). If not, single level structural equation models will be used.

20