NE 5.1.3 MA Supporting Statement B rev

Regional Education Laboratory Northeast & Islands

Supporting Statement B

Section B. Data Collection Procedures and Statistical Methods

1. Respondent Universe and Sampling Methods

Universe of Schools and Eligibility Requirements

Schools must meet the following eligibility criteria to participate in the study:

• They must serve grade 4 and 5 students in general education classroom settings

• They must be public schools located in Massachusetts

• They must be situated in school districts in the state of Massachusetts

• Most intervention students in a school must receive a minimum of one hour of math intervention per week for at least one quarter.

• Schools must incorporate the intervention block into an extended general math class.

Using the Common Core of Data (CCD), approximately 525 public schools in Massachusetts met the eligibility criteria at the end of the 2021 school year.

Universe of Teachers Within Participating Schools

To be eligible to participate in this study, teachers must meet the following eligibility criteria:

• They must teach a math intervention classes for grade 4 and 5 students

Among the 60 participating schools, all math intervention teachers will be recruited. This amounts to a sample of approximately 90 teachers. Teachers considering participation in the study will indicate their willingness to provide logistical support for the study’s data collection activities. Teacher measures will be administered to all participating teachers. No sampling of teachers will take place. We expect close to 100% response rates because teachers are opting into the study and intervention teachers often have fewer PD opportunities than general education teachers.

Universe of Students Within Participating Schools

The evaluation team will include all grade 4 and 5 students in participating teachers’ math intervention classes. Students who score in the bottom two levels on the state standardized math assessment—Not Meeting Expectations or Partially Meeting Expectations—are typically designated as in need of intervention. We expect this to amount to 1,800 students across all 60 schools. No sampling of students will take place, and no subgroups will be excluded. Student-level measures will be collected during regular school hours. Student nonresponse could occur if a student is absent on the days of the student survey and math assessment, if he or she leaves the school during the study year, or if a child’s parents do not give consent for the student to participate. It is expected that parents will not have significant concerns that will result in withdrawal of their children from the study. Therefore, we expect response rates of at least 85%.

2. Procedures for Collection of Information

The study will include all schools that meet eligibility criteria and have at least one teacher that volunteered to participate in the study, estimated to be 60 schools in total. The types of study data to be collected are:

1. Student responses to three measures, collected electronically

2. Teacher responses to five measures, collected electronically

3. Administrative student records

4. Administrative teacher records

5. Attendance and archival data related to the professional development course intervention participation, engagement, and implementation

6. Teacher observations

Data collection processes are further elaborated in the following sections.

Preliminary Activities

The WestEd study team and REL-NEI staff will collaborate on project recruitment activities. After the sample has been recruited, the WestEd study team will communicate the data expectations to schools and obtain staff and parental consent.

1. Communicating data needs to schools. The WestEd study team will create and share a schedule of all study data collection activities with participating teachers and other relevant staff. As needed, the study team will conduct a phone or video conference with the point of contact at each school to elaborate on the data collection schedule and confirm the presence of necessary student records.

2. Obtain the necessary school staff and parental consent. The processes for obtaining consent from these two groups are as follows:

   1. Teacher consent: The study team will share “Active consent” with prospective study teachers. This electronic form (See Attachment #) informs teachers of the study’s purpose, the data collection activities in which they will participate, the risks and benefits of participation, their freedom to withdraw from the study without repercussions, and contact information for the Institutional Review Board and the principal investigator. Teachers have the choice of indicating “Yes, I agree to participate” or “No, I do not wish to participate.” Teachers will complete these forms electronically, with a copy of the form and their response automatically emailed to them upon completion for their own records. After all teachers have returned their consent forms, the WestEd study team will request the state-issued unique identifier for teacher records. These unique identifiers will allow the study team to merge teacher data from administrative records with primary data collected by the research team, and also determine response rates for teacher data collections. A separate crosswalk data file containing teachers’ names and their ID numbers. This file will be encrypted, and only shared with study team staff who are (1) e-mailing staff to request that they complete a data collection instrument, or (2) processing the data provided by the Massachusetts Department of Elementary and Secondary Education (DESE).

   2. Parental consent: In August of 2024, the WestEd study team will identify whether study school districts require active parental consent. For school districts that do not require active parental consent, the study team will provide the district’s schools with the necessary “opt out” information letters (passive consent documents; see Attachment #) to distribute to all guardians of grade 4 and 5 students identified for math intervention. The letters give parents all of the information found on a consent form and allow parents to “opt out” by indicating they do not consent to any collection of data from their child for study purposes. The study team received IRB approval for passive consent in cases where the school districts themselves do not require active consent under the justification that research is exempt because it involves established educational settings and normal educational practices (i.e. professional development) that are not likely to adversely impacts students' opportunities to learn required educational content. Additionally, requiring active consent where not necessary increases burden on parents and can jeopardize sample size, and thus, by extension, ability to detect significant effects if the intervention has a positive impact on student outcomes. For school districts that do require active parental consent, the WestEd study team will provide study schools with an active consent form (see Attachment #). The form includes the same parent information letter provided in the passive consent form but indicates that parents need to consent to any collection of data from their child for study purposes. Schools that use passive consent forms will collect and keep the letters that are returned, and in mid-September 2024 they will provide the study team with the state issued unique identifier for students in grade 4 and 5 whose guardians did not withhold consent for their child to participate in the study. Similarly, schools that were given active consent forms will collect and keep the letters that are returned, and in mid-September 2024 they will provide the study team with the state issued unique identifier for students in grade 4 and 5 whose guardians provided consent for their child to participate in the study. These unique identifiers will allow study team staff to merge student data from administrative records with primary data collected by the research team, and also determine response rates for student data collections. The study will create a separate crosswalk data file containing student ID numbers and their associated teacher and school. This encrypted file will only be shared with study team staff processing the data provided by the DESE.

Collection of Teacher Measures

All teachers participating in the study will complete each measure electronically. The monthly logs and the pre/post surveys will be completed electronically via SurveyMonkey. Participating teachers will receive an email with a unique link to complete these measures in October 2024 and again in April 2025. Intervention teachers will receive links to the instructional logs following the expected completion of each module in the PD course, which will occur between October 2024 and April 2025.

Collection of Student Measures

The WestEd study team will work with participating teachers to coordinate the administration of student measures. The math assessment (Test of General Fraction Knowledge), self-efficacy, and teacher instructional practices measures will all be completed electronically via SurveyMonkey. Students whose guardians did not withhold consent for their child to participate in the study will receive an email with a unique link to complete these measures in October 2024 and again in April 2025. School-issued email addresses will be used to ease the logistical burden on participating teachers to administer these measures. All measures will be administered electronically, unless technology resource constraints lead a participating teacher to request paper versions of measures to administer to their students.

Administrative Data Request

The Wested study team will work with the state to execute a data sharing agreement for the administrative records request. It is expected that this data sharing agreement will be executed in December 2024. After this agreement is in place, DESE staff will assemble two data files, one containing teacher level records, and another containing student-level records. They can then upload the data files to the WestEd study team’s secure FTP site.

3. Methods to Maximize Response Rates and To Deal With Non-Response

The ED seeks complete data for this study. To ensure the acquisition of complete data, the WestEd study team will maintain e-mail and telephone contact with the appropriate parties in participating sites. All data received for the study will be reviewed for completeness. When data contain too many missing values, the study team will reach out to designated staff at schools or state-level educational organizations to ascertain a more complete understanding of the sources of missing data elements.

Because teachers are volunteering for the study and receiving incentives for participation, the study team expects high response rates and will follow up as needed to ensure adequate response rates of 85% percent or higher, consistent with the National Center for Education Statistics’ Statistical Standards (Seastrom, 2002). Student nonresponse might occur if a student is absent on the day of the student survey and readiness assessment, if he or she leaves the school during the study year, or if a child’s parents do not give consent for the student to participate. It is expected that parents will not have significant concerns that will result in withdrawal of their children from the study.

The study team will take several additional steps to maximize response rates. The study team will notify teacher participants and student guardians about the study before it is implemented. The staff consent form will provide (1) teacher participants with full knowledge of the data collection activities involving them and the financial compensation for the data collection burden; and (2) assurance that the data will remain confidential. The study team will provide staff members will copies of the consent form, and the form will list contact information to receive additional information about the study or to resolve any questions. The guardian information letter and consent form will provide parents with full knowledge of the data collection activities involving students and the assurance that data will remain confidential. The study team will track teacher and student completion rates and follow up with teachers who have not completed data collection activities – including teacher and student measures – in the allotted timeframes.

In the event that a response rate lower than 85 percent is achieved, the study team will compare the characteristics of respondents with those of nonrespondents to determine bias. If it proves difficult to compare staff characteristics on the teacher survey, the study team will assess the extent to which schools with high response rates differ from schools with low response rates by testing (using a t-test) for statistical differences in school-level characteristics such as percentage of multilingual learners. For the analysis of student outcomes, a bias analysis will also be conducted for response rates lower than 85 percent. These analyses will compare characteristics from administrative data of students who responded to the measure with those who did not. The study team will use univariate statistical analysis (t-tests) to compare respondents on individual characteristics (e.g., the test scores of students who did and did not respond), and a multivariate statistical analysis (logistic regression analysis) to determine whether a combination of student characteristics predicts nonresponse.

All study reports will clearly indicate any differences found between respondents and nonrespondents.

4. Tests of Procedures or Methods to be Undertaken

The study employs measures that have been previously utilized in empirical research, with existing evidence of reliability and validity (Charalambous et al., 2020; Ferguson & Danielson, 2015; Seethaler et al., 2023; Le et al., 2006; Page et al., 2014; Panorama Education, 2020). With exception to measures related to fidelity of implementation, none of the outcome measures administered in this study were developed specifically for the WWC practice guide or the toolkit or for the purposes of this study.

The implementation study includes data collection using a service contrast embedded in the teacher background survey, archival data collected from PD course modules, observations of teachers, and a teacher instructional log that will follow the completion of each module. The survey protocols were reviewed by the PD developers at EDC to ensure these implementation measures align with the PD course.

5. Individuals Consulted on Statistical Aspects of the Design

Several individuals consulted on the statistical, data collection, and analytic aspects of the proposed Mathematics Intervention Toolkit for the Elementary Grades Efficacy Study:

• Kirk Walters, Senior Managing Director, WestEd

• Scott Strother, Senior Research Associate, WestEd

• Mingyu Feng, Senior Research Associate, WestEd

• Noman Khanani, Research Associate, WestEd

• Amy Brodesky, Project Director, Education Development Center

• Sarah Ryan, Research Scientist, Education Development Center

• Mark Dynarski, Founder, Pemberton Research

Electronic file transfers will be conducted by

• Noman Khanani, Research Associate, WestEd

Online data collection and analysis to be overseen by

• Noman Khanani

• Maria Salciccioli

6. References

Charalambous, C. Y., Hill, C. Y., & Chin, M. (2020). Mathematical content knowledge and knowledge for teaching: Exploring their distinguishability and contribution to student learning. Journal of Mathematics Teacher Education, 23, 679–613. https://doi.org/10.1007/s10857-019-09443-2

Ferguson, R., & Danielson, C. (2015). How framework for teaching and tripod 7Cs evidence distinguish key components of effective teaching. In T. J. Kane, K. A. Kerr, & R. C. Pianta (Eds.), Designing teacher evaluation systems: New guidance from the Measures for Effective Teaching Project (Chapter 4). John Wiley & Sons. https://doi.org/10.1002/9781119210856.ch4

Seethaler, P.M., Espinas, D.R., and Fuchs, L.S. (2023). Fourth-grade fractions battery. Vanderbilt University.

Le, V. N., Stecher, B. M., Lockwood, J. R., Hamilton, L. S., & Robyn, A. (2006). Improving mathematics and science education: A longitudinal investigation of the relationship between reform-oriented instruction and student achievement. RAND Corporation. https://www.rand.org/pubs/monographs/MG480.html

Page, S., Pendergraft, B., & Wilson, J. (2014). Examining elementary teachers’ sense of efficacy in three settings in the Southeast. Journal of Inquiry and Action in Education, 5(3). https://digitalcommons.buffalostate.edu/jiae/vol5/iss3/3

Panorama Education (2020). Reliability and validity of panorama’s survey topics for students: 2020 udpate.

Seastrom, M. M. (2002). NCES statistical standards. Washington, DC: US Government Printing Office, US Department of Education, Institute of Education Sciences. Retrieved October 4, 2022.