Impact Evaluation of Math Professional Development

Appendix B

Data Collection Instruments

Appendix B-1

Math Professional Development Study

End-of-Year Teacher Survey

June 2014

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid Office of Management and Budget (OMB) control number. The valid OMB control number for this information collection is XXXX-XXXX. The time required to complete this information collection is estimated to average 30 minutes per response. This information collection is voluntary. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202–4651. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Thomas Wei, U.S. Department of Education, Institute of Education Sciences, Room 500H, 555 New Jersey Ave. NW, Washington, DC 20208-5500.

Dear Teacher:

The Math Professional Development Study (MPD) is an important national study designed to test innovative models of professional development for mathematics instruction in the fourth grade. Your participation is important and appreciated, but you do have the right to skip any question that you do not wish to answer. The following are responses to some general questions concerning your participation.

What is the purpose of this survey?

The purpose of this survey is to obtain background information about the teachers within the study as well as information about their instructional contexts and professional development experiences in mathematics.

Who is conducting this survey?

The U.S. Department of Education’s Institute of Education Sciences commissioned the MPD study, and American Institutes for Research is administering the study survey.

Why should you participate in this survey?

Policymakers and educational leaders rely on findings from studies, such as the MPD study, to inform their decisions. The current study will fill a critical gap in the mathematics and professional development research base regarding what is effective for improving fourth-grade mathematics achievement.

Will your responses be kept confidential?

American Institutes for Research follows the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for statistical purposes. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific school or individual. We will not provide information that identifies respondents to anyone outside the study team, except as required by law. Additionally, no one in your school or district will see your responses.

How will your information be reported?

The information you provide will be combined with the information provided by others in statistical reports. No individual data that link your name or e-mail address with your responses will be included in the statistical reports.

Thank you for your cooperation in this very important effort!

Professional Development Experiences

Related to Mathematics and Mathematics Teaching and Learning

Professional development refers to a variety of activities intended to enhance your professional knowledge and skills, including workshops, seminars, institutes, college courses, coaching, mentoring, teacher networks, getting observed and receiving feedback on your teaching, and committee work.

1. Counting only professional development activities that focused on mathematics or mathematics teaching and learning, what is the total number of hours you participated in each of the following activities during summer 2013 and the 2013-2014 school year?

For each activity, write the total number of HOURS you spent, rounding up to the nearest whole hour.

Mark “0” if you participated in none.

Please do NOT count the same hours in more than one row.

Professional Development Activity	Hours of Professional Development
a. Attended traditional course/workshop/seminar-style professional development that focused on mathematics or mathematics teaching and learning (e.g., Intel Math)		hours
b. Participated in collaborative professional learning communities or planning groups with other teachers that focused on mathematics or mathematics teaching and learning (e.g., Mathematics Learning Communities)		hours
c. Classroom teaching was observed and was given feedback that focused on mathematics or mathematics teaching and learning by someone with knowledge of mathematics instruction (e.g., a coach, department chair, or mentor)		hours
d. Participated in other types of professional development that focused on mathematics or mathematics teaching and learning. Please explain: ______________________________________________________		hours

The next ten questions focus on the professional development activities you reported in Question 1.

If you responded with a number greater than 0 to Question 1 Row a (attended traditional course/workshop/seminar-style professional development activities that focused on mathematics or mathematics teaching and learning), please respond to Questions 2, 3, and 4.

If you responded with 0 hours to Question 1 Row a, please skip to Question 5.

2. In your traditional course/workshop/seminar-style professional development activities that focused on mathematics or mathematics teaching and learning during summer 2013 and the 2013-2014 school year, how often did you do the following activities?

Check one box for each row.

	Activity During Professional Development	Never/ Rarely	Sometimes	Often	Most or All of the Time
a.	Solve mathematics problems				
b.	Share your solutions to mathematics problems with other teachers in small- or large-group settings				
c.	Discuss mathematics topics that appear in mathematics curricula above or below the current grade you teach				
d.	Explore the conceptual underpinning of K–8 mathematics concepts (e.g., understanding why the multiplication algorithm works)				
e.	Practice using a variety of representations to illustrate a given mathematical concept				
f.	Practice analyzing student work on mathematics problems				
g.	Practice writing story problems for students you teach				
h.	Explore the connections between mathematics topics and solution methods				
i.	Explore the ways students commonly approach mathematics problems				
j.	Explore common student misconceptions and errors in mathematics				

3. In your traditional course/workshop/seminar-style professional development activities that focused on mathematics or mathematics teaching and learning during summer 2013 and the 2013-2014 school year, how often were the following topics a focus?

Check one box for each row.

	Topic Area	Never/ Rarely	Sometimes	Often	Most or All of the Time
a.	Addition, subtraction, multiplication, and division				
b.	Connections between addition, subtraction, multiplication, and division				
c.	Fractions and operations with fractions				
d.	Algebra				
e.	Connections between algebra and arithmetic				
f.	Decimals				
g.	Exponents				

4. To what extent do you agree or disagree with the following statements about your experiences in your traditional course/workshop/seminar-style professional development activities that focused on mathematics or mathematics teaching and learning during summer 2013 and the 2013-2014 school year?

Check one box for each row.

	Your Experience Was…	Strongly Disagree	Somewhat Disagree	Somewhat Agree	Strongly Agree
a.	Consistent with your own goals for your professional development				
b.	Complementary to your use of district-adopted curricular materials				
c.	Related to the mathematics content you taught this year				
d.	Logically connected from one day or session to the next				
e.	Clear about how you could use what you learned from the professional development experience in your classroom				
f.	Focused on practices that district or school leaders expect you to demonstrate in your classroom				

If you responded with a number greater than 0 to Question 1 Row b (attended collaborative professional learning communities or planning group activities that focused on mathematics or mathematics teaching and learning), please respond to Questions 5, 6, and 7.

If you responded with 0 hours to Question 1 Row b, please skip to Question 8.

5. In your collaborative professional learning communities or planning group activities that focused on mathematics or mathematics teaching and learning during summer 2013 and the 2013-2014 school year, how often did you do the following activities?

Check one box for each row.

	Activity During Professional Development	Never/ Rarely	Sometimes	Often	Most or All of the Time
a.	Solve mathematics problems				
b.	Share your solutions to mathematics problems with other teachers in small- or large-group settings				
c.	Discuss mathematics topics that appear in mathematics curricula above or below the current grade you teach				
d.	Explore the conceptual underpinning of K–8 mathematics concepts (e.g., understanding why the multiplication algorithm works)				
e.	Practice using a variety of representations to illustrate a given mathematical concept				
f.	Practice analyzing student work on mathematics problems				
g.	Practice writing story problems for students you teach				
h.	Explore the connections between mathematics topics and solution methods				
i.	Explore the ways students commonly approach mathematics problems				
j.	Explore common student misconceptions and errors in mathematics				

6. In your collaborative professional learning communities or planning group activities that focused on mathematics or mathematics teaching and learning during summer 2013 and the 2013-2014 school year, how often were the following topics a focus?

Check one box for each row.

	Topic Area	Never/ Rarely	Sometimes	Often	Most or All of the Time
a.	Addition, subtraction, multiplication, and division				
b.	Connections between addition, subtraction, multiplication, and division				
c.	Fractions and operations with fractions				
d.	Algebra				
e.	Connections between algebra and arithmetic				
f.	Decimals				
g.	Exponents				

7. To what extent do you agree or disagree with the following statements about your experiences in collaborative professional learning communities or planning groups that focused on mathematics or mathematics teaching and learning during summer 2013 and the 2013-2014 school year?

Check one box for each row.

	Your Experience Was…	Strongly Disagree	Somewhat Disagree	Somewhat Agree	Strongly Agree
a.	Consistent with your own goals for your professional development				
b.	Complementary to your use of district-adopted curricular materials				
c.	Related to the mathematics content you taught this year				
d.	Logically connected from one day or session to the next				
e.	Clear about how you could use what you learned from the professional development experience in your classroom				
f.	Focused on practices that district or school leaders expect you to demonstrate in your classroom				

If you responded with a number greater than 0 to Question 1 Row c (feedback you received after a video or in-person observation of your classroom teaching that focused on mathematics or mathematics teaching and learning), please respond to Questions 8.

If you responded with 0 hours to Question 1 Row c, please skip to Question 9.

8. In the feedback you received after a video or in-person observation of your classroom teaching that focused on mathematics or mathematics teaching and learning during summer 2013 and the 2013-2014 school year, how often were the following topics a focus of the feedback?

Check one box in each row

	Topic Area	Never/ Rarely	Sometimes	Often	Most or All of the Time
a.	How you made connections between mathematics topics and/or representations				
b.	How you conveyed the meaning of mathematical procedures				
c.	How you remediated student errors				
d.	How you responded to student thinking				
e.	The clarity, precision, and correctness of your mathematics presentations				
f.	Providing you with useful feedback about your teaching of mathematics				

If you responded with a number greater than 0 to Question 1 Row d (other types of professional development that focused on mathematics or mathematics teaching and learning, i.e., excluding traditional course/workshop/seminar-style professional development, collaborative professional learning communities or planning group activities, and classroom observations and feedback), please respond to Questions 9, 10, and 11.

If you responded with 0 hours to Question 1 Row d, please skip to Question 12.

9. In your other types of professional development (i.e., excluding traditional course/workshop/seminar-style professional development, collaborative professional learning communities or planning group activities, and classroom observations and feedback) that focused on mathematics or mathematics teaching and learning during summer 2013 and the 2013-2014 school year, how often did you do the following activities?

Check one box for each row.

	Activity During Professional Development	Never/ Rarely	Sometimes	Often	Most or All of the Time
a.	Solve mathematics problems				
b.	Share your solutions to mathematics problems with other teachers in small- or large-group settings				
c.	Discuss mathematics topics that appear in mathematics curricula above or below the current grade you teach				
d.	Explore the conceptual underpinning of K–8 mathematics concepts (e.g., understanding why the multiplication algorithm works)				
e.	Practice using a variety of representations to illustrate a given mathematical concept				
f.	Practice analyzing student work on mathematics problems				
g.	Practice writing story problems for students you teach				
h.	Explore the connections between mathematics topics and solution methods				
i.	Explore the ways students commonly approach mathematics problems				
j.	Explore common student misconceptions and errors in mathematics				

10. In your other types of professional development (i.e., excluding traditional course/workshop/seminar-style professional development, collaborative professional learning communities or planning group activities, and classroom observations and feedback) that focused on mathematics or mathematics teaching and learning during summer 2013 and the 2013-2014 school year, how often were the following topics a focus?

Check one box for each row.

	Topic Area	Never/ Rarely	Sometimes	Often	Most or All of the Time
a.	Addition, subtraction, multiplication, and division				
b.	Connections between addition, subtraction, multiplication, and division				
c.	Fractions and operations with fractions				
d.	Algebra				
e.	Connections between algebra and arithmetic				
f.	Decimals				
g.	Exponents				

11. To what extent do you agree or disagree with the following statements about your experiences in other types of professional development (i.e., excluding traditional course/workshop/seminar-style professional development, collaborative professional learning communities or planning group activities, and classroom observations and feedback) that focused on mathematics or mathematics teaching and learning during summer 2013 and the 2013-2014 school year?

Check one box for each row.

	Your Experience Was…	Strongly Disagree	Somewhat Disagree	Somewhat Agree	Strongly Agree
a.	Consistent with your own goals for your professional development				
b.	Complementary to your use of district-adopted curricular materials				
c.	Related to the mathematics content you taught this year				
d.	Logically connected from one day or session to the next				
e.	Clear about how you could use what you learned from the professional development experience in your classroom				
f.	Focused on practices that district or school leaders expect you to demonstrate in your classroom				

Mathematics Instruction

12. How much total time did you spend teaching mathematics each day to your regular fourth-grade class LAST WEEK?

If last week was not a typical week or did not have a full schedule of classes, answer for the most recent week that did.

Monday:		Minutes teaching mathematics
Tuesday:		Minutes teaching mathematics
Wednesday:		Minutes teaching mathematics
Thursday:		Minutes teaching mathematics
Friday:		Minutes teaching mathematics

Certification

13. Which of the following describes the teaching certificate you held in June 2013 (i.e., the beginning of the summer prior to this past school year) in this state?

Check only one box.

	Regular or standard state certificate or advanced professional certificate
	Certificate issued after satisfying all requirements except the completion of a probationary period
	Certificate that requires some additional coursework, student teaching, or passage of a test before regular certification can be obtained
	Certificate issued to persons who must complete a certification program in order to continue teaching
	I do not have any of the above certifications in THIS state

14. Did you enter teaching through an alternative certification program? (An alternative program is a program that was designed to expedite the transition of non-teachers to a teaching career, for example, a state, district, or university alternative certification program.)

 Yes

 No

Education and Preservice Training

15. By June 2013 (i.e., the beginning of the summer prior to this past school year) had you earned any of the following degrees, diplomas, or certificates?

Please check No or Yes in each row then, if applicable, write in the major code from the List of Codes that follows.

If you have earned more than one credential per row, report the one most relevant to your current teaching assignment (e.g., if you have earned two different master’s degrees, use row b to report the master’s degree that is most relevant to your current teaching assignment).

	Degree	Earned	Major Field Code (From List of Codes)
a.	Bachelor’s degree	 No  Yes 
b.	Master’s degree	 No  Yes 
c.	Educational specialist, professional diploma, or certificate of advanced graduate studies (at least one year beyond master’s level)	 No  Yes 
d.	Doctorate or professional degree (e.g., Ph.D., Ed.D., M.D., L.L.B, J.D., D.D.S.)	 No  Yes 

List of Codes: Major Field of Study

Code	Major Field
01	Elementary education or early childhood education
02	Secondary or middle grades education, general
03	Mathematics education
04	Mathematics
05	Other mathematics-related subject (e.g., statistics, economics, engineering)
06	English language arts
07	Natural science (e.g., biology, chemistry, physics, earth science)
08	Social science (e.g., anthropology, geography, history, psychology)
09	Special education
10	English as a second language
11	Other (e.g., foreign language, health education, business, computer science)

16. By June 2013 (i.e., the beginning of the summer prior to this past school year) had you completed at least one college course (undergraduate or graduate) in any of the following mathematics or mathematics-related subjects?

Please check the number of courses completed in each row. Do not count any course twice.

Include Advanced Placement courses completed in high school for which you received college credit.

Content of Course		Number of Courses Completed
		0	1	2	3 or More
a.	College algebra (not abstract or linear algebra)				
b.	Geometry or trigonometry				
c.	Precalculus				
d.	Calculus				
e.	Mathematics for elementary teachers (i.e., a course for teachers in which the majority of time was focused on mathematics, not on how to teach mathematics or how students learn mathematics)				
f.	Mathematics education or methods of teaching mathematics (i.e., a course for teachers in which the majority of time was focused on how to teach mathematics and how students learn mathematics, not mathematics, itself)				
g.	Statistics or probability, including statistical methods in the social sciences (e.g., econometrics or psychometrics)				
h.	Other mathematics or mathematics-related courses (e.g., abstract algebra, linear algebra, differential equations, number theory, mathematical logic, set theory, foundations of mathematics). Please specify: _______________________________________				

Experience

17. How many years of teaching experience do you have in each of the following settings?

For each row, write the number of years, using whole numbers only.

Count school year 2013-2014 as one full year.

	Setting	Number of Years
a.	Teaching in total		Years
b.	Teaching at this school		Years
c.	Teaching an elementary school class where your responsibilities included instruction in multiple subjects, including mathematics		Years
d.	Teaching an elementary class where your responsibilities included solely mathematics instruction		Years
e.	Teaching mathematics content that would be considered by your district to be above the fourth-grade curriculum		Years

Demographics

18. Are you male or female?

 Male

 Female

19. Are you of Hispanic or Latino origin?

 Yes

 No

20. What is your race?

Check one or more box.

 White

 Black or African American

 Asian

 Native Hawaiian or other Pacific Islander

 American Indian or Alaska Native

Appendix B-2

Impact Evaluation of Math Professional Development

District Archival Records Collection Protocol

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is XXXX–XXXX. This information collection is voluntary. The estimated time required to complete this information request is 20 hours per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202–4537. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Thomas Wei, U.S. Department of Education, Institute of Education Sciences, Room 500H, 555 New Jersey Avenue, NW, Washington, DC 20208-5500.

District Archival Records Collection Protocol

The Math Professional Development Study requests district administrative records for the students in participating teachers’ classrooms.

In this protocol we specify the data elements requested for all students in the participating teachers’ classrooms at three points in time: (1) summer/fall 2013 when classroom assignments are formed; (2) in March 2014; and (3) at the time that the spring 2014 state assessment is administered.

We will work with the district to determine the most efficient process for providing the requested data. The data elements that will be requested are listed below.

1. Classroom Roster Records

At three points in time: (1) summer/fall 2013 when classroom assignments are formed; (2) in March 2014; and (3) at the time that the spring 2014 state assessment is administered, the study team will request files that include the following:

• Date

• School

• Teacher ID (participating teachers)

• Student ID (students in participating teachers’ classrooms)

Student records will be requested for the students in these classroom roster records, as specified in the next section.

2. Student Records

Student assessment and demographic records will be used to describe the student sample, conduct baseline equivalence analyses, and conduct impact analyses for study reports. More specifically, the purpose of the archival student achievement and demographic records is to provide:

• Demographic variables used to describe the characteristics of students who were in the participating teachers’ classes;

• Demographic and prior achievement variables used to test for baseline equivalence and as covariates (i.e. control variables) in analyses of the impact of the study’s PD intervention on student achievement scores.

• Achievement variables (i.e., students’ fourth grade mathematics achievement on state assessment) used as outcome measures in analyses of the impact of the study’s PD intervention on student achievement scores.

Requested Data Elements in Student Achievement and Demographic Records

1. Identifiers

A student identifier is needed to link student data across years and to link data across files if achievement, demographic, and other data are provided in separate files. The identifier need not be the actual identification (ID) number used by the district. We anticipate creating a “crosswalk” file, which contains the study ID linked to local IDs and would remain within the district. This file would permit the necessary links while at the same time ensuring that student names and local IDs do not leave the district office. In addition, by making it possible for the project team to carry out the linking of files, an identifier would reduce the burden on the district. We plan to work with the districts involved in the study to develop a strategy for collecting the required achievement data that meets the needs of the study and is feasible for the districts, while meeting confidentiality requirements.

2. Student Background Variables

• Month and year of birth (MM/YYYY)

• Gender

• Ethnicity (current U.S. census categories)

• Race

• Enrolled grade in SY2012-13 and SY2013-14

• National School Lunch Program status in SY2012-13 and SY2013-14

• Free

• Reduced

• Neither

• English language learner status in SY2012-13 and SY2013-14

• Individualized Education Plan status in SY2012-13 and SY2013-14

• School entry and exit dates in SY2012-13 and SY2013-14

• Attendance for SY2012-13 and SY2013-14

• Number of days absent

• Number of days attended

3. Student Achievement Variables for SY2012-13 and SY2013-14

• Mathematics score code (e.g., A = absent, D = no info for student)

• Mathematics language version (e.g., E = English, S = Spanish)

• Mathematics test version (e.g., S = regular, M = modified, T = alternate)

• Mathematics performance level (provide corresponding level description)

• Mathematics scale score

• Mathematics scale score standard error of measurement

Student records will be requested for all students in the participating teachers’ classrooms at the three specific timepoints: (1) summer/fall 2013 when classroom assignments are formed; (2) in March 2014; and (3) at the time that the spring 2014 state assessment is administered.

We know that the SY2013-14 achievement and attendance data will not be available until summer 2014.

As noted earlier, we will work with each district to determine the most efficient process for providing the requested data.