Technical Working Group Suggestions - Attachment A- 9

Attachment A-9. Technical Working Group Suggestions
In June 2014, Southwest REL researchers obtained comments from two TWG reviewers: Dan
Goldhaber and Geoff Borman. This Appendix summarizes in bullet form the conceptual
suggestions that Southwest REL researchers received. Line edits are omitted from this
summary. The researchers responses are noted under each bullet prefaced by “RESPONSE.”

Description of Topic and Articulation of Need
Comments:
 This requires a stronger argument. “We believe that” and “there is little reason to
expect” are not good arguments. Instead, describe what events/trends/developments
could mess things up and how your study will address these. Maybe you are doing this
later, but it’s worth doing in the introduction too.


This statement may be a bit too strong, in my opinion. This sounds like something one
might say about the experimental manipulation of such a policy, which is clearly not
the case here.
RESPONSE: We’ve added trend data for the past five years showing the percentage of
students who completed the pre-policy degree plans that required algebra II. There is
very little change across time in the percentage of students completing algebra II. We
have also altered the wording to make it sound less like we are using a quasiexperimental design.

Research Questions


I think it would be much better to focus on Enrollment and Completion rather than
Completion and Failure. The problem with failure is that you only observe it for
students who choose to enroll. That creates a huge potential selection bias.
Enrollment and Completion are both observed for everyone and by differencing them
you can back out effects on Failure without calling them that.
RESPONSE: We have added enrollment in algebra II as an outcome, and we have
changed the failure outcome. We are now looking at the percentage of students who
fail their third mathematics course, regardless of course content—so we will be looking
at the mathematics failure rates for all students, not just the ones who enroll in algebra
II.



As we discussed at the TWG meeting, if there is a way to assess whether students are
taking more “career- and vocational-type” classes after HB5, that would seem like a
key policy-relevant hypothesis to test.
RESPONSE: This is what we are planning to do as part of our descriptive analysis looking
at the types of mathematics courses that students are taking post-policy. We’ve added
some language to make this clearer.

Literature Review


This entire section is overly biased in favor of Algebra II. While I agree that HB5 is silly
and incomprehensible, you have to try to at least come up with some potential
counterarguments. Why did people want to get rid of Algebra II and dumb down their
kids? What are these other mathematics courses that kids can now take to meet the
requirements and how potentially beneficial are they? Are there any serious
education researchers who agree with Texas? Are there any potential positive effects
of HB5? (Students less likely to drop out or become discouraged, fewer underprepared students qualifying for UT, career students being better prepared through
specific courses they now have the freedom to take). Are there any states or countries
that have done something like this? What were their experiences?
RESPONSE: We have reconstructed the literature review to present better present both
sides of the algebra II debate and provide a more objective review of previous research.



I don’t think you can say that. The NUMBER of courses they take may not be affected
but the TYPE of courses could still be affected even for students who take more
courses than required.
RESPONSE: The graduation requirements at this time were simply increased; students
were required to take more courses in order to graduate. Students already taking the
required number of courses were not affected by the policy change.

Power


This would seem to be a tiny effect. What would be the practical policy relevance of
such an effect?
RESPONSE: We do not believe that an effect size this small would have practical policy
relevance. However, it is the minimum detectable effect size for this study, which we
are required to calculate per IES requirements. We have added language specifying
what we believe would be a policy relevant difference.

Outcomes Analyses


I see that success must mean attempted and passed (would any grade other than an
“F” qualify as a success?), but does failure mean not taken, or taken and not passed,
or both? I would think that taken and not passed rates would be higher pre-HB5,
because a larger and more diverse group, presumably, took the course when it was
required. How would this affect your analysis?
RESPONSE: Failure refers to not passing the course, which in Texas means earning a
grade of “F”. We will retain this definition of failure for our analyses; however, we will
look at failure rates for all students’ third mathematics course, rather than just the
failure rates for those who enroll in algebra II.

Limitations and Possible Solutions


Why use only one year? If you are going to do an interrupted time series study like
this you need to have at least a couple of pre-HB5 data points to account for history
and maturation bias problems. What if Algebra II enrollment was on an upward trend
independently already? That could cause you to draw completely the wrong
conclusion.
RESPONSE: We are not conducting an interrupted time series analysis.