TWG Suggestions

Attachment A-13. Technical Working Group Suggestions

REL Southwest

Project Title: 4.7 Assessing the Role of Noncognitive and School Environmental Factors in Students’ Transitions to High School in New Mexico

Description of Topic and Articulation of Need

Comment Hans Bos:

I think this is a very good idea but would this really help close the achievement gap as it is normally defined? If anything, reducing dropout will increase the achievement gap in subsequent grades because you’re adding more marginal students. This is something to acknowledge in a footnote.

RESPONSE: The focus of this study isn’t necessarily on reducing dropout. It is on investigating whether noncognitive factors are related to measures of success in grade 9, which include grade 9 attendance, grade 9 course failures, and grade 9 GPA. The idea is to look at ways to help students improve their academics, not to keep marginal kids in school. Thus, this study does target the achievement gap as it is normally defined.

Research Questions

Comment Hans Bos:

I think you should add a question that looks at the correlation between 1 and 2 before you start correlating them separately with successful transitions.

RESPONSE: We are conducting analyses that look at correlations between students’ perceptions of their noncognitive skills and school environmental factors. And, these are modeled in the structural equation model. These analyses are described in the analysis section. I’m not sure that there needs to be a separate research question focused on this.

Literature Review

Comment Hans Bos:

Just acknowledge that that is no evidence per se of the importance of the transition. The point is that it is difficult to catch up from a bad ninth grade but that does not mean that earlier grades are somehow less important.

RESPONSE: I’m not sure that I understand this comment. Throughout his comments, Dr. Bos seems to be implying that the transition to high school is not important. We don’t believe that earlier grades are not important. However, research shows, and is cited later in this paper, that even students who performed well in earlier grades do not perform well in high school. Students do not perform poorly in high school just because they are not academically prepared.

Comments Hans Bos:

Now you’re getting somewhere. So how about 8th grade indicators?

RESPONSE: We originally proposed a study that would look at relationships between grade 8 achievement indicators and grade 9 outcomes and were told by our COR that this has already been done.

Comment Hans Bos:

I think it would also be interesting to know what percentage of those grade 8 stars fall into this category. Is it 5% or 25% or more? It makes a big difference whether you are talking about a couple of students who fall off the wagon or a major shift in on track status between Grade 8 and 9. The implications of the latter would be much more serious!

RESPONSE: This is difficult to respond to because the on-track indicators exist only for high school students, as high school graduation has different requirements than middle school. Also, we have been told not to look at specifically at relationships between grade 8 achievement and measures of a successful grade 9 transition in this study.

Comment Hans Bos:

Sorry, for that argument to hold, being “on track” in a low-performing middle school would have to be the same thing as being “on track” in a high-performing school.

RESPONSE: I’m not sure whether or not Dr. Bos is implying that being on-track means something different at low-performing and high-performing schools. If he is, then we disagree. In our experience, and in the literature that we have read, on-track has one, consistent meaning across schools.

Comment Hans Bos:

I think you need to be careful not to mix up attitudes and skills here. They are not equivalent even if they may be related in practice. The ability to focus and prioritize, for example, is something else than the willingness to focus on learning or the belief that education is important.

RESPONSE: This is true. However, in the literature, attitudes and skills have been grouped together under the term noncogntive factors.

Comment Hans Bos:

I liked this section until this sentence. I again think you are mixing concepts here. If you are in a bad school or have a bad teacher, it is quite possible to be a student with a growth mindset who believes that her classwork is useless. Some of the most accomplished people would fall into this category.

RESPONSE: The sentence referred to above is this: “Likewise, research shows that when students believe their classwork has a purpose and has relevance for their future, they are more likely to engage in academic behaviors, employ learning strategies, and demonstrate self-control and grit (Wigfield & Eccles, 1992, 2000).” While I agree that it is true that students who have growth mindsets may have bad teachers and believe their coursework is useless, the sentences states that students with growth mindsets are more likely to do these things, not that all students with growth mindsets do. Regardless, there is research by Wigfield and Eccles that demonstrating relationships between growth mindsets and these other noncogitive factors, it is not a personal opinion.

Comment Hans Bos:

I think this section is strong but one thing you may want to consider is that academic behaviors are more proximal to academic outcomes than the attitudes that accompany these behaviors. IOW, students first feel that it matters to show up and then they show up and then it turns out that they were right and it mattered.

RESPONSE: Yes. Academic behaviors are likely to be stronger predictors of grade 9 outcomes than some of the other types of noncognitive factors.

Comment Hans Bos:

Not being a native English speaker, I always wonder what this word really means. Maybe define it? Is it similar to resilience?

RESPONSE: Dr. Bos is referring to the word “grit”. It has now been better defined in the proposal.

Comment Hans Bos:

This section needs a lot of work or should be dropped. Most importantly, I think that most learning strategies are very much cognitive and can be taught and practiced.

RESPONSE: This section has been restructured.

Comment Hans Bos:

This sentence sounds a bit strange. I think learning strategies are more than a demonstration of non-cognitive factors. I see them more as a mediating outcome between “non-cognitive factors” and academic success.

RESPONSE: Yes. The sentence was unclear and has been rewritten.

Comment Hans Bos:

Those are just good old peer effects. I don’t think this has anything to do with a sense of belonging or social skills. In fact, many high performing high school students find peers who share their feeling that high school is mostly stupid and a waste of time.

RESPONSE: The sentence Dr. Bos is referring to has been deleted.

Comment Hans Bos:

I agree with most of what you say here but you do put a lot of stuff in one short paragraph. I think you need to disentangle some of these things and also discuss more explicitly how they are perceived by disadvantaged students. Fairness, inclusiveness, and support can feel very different to an academically struggling student entering ninth grade than to a student who has been successful and well-adjusted through middle school. The challenge is to create a supportive and effective environment for at-risk students. Even the best high schools often have difficulty with this.

RESPONSE: This section has been expanded. We are unaware of any empirical literature that suggests that fairness, inclusiveness, and support can feel very different to an academically struggling student entering ninth grade than to a student who has been successful and well-adjusted through middle school.

Study Design and Data

Comment Geoffrey Borman:

CLARIFY: This seems like an important caveat you mention here. These discussions are anecdotal, but are there clear interpretations for how these cultural differences could affect your measurement approach? Is there existing research evidence that might help navigate these potential differences and, potentially, adjustments to the survey and overall measurement approach?

RESPONSE: We have worked with alliance members to ensure that the survey items are appropriate for students from Hispanic and Native American backgrounds. We have also conducted cognitive interviews with grade 9 Hispanic, Native American, and White students to make sure that the survey items are understandable and relevant to them. There is not much literature available on Hispanic and Native American students’ noncognitive skills.

Comment Hans Bos:

Why not statewide or at least in as many districts as possible? I think you’re being too modest here. Especially the school environment variables could benefit from having more than 10-15 degrees of freedom!

RESPONSE: We do not believe that it would be feasible to administer this survey statewide. Until we have evidence showing that the scales on our survey are related to measures of successful grade 9 outcomes, we do not believe it will be possible to convince the state to administer the survey on a wide scale. We have altered our analyses a bit, so it should not make a difference that there will only be about 10 degrees of freedom at level-2 of our models.

Comment Hans Bos:

I think you should at least consider online administration. It would make scaling this up much easier and also would leave behind a system that the state could use to repeat the survey regularly.

RESPONSE: We have considered online administration, and we do not believe it will be feasible. Most schools do not have computers for each student in all classrooms, and it is difficult to get teachers to arrange times for students to work in the computer lab and/or for the laptop cart. This would place a level of burden on teachers and schools that we believe will prohibit districts/schools from agreeing to participate in the study.

Comment Hans Bos:

I am not an expert on this but I think you can set it up in such a way that the state or the participating districts administer this for their own purposes and share the data with you. That way you wouldn’t need OMB approval.

RESPONSE: We are going to proceed with seeking OMB approval. We are concerned about administration and response rates, if we have the districts administer the surveys themselves without support.

Outcomes Analyses

Comment Geoffrey Borman:

CONSIDER: Should this be framed as an additional research question? Also, if the concerns of cultural differences noted on p. 9 are valid, then shouldn’t you look at the measurement properties of the survey separately by student race/ethnicity in order to assess whether the survey is performing similarly across different groups?

RESPONSE: Good question. We have added analyses that investigate the measurement properties of the survey by race/ethnicity.

Comment Hans Bos:

Why? You could lose tons of observations that way, especially if you have some sensitive questions that many students refuse to answer. I would follow IES guidance (Puma et al) and use dummy imputation.

RESPONSE: We will not be imputing data, as we have been informed by our COR that missing data imputation cannot be used in correlational studies.

Comment Hans Bos:

You should include measures of middle school performance as moderators as well.

RESPONSE: Middle school performance indicators are included in the models as controls. They are not included as moderators. There is already literature showing that middle school achievement predicts grade 9 outcomes. In this study, we are investigating relationships between noncognitive factors and grade 9 outcomes, holding achievement constant.

Comment Hans Bos:

So you WERE planning to do these! Please also include them in the RQs.

RESPONSE: We do not think that the correlations among noncognitive factors and school environment factors need their own research question.

Comment Hans Bos:

Again, you will need a lot more schools to have sufficient power at the school level.

RESPONSE: We’ve changed the analyses, so power at level-2 is not a big concern in this study.

Comment Geoffrey Borman:

CONSIDER: I wonder if this analysis may be too difficult to interpret and whether it will hold much meaning. This will be modeled, as I understand it, as a cross-level interaction of a student-level interaction term. That may be a difficult result to interpret!! Also, I’m not clear why one would think, theoretically or practically, that the perceived school context would shape the relationships between noncognitive factors and the outcomes. Perhaps greater detail is needed or some examples are needed to make this more apparent. If there is no obvious theory or practical reason that these school environment measures would interact with the slopes for the student-level noncognitive predictors, you might consider dropping such analyses of cross-level interactions for these slope estimates.

RESPONSE: We have worked to make the analyses less unwieldy and confusing. We believe the latest version of the proposal is clearer with regard to data analysis and results display.

Limitations and Possible Solutions

Comment Hans Bos:

Why not try to compel them now?

RESPONSE: Dr. Bos is referring to us trying to compel the state to administer the survey to all students. As described above, we do not see this as being feasible.

Possible Challenges and How They Will Be Addressed

Comment Hans Bos:
This section requires more work. It is the achilles heel of these data collection efforts. Ideally you would identify examples of successful administration (with high response rates) of similar surveys and describe what those researchers did to get those results and how you will adapt those efforts for your study. Again, I think large-scale electronic administration is your solution here. You can randomly chop up your survey into four short (10 minute) sets of questions and randomly give each student two of those sections. Then you combine all the results using the overlapping samples. You have to do something innovative like that or you may end up with a 40% response rate and not much to say. (Not to mention a lot of frustration on your part).

RESPONSE: Online administration of the survey does not seem to be feasible. As outlined in the proposal, we will work with schools to administer the surveys and achieve higher response rates.

TWG Summary Comments

Geoffrey Borman:

I believe that this could provide educators with more detailed information to help predict students’ progress in school and to forecast which students may need additional support or help. I have several concerns that I have expressed in my comments. In addition, I am not clear on whether (and by whom) the various scales have been previously validated. A more detailed crosswalk between the survey constructs and the authors and the citations that have validated them would be helpful. Also, is there a composite “on track” measure that you will develop or will you use single dependent variables, such as grades, and test scores? Perhaps a composite “on track to graduation” measure could also be considered. There are many measures to be gleaned from the surveys and many potential outcomes to be evaluated. It seems that there could be a daunting and potentially confusing array of analyses. In this sense, a more refined and simple set of independent and dependent variables might be specified. As written, I am having some difficulty envisioning what the overall collection of correlational analyses would look like.

RESPONSE: The sources of the scales have been added to the survey crosswalk in the appendix. All of the scales have been validated in previous research. We are not going to create a single on-track measure. We are planning to look at each of the three outcomes separately—grade 9 attendance, grade 9 course failures, and grade 9 GPAs. We have worked to make the analyses less unwieldy and confusing. We believe the latest version of the proposal is clearer with regard to data analysis and results display.

Hans Bos:

I have attached my comments. I like the study you are proposing and I think it will produce useful data and information. However, I do think that the proposal still needs a lot of work, mostly in the conceptual discussion of “non cognitive factors”. I realize that this is a popular and convenient term but when you dig deeper like you do in this proposal you find yourself running into the limitations of this concept pretty quickly. As a residual term for things that aren’t measured in standardized tests it may be useful and appropriate but when you start to break it down into different components, it doesn’t work so well. For example, most learning strategies are cognitive and can be learned and taught just like academic subjects. (In fact, some academic subjects require specific learning strategies to be learnable). I just don’t think you can separate these academic and non-academic aspects of learning like that. Moreover, I think your conceptualization of non-cognitive factors includes outcomes that are successive in a causal framework rather than complementary, with some factors (e.g., attitude) preceding others (e.g., attendance). So rather than complementary “factors” you are sometimes combining causes with their immediate proximal outcomes. It is fine to mix these things in a survey but you need to distinguish them conceptually and analytically. Another problem I see is the mixing of skills and attitudes. While these are related within a person, they require different approaches to build and sustain them. An attitude can change quickly (e.g., feeling safe) but a skill takes a long time to build (e.g., social skills). This is true for contextual issues in students’ lives as well. You can make a student feel better about being in school without solving their difficulty getting to school in the morning (and vice versa). Another conceptual challenge is the mixing of environmental variables (school climate, peer support, safety) and personal feelings and skills (social skills, “belonging”). And finally, some concepts aren’t defined or supported well enough (e.g., “grit”). Both for the proposal and for the project itself it is important to sort out these conceptual challenges, present them clearly, and document their conceptual and empirical importance for students’ academic success. 

RESPONSE: The goal of this study is to look at relationships among different types of noncognitive and school environmental factors and grade 9 outcomes. The concept of noncognitive factors is defined as a mix of attitudes, skills and dispositions. There are a variety of factors that fall under this term. As outlined in the literature review, they can generally be divided into five categories: academic behaviors, academic mindsets, perseverance, learning strategies, and sense of belonging. We have added school environment to this study, as we believe that students’ perceptions of their school environments also matter for high school transitions. The literature does not support a hierarchy of these factors. That is, the research itself does not suggest that some of these factors come before other. Rather, it is a bit of a “chicken and the egg” scenario in which it is difficult to determine which came first. For that reason, in our structural equation model, we have the noncognitive and school environmental factors being correlated but not predictive of each other.

As for the survey itself, I think you should be more ambitious in its administration, especially if you want to look at school-level factors. I would try to convince the state or a large group of districts to administer this instrument universally. That way you have sufficient variation at the school level to support the more ambitious analyses you propose. This will also allow you to be more creative about reducing the burden to students (for example, by administering smaller sub-surveys to random subsets of students) and it could help you avoid OMB. Most importantly, by administering the survey statewide and building the infrastructure to do so routinely, you open the door for longitudinal data collection across different cohorts of students, which will make all of this much more useful in the long term. 

RESPONSE: As described above, we do not believe it will be feasible to administer this survey at the scale suggested by Dr. Bos. We also do not believe that it will be feasible to administer the survey online.

Finally, I think both the discussion of the survey and the analyses you propose should more strongly focus on “at risk” students. It is remarkable that you don’t propose to look at low-performing eight graders anywhere in the proposal. Why the focus on ethnicity as the main student-level moderator? How about identifying students who are likely to drop out in ninth grade using their academic performance in middle school and seeing how much non cognitive factors matter for them? How about specific analyses focusing on students in that group for whom ninth grade is a positive turnaround year? As dropout rates continue to drop, it becomes more and more feasible to identify potential dropouts early and  intensively intervene with them. What does the literature about those types of interventions say about non cognitive factors, which ones are malleable at a student and school level, and how persistent impacts on those factors are?  You don’t want to end up with a generic laundry list of factors that appear to be correlated with HS dropout but offer few opportunities for improvement. (How do you increase grit among rising eight graders who read at a fifth grade level?) By at least adding an explicit focus on at risk students as one of the main subgroup breakdowns you are more likely to produce results that inform interventions that will actually make a difference. 

RESPONSE: The New Mexico Achievement Gap Research Alliance is particularly interested in looking at differences in noncognitive and school environmental factors by race/ethnicity. In New Mexico, race/ethnicity is highly correlated with achievement. We’re not sure that we understand the rest of Dr. Bos’ argument. It appears that he is suggesting an entirely different study. The literature suggests that the different types of noncognitive factors outlined in the proposal are all malleable.