The Faulty Logic Of Using Student Surveys In Accountability Systems

In a recent post, I discussed the questionable value of student survey data to inform teacher evaluation models. Not only is there little research support for such surveys, but the very framing of the idea often reflects faulty reasoning.

A quote from a recent Educators 4 Excellence white paper helps to illustrate the point:

For a system that aims to serve students, young people’s interests are far too often pushed aside. Students’ voices should be at the forefront of the education debate today, especially when it comes to determining the effectiveness of their teacher.

This sounds noble… but seriously, why should students’ opinions be "at the forefront of the education debate"? Are students’ needs better served when we ask students what they need directly? Research on this is explicit: no, not really.

In addition, the assertion that students’ judgments should be paramount makes the implicit assumption that those who care most about a topic are most correct in their opinions about it. Since when did caring the most or being the most directly affected – the case with children and the education system – become a viable substitute for factual knowledge and rational decision-making?

In the case of parents’ discipline policies, children are also the most directly affected – does this mean that they are also the best judge of what’s best for them? In the case of the healthcare system, patients are the most directly affected and certainly care the most deeply about their own outcomes – so, should patients’ opinions of their doctors be "at the forefront" of today’s health policy debate? The logic seems weak to me.

According to author and philosopher Jamie Whyte "it seems you can’t argue with the morally sincere" – see here. But research tells us that, if we really care to incorporate children’s opinions in teacher evaluation, the surveys they take must be made and used in a manner that circumvents the biases they will inevitably, unintentionally, and unconsciously display when assessing their teachers.

And, if we really care about children and the quality of their education, we should not let sincere concern become a license for irrationality. Rationality and evidence, however imperfect, are the tools that, as Whyte explains, give our beliefs "the greatest chance of being true."

- Esther Quintero

Permalink

I'm not sure that I find your argument against student surveys all that convincing. You are basing your reasoning on the premise that student surveys are gathering student judgment on "what's best for them." But that's not the data that a well-designed survey (such as The Tripod Project surveys) is intended to gather. What is learned from student surveys is how the student perceives their learning environment. This is incredibly useful and important data to any educator that gives a hoot about effective and caring instruction.

Let's go to your example of the healthcare system, in which you rhetorically question "should patients’ opinions of their doctors be 'at the forefront' of today’s health policy debate?" Perhaps not. But patients' opinions of their doctors should certainly be critically considered data. Doctor's provide a service, and how they talk to and treat their patients is a huge component of that service. I know that I will choose to leave a doctor if I don't feel that I can communicate effectively with them or vice versa. And if I see that a doctor's office has received a number of complaints (such as on Yelp or some other rating service), I will not waste my time giving them my business. Students obviously don't have that choice to the same extent, but if anything, that seems to me all the more reason we should take into consideration how they perceive their learning environment.

Certainly any data gathered by a student survey will contain a degree of bias. Which is why such data should only be considered when gathered over an extended period of time (3 years) and when correlated with multiple points data, such as observations and progress assessments.

Permalink

Hi Mark,

Thank you for your comment.

My post does not attempt to cover the overall value of student surveys. A previous post did go through a number of studies, explaining how and why student surveys are so easily biased. Most of the evidence that I reviewed comes from college student surveys, but I believe the lessons are transferable to the school context. In this specific post, I take issue with the notion that those most affected by something automatically know the most about it, and so should have opinions that are to be given extra weight. I think this reasoning is faulty.

I am sure there are survey instruments that are better designed than others – I am not aware, however, of surveys designed to capture stereotype biases which are the most important when we evaluate people. I agree that students’ or patients’ opinions are data, valuable data indeed, but they are also subject to demonstrable biases. If we stop at the first part of the sentence we have a problem.

Opinions about people are never 100% objective so my point is (1) we should acknowledge that, (2) we should be at least minimally educated on the ways in which another person’s characteristics are likely to be influencing (or biasing) our perceptions of his/her actions, and (3) we should not base important decisions on opinions/evaluations (gathered by surveys, observations etc.) without understanding both (1) and (2). In my experience, we tend to stop at (1) and this is insufficient. In fact, it can be counterproductive in the sense that admitting something is imperfect doesn't help you to improve it, but can somehow help to justify its continued use.

Esther

Permalink

Esther, thanks for your considerate and reflective response.

The research which you presented in your first post on this matter falls under the same point of scrutiny I presented above, which is that student survey data should not be about student judgment of the teacher per se, but rather about their perceptions of their learning environment. In other words, a well-designed survey should not be a popularity contest. But I agree that designing such a survey is certainly not an easy task and may even be well nigh impossible to achieve with perfection.

I also agree with your point that any survey instrument (or observation instrument, no matter how well-designed) is subject to a potentially great degree of bias, and that we must take this into consideration when using such data. This again comes back to my point about using such data only when gathered over extended periods of time and when triangulated with multiple points of data.

I think that if a teacher has received poor feedback from student surveys (over 3 years), has been rated poorly by observations by both peers and administrators during that time, and whose students have been performing poorly on whatever measure of performance that school district is using, we can probably safely say that that teacher is either not in the right position or not receiving the right amount of support.

I would like to give a bit of pushback on your statement about the faulty reasoning of thinking that "those most affected by something automatically know the most about it." That's such a broad meta-level statement to make, and I don't think it stands as a viable take down to the potential use of student surveys in teacher evaluations, because it takes the practical use of such data far out of context. In the Educators 4 Excellence policy recommendations for teacher evaluations (I'm not part of their collective, by the way, I just find this stuff interesting since I made some recommendations of my own along with the VIVA Project: http://vivateachers.org, and we also recommended student survey use), they aren't saying that student surveys are the end-all-be-all of teacher evaluations--they are only recommending using them as 10% of the total weight, in conjunction with 5 other points of data. When used in such a way, I think that the biases of students can be better evaluated in the context of other measures.

Your note of caution is well taken, but I think that student surveys should be an important source of information. As a teacher, I consider my students' feedback extremely important in the end-of-the-year reflective process. I administered a student survey this year, and I felt like the results helped me target some areas to focus on in my next year.

Best,
Mark