Smear Review

A few weeks ago, the National Education Policy Center (NEPC) issued a review of the research on virtual learning. Several proponents of online education issued responses that didn't offer much substance beyond pointing out NEPC’s funding sources. A similar reaction ensued after the release last year of the Gates Foundation's preliminary report on the Measures of Effective Teaching Project. There were plenty of substantive critiques, but many of the reactions amounted to knee-jerk dismissals of the report based on pre-existing attitudes toward the foundation's agenda.

More recently, we’ve even seen unbelievably puerile schemes in which political operatives actually pretend to represent legitimate organizations requesting consulting services. They record the phone calls, and post out-of-context snippets online to discredit the researchers.

Almost all of the people who partake in this behavior share at least one fundamental characteristic: They are unable to judge research for themselves, on its merits. They can’t tell the difference, so they default to attacking substantive work based on nothing more than the affiliations and/or viewpoints of the researchers.

The truth is that there’s plenty of questionable research flying around, and much of it does come from organizations and think tanks that have a clear point of view. But the quality of this work varies between (and within) organizations, not by which "side" they're on in the education debate. None of them is perfect, but several “pro-reform” organizations, such as Education Next and CALDER, regularly produce/sponsor good work in the field of education, as do organizations that are more skeptical of market-based reform, like the Economic Policy Institute and NEPC.

Needless to say, only a fool thinks that empirical research is completely detached from the premises and biases of the researchers, and even the best, most "objective" outfits slip up from time to time. It’s perfectly normal - indeed, necessary - to be skeptical. That’s a good, healthy approach.

But criticizing research based on where it comes from, not what it says, is the laziest, most unproductive form of discourse. It is the preferred tool of the ignorant – those who cannot discern quality for themselves have only one recourse, and that is to attack the messenger. When they do so, it cheapens the dialogue, and threatens the critical role of research in policymaking.

So, a piece of advice to those whose first step in assessing reports and papers is to check the author's credentials or the organization's funding: Take a research methods course or three, and come back to the table when you actually have something to offer.

- Matt Di Carlo

Permalink

good post and i admire the sentiment but yet i still can think of precious few think tank / advocacy group reports that don't confirm their funders' views or the reporting organization's ideology. can you give us a few examples? KIPP and New Leaders have both put out / allowed publication of studies showing limited effects, for example, and the recent CRPE report on CMOs gave a mixed view at best. after that i'm sort of out.

Permalink

Good post, although once again, it's odd that Diane Ravitch tweeted it as if she isn't one of the premier offenders herself (consider how often she criticizes research as "Gates-funded").

Permalink

Matt..This is a good, and very reasonable post.

Our organization has a clear point of view in favor of school choice/vouchers, and so our motivations are constantly questioned (often by NEPC..) but that's just the way it is.. Our responses have to be based on observations and explaining methods.

Alexander.. I would put Fordham's charter school work (esp. in OH) in the category of non-self-confirming(?) work.. You make a good point though, and something that think tanks and advocacy outfits have to respond to without being defensive.

Unfortunately I think many researchers (outside of education, included) also likely face the same kind of confimation temptation once they're invested (time, money, career, etc.) in a line of work and series of findings. And it is a challenge for them as well.

Permalink

You know, there's a certain point where attempting, or claiming, to be extremely objective drives you over the cliff of incoherence. There are certain organizations that so consistently produce one-sided, and biased "studies" (press releases actually), that it is a waste of time to try and parse whether one particular piece of propaganda has more credibility than another probable piece of propaganda. It is possible to take a data set and describe it relentlessly as showing the glass half-empty when it is equally half full. There are studies coming from Gates, Heritage, Hoover, Manhattan, et al that deserve to be dismissed out of hand. Whether the broken clock happens, via coincidence mostly, to be correct twice in 24 attempts should not sway those who have better, and more noble, duties to attend to.

Permalink

While I don't stand for ad hominem attacks, I think that "consider the source" is an improtant step when evalauting research -- or even deciding whether or not to evaluate research.

There are sources which I dismiss out of hand. I have examined their methodology close enough, and times enough, to have decided (realized?) that it's not worth my trouble anymore. Either they are intentionally using poor design to get their desired results or they are ignorantly using poor designed and don't realize that their results lack credibility. Either of those options leaves me unwilling to wade through more papers from them in the future.

Permalink

Mr Buck -- not surprisingly -- has descending to making silly attack that ignores there reality of the situation.

The Gates Foundation has an instutional perspective on particular solutions, particular policies and particular methodologies. It is as much as advocacy organization in the eduction sector as it is anything else. It has show over and over again that it puts its ideology ahead of research.

The first example? It's disasterous waste of it's own and the public's money to push for "small schools." They've long since admitted that this was a mistake, and what led to the mistake.

The latest example (that I know of) is their search for effective teacher practices. But this search is not about effectivenss or anything ideologically different than they have done before. They are still chasing standardized test scores.

"Gates funded" does not mean that it is bad. Rather, it means that it is done with a particular ideology and limited idea of learning and the purpose of schooling.

Permalink

Diane Ravich tweets today: "Always important to consider the source when looking at "new data" and "findings."" She also tweets, "Dobbie and Fryer paid for by Eli Broad Foundation, believe in market-based reforms."

She's insinuating that Roland Fryer's new study is suspect (to be sure, she isn't intellectually capable of analyzing Fryer's studies in any terms more sophisticated than this).