Teacher Value-Added Scores: Publish And Perish

On the heels of the Los Angeles Times’ August decision to publish a database of teachers’ value-added scores, New York City newspapers are poised to do the same, with the hearing scheduled for late November.

Here’s a proposition: Those who support the use of value-added models (VAM) for any purpose should be lobbying against the release of teachers’ names and value-added scores.

The reason? Publishing the names directly compromises the accuracy of an already-compromised measure. Those who blindly advocate for publication – often saying things like “what’s the harm?" – betray their lack of knowledge about the importance of the models’ core assumptions, and the implications they carry for the accuracy of results. Indeed, the widespread publication of these databases may even threaten VAM’s future utility in public education.

More On What "Superman" Left Out

Our guest author today is Diane Ravitch, Research Professor of Education at New York University and an historian of education. In addition, she is a nonresident senior fellow at the Brookings Institution in Washington, D.C..  Her latest book is The Death and Life of the Great American School System.

In my recent article for the New York Review of Books about “Waiting for Superman," I praised the SEED Charter School in Washington, D.C. (one of the schools featured in the movie) for their high graduation and college acceptance rates.  I also pointed out, however, that they spend about $35,000 per student, three times as much as normal schools spend.  This fact was not mentioned in the movie.

Nor was the school’s incredibly high attrition rate.  Take a quick look at the graph below (hat tip to Leigh Dingerson).  They start out with about 150 students in seventh grade, but their enrollment slowly declines to around 30 in grade twelve.  This level of attrition is alarming, and it makes any simple evaluation of SEED’s results impossible. 

Waiting For Methodology Man

I finally saw the entire “Waiting for Superman” movie last weekend, in a mostly-empty Georgetown theatre. I went with my mother, not just because she’s a great public school teacher, but also because I needed someone to comfort me while I watched.

We both had strong reactions to dozens of things about the film, and you almost have to admire the chutzpah. It is about education – with a primary focus on teachers – and includes sit-down interviews with superintendents, parents, students, businessmen, economists, and journalists, but not one teacher.

Given all the attention that has already been lavished on it, I’ll discuss just one other thing that struck me, one which I keep hearing elsewhere.  There is exactly one sentence in the whole film in which director Davis Guggenheim addresses the research on charter school effects beyond the anecdotal evidence that dominates his narrative.  He notes, “Only one in five charters is excellent," with the implication that these charters show that it can be done.

He is presumably referring to the CREDO study released last year, which is the largest (15 states plus D.C.) and arguably the most overplayed charter analysis in history (for other multi-state studies showing no charter effects, see here, here, here, and here).  The CREDO authors understandably framed their results in a “media-friendly” manner – by reporting the percentage of charters that did better than comparable regular public schools (17 percent), along with the proportion that did worse (37 percent).

My first point is that 17 percent is equivalent to one in six, not one in five. But beyond that, some charter advocates have taken the remarkable step of turning the finding that twice as many charters do worse than regular publics into “evidence” that the former should be expanded.  The rationale is, as Guggenheim puts it, that these “one in five” charters are “excellent," and if we can increase that proportion, we can fix our public education system. There is only one problem: That’s not what the study says. Guggenheim is either deliberately misleading his viewers or, more likely, just doesn’t know what he’s talking about.

Teacher Contracts: The Phantom Menace

In a previous post, I presented a simple tabulation of NAEP scores by whether or not states had binding teacher contracts.  The averages indicate that states without such contracts (which are therefore free of many of the “ill effects” of teachers’ unions) are among the lowest performers in the nation on all four NAEP exams. 

The post was largely a response to the constant comparisons of U.S. test scores with those of other nations (usually in the form of rankings), which make absolutely no reference to critical cross-national differences, most notably in terms of poverty/inequality (nor to the methodological issues surrounding test score comparisons). Using the same standard by which these comparisons show poor U.S. performance versus other nations, I “proved” that teacher contracts have a positive effect on states’ NAEP scores.

As I indicated at the end of that post, however, the picture is of course far more complicated. Dozens of factors – many of them unmeasurable – influence test scores, and simple averages mask them all. Still, given the fact that NAEP is arguably the best exam in the U.S. – and is the only one administered to a representative sample of all students across all states (without the selection bias of the SAT/ACT/AP) – it is worth revisiting this issue briefly, using tools that are a bit more sophisticated. If teachers’ contracts are to blame for low performance in the U.S., then when we control for core student characteristics, we should find that the contracts’ presence is associated with lower performance.  Let’s take a quick look.

It's A Bird! It's A Plane! No, It's The Superman Movement!

Yes, it’s the Superman Movement. Most filmmakers must secretly dream of a sequel that is bigger, better, and more important than the original. The makers of Waiting for Superman are apparently no different. "For us, the theatrical release is just the start of social action," says Jim Berk, CEO of the aptly named Participant Media, the studio behind the movie (see here). "When I started the company, it was to motivate the grass roots and really get people to embrace an issue, and the idea was that the politics would follow," confirms Jeff Skoll, Participant’s founder and chairman.

In 2009, these leaders decided Participant needed its own organizing arm, so they invented TakePart.com, a website tied to an extensive network of social action websites. TakePart, which constructs a special operation for every film, also offers advice to potential activists on their chosen issues – what to do and how to do it.

Charter school funders gloated and applauded when an early preview clip of Superman was shown at a Grantmakers for Education (GFE) conference in Baltimore last fall. GFE is made up of a wide array of education funders, ranging from powerhouses like Gates, Broad, and Walton, to community and family foundations of every stripe. (Full disclosure: The Albert Shanker Institute is an active member.) Participant was already drawing the foundation world into Superman’s policy and action orbit, hoping its dollars would follow the movie’s message.

We Interrupt This Message

So, I’m reading an opinion piece by Harold Meyerson in the online edition of yesterday’s Washington Post. Meyerson starts by talking about how teachers’ unions get blamed for everything. All of a sudden, in the middle of the text, right after the second paragraph, the piece is interrupted by the following message:

(Watch a video of D.C. Schools Chancellor Michelle Rhee discussing the D.C. Public School system.)
Strange, I thought. Then, right after Meyerson gets going again, criticizing “Waiting for Superman” and hailing the Baltimore teachers’ contract as meaningful progress, I am interrupted yet again:
(For more opinions on the trouble with America's education system, read Jo-Ann Armao's "Is the public turning against teacher unions?" and a Post editorial "Education jobs bill is motivated by politics.")
Now I am taken aback. I’m reading this piece defending teachers’ unions, and at two separate points, in the middle of the text, the Post inserts links: one to an editorial implying that the education jobs bill is a gift to teachers’ unions; one to a video of Michelle Rhee; and the third a short article by Armao that is fair but has undertones. Opinions within opinions, it seems.

Talking About But Not Learning From Finland

Finland’s education system has become an international celebrity. Their remarkable results are being trumpeted, usually in the “What can we learn from them?" context. Yet a lot of the recent discussion about what we can learn – as far as concrete policies – has been rather shallow. 

Right now, the factoid that is getting the most play is that Finnish teachers come from the “top ten percent” of those entering the labor force, whereas U.S. teachers don’t. But without knowing the reasons behind this difference, this fact is not particularly useful.

Although there has been some interesting research on these issues (see here, here, here, here, here, and here), I still haven’t really seen a simple comparison of Finnish vs. American policies that can help us understand what they’re doing right (and perhaps what we’re doing wrong). I am not an expert in comparative education, but I have assembled a few quick lists of features and policies. Needless to say, I am not suggesting that we do everything Finland does, and cease doing everything they don’t. It's very difficult to isolate the unique effects of each of these policies. Also, more broadly, Finland is small (less than six million residents), homogeneous, and their welfare state keeps poverty and inequality at one of the lowest levels among all developed nations (the U.S. is among the highest).

But if we are going to learn anything from the Finnish system, it is important to lay out the concrete differences (I inevitably missed things, so please leave a comment if you have additions).

Performance-Enhancing Teacher Contracts?

** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post.

Please check out our two other posts (here and here), which present summaries and discussions of the actual evidence on the relationship between unions and test scores.

For years, some people have been determined to blame teachers’ unions for all that ails public education in America. This issue has been around a long time (see here and here), but, given the tenor of the current debate, it seems to bear rehashing.  According to this view, teachers unions negatively affect student achievement primarily through the mechanism of the collective bargaining agreement, or contract. These contracts are thought to include “harmful” provisions, such as seniority-based layoffs and unified salary schedules that give raises based on experience and education rather than performance.

But a fairly large proportion of public school teachers are not covered under legally-binding contracts.  In fact, there are ten states in which there are no legally binding K-12 teacher contracts at all (AL, AZ, AR, GA, LA, MS, NC, SC, TX, and VA). Districts in a few of these states have entered into what are called “meet and confer” agreements about salary, benefits, and other working conditions, but administrators have the right to break these agreements at will. For all intents and purposes, these states are free of many of the alleged “negative union effects."

Here’s a simple proposition: If teacher union contracts are the problem, then we should expect to see higher achievement outcomes in the ten states where there are no binding teacher contracts.

So, let’s take a quick look at how states with no contracts compare with the states that have them.

Three Questions For Those Who Dismiss The Nashville Merit Pay Study

The reaction from many performance pay advocates to the Nashville evaluation released last week has been that the study is relatively meaningless (see here and here for examples).  The general interpretation: The results show that the pay bonuses do not improve student achievement, but short-term test score gains are not the "true purpose" of these incentive programs. What they are really supposed to improve, so the line goes, is the quality of people who pursue teaching as a career, as well as their retention rates.

While I disagree that the findings are not important (they are, if for no other reason than they discredit the idea that teachers are holding their effort hostage to more money), I am sympathetic towards the view that the study didn’t tackle the big issues. Attracting the best possible people into the profession – and keeping them there – are much more meaningful goals than short-term test score gains, and they are not addressed in this study (though some results for retention are reported).

But this argument also begs a few important questions that I hope we can answer before the Nashville study fades into evaluation oblivion.  I have three of them.

Remorse Code (Or Comments From A Crib Strangler)

Those who publicly advocate for the kind of education policies put forth in "Waiting for Superman" are now seeing the equivalent of a letter-high fastball down the middle. They can wait on it and crank it out, using the buzz created by a major motion picture to advance the movie’s/campaign’s arguments at face value. I'm a little late on this (at least by blog standards), but over at Fordham’s Flypaper blog, Mike Petrilli saw this fastball, yet instead of unloading, he sacrifice bunted the runner into scoring position for the good of the team.

Responding to an interview in which Davis Guggenheim, the film’s director, claims that charter schools have "cracked the code" on how to educate even the poorest kids, Petrilli warns against the hubris of thinking that we are anywhere beyond first steps when it comes to fixing urban schools. He points out that charters like KIPP benefit from selection effects (more motivated and informed parents seek out the schools), and that the degree to which these schools have actually "closed the gap" between poor and affluent schools has been somewhat oversold. Petrilli also notes that while some of these schools seem to have "cracked the code," there is still little idea of how to expand them to serve more than a tiny minority of poor kids.

Thoughtful comments like these should remind those of us who care about expanding quality education that, although we may have canyon-sized differences between us on what needs to be done (Petrilli claims that those who disagree with him are trying to "strangle" reforms "in their crib"), there may be a few important respects in which we are closer than we may appear. Still (in addition to the crib-strangling allegations), I would take issue with one of Petrilli’s central points – that charters like KIPP may have "cracked the code," and the main problem now is how to scale them up. From my perspective, the "code" is specific policies and practices that produce results. And on this front, we’re practically still using decoder rings from cereal boxes.