According to the principles of market-based education reform, there’s at least one large, urban public school district operating at max power: District of Columbia Public Schools.

For the past 2-3 years, DCPS has been a reformer’s paradise. The district has a new evaluation system (IMPACT), which it designed by itself. The system includes heavily-weighted value-added estimates (50 percent for teachers in tested grades/subjects), and the results of teachers’ evaluations are used every year to fire the teachers who receive the lowest evaluation ratings, or receive the second lowest score for two consecutive years. “Ineffective teachers” are being weeded out – no hearing, no due process, no nothing.

Furthermore, these evaluation scores are also used to award performance bonuses, and very large ones at that – up to $25,000. This should, so the logic goes, be attracting high-achieving people to DCPS, and keeping them around after they arrive. And, finally, as a result of many years of growth, the city has among the largest charter school sectors in the nation, with almost half of public school student attending charters. Theoretically, this competition should be upping the game of all schools, charter and regular public alike.

Basically, almost everything that market-based reformers think needs to happen has been the reality in DCPS for the past 2-3 years. And the staff  has been transformed too. The majority of principals, and a huge proportion of teachers, were hired during the tenure of either Michelle Rhee or her successor, Kaya Henderson.

The district should be in overdrive right about now. Is it?

I’ve heard several people comment that DCPS’s flat proficiency rates over the past two years represent evidence that the new policies – especially IMPACT – are failures. This is of course unfair. First, the data that DCPS makes public are not particularly good measures of performance over time, since they don’t follow students over time, and DCPS is a high-mobility district undergoing rapid change in the students it serves. Second, even if we had better data and analysis, policies take time to work. IMPACT and the other DC reforms are quite young by policy analysis standards. Third and finally, if, hypothetically, scores were to increase over the next few years, this would at best represent tentative evidence that the policies are “working," since scores can change for many reasons besides personnel policies.

So, no – we still don’t know if these policies are “working," even by the narrow measure of their effect on test scores.

Still, after two years of flat rates, I wouldn’t be surprised if supporters of DCPS-style reforms are more than a little nervous. As I’ve said before, the reasonable expectation for even the most perfectly-designed teacher-focused policy changes is that they’ll take a while to have an effect, and that the eventual improvement will be slow, steady and will likely have a ceiling, perhaps a relatively low one. Most sensible people in the market-based reform crowd are well aware of the fact that policy effects are gradual and won’t, by themselves, get us where we need to be.

Yet so many others – advocates, officials and commentators - have spent the past few years making grand promises and celebrating short-term testing gains as evidence supporting their approach.

We hear regularly about “miracle schools," and are told that charter schools can close huge achievement gaps in just a few years. In many states, short-term testing results are judge and jury for schools and districts - they are closed or sanctioned when results are bad, and rewarded when they're good. Every year, some district officials tell the public they’re making huge progress, even when they’re not, and they claim, without evidence, that the “progress” is directly attributable to their reforms. For her part, Rhee travels the nation and promises huge, immediate gains based on massive misinterpretations of testing results that actually occurred prior to the implementation of the policies for which she is pushing. By these standards, the DC policies are a failure.

Look – I like high expectations as much as the next person, but in the real world of policymaking, realism and consistency must be the primary dispositions. Maybe the drastic changes in DCPS (and elsewhere) will eventually produce results, maybe not. Either way, those who believe in them do their cause a disservice by applying selectively a short-term, test-based standard for success and failure, and by tolerating the selling of a panacea when none exists.

- Matt Di Carlo