Value-Added In Teacher Evaluations: Built To Fail

With all the controversy and acrimonious debate surrounding the use of value-added models in teacher evaluation, few seem to be paying much attention to the implementation details in those states and districts that are already moving ahead. This is unfortunate, because most new evaluation systems that use value-added estimates are literally being designed to fail.

Much of the criticism of value-added (VA) focuses on systematic bias, such as that stemming from non-random classroom assignment (also here). But the truth is that most of the imprecision of value-added estimates stems from random error. Months ago, I lamented the fact that most states and districts incorporating value-added estimates into their teacher evaluations were not making any effort to account for this error. Everyone knows that there is a great deal of imprecision in value-added ratings, but few policymakers seem to realize that there are relatively easy ways to mitigate the problem.

This is the height of foolishness. Policy is details. The manner in which one uses value-added estimates is just as important – perhaps even more so – than the properties of the models themselves. By ignoring error when incorporating these estimates into evaluation systems, policymakers virtually guarantee that most teachers will receive incorrect ratings. Let me explain.

The High Cost Of Caring

The field of early childhood education (ECE) is riddled with contradictions. Bluntly, when those we love the most—our children—are at the most consequential stage of their cognitive, social, and emotional development, we leave them in the hands of the people we pay the least. According to the latest data from the U.S. Bureau of Labor Statistics, for example, childcare workers earn about 4 percent less than animal caretakers—$20,940 and $21,830 per year, respectively.

I am far from the first to make this embarrassing comparison; more than a decade ago, Marci Whitebook provided an extensive overview. Unfortunately, the comparisons still hold.

Over the intervening years, there have been many determined efforts to regulate and improve the working conditions of early childhood educators, including raising the qualifications and wages for the profession. Indeed, the demand for worthy salaries is often discussed in combination with workforce development efforts. In other words, we want early childhood workers to be both better trained and better paid. While this may seem to be a perfectly reasonable approach, it suggests that the low wages are a result of inadequate qualifications. Perhaps. But I believe that this obscures another important explanation for these workers’ persistently meager pay.

A Response To Joel Klein

Our guest author today is Edith (Eadie) Shanker, Albert Shanker’s widow and a retired New York City teacher.

A few months ago, in the Wall Street Journal (WSJ), Joel Klein invoked Al Shanker’s name as an educator in support of today’s charter school “reform” efforts. Klein wanted the public to believe that Al was the originator of the charter school concept (he wasn’t) and that he would today be supportive of the charter school ”reform” ideology now being spread around New York City and the country as a panacea for low student achievement. Conveniently, Klein did not indicate that Al denounced the idea of charters when it became clear that the concept had changed and was being hijacked by corporate and business interests. In Al’s view, such hijacking would result in the privatization of public education and, ultimately, its destruction - all without improving student outcomes.

Now, in his recent Atlantic magazine article, Klein trots out a quotation attributed to Al (said in jest if at all) to support the stereotype that, as a union leader, Al cared only about “protecting” the union’s members, including “bad” teachers. Using this alleged quotation – “when school children start paying union dues, that’s when I’ll start representing the interests of children” - Klein not only plays fast and loose with Al’s reputation as a union leader but also as a sterling educator. (To be a true expert on Al’s views on how to improve education for children - and how to be a union leader - Klein could check out 27 years’ worth of his “Where We Stand” columns in the New York Times.)

What Do Teachers Really Think About Education Reform?

There has recently been a lot of talk about teachers’ views on education policy. Many teachers have been quite vocal in their opposition to certain policies (also here) and many more have expressed their views democratically – through their unions – especially in states where teachers have collective bargaining rights.

We should listen carefully to these views, but it’s also important to bear in mind that there are millions of public school teachers out there, with a wide variety of opinions on any particular education policy, and not all of their voices might be getting through.

So, the question remains: How do most teachers feel about the current wave of education policy reforms spreading throughout states and districts, including (but not at all limited to) merit pay, eliminating tenure and incorporating test-based measures into teacher evaluations?

The logical mechanism by which we might learn more about teachers’ views on these policies is, of course, a survey. Unfortunately, useful national surveys are quite rare. In order to get accurate estimates, you need an unusually large number of teachers to take the survey (a deliberate "oversample"), and they must be randomly polled (lest there be selection bias). In my last post, I suggested that states/districts conduct their own teacher surveys.  In the meantime, some national evidence is already available, and if the data make one thing clear, it’s that we need more. When it comes to supporting or opposing different policies, teachers’ opinions, like everyone’s, depend a great deal on the details.

To Understand The Impact Of Teacher-Focused Reforms, Pay Attention To Teachers

You don’ t need to be a policy analyst to know that huge changes in education are happening at the state- and local-levels right now – teacher performance pay, the restriction of teachers’ collective bargaining rights, the incorporation of heavily-weighted growth model estimates in teacher evaluations, the elimination of tenure, etc. Like many, I am concerned about the possible consequences of some of these new policies (particularly about their details), as well as about the apparent lack of serious efforts to monitor them.

Our “traditional” gauge of “what works” – cross-sectional test score gains – is totally inadequate, even under ideal circumstances. Even assuming high quality tests that are closely aligned to what has been taught, raw test scores alone cannot account for changes in the student population over time and are subject to measurement error. There is also no way to know whether fluctuations in test scores (even fluctuations that are real) are the result of any particular policy (or lack thereof).

Needless to say, test scores can (and will) play some role, but I for one would like to see more states and districts commissioning reputable, independent researchers to perform thorough, longitudinal analyses of their assessment data (which would at least mitigate the measurement issues). Even so, there is really no way to know how these new, high-stakes test-based policies will influence the validity of testing data, and, as I have argued elsewhere, we should not expect large, immediate testing gains even if policies are working well. If we rely on these data as our only yardstick of how various policies are working, we will be getting a picture that is critically incomplete and potentially biased.

What are the options? Well, we can’t solve all the measurement and causality issues mentioned above, but insofar as the policy changes are focused on teacher quality, it makes sense to evaluate them in part by looking at teacher behavior and characteristics, particularly in those states with new legislation. Here’s a few suggestions.

Quote, Unquote

This post is co-authored by Matt Di Carlo and Esther Quintero.

Update: Please see this May 2012 "Fact Checker" piece on the Shanker quote in the Washington Post.

***

This week, in an Atlantic article, former New York City Public Schools Chancellor Joel Klein dropped an incendiary Albert Shanker quote that you’ve probably heard before:

When school children start paying union dues, that’s when I’ll start representing the interests of school children.

The negative implications of this statement are obvious, which is why it is so frequently quoted by (mostly) conservative pundits and journalists.

We didn’t know Al Shanker personally. He died while we were still college undergraduates. So, we were surprised to learn that the people who knew and worked with Shanker have long thought this quote to be apocryphal.

We were skeptical but intrigued, and decided to do a little detective work.

Fordham Responds To The Common Core "Counter-Manifesto"

The following post was written by Chester E. Finn Jr., President, and Michael J. Petrilli, Executive Vice-President, of the Thomas B. Fordham Institute in Washington, D.C.  It was originally posted here, on the Fordham Institute’s blog. We have reprinted it with the permission of the authors.

The "counter-manifesto" released this week in opposition to national testing and a national curriculum is full of half-truths, mischaracterizations, and straw men. But it was signed by a lot of serious people and deserves a serious response.

First, let us dispatch some silliness. To the best of our knowledge, and based on all evidence that we’re aware of, neither the signers of the Shanker Institute manifesto, nor leaders in the Obama/Duncan Education Department, advocate a “nationalized curriculum” that would “undermine control of public school curriculum and instruction at the local and state level” and “transfer control to an elephantine, inside-the-Beltway bureaucracy." Nor is anybody calling for “a one-size fits all, centrally controlled curriculum for every K-12 subject." We certainly wouldn’t support such a policy—and can understand why the conservative luminaries who signed the counter-manifesto wouldn’t want it, either. As parents, grandparents, charter-school authorizers, and champions of school choice in almost all its forms, we believe deeply in the importance of schools having the freedom to shape their own unique educational approaches.

So let us be clear: While the assessments linked to the Common Core State Standards will be mandatory (for schools and districts in states that choose to use them), the use of any common curricular materials will be purely voluntary. We don’t see any evidence to indicate otherwise.

Where Al Shanker Stood: Common Content

The recent, breathless opposition to the idea of common curricular content led us to reflect on just how long educators have been asking for this practical tool for better schooling - only to be rebuffed by those more interested in playing politics. It’s been generations. More than 20 years ago, Al Shanker waded into the fray. The following, entitled “An American Revolution in Education: Developing a Common Core," was published by Al in his weekly Where We Stand column on Feb. 24, 1991.

If anyone had talked about a common curriculum for US schools a few years ago, people would have said he was crazy. Sure, that's the way they do it in most other industrialized countries; and, sure, their students achieve at a much higher level than ours. But the education system in those countries are under the control of their central governments, and the idea of our federal government dictating what children learn in schools was out of the question. Now, we have begun to understand the price we pay for our fragmented curriculum. We've also begun to find ways of building a common curriculum in a typically American way — through voluntary effort rather than government intervention.

Why should we be so eager for a common curriculum? Exactly what difference does it make in an education system — and, ultimately in what children learn?

Shanker Institute Counters Efforts To Undermine Common Core State Standards; Repeats Call For Matching Curricula

The “Closing the Door on Innovation” manifesto issued today by a group of conservatives distorts the purpose of the nonpartisan Albert Shanker Institute-sponsored Call for Common Content statement released in March. The statement was signed by a diverse group of education and other leaders from across the political spectrum - and emphasized that teachers must have access to voluntary curriculum guidelines in order to teach effectively to the new state-led common core standards. Aligning the new standards to high quality curriculum is critical to ensuring that all children in the U.S. receive a rigorous education.

“While we agree that curriculum should be designed before assessments, their claim that the ‘Call for Common Content’ is about creation of a ‘national curriculum’ and ‘national standards’ is just plain wrong," said Randi Weingarten, president of the American Federation of Teachers (AFT), one of the signatories to the Shanker Institute statement.

“What we argued then, and what the AFT’s own committee on implementation of common core standards will reinforce in its upcoming recommendations, was that educators need and want a set of curricular roadmaps that are aligned to common standards and developed from various high-quality, content rich, multiple curriculum resources, with strong input from teachers themselves and other curriculum experts."

“And," Weingarten said, “Without these resources, especially in a time of tight education budgets, it will be up to teachers to make up all of this content aligned to standards as they go along, under the guise of local autonomy. That is a recipe for failure and unfair to both students and teachers."

The Importance of STEM In The Early Grades

Our guest author today is Stan Litow, Vice President of Corporate Citizenship and Corporate Affairs at IBM, President of the IBM Foundation, and a member of the Shanker Institute's board of directors.

This is a difficult year for city and state leaders. They are struggling mightily with how to cope with both declining revenues and escalating costs, resulting in painful short term decisions about what to cut, how to cut, and ways in which basic or vital services can be maintained. Sadly, we have heard far too little these days about where to invest and how to invest in order to produce longer term benefit and mitigate longer term costs.

As people focus on education, it has been common wisdom that business leaders and those concerned with the bottom line have an interest in education too, but that interest is focused solely on STEM, or Science Technology, Engineering and Math. And that focus is placed on the later grades such as middle and high schools. It is undeniable that STEM is important, especially if we are to nurture the next generation of innovators. To do so, we must invest more creatively to improve teacher quality and student outcomes. But we can not address these challenges by limiting our focus to secondary education. While career pathways are great motivators for teenagers and young adults, we simply can not wait until high school - or even middle school - to prepare students and capture their imaginations. We must start earlier, much earlier. In that effort, early childhood education is vitally important.