Shanker Blog

Three Questions For Those Who Dismiss The Nashville Merit Pay Study

The reaction from many performance pay advocates to the Nashville evaluation released last week has been that the study is relatively meaningless (see here and here for examples).  The general interpretation: The results show that the pay bonuses do not improve student achievement, but short-term test score gains are not the "true purpose" of these incentive programs. What they are really supposed to improve, so the line goes, is the quality of people who pursue teaching as a career, as well as their retention rates.

While I disagree that the findings are not important (they are, if for no other reason than they discredit the idea that teachers are holding their effort hostage to more money), I am sympathetic towards the view that the study didn’t tackle the big issues. Attracting the best possible people into the profession – and keeping them there – are much more meaningful goals than short-term test score gains, and they are not addressed in this study (though some results for retention are reported).

But this argument also begs a few important questions that I hope we can answer before the Nashville study fades into evaluation oblivion.  I have three of them.

Standing Up For The Rights Of Others

"...part of the price of our own freedom is standing up for the freedom of others..."   - President Barack Obama

In an extraordinary speech at the United Nations last Thursday, President Obama asserted his leadership and the leadership of the U.S. in the promotion of democracy and human rights around the world. Think that’s a "no news" story? You’d be wrong. The Bush administration’s effort to frame the Iraq invasion as an effort to bring democracy to the region has had the effect of linking traditional U.S. democracy promotion to military intervention in the minds of many people, in the U.S and abroad. And, although Mr. Obama campaigned in support of democracy promotion, his administration has approached the issue cautiously. In fact, the administration has been criticized for backing away from a tough democracy and human rights line in its bilateral relations, especially in the Middle East and China. Moreover, although he promised to increase the budget for the National Endowment for Democracy, in his first budget, the President actually proposed a funding reduction, but in the subsequent compromise legislation, signed off on a small increase.

In this context, apparently anticipating a skeptical reaction to the speech, the White House released a "fact sheet" outlining activities and initiatives to illustrate its commitment to promoting democratic ideals.

Remorse Code (Or Comments From A Crib Strangler)

Those who publicly advocate for the kind of education policies put forth in "Waiting for Superman" are now seeing the equivalent of a letter-high fastball down the middle. They can wait on it and crank it out, using the buzz created by a major motion picture to advance the movie’s/campaign’s arguments at face value. I'm a little late on this (at least by blog standards), but over at Fordham’s Flypaper blog, Mike Petrilli saw this fastball, yet instead of unloading, he sacrifice bunted the runner into scoring position for the good of the team.

Responding to an interview in which Davis Guggenheim, the film’s director, claims that charter schools have "cracked the code" on how to educate even the poorest kids, Petrilli warns against the hubris of thinking that we are anywhere beyond first steps when it comes to fixing urban schools. He points out that charters like KIPP benefit from selection effects (more motivated and informed parents seek out the schools), and that the degree to which these schools have actually "closed the gap" between poor and affluent schools has been somewhat oversold. Petrilli also notes that while some of these schools seem to have "cracked the code," there is still little idea of how to expand them to serve more than a tiny minority of poor kids.

Thoughtful comments like these should remind those of us who care about expanding quality education that, although we may have canyon-sized differences between us on what needs to be done (Petrilli claims that those who disagree with him are trying to "strangle" reforms "in their crib"), there may be a few important respects in which we are closer than we may appear. Still (in addition to the crib-strangling allegations), I would take issue with one of Petrilli’s central points – that charters like KIPP may have "cracked the code," and the main problem now is how to scale them up. From my perspective, the "code" is specific policies and practices that produce results. And on this front, we’re practically still using decoder rings from cereal boxes.

Educational Unilateralism

In New York City this week, a special "plenary summit" of the UN General Assembly met to encourage the world to step up support for the Millennium Development Goals (MDG) first okayed by the UN in 2000. These eight goals – which include slashing poverty, combating disease, fighting hunger, protecting the environment, and boosting education – had a 2015 target date for their achievement. Ten years on, the summit reviewed progress and urged participants to speed up the pace.While the eradication of disease and hunger was named as the key priority, the nations of the world also recognized the crucial importance of education. Goal 2 focuses on the right of all kids to at least a primary school education. Goal 3 promotes the right of girls to have the same access to education as boys – a major problem in much of the developing world. 

Although the U.S. is the world’s largest donor country, surveys show that few Americans have heard about it. President Obama, who during his campaign pledged to fund a $2 billion Global Fund for Education, has done little – what with the financial crisis and debates over both the means and ends of foreign assistance programs getting in the way. In the meanwhile, critics call MDG little more than a laundry list of needs, with no real strategy on how to achieve them. Still, the goals are well worth reading, if only as a reflection of what the world believes (at least on paper) are the rock-bottom problems facing humanity in the 21^st century.

Persistently Low-Performing Incentives

Today, the National Center on Performance Incentives (NCPI) and the RAND Corporation released a long-awaited experimental evaluation of teacher performance pay in Nashville, Tenn. It finds that performance bonuses have virtually no effect on student math test scores (there were small but significant gains by fifth graders, but only in two of the three years examined, and the gains did not last into sixth grade).

Since this is such a politically contentious issue, these findings are likely to spark a lot of posturing and debate. So it’s worth trying to put them in context. As I discussed in a prior post, we now have at least preliminary results from three randomized experimental evaluations of merit pay in the U.S., the first contemporary, high-quality evidence of its kind.  This Nashville report and the two previously-released studies – one from Chicago and one from New York City's schoolwide bonus program – reached the same conclusion: Performance bonuses for teachers have little or no discernable effect on student test scores. 

Although the NYC and Chicago findings are preliminary (the evaluations are still in progress), the NYC program provides schoolwide and not individual bonuses, and one additional study (Round Rock, Tex.) is yet to be released, the three already-released reports do represent a fairly impressive, though still very tentative, body of evidence on merit pay’s utility as a means to improve test scores.

And at this point, it’s a good bet that, when all the evaluations are final and the smoke has cleared, we will have to conclude that performance bonuses are, at the very least, a very unpromising policy for producing short-term test score gains. 

Research Wars

Weeks before the fact, a Sept. 29 forum sponsored by the Economic Policy Institute and the National Education Policy Center has sparked some interesting debate over at the National Journal. The event, centered around the recent book, Think Tank Research Quality: Lessons for Policy Makers, the Media and the Public, is an effort to separate "the junk research from the science."

The crux of the debate is whether the recent explosion of self-published reports by various educational think tanks has helped or hindered the effort to improve the quality of educational research. (Full disclosure: The Albert Shanker Institute is often called a "think tank" and we frequently self-publish.) The push and pull of dueling experts and conflicting reports, say some, has turned education research into a political football—moved down the field by one faction, only to be punted all the way to the other end by a rival faction—each citing "research" as their guide.

"My research says this works and that doesn’t," can always be countered by, "Oh yeah, well my research says that works and this doesn’t." There are even arguments about what "what works" means because, except for performance on standardized tests, our goals remain diverse, decentralized and subject to local control. As a result, public education is plagued by trial and error policies that rise and fall, district by district and state by state, like some sort of crazed popularity contest.

Teacher Quality On The Red Carpet; Accuracy Swept Under The Rug

The media campaign surrounding “Waiting for Superman," which has already drawn considerable coverage, only promises to get bigger. While I would argue – at least in theory – that increased coverage of education is a good thing, it also means that this is a critically important (and, in some respects, dangerous) time. Millions of new people will be tuning in, believing that they are hearing serious discussions about the state of public education in America and “what the research says” about how it can be improved.

It’s therefore a sure bet that what I’ve called the “teacher effects talking point” will be making regular appearances. It goes something like this: Teachers are the most important schooling factor influencing student achievement. This argument provides much of the empirical backbone for the current push toward changes in teacher personnel policies. It is an important finding based on high-quality research, one with clear policy implications. It is also potentially very misleading.

The same body of evidence that shows that teachers are the most important within-school factor influencing test score gains also demonstrates that non-school factors matter a great deal more. The first wave of high-profile articles in our newly-energized education debate not only seem to be failing to provide this context, but are ignoring it completely. Deliberately or not, they are publishing incorrect information dressed up as empirical fact, spreading it throughout a mass audience new to the topic, to the detriment of us all.

More Than One Way Of Winning

Our guest author today is James R. Stone, professor and director of the National Research Center for Career & Technical Education at the University of Louisville.

In recent years, a consensus has emerged among education researchers and policymakers that all students should graduate from high school both "college- and career-ready." President Obama has made this part of his education agenda. And numerous advocacy organizations have championed the notion. But what does the phrase actually mean?

"College-ready" usually means not needing remedial courses once in college, and "career-ready" is usually equated with college-ready. High standards and expectations are the means recommended to prepare college-ready graduates. This means rigorous courses aligned with standards, and tests to ensure that students meet those standards. Presumably, career-readiness comes with the same requirements. The evidence contradicts the rhetoric, however. Paul Barton at ETS, Peter Cappelli at the Wharton School, and other labor market experts argue that being prepared for college is not the same as being prepared for a successful transition into the workforce.

Perhaps we ought to consider an alternative framework that more clearly defines what college- and career-ready means.

Extra Time: More From The Magazine's Education Poll

** Also posted here on "Valerie Strauss’ Answer Sheet" in the Washington Post.

A recent education poll conducted by Time Magazine has gotten a lot of attention. Many of the questions are worded so badly that the results are rather meaningless. The question on merit pay, for example, defines the practice as “paying teachers according to their effectiveness” (who would oppose that, if it could be accurately measured?). Other questions are very interesting, such as the one asking whether respondents would pay higher taxes to improve public schools (56 percent would). Or the finding that, when asked what will “improve student achievement the most," more than twice as many people choose “more involved parents” (54 percent) over “more effective teachers” (24 percent).

But, as is sometimes the case, a few of the survey’s most interesting results were not included in the published article, which highlighted only 11 out of 40-50 or so total questions (the full set of results is available here). Here are three or four unpublished items that caught my eye (the sample size is 1,000, with a margin of error of +/- 3 percent):

Labor Day In Hell

The new Albert Shanker Institute-supported report, The Global State of Workers’ Rights: Free Labor in a Hostile World, released on Labor Day by the human rights organization Freedom House, has received some notable attention in the press, both here and around the world. One photo essay in Foreign Policy, titled "Labor Day in Hell," illustrates 14 of the worst-offending nations, among them Belarus, North Korea, and Sudan (see the screenshot below).

Indeed, the report, which examined the state of labor rights in the world for the year 2009, found serious violations of workers’ freedoms in all parts of the world except Western Europe.　Countries were ranked on a five-category scale of Free, Mostly Free, Partly Free, Repressive, and Very Repressive.

The United States was rated as Mostly Free—the same rank accorded to Bolivia, Mongolia, Romania, and Zambia—less free than all of Western Europe and such nations as Australia, Canada, Chile, South Africa, and South Korea. As the report notes, although American law recognizes core labor rights, the U.S. political environment is "distinctly hostile to unions, collective bargaining, and labor protest." So not Hell, but not Heaven either.