Shanker Blog

How To Make A Misleading Public/Private Earnings Gap Disappear

USA Today last week published yet another story claiming that public sector workers make more that their private sector counterparts - this one saying that Wisconsin is one of many states where this is the case. Their “analysis” used data from the Bureau of Economic Analysis, and compared total compensation (salary+benefits) between workers in the private sector and state/local government.

No matter how many times they are told that you can’t just make a straight comparison of dissimilar groups of workers, apparently they still don’t get it. Incredibly, this particular article admits as much, and even quotes economist Jeffrey Keefe, who tells them that the gross comparisons don’t account for important sectoral differences in education and other factors. In other words, their numbers don’t tell us much of anything about public versus private sector compensation. Still, there is the headline: "Wisconsin one of 41 states where public workers earn more." How many people saw that headline, and now believe that public workers are “overpaid?"

USA Today, of course, is not alone. These assertions have lately become insidious, coming from governors, commentators, and others. But when a major national newspaper decides to run this story at this politically-charged time, based on their very own “analysis," a separate response seems in order.

I’ve discussed this issue before, but maybe it would be more helpful to show how the data are more properly analyzed in a step-by-step fashion, using 2009 U.S. Census microdata (the American Community Survey, available from the wonderful organization IPUMS.org). Here’s how you make a false earnings gap disappear in five minutes.

Seize The Day?

Wisconsin Gov. Scott Walker’s determination to destroy collective bargaining rights for his state’s public employees has generated a lot of hyperbolic rhetoric from both sides. Some conservatives have taken particular umbrage at demonstrators’ signs likening Walker to Adolf Hitler, Benito Mussolini, and Hosni Mubarak. They are right that Walker is not akin to these brutal, murderous dictators, who solidified power by crushing independent unions. Indeed, they need not look overseas at all to find anti-union inspiration. The U.S. has its own rich tradition of union-busting – albeit considerably less fierce than in these particular dictatorial regimes.  

This information is just a mouse-click away. Anyone with access to the internet can easily trace the history of violent state and business response to unions and union organizing in America, dating back 150 years. It’s not just the infamous Pinkertons and other thugs hired by business. Police, the National Guard, even federal troops have been used to brutally suppress workers’ efforts to form their own unions. Homestead, Haymarket, Ludlow, Pullman, the 1937 Battle of the Overpass – all are storied examples of incredibly violent action against workers and their organizations.

This sort of drama, punctuated by carnage and death, is pretty much a thing of the past. With the passage of the 1947 Taft-Hartley Act and 1959 Landrum-Griffin Act, anti-union judicial decisions, global outsourcing, and the emergence of union-busting consultants, quashing unions has become, well, child’s play. America’s private sector unions have been on the defensive for better than half a century, with membership eroded to only seven percent of the private sector workforce. With Wisconsin, the attack against public service unions is well and truly launched.

Would The New York City Layoffs Hurt Poor Schools More?

As first reported by the New York Times, the New York City Department of Education released a dataset this past Sunday, which lists the number of potential teacher layoffs that would occur in each school absent a budget infusion.

Layoffs are a terrible thing for schools and students, and this list is sobering. But the primary impetus for releasing for this dataset appears to be the city’s ongoing push to end so-called seniority-based layoffs, and its support for seniority-ending legislation that is now making its way through the state legislature. One of the big talking points on this issue has always been that layoffs that take experience into account would hurt high-poverty schools the most, because these schools tend to have the least experienced teachers. As I discussed in a prior post, Michelle Rhee is making this argument everywhere she goes, and it was one of the primary themes in a new report by the New Teacher Project (released last week). Although I have not heard city officials use the argument since the database was released over the weekend, similar assertions have very recently been made by Mayor Bloomberg, former Chancellor Joel Klein, and current Chancellor Cathie Black.

I find all this a bit curious, given that the best research on the topic finds that the argument is untrue (including a study of New York City, and a statewide analysis of Washington [also here]). Now, it is at least possible that, if layoffs were conducted strictly on the basis of seniority, higher-poverty schools could end up bearing the brunt of dismissals. This is almost never the case, however – layoffs in almost every district proceed based on a variety of criteria, among which seniority is only one (albeit often the most important).

It is fortuitous, then, that the city’s dataset provides an opportunity to test the claim that the “worst-case scenario” – over 4,500 layoffs using current New York City procedures – would hurt high-poverty schools the most. Let’s take a look.

A List Of Education And Related Data Resources

We frequently present quick analyses of data on this blog (and look at those done by others). As a close follower of the education debate, I often get the sense that people are hungry for high-quality information on a variety of different topics, but searching for these data can be daunting, which probably deters many people from trying.

So, while I’m sure that many others have compiled lists of data resources relevant to education, I figured I would do the same, with a focus on more user-friendly sources.

But first, I would be remiss if I didn’t caution you to use these data carefully. Almost all of the resources below have instructions or FAQ’s, most non-technical. Read them. Remember that improper or misleading presentation of data is one of the most counterproductive features of today’s education debates, and it occurs to the detriment of all.

That said, here are a few key resources for education and other related quantitative data. It is far from exhaustive, so feel free to leave comments and suggestions if you think I missed anything important.

Bahrain: Workers Lead The Way

Our guest author today is Heba F. El-Shazli.  She has 25 years of experience in the promotion of democracy, independent trade unions, political and economic development. She has worked with institutions and leaders throughout the Middle East and North Africa (MENA) to challenge authoritarian regimes. Currently she is a visiting professor of international studies and modern languages at the Virginia Military Institute. The views expressed here are her own.

Bahrain has been rocked by turmoil since Feb. 14 – with protesters calling for political reforms from Pearl Square’s "towering monument of a pearl," in the heart of Manama, Bahrain’s capital city. It is the country’s Tahrir Square, its own seat of Liberation. In contrast to Egypt, though, Bahrain’s path to freedom been slower and more violent. On Feb. 17, the government brutally attacked protesters, killing four and injuring dozens. The next day, security forces opened fire on a crowd of thousands marching in funeral processions for the previous day’s victims.

In the midst of this chaos, a young and independent Bahraini labor movement is finding its voice. In response to the government’s violence, the General Federation of Bahrain Trade Unions (GFBTU), with a membership of 66 unions – around 25% of the workforce – threatened a general strike if the government did not back off, start talking to demonstrators, and permit peaceful protest to continue.

And the government backed off.

Students First, Facts Later

On Wednesday, Michelle Rhee’s new organization, Students First, rolled out its first big policy campaign: It’s called “Save Great Teachers," and it is focused on ending so-called “seniority-based layoffs."

Rhee made several assertions at the initial press conference and in an accompanying op-ed in the Atlanta Constitution Journal (and one on CNN.com). At least three of these claims address the empirical research on teacher layoffs and quality. Two are false; the other is misleading. If history is any guide, she is certain to repeat these “findings” many times in the coming months.

As discussed in a previous post, I actually support the development of a better alternative to seniority-based layoffs, but I am concerned that the debate is proceeding as if we already have one (most places don't), and that there's quite a bit of outrage-inspiring misinformation flying around on this topic. So, in the interest of keeping the discussion honest, as well as highlighting a few issues that bear on the layoff debate generally, I do want to try and correct Rhee preemptively.

A Wisconsin Moment For Our Education Policy Debate

There is an obvious, albeit somewhat uncomfortable connection between what’s happening in Wisconsin and what’s been happening in education policy discussions.

A remarkably high proportion of the discussion is focused – implicitly or explicitly – on the presumed role of teachers’ unions. The public is told that our school systems are failing, and that teachers’ unions are at least partially to blame because they protect bad teachers and block “needed” reforms such as merit pay. In this storyline, unions are faceless villains that put the interests of adults above those of children.

Wisconsin represents a threat to this perspective in at least three important manners.

Revisiting The Effect Of Teachers' Unions On Student Test Scores

The Wisconsin protests have predictably spurred a great deal of information-seeking, with union supporters and opponents alike searching for evidence that supports their cases. One of the most prevalent topics over the past week or so is the effect of teacher collective bargaining on student test scores. As a result, a couple of our previous posts have been shared widely. The first (also republished here) compares NAEP scores in states that allow binding teacher contracts with those in states that do not (or have only one or two); the second, follow-up post offers some additional, multivariate analysis.

Although it is true that the first post (which was at least partially satirical - see the last few sentences) shows that states without binding contracts are among the lowest-performing in the nation, I want to clear something up: As I noted in both posts, neither the data nor my argument offer any conclusive proof that teacher contracts act to increase student test scores. The intention of those posts was to address the age-old counter claim – that teacher contracts are somehow injurious to student achievement – and to provide very tentative evidence that the contracts appear to have little discernible impact either way (which is what the follow-up post, using state-level models that controlled for basic student characteristics, indicated, along with the requisite caveats).

This speaks directly to those who seek to blame unions for poor achievement in the U.S. - if union contracts were in fact a major contributing cause of low test performance, it might be reasonable to expect to find at least some clear differences between states that did and did not allow them. Although my analysis was extremely limited, I found no such evidence.

But this also applies to those who have been claiming recently – many in the Wisconsin context – that teacher bargaining clearly improves these outcomes.

Value-Added: Theory Versus Practice

** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post

About two weeks ago, the National Education Policy Center (NEPC) released a review of last year’s Los Angeles Times (LAT) value-added analysis – with a specific focus on the technical report upon which the paper’s articles were based (done by RAND’s Richard Buddin). In line with prior research, the critique’s authors – Derek Briggs and Ben Domingue – redid the LAT analysis, and found that teachers’ scores vary widely, but that the LAT estimates would be different under different model specifications; are error-prone; and conceal systematic bias from non-random classroom assignments.  They were also, for reasons yet unknown, unable to replicate the results.

Since then, the Times has issued two responses. The first was a quickly-published article, which claimed (including in the headline) that the LAT results were confirmed by Briggs/Domingue – even though the review reached the opposite conclusions. The basis for this claim, according to the piece, was that both analyses showed wide variation in teachers’ effects on test scores (see NEPC’s reply to this article). Then, a couple of days ago, there was another response, this time on the Times’ ombudsman-style blog. This piece quotes the paper’s Assistant Managing Editor, David Lauter, who stands by the paper’s findings and the earlier article, arguing that the biggest question is:

...whether teachers have a significant impact on what their students learn or whether student achievement is all about ... factors outside of teachers’ control. ... The Colorado study comes down on our side of that debate. ... For parents and others concerned about this issue, that’s the most significant finding: the quality of teachers matters.

Saying “teachers matter” is roughly equivalent to saying that teacher effects vary widely - the more teachers vary in their effectiveness, controlling for other relevant factors, the more they can be said to “matter” as a factor explaining student outcomes. Since both analyses found such variation, the Times claims that the NEPC review confirms their “most significant finding."

The review’s authors had a much different interpretation (see their second reply). This may seem frustrating. All the back and forth has mostly focused on somewhat technical issues, such as model selection, sample comparability, and research protocol (with some ethical charges thrown in for good measure). These are essential matters, but there is also an even simpler reason for the divergent interpretations, one that is critically important and arises constantly in our debates about value-added.

The Legend Of Last Fall

The subject of Michelle Rhee’s teaching record has recently received a lot of attention. While the controversy has been interesting, it could also be argued that it’s relatively unimportant. The evidence that she exaggerated her teaching prowess is, after all, inconclusive (though highly suggestive). A little resume inflation from a job 20 years ago might be overlooked, so long as Rhee’s current claims about her more recent record are accurate. But are they?

On Rhee’s new website, her official bio - in effect, her resume today (or at least her cover letter) - contains a few sentences about her record as chancellor of D.C Public Schools (DCPS), under the header "Driving Unprecedented Growth in the D.C. Public Schools." There, her test-based accomplishments are characterized as follows:

Under her leadership, the worst performing school district in the country became the only major city system to see double-digit growth in both their state reading and state math scores in seventh, eighth and tenth grades over three years.

This time, we can presume that the statement has been vetted thoroughly, using all the tools of data collection and analysis available to Rhee during her tenure at the helm of DCPS.

But the statement is false.