The Wisconsin protests [1] have predictably spurred a great deal of information-seeking, with union supporters and opponents alike searching for evidence that supports their cases. One of the most prevalent topics over the past week or so is the effect of teacher collective bargaining on student test scores. As a result, a couple of our previous posts have been shared widely. The first [2] (also republished here [3]) compares NAEP scores in states that allow binding teacher contracts with those in states that do not (or have only one or two); the second [4], follow-up post offers some additional, multivariate analysis.

Although it is true that the first post (which was at least partially satirical - see the last few sentences) shows that states without binding contracts are among the lowest-performing in the nation, I want to clear something up: As I noted in both posts, neither the data nor my argument offer any conclusive proof that teacher contracts act to increase student test scores. The intention of those posts was to address the age-old counter claim [5] – that teacher contracts are somehow injurious to student achievement – and to provide very tentative evidence that the contracts appear to have little discernible impact either way (which is what the follow-up post, using state-level models that controlled for basic student characteristics, indicated, along with the requisite caveats).

This speaks directly to those who seek to blame unions for poor achievement in the U.S. - if union contracts were in fact a major contributing cause of low test performance, it might be reasonable to expect to find at least some clear differences between states that did and did not allow them. Although my analysis was extremely limited, I found no such evidence.

But this also applies to those who have been claiming recently – many in the Wisconsin context – that teacher bargaining clearly improves these outcomes.

A few such arguments made the rounds over the past week.  They were addressed thoughtfully on one blog [6], which examined the validity of the data [7], and pointed out [8] (correctly) that it remained unclear whether differences were attributable to “union effects” or to student characteristics.  The situation, however, is actually even more complicated that that.

Establishing a direct causal relationship between union contracts and achievement is extremely difficult, especially at the state-level. There is tremendous variation within and between states in the strength of individual locals (there are thousands) and in the terms of the contracts they have negotiated, to say nothing of all the other factors that might influence achievement (e.g., student/family characteristics, school-level policies/practices, resources, etc.). Such differences usually seem to elude those who seek to blame "the teachers' union" for various problems.

Making things worse, even some states that don’t allow bargaining or contracts do have policies commonly associated with unions, such as seniority [9], and unions can affect policy [10] outside of the collective bargaining process (for example, by waging public campaigns). Finally, of course, teachers in many unionized districts aren’t union members, so a simple “yes/no” categorization is a rather imperfect measure of union strength, even at the district-level. The manner in which one measures union presence or strength is critically important.  (For our posts, I divided states by whether or not states allow binding contracts, since there are a few states in which districts are not required to bargain with teachers, but contracts are legally-binding if the two sides complete the process; simpler schemes, such as CB/non-CB, miss this.)

Nevertheless, there is actually a bunch of research studies on the union/test score relationship.  For instance, one review [11] of 17 previous analyses concluded that most (but not all) show unions with a small positive benefit overall, but a possible negative effect for the lowest-performing students [12]. Results varied by the outcome used (e.g., graduation rates, SAT/ACT, etc.), and by other characteristics, such as race [13].

Generally, though, the body of evidence is inconclusive (see chapter six in this book [14]) and there is little basis for strong arguments either way.

(Side note: Even if there was solid evidence, one could not reasonably blame unions without placing equal blame on the administrators who bargained and approved the contracts. It is remarkably ironic – and hypocritical – for some superintendents [15] to criticize unions for the policies that they themselves negotiated. Similarly, it is important to realize that many policies supported by teachers’ unions, such as peer review, might have a positive effect if implemented more widely.)

Some union opponents [16], on the other hand, focus less on achievement and more on their fiscal impact – i.e., teacher contracts lead to higher pay and benefits, which boosts costs for states and districts (or shifts costs to less productive inputs). There are also, of course, serious impediments to this line of research. Differences within and between states in the cost of living, state pension systems, membership structures, and other factors are difficult to control for (i.e., more aggregate data bias results [17]). In addition, even if unions did increase costs, this might still be offset by positive financial (and educational) benefits – for example, improved organizational efficiency [18], better communication [19], or more standardized practices [12].

Accordingly, most studies of union effects on teacher compensation and overall costs find at least some positive association (see the review of in this older paper [20]), but it is not entirely clear the degree to which this is a direct causal effect of unions per se. For instance, besides the aforementioned measurement and contextual issues, it is tough to prove that unions increase costs or affect productivity unless you can look at outcomes before and after unionization, both within states [21] and between them. One very influential study [22] that examined the union/cost relationship (as well as dropout rates) over time found that unionization increased pay and overall expenditures, while also increasing dropout rates. A more recent, important study [23], in contrast, employed hand-gathered union certification data (which the authors argue is more accurate), and found little effect on either pay or dropout rates, along with only short-run effects on overall costs.

So, overall, it is remarkably difficult to isolate union effects or how they might arise. The evidence is mixed and inconclusive, especially for student achievement, and any strong, blanket statements – whether for or against unions – should be taken with a grain of salt. They make for good talking points, but their evidentiary basis is, in most cases, shaky.

What is clear is that unions do accomplish other goals – giving teachers (and other workers) a voice in their profession, their compensation, and their working conditions. On this score, as the Wisconsin protests demonstrate, the evidence is rather compelling.