The IMPACT Of Teacher Turnover In DCPS

Teacher turnover has long been a flashpoint in education policy, yet these debates are rife with complications. For example, it is often implied that turnover is a “bad thing,” even though some turnover, as when low-performing teachers leave, can be beneficial, whereas some retention, as when low-performing teachers stay, can be harmful. The impact of turnover also depends heavily on other factors, such as the pool of candidates available to serve as replacements, and how disruptive turnover is to the teachers who are retained.

The recent widespread reform of teacher evaluation systems has made the turnover issue, never far below the surface, even more salient in recent years. Critics contend that the new evaluations, particularly their use of test-based productivity measures, will cause teachers to flee the profession. Supporters, on the other hand, are in a sense hoping for this outcome, as they anticipate that, under the new systems, voluntary and involuntary separations will serve to improve the quality of the teacher workforce.

A new working paper takes a close look the impact of teacher turnover under what is perhaps the most controversial teacher evaluation system in the nation – that used in the District of Columbia Public Schools (DCPS). It's a very strong analysis that speaks directly to policy in a manner that does not fit well into the tribal structure of education debates today.

The paper’s authors, Melinda Adnot, Thomas Dee, Veronica Katz, and James Wyckoff, analyze DCPS data between 2009-10 and 2012-13, specifically how the evaluation scores of teachers leaving their schools compared with those of their replacements, and how and whether this eventually influenced student achievement (the evaluation system is called IMPACT). During this time period, overall attrition in DCPS (teachers leaving the district entirely) was around 18 percent. There was, however, a great deal of variation in this rate by IMPACT rating. For example, among teachers rated “effective” or “highly effective,” attrition was roughly 13 percent, compared with over 45 percent among teachers rated “minimally effective” or “ineffective.” Some of the teachers in the latter group left involuntarily (i.e., they were fired for low ratings), whereas others left on their own.

The impact of this turnover on overall teacher effectiveness depends on the difference in effectiveness (measured here with IMPACT scores) between exiting teachers and their replacements, while the effect on student achievement is further dependent upon the relationship between IMPACT scores and test-based effectiveness. These are the primary questions of interest for Adnot et al., who, put very simply, compare grade/school units that experienced turnover to those that did not.

They find, in brief terms, that teachers who left were replaced by teachers with higher IMPACT scores the following year. The difference is large in magnitude (about one third of one standard deviation), and corresponds to an educationally meaningful increase in achievement scores – about eight percent of a standard deviation in math and five percent in reading, though the latter estimate is only significant at the 90 percent confidence level. (Note that this does not mean that districtwide achievement scores improved by these amounts, as these estimates represent the year-to-year change in achievement in school/grade units that experienced turnover, compared to those that did not.)

There is, however, substantial variation by measured teacher effectiveness, as well as by school poverty. The estimated impact on both evaluation scores and student achievement is positive, large in magnitude, and statistically discernible among low-rated teachers who were replaced, but negative (though not statistically significant) when higher-rated teachers are replaced. This means, as would be expected, that exits of low-rated teachers may be beneficial, whereas exits of high-rated teachers are, on balance, harmful, or at least not helpful.

In addition, there is a rather large difference in the impact of turnover by student poverty (i.e., subsidized lunch eligibility rates). In high poverty schools (which means most DCPS schools), the estimated impact was large and statistically discernible. In low poverty schools, though, the effect was basically zero.

(Before discussing these results, I feel obliged to mention that DCPS has been very accommodating about providing their data to some researchers, but we at the Shanker Institute, despite extensive efforts over the course of almost two years, were unable to obtain data on teacher race and ethnicity for our recent teacher diversity study, despite the fact that the data we requested were far less sensitive and complicated than those used for this and other DCPS studies.)

Having said that, there are a few implications of these important results that bear articulating.

Teacher turnover/attrition is neither good nor bad by itself. This first and most obvious point is often ignored in our education policy discourse, but when a teacher leaves, the impact of this departure depends primarily on how good she is vis-à-vis the performance of her replacement. There is no reason to believe that aggregate turnover would necessarily have a positive or negative impact on meaningful outcomes – it all depends on who leaves, who replaces them, and how the churn affects those who remain. Prior research on this question, though not abundant, finds that the test-based impact of turnover may be negative (e.g., Ronfeldt et al. 2013).

The results of this paper, in contrast, suggest that, in DCPS, at least during this time period and measuring outcomes in terms of IMPACT scores and student achievement tests, the effect of teachers leaving, while heterogeneous, may have been positive overall. This certainly is not to say that schools and districts should ignore or encourage turnover, but it does illustrate empirically the underappreciated idea that the impact of teachers leaving can cut both ways. The degree to which, in this particular study, the estimated impact is due to the IMPACT system itself is an open question, one discussed a bit further below.

The (prospective) teacher labor pool in DC appears quite deep, at least for now. This is a very interesting and important finding. Now, in theory, dismissing and paying teachers based on IMPACT scores, which include test-based effectiveness measures, should produce higher IMPACT scores and, ultimately, higher achievement as well. The reality, of course, is that any number of factors can intervene in this process and create problems. In addition to and the effectiveness of teachers who choose voluntarily to leave, one huge (and unpredictable) factor is the pool of available replacements.

In DCPS, even though almost one in five teachers must be replaced every year, it seems that the labor market was sufficient not only to maintain teacher quality (at least as measured by these outcomes), but perhaps even to increase it. And this is even more significant considering the fact that, during the period of time in this study, the average measured performance of exiting teachers improved. This might have attenuated the postive impact of replacing them, but for the fact that the performance of their replacements also improved. Whether this is sustainable, and whether the same situation applies elsewhere, remain open questions to say the least.

The impact of turnover, and not just the level, may vary by school context. It is well established that teacher turnover is higher in schools serving larger proportions of disadvantaged students (e.g., Ingersoll 2001), but the results from Adnot et al. indicate that the impact of that turnover on meaningful outcomes also may vary by contextual factors, in this case student characteristics. Specifically, as mentioned above, replacements in low poverty schools were, on the whole, no different from their predecessors, whereas there was a rather large discrepancy in high poverty schools. This is important to keep in mind when, for example, looking at aggregate statistics on teacher turnover at the district level, and especially at the state or national levels.

Is this paper an evaluation of IMPACT itself? No, but that's not a black and white answer. The authors are very careful to note that the paper should not be interpreted that way (although that didn’t stop several commentators and reporters from doing so). This is an example of great researchers taking steps to ensure that their work is interpreted correctly.

The primary problem with seeing this analysis as an evaluation of IMPACT per se is that most attrition is voluntary, and the level and nature of teacher leaving during this time period cannot be attributed to IMPACT directly (nor, of course, can the quality of replacements). It may have occurred under a different system, or may even have been occurring to an extent before IMPACT was implemented. This may seem like a trivial issue, but it is not - this is an important distinction between, on the one hand, evidence on the effect of turnover under IMPACT and, on the other hand, the effect of turnover because of IMPACT (see Dee and Wyckoff 2015 for better evidence).

That said, IMPACT was directly intended to influence the quality composition of the DCPS teacher workforce, and the results from Adnot et al., combined with those from previous studies, certainly suggest that turnover under IMPACT may be having a positive impact. In that sense, these results can hardly be viewed as agnostic on the effect of the system.

Issues Areas
Permalink

What does this mean in the real world?

educationally meaningful increase in achievement scores – about eight percent of a standard deviation in math and five percent in reading, though the latter estimate is only significant at the 90 percent confidence level

How could such a net gain be worth all the costs - ranging from money costs to turmoil?

So, of course, exiting low-performers raised scores. But, predictably, IMPACT increased frustration and attrition by high performers, dropping those net gains.

So, doesn't that lead to the real question - that the researchers ignored? What were the results of IMPACT on the majority of teachers?

IMPACT advocates have ignored their burden of proof. Why do they believe that their efforts to exit low-performers didn't hurt the effectiveness - and undermine the education values - of the majority?
John Thompson

Permalink

Why didn't the authors of this "paper" disclose their relationships to organizations influential in creating/ preserving IMPACT with the release of their "findings". Also, it appears as though their "findings" surreptitiously created metrics that would offer positive calculations on behalf of their intended outcome. In other words, when a former TFA staffer writes a paper to defend the program created by another TFA alum, under the leadership of a former TFA big wig, they may have had some implicit bias from the outset, or an intended goal to achieve. It is VERY easy to rearrange/redesign a mathematical formula to produce the results that are intended. As a colleague of several DCPS Teacher selection ambassadors, it is ludicrous to say that there is a plethora of educators waiting in the wings, in fact as of 2/4/2016, there are still 50 vacant positions in DCPS. Each year we replace around 700 teachers, and throw them into an IMPACT system without ANY (actually one whole hour) of explicit training on the expectations of the IMPACT rubric or its application. The fact that IMPACT scores changed from one set of teachers to another could also be attributed to the relationship Principals have with their teachers. If a Principal doesn't "like" a teacher, they can bomb their IMPACT score (after all, it's a he said/she said scenario) which the teachers' only recourse is a grievance at the end of the year. This "study" speaks nothing to the relatively high level of discretion and personal biases that are also included in the IMPACT system. As a former ME (Master Educator) explicitly explained to me: "The first 5 years of IMPACT were entirely designed and purposed to fire people. People who had been in the district a long time cost a lot of money, and the central office understood that the only way to remove them was to create a system which they could control almost all of the data being entered (through principals and MEs- many of whom have long since been fired or departed because they were informed by central office that "too many people were receiving too high scores") that would cut away at the targeted teachers' professional integrity."

It is interesting to me that the people who oversaw the creation of IMPACT were not long standing educators with skin in the game (after all, Rhee proudly boasts of taping the mouths of children in Baltimore in her less than 2 year stint teaching) but rather "policy" people concerned with the corporatization of education. Imagine is the President nominated a Surgeon General with two years of actual experience, they would be laughed out of the Senate, but somehow the "ed reform" movement adores and welcomes such a lack of actual experience. Maybe, just maybe, we should read the tea leaves and determine that it is crippling POVERTY that surrounds students in the 7500 hours a year that they are away from school (compared to the 1200 hours or so in school per year). But, the dirty secret is that "ed reform" is INCREDIBLY profitable, and addressing poverty in a meaningful way would meant that companies would actually not profit as much because taxes would most likely have to be levied to offer more comprehensive social services (wrap around and such). Ask yourself one question, who benefits the most from these evaluation systems. If a teacher works thirty years and retires, they may be lucky in their life to be compensated nearly $2,000,000 over their career plus retirement. Those teachers know all their students, the families, their living situations, and have close ties. Michelle Rhee has made that in the last 3 years by trumpeting her failed view of only analyzing data and treating human children as little more than a data metric to be used however needed to bring about more money to enrich her and her friends (see also Kaya Henderson and TNTP) while still leaving students well behind.